Jobsub ID 94175.163@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94175.163@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:36:01
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:42:18
From worker node	Hostname	r22-n09.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:37:54
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42911571_96_20210414T100234Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42911571_18_20210414T100112Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43492941_37_20210414T191219Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43492941_60_20210414T191207Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42920571_84_20210414T223300Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18815731_114_20210414T210322Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42920571_78_20210415T024958Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42920571_71_20210415T033753Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_123_20210415T072501Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_12_20210415T094914Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:42:18
Saved logs	justin-logs:94175.163-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

441 ; subrun: 100 ; event: 129
evttime: 376831814
########## EvtNo.129
43
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.8551e-06 -1.1069e-07  1.7624e+00  2.2376e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002244 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 36481.5 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.2259e+00  9.9996e-01  1.0001e+00  6.2258e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.232113 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.4079e+14 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6939e+06  1.0002e+00  1.0082e+00  2.6939e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2696587.650458 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.1243e-07  1.2215e-07  9.0121e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.8101e-07  1.7917e-06  5.4212e-04  3.9995e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.5844e-06 -1.5487e-07  1.7624e+00  2.2376e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002247 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 16103.1 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.4901e+00  9.9993e-01  1.0001e+00  4.4901e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.494615 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.47306e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.3533e+06  1.0002e+00  1.0077e+00  1.3533e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1354688.613868 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.6200e-07  1.2163e-07  8.9878e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.0421e-07  2.3341e-07  9.4476e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:41:11 GMT  run: 43511441 subRun: 100 event: 130
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43511441 ; subrun: 100 ; event: 130
evttime: 376831815
run: 43511441 ; subrun: 100 ; event: 130
evttime: 376831815
########## EvtNo.130
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
1
2212
Weights; 2
11-Nov-2024 07:41:43 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/d8/08/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_12_20210415T094914Z.root"
11-Nov-2024 07:41:43 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/d8/08/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_12_20210415T094914Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.861109       22.4567       58.9706       27.9696       20.6125        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000878637     0.10213       2.35156     0.00447869      0.27074        100    
reco:pandoracali:CalibrationdEdXPDSP           0.282762       1.41612       8.82708      0.815539       1.86136        100    
reco:pandoracalipid:Chi2ParticleID            0.00264526    0.00457757     0.0085138    0.00437143    0.00102314       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.233462       1.01789       5.07663      0.762179      0.735167        100    
reco:pandora2calipid:Chi2ParticleID           0.00539041    0.00894951     0.0137267    0.00861067    0.00176801       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000212381    0.0277673     0.383912     0.00288989     0.069097        100    
[art]:TriggerResults:TriggerResultInserter    1.9997e-05    2.75013e-05   0.000108664   2.4373e-05    1.14754e-05      100    
end_path:pduneana:PDSPAnalyzer                1.8787e-05      19.8787       58.0089       27.1812       20.3843        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.861109       22.4567       58.9706       27.9696       20.6125        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000878637     0.10213       2.35156     0.00447869      0.27074        100    
reco:pandoracali:CalibrationdEdXPDSP           0.282762       1.41612       8.82708      0.815539       1.86136        100    
reco:pandoracalipid:Chi2ParticleID            0.00264526    0.00457757     0.0085138    0.00437143    0.00102314       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.233462       1.01789       5.07663      0.762179      0.735167        100    
reco:pandora2calipid:Chi2ParticleID           0.00539041    0.00894951     0.0137267    0.00861067    0.00176801       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000212381    0.0277673     0.383912     0.00288989     0.069097        100    
[art]:TriggerResults:TriggerResultInserter    1.9997e-05    2.75013e-05   0.000108664   2.4373e-05    1.14754e-05      100    
end_path:pduneana:PDSPAnalyzer                1.8787e-05      19.8787       58.0089       27.1812       20.3843        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4535.57 MB
  Peak resident set size usage (VmHWM): 3370.49 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4535.57 MB
  Peak resident set size usage (VmHWM): 3370.49 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         52         52          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3005.848258 Real = 3750.462667

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4535.57 VmHWM = 3370.49

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END