Jobsub ID 94181.93@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94181.93@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:02:04
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:07:51
From worker node	Hostname	r22-n15.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:04:26
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_94_20210702T012102Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_69_20210702T041220Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_143_20210702T014824Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_119_20210702T012658Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_100_20210702T041258Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_59_20210702T040630Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_86_20210702T041252Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46319221_25_20210702T065903Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_42_20210702T040338Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46319221_101_20210702T123017Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:07:51
Saved logs	justin-logs:94181.93-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

r Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 3.5505e+12 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.0867e+06  1.0002e+00  1.0008e+00  2.0867e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2088811.954251 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.0565e-07  3.9538e-07  9.2209e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.1751e-07  3.9563e-07  9.2280e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
-13
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:05:57 GMT  run: 46319221 subRun: 447 event: 1018
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46319221 ; subrun: 447 ; event: 1018
evttime: 378403230
run: 46319221 ; subrun: 447 ; event: 1018
evttime: 378403230
########## EvtNo.1018
404
2
 MINUIT WARNING IN HESSE   
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN HESSE   
 ============== 1.05652 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.2170e-07  2.9172e-06  1.8627e+00  2.1373e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002137 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 11.0839 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.1855e+00  1.0303e+00  1.3591e+00  5.7961e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.191342 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -8.2959e-08  2.3827e-06  1.8626e+00  2.1374e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002138 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.24708e+06 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.3003e+02  1.0000e+00  1.0000e+00  1.3203e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 130.158576 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:06:56 GMT  run: 46319221 subRun: 447 event: 1019
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46319221 ; subrun: 447 ; event: 1019
evttime: 378403232
run: 46319221 ; subrun: 447 ; event: 1019
evttime: 378403232
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:07:14 GMT  run: 46319221 subRun: 447 event: 1020
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46319221 ; subrun: 447 ; event: 1020
evttime: 378403234
run: 46319221 ; subrun: 447 ; event: 1020
evttime: 378403234
11-Nov-2024 08:07:17 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/1e/93/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46319221_101_20210702T123017Z.root"
11-Nov-2024 08:07:17 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/1e/93/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46319221_101_20210702T123017Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.878059       20.8964        72.24        8.67825       20.7901        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000666296    0.134095       2.05589     0.00500036     0.314909        100    
reco:pandoracali:CalibrationdEdXPDSP           0.300479       1.47644       8.6137       0.836285       1.80933        100    
reco:pandoracalipid:Chi2ParticleID             0.0028501    0.00476028     0.0110662    0.00448608    0.00122195       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.462682       1.23369       5.75354      0.868672      0.931455        100    
reco:pandora2calipid:Chi2ParticleID            0.0051251    0.00897163     0.0142253    0.00870296    0.00173795       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000187303    0.137224       4.79822     0.00339078     0.582863        100    
[art]:TriggerResults:TriggerResultInserter    2.0587e-05    3.36676e-05   0.000145425   2.75985e-05   1.74528e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9729e-05      17.9007       61.2158     4.72245e-05     20.9331        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.878059       20.8964        72.24        8.67825       20.7901        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000666296    0.134095       2.05589     0.00500036     0.314909        100    
reco:pandoracali:CalibrationdEdXPDSP           0.300479       1.47644       8.6137       0.836285       1.80933        100    
reco:pandoracalipid:Chi2ParticleID             0.0028501    0.00476028     0.0110662    0.00448608    0.00122195       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.462682       1.23369       5.75354      0.868672      0.931455        100    
reco:pandora2calipid:Chi2ParticleID            0.0051251    0.00897163     0.0142253    0.00870296    0.00173795       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000187303    0.137224       4.79822     0.00339078     0.582863        100    
[art]:TriggerResults:TriggerResultInserter    2.0587e-05    3.36676e-05   0.000145425   2.75985e-05   1.74528e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9729e-05      17.9007       61.2158     4.72245e-05     20.9331        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4443.49 MB
  Peak resident set size usage (VmHWM): 3255.1 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4443.49 MB
  Peak resident set size usage (VmHWM): 3255.1 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         45         45          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2817.772342 Real = 3698.238009

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4443.49 VmHWM = 3255.1

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END