Jobsub ID 286903.170@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286903.170@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:39:01
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:47:49
From worker node	Hostname	r22-n09.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:40:31
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42936861_94_20210416T003725Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_114_20210416T020231Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_127_20210415T205859Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_119_20210415T231830Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_20_20210416T061609Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586751_107_20210416T160004Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_10_20210416T163845Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42961101_113_20210416T095321Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_97_20210416T164305Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_17_20210416T145601Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:47:49
Saved logs	justin-logs:286903.170-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

ARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.383517 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.4782e-07  1.3401e-06  1.5847e+00  2.4153e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002416 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.52994 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6264e+00  1.1397e+00  1.3629e+00  3.1238e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.629548 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 21612.3 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.6680e+01  1.0000e+00  1.0003e+00  4.8680e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 46.728967 TO DIAGONAL.
Getting reco beam calo
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 07:45:28 GMT run: 43586281 subRun: 160 event: 175
Invalid track range -999 return -1
%MSG
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 07:45:28 GMT run: 43586281 subRun: 160 event: 175
Invalid track range -999 return -1
%MSG
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 07:45:28 GMT run: 43586281 subRun: 160 event: 175
Invalid track range -999 return -1
%MSG
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 07:45:28 GMT run: 43586281 subRun: 160 event: 175
Invalid track range -999 return -1
%MSG
Getting abscex weigths
Making trajs 
1
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:46:22 GMT  run: 43586281 subRun: 160 event: 176
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586281 ; subrun: 160 ; event: 176
evttime: 376857034
run: 43586281 ; subrun: 160 ; event: 176
evttime: 376857034
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:46:37 GMT  run: 43586281 subRun: 160 event: 177
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586281 ; subrun: 160 ; event: 177
evttime: 376857035
run: 43586281 ; subrun: 160 ; event: 177
evttime: 376857035
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:46:50 GMT  run: 43586281 subRun: 160 event: 178
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586281 ; subrun: 160 ; event: 178
evttime: 376857035
run: 43586281 ; subrun: 160 ; event: 178
evttime: 376857035
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:47:06 GMT  run: 43586281 subRun: 160 event: 179
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586281 ; subrun: 160 ; event: 179
evttime: 376857036
run: 43586281 ; subrun: 160 ; event: 179
evttime: 376857036
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:47:17 GMT  run: 43586281 subRun: 160 event: 180
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586281 ; subrun: 160 ; event: 180
evttime: 376857037
run: 43586281 ; subrun: 160 ; event: 180
evttime: 376857037
11-Nov-2024 07:47:19 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/13/90/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_17_20210416T145601Z.root"
11-Nov-2024 07:47:19 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/13/90/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_17_20210416T145601Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.844028       24.0898       63.7006        29.24        21.5342        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000700885    0.111751       2.05566      0.0057132     0.295034        100    
reco:pandoracali:CalibrationdEdXPDSP           0.412116       1.67112       8.61187      0.793342       2.07334        100    
reco:pandoracalipid:Chi2ParticleID            0.00278124    0.00457048    0.00943067    0.00439454    0.00106404       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.307244       1.03904       5.10757      0.756637      0.808711        100    
reco:pandora2calipid:Chi2ParticleID           0.00486336    0.00879193     0.0143354    0.00865192    0.00176502       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00023082     0.030269      0.423117     0.00311783     0.0669527       100    
[art]:TriggerResults:TriggerResultInserter    2.0508e-05    2.87685e-05   9.2062e-05    2.41435e-05   1.17955e-05      100    
end_path:pduneana:PDSPAnalyzer                1.8857e-05      21.2237       59.9424       27.3273       21.2691        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.844028       24.0898       63.7006        29.24        21.5342        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000700885    0.111751       2.05566      0.0057132     0.295034        100    
reco:pandoracali:CalibrationdEdXPDSP           0.412116       1.67112       8.61187      0.793342       2.07334        100    
reco:pandoracalipid:Chi2ParticleID            0.00278124    0.00457048    0.00943067    0.00439454    0.00106404       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.307244       1.03904       5.10757      0.756637      0.808711        100    
reco:pandora2calipid:Chi2ParticleID           0.00486336    0.00879193     0.0143354    0.00865192    0.00176502       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00023082     0.030269      0.423117     0.00311783     0.0669527       100    
[art]:TriggerResults:TriggerResultInserter    2.0508e-05    2.87685e-05   9.2062e-05    2.41435e-05   1.17955e-05      100    
end_path:pduneana:PDSPAnalyzer                1.8857e-05      21.2237       59.9424       27.3273       21.2691        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4525.89 MB
  Peak resident set size usage (VmHWM): 3384.98 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4525.89 MB
  Peak resident set size usage (VmHWM): 3384.98 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         52         52          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3128.033668 Real = 3929.898453

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4525.89 VmHWM = 3384.98

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END