Jobsub ID 286903.155@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286903.155@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:39:01
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:47:49
From worker node	Hostname	r22-n09.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:40:39
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_11_20210416T163848Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586751_50_20210416T154139Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_101_20210416T145609Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_118_20210416T164428Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_139_20210416T164732Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_59_20210416T164032Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_56_20210416T163937Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_134_20210416T145616Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_6_20210419T044353Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_23_20210421T011632Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:47:49
Saved logs	justin-logs:286903.155-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

4:40 GMT  run: 42981851 subRun: 171 event: 236
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981851 ; subrun: 171 ; event: 236
evttime: 376945344
run: 42981851 ; subrun: 171 ; event: 236
evttime: 376945344
########## EvtNo.236
59
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.7351e-02  1.6455e-03  1.7351e+00  2.3106e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.049661 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 33.617 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1294e+01  9.1265e-01  1.0967e+00  1.3285e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 11.307623 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1171e-05  1.2234e-06  1.6292e+00  2.3708e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002382 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.9866e-02  1.4006e-03  1.7314e+00  2.3071e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.042173 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 34.3059 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0428e+01  9.4119e-01  1.0994e+00  1.2387e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 10.440241 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 4013.79 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.7991e+01  1.0002e+00  1.0015e+00  2.9989e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 28.021074 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
3
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:45:57 GMT  run: 42981851 subRun: 171 event: 237
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981851 ; subrun: 171 ; event: 237
evttime: 376945345
run: 42981851 ; subrun: 171 ; event: 237
evttime: 376945345
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:46:15 GMT  run: 42981851 subRun: 171 event: 238
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981851 ; subrun: 171 ; event: 238
evttime: 376945346
run: 42981851 ; subrun: 171 ; event: 238
evttime: 376945346
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:46:30 GMT  run: 42981851 subRun: 171 event: 239
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981851 ; subrun: 171 ; event: 239
evttime: 376945347
run: 42981851 ; subrun: 171 ; event: 239
evttime: 376945347
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:46:43 GMT  run: 42981851 subRun: 171 event: 240
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981851 ; subrun: 171 ; event: 240
evttime: 376945348
run: 42981851 ; subrun: 171 ; event: 240
evttime: 376945348
########## EvtNo.240
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
0
2212
Weights; 2
11-Nov-2024 07:47:16 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/bc/dc/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_23_20210421T011632Z.root"
11-Nov-2024 07:47:16 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/bc/dc/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_23_20210421T011632Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.09374       22.6298       64.137        7.11033       22.0124        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000759389    0.136197       2.07676     0.00492475     0.330236        100    
reco:pandoracali:CalibrationdEdXPDSP           0.458808       1.63339       21.4716      0.943134       2.4962         100    
reco:pandoracalipid:Chi2ParticleID            0.00300291    0.00500769     0.0094691    0.00460916    0.00133712       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.420344       1.19909       5.00445      0.904263      0.893683        100    
reco:pandora2calipid:Chi2ParticleID           0.00610855    0.00885859     0.0137269     0.0087772    0.00161936       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000298273    0.0286133     0.629737     0.00304909     0.0795873       100    
[art]:TriggerResults:TriggerResultInserter     2.117e-05    3.25185e-05   9.7036e-05    2.8264e-05    1.24337e-05      100    
end_path:pduneana:PDSPAnalyzer                2.2782e-05      19.618        61.494      5.0801e-05      21.8122        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.09374       22.6298       64.137        7.11033       22.0124        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000759389    0.136197       2.07676     0.00492475     0.330236        100    
reco:pandoracali:CalibrationdEdXPDSP           0.458808       1.63339       21.4716      0.943134       2.4962         100    
reco:pandoracalipid:Chi2ParticleID            0.00300291    0.00500769     0.0094691    0.00460916    0.00133712       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.420344       1.19909       5.00445      0.904263      0.893683        100    
reco:pandora2calipid:Chi2ParticleID           0.00610855    0.00885859     0.0137269     0.0087772    0.00161936       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000298273    0.0286133     0.629737     0.00304909     0.0795873       100    
[art]:TriggerResults:TriggerResultInserter     2.117e-05    3.25185e-05   9.7036e-05    2.8264e-05    1.24337e-05      100    
end_path:pduneana:PDSPAnalyzer                2.2782e-05      19.618        61.494      5.0801e-05      21.8122        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4507.16 MB
  Peak resident set size usage (VmHWM): 3286.47 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4507.16 MB
  Peak resident set size usage (VmHWM): 3286.47 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         48         48          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3036.327692 Real = 3927.180381

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4507.16 VmHWM = 3286.47

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END