Jobsub ID 286905.124@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286905.124@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:48:23
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:00:55
From worker node	Hostname	r22-n09.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:51:29
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_9_20210415T121826Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_5_20210415T094914Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42920571_40_20210415T131829Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510331_132_20210415T081135Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_64_20210415T094920Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_115_20210415T094925Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_56_20210415T071053Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18803601_84_20210415T180941Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_8_20210415T225656Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_67_20210415T204129Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:00:55
Saved logs	justin-logs:286905.124-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

SG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:58:27 GMT  run: 18828981 subRun: 124 event: 676
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 676
evttime: 376840919
run: 18828981 ; subrun: 124 ; event: 676
evttime: 376840919
########## EvtNo.676
160
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.4147e-01  5.9887e-03  1.8342e+00  2.4013e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.243873 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4222e-02  1.7289e-01  1.1638e+00  2.6775e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 0.016900 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -8.0945e-07  4.7089e-06  1.6145e+00  2.3855e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002386 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 8.17068e+06 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.0725e+02  1.0000e+00  1.0002e+00  5.0925e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 507.758801 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.9141e-01  5.0488e-03  1.8188e+00  2.3676e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.193782 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 89.2474 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.1572e+00  9.9301e-01  1.0179e+00  6.1463e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.163304 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.0660e-06  1.9397e-06  1.6138e+00  2.3862e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002388 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:59:11 GMT  run: 18828981 subRun: 124 event: 677
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 677
evttime: 376840920
run: 18828981 ; subrun: 124 ; event: 677
evttime: 376840920
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:59:26 GMT  run: 18828981 subRun: 124 event: 678
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 678
evttime: 376840922
run: 18828981 ; subrun: 124 ; event: 678
evttime: 376840922
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:59:38 GMT  run: 18828981 subRun: 124 event: 679
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 679
evttime: 376840923
run: 18828981 ; subrun: 124 ; event: 679
evttime: 376840923
########## EvtNo.679
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
3
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:00:21 GMT  run: 18828981 subRun: 124 event: 680
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 680
evttime: 376840924
run: 18828981 ; subrun: 124 ; event: 680
evttime: 376840924
11-Nov-2024 08:00:24 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/09/4a/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_67_20210415T204129Z.root"
11-Nov-2024 08:00:24 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/09/4a/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_67_20210415T204129Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.92334       23.3189       67.9912       20.9866       21.3285        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000974749    0.110565       3.5356      0.00536067     0.382674        100    
reco:pandoracali:CalibrationdEdXPDSP           0.388217       1.68324       11.4457      0.835665       2.06018        100    
reco:pandoracalipid:Chi2ParticleID            0.00291864    0.00464684    0.00819257    0.00440331    0.00103913       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.310625       1.35682       7.15996      0.941788       1.20951        100    
reco:pandora2calipid:Chi2ParticleID           0.00568491    0.00907483     0.0152627    0.00885528    0.00190151       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000270157    0.0556098      1.30215     0.00346569     0.171352        100    
[art]:TriggerResults:TriggerResultInserter    2.0186e-05    3.59839e-05   0.000123543   2.9673e-05    1.71853e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9692e-05      20.0983       60.8858       13.604        21.0631        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.92334       23.3189       67.9912       20.9866       21.3285        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000974749    0.110565       3.5356      0.00536067     0.382674        100    
reco:pandoracali:CalibrationdEdXPDSP           0.388217       1.68324       11.4457      0.835665       2.06018        100    
reco:pandoracalipid:Chi2ParticleID            0.00291864    0.00464684    0.00819257    0.00440331    0.00103913       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.310625       1.35682       7.15996      0.941788       1.20951        100    
reco:pandora2calipid:Chi2ParticleID           0.00568491    0.00907483     0.0152627    0.00885528    0.00190151       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000270157    0.0556098      1.30215     0.00346569     0.171352        100    
[art]:TriggerResults:TriggerResultInserter    2.0186e-05    3.59839e-05   0.000123543   2.9673e-05    1.71853e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9692e-05      20.0983       60.8858       13.604        21.0631        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4460.76 MB
  Peak resident set size usage (VmHWM): 3363.76 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4460.76 MB
  Peak resident set size usage (VmHWM): 3363.76 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3099.297012 Real = 4063.062806

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4460.76 VmHWM = 3363.76

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END