Jobsub ID 94172.119@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94172.119@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:22:13
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:30:59
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:24:47
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_124_20210416T145613Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_95_20210416T145613Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_66_20210416T164017Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_72_20210416T164100Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_66_20210416T145608Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_74_20210416T164101Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_63_20210416T145608Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_46_20210416T163924Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_119_20210416T164428Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_21_20210416T163920Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:30:59
Saved logs	justin-logs:94172.119-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

VE MATRIX:
        -3.1848e-01  7.8127e-01  1.2170e+00  2.3202e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 0.320802 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.3451e-07  1.0051e-06  1.8122e+00  2.1878e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002188 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5102e-07  3.7985e-07  1.9074e+00  2.0926e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002093 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
-13
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:28:30 GMT  run: 42981491 subRun: 166 event: 219
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981491 ; subrun: 166 ; event: 219
evttime: 376857668
run: 42981491 ; subrun: 166 ; event: 219
evttime: 376857668
########## EvtNo.219
234
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7093e-01  5.0627e-03  1.8414e+00  2.3245e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.173252 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.7073e-07  4.1813e-06  1.7355e+00  2.2645e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002265 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 536380 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.8921e+01  1.0000e+00  1.0001e+00  5.0921e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 48.972296 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6612e-01  4.8131e-03  1.8391e+00  2.3222e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.168442 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 30.773 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.9803e+00  9.6766e-01  1.0434e+00  3.9692e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.984237 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.4594e-07  2.8133e-06  1.7349e+00  2.2651e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002266 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 85466.7 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.8710e+01  1.0000e+00  1.0001e+00  2.0710e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 18.730328 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:29:45 GMT  run: 42981491 subRun: 166 event: 220
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981491 ; subrun: 166 ; event: 220
evttime: 376857670
run: 42981491 ; subrun: 166 ; event: 220
evttime: 376857670
########## EvtNo.220
Getting abscex weigths
Making trajs 
3
2212
Weights; 2
11-Nov-2024 07:30:26 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/08/f2/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_21_20210416T163920Z.root"
11-Nov-2024 07:30:26 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/08/f2/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_21_20210416T163920Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.844466       23.1636       56.8347       31.4314       20.602         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000869509    0.132167       4.81553     0.00516792     0.501925        100    
reco:pandoracali:CalibrationdEdXPDSP           0.425744       1.4067        7.7756       0.869411       1.60043        100    
reco:pandoracalipid:Chi2ParticleID            0.00278794    0.00514591     0.0093095    0.00488541    0.00119613       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.39238       1.13843       4.68174      0.856552      0.786184        100    
reco:pandora2calipid:Chi2ParticleID           0.00472319    0.00961443     0.0163864    0.00949575    0.00216981       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000215717    0.0321229     0.401218     0.00344828     0.0705545       100    
[art]:TriggerResults:TriggerResultInserter    2.0967e-05    3.02244e-05   9.1809e-05    2.5954e-05    1.09797e-05      100    
end_path:pduneana:PDSPAnalyzer                2.0011e-05      20.4389       55.291        30.0138       20.5997        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.844466       23.1636       56.8347       31.4314       20.602         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000869509    0.132167       4.81553     0.00516792     0.501925        100    
reco:pandoracali:CalibrationdEdXPDSP           0.425744       1.4067        7.7756       0.869411       1.60043        100    
reco:pandoracalipid:Chi2ParticleID            0.00278794    0.00514591     0.0093095    0.00488541    0.00119613       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.39238       1.13843       4.68174      0.856552      0.786184        100    
reco:pandora2calipid:Chi2ParticleID           0.00472319    0.00961443     0.0163864    0.00949575    0.00216981       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000215717    0.0321229     0.401218     0.00344828     0.0705545       100    
[art]:TriggerResults:TriggerResultInserter    2.0967e-05    3.02244e-05   9.1809e-05    2.5954e-05    1.09797e-05      100    
end_path:pduneana:PDSPAnalyzer                2.0011e-05      20.4389       55.291        30.0138       20.5997        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4658.15 MB
  Peak resident set size usage (VmHWM): 3560.61 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4658.15 MB
  Peak resident set size usage (VmHWM): 3560.61 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         51         51          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3143.026573 Real = 3846.540036

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4658.15 VmHWM = 3560.61

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END