Jobsub ID 94209.58@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94209.58@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 08:38:08
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 09:46:05
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 08:40:12
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_19_20210416T033005Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_75_20210415T204135Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_24_20210416T035749Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42961101_122_20210416T095605Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_88_20210416T164226Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_118_20210416T145612Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_126_20210630T023604Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_79_20210630T114326Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_41_20210630T165521Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_68_20210630T145216Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 09:46:05
Saved logs	justin-logs:94209.58-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

RIX:
        -2.9500e+07  1.0022e+00  1.0094e+00  2.9500e+07
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 29529291.038787 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.8421e-07  1.6037e-07  9.1748e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.2110e-07  2.5700e-07  9.6029e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.20929e+16 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1257e+26  1.0000e+00  1.0001e+00  1.1257e+26
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 112681611346628272654909440.000000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4720e-09  2.3631e-07  1.5131e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.75771e+15 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.5210e+07  1.0005e+00  1.0022e+00  4.5210e+07
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 45255362.689552 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.0839e-07  2.2177e-07  9.4498e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.4473e-07  2.4291e-07  9.5430e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1.12325e+15 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.9878e+06  1.0000e+00  1.0015e+00  1.9878e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1989756.325680 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.7959e-07  8.1117e-09  1.5165e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.4867e-07  5.8776e-09  1.4961e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:45:30 GMT  run: 46273851 subRun: 296 event: 690
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 690
evttime: 378364994
run: 46273851 ; subrun: 296 ; event: 690
evttime: 378364994
11-Nov-2024 09:45:32 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/ce/5b/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_68_20210630T145216Z.root"
11-Nov-2024 09:45:32 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/ce/5b/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_68_20210630T145216Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.762245       21.8854       65.9831       8.21198       21.6755        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000758981    0.131811       4.22045     0.00494697     0.459884        100    
reco:pandoracali:CalibrationdEdXPDSP           0.362844       1.82635       14.8169      0.834268       2.72887        100    
reco:pandoracalipid:Chi2ParticleID            0.00241282    0.00459835    0.00813466    0.00441722    0.00107035       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.272625       1.14375       11.6168      0.733846       1.33349        100    
reco:pandora2calipid:Chi2ParticleID           0.00464491     0.0090684     0.0160426    0.00884758    0.00186329       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000213997    0.0509627      2.45781     0.000695263    0.249307        100    
[art]:TriggerResults:TriggerResultInserter    1.9858e-05    2.75801e-05   9.4088e-05     2.378e-05    9.99519e-06      100    
end_path:pduneana:PDSPAnalyzer                1.8065e-05      18.7184       54.3953     4.69325e-05     21.142         100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.762245       21.8854       65.9831       8.21198       21.6755        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000758981    0.131811       4.22045     0.00494697     0.459884        100    
reco:pandoracali:CalibrationdEdXPDSP           0.362844       1.82635       14.8169      0.834268       2.72887        100    
reco:pandoracalipid:Chi2ParticleID            0.00241282    0.00459835    0.00813466    0.00441722    0.00107035       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.272625       1.14375       11.6168      0.733846       1.33349        100    
reco:pandora2calipid:Chi2ParticleID           0.00464491     0.0090684     0.0160426    0.00884758    0.00186329       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000213997    0.0509627      2.45781     0.000695263    0.249307        100    
[art]:TriggerResults:TriggerResultInserter    1.9858e-05    2.75801e-05   9.4088e-05     2.378e-05    9.99519e-06      100    
end_path:pduneana:PDSPAnalyzer                1.8065e-05      18.7184       54.3953     4.69325e-05     21.142         100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4452.1 MB
  Peak resident set size usage (VmHWM): 3321.02 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4452.1 MB
  Peak resident set size usage (VmHWM): 3321.02 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         45         45          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2885.355122 Real = 3844.559198

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4452.1 VmHWM = 3321.02

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END