Jobsub ID 286945.92@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286945.92@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 09:36:36
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:46:50
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 09:37:48
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_93_20210630T022512Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_76_20210630T024951Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_44_20210630T144938Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_70_20210630T114202Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_114_20210630T111836Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_109_20210630T114740Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46283101_37_20210701T023919Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46277861_61_20210701T011622Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46283101_9_20210701T023606Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46285521_94_20210701T072714Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:46:50
Saved logs	justin-logs:286945.92-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:45:17 GMT  run: 46285521 subRun: 320 event: 949
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46285521 ; subrun: 320 ; event: 949
evttime: 378378907
run: 46285521 ; subrun: 320 ; event: 949
evttime: 378378907
########## EvtNo.949
70
2
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 269190 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.2111e+02  1.0000e+00  1.0000e+00  3.2311e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 321.433425 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002233 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002075 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0756e-07  2.2864e-06  1.7737e+00  2.2263e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002226 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 3.88822e+06 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -8.2060e+02  1.0000e+00  1.0000e+00  8.2260e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 821.427229 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 369471 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.3439e+02  1.0000e+00  1.0000e+00  5.3639e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 534.928917 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.3806e-08  1.8664e-06  1.7734e+00  2.2266e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002227 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 400082 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6350e+02  1.0000e+00  1.0000e+00  2.6550e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 263.766569 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
3
-13
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:46:09 GMT  run: 46285521 subRun: 320 event: 950
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46285521 ; subrun: 320 ; event: 950
evttime: 378378909
run: 46285521 ; subrun: 320 ; event: 950
evttime: 378378909
11-Nov-2024 10:46:13 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/34/d9/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46285521_94_20210701T072714Z.root"
11-Nov-2024 10:46:13 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/34/d9/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46285521_94_20210701T072714Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.16687       23.0417       68.6977       18.8601       21.0969        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000793481    0.138862       2.8588      0.00490624     0.351011        100    
reco:pandoracali:CalibrationdEdXPDSP           0.432682       1.68567       12.0703      0.856855       2.15293        100    
reco:pandoracalipid:Chi2ParticleID            0.00300396    0.00490316     0.0116171    0.00455344     0.0013665       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.447297       1.48133       12.3518      0.923079       2.00441        100    
reco:pandora2calipid:Chi2ParticleID           0.00526557    0.00952714     0.0160253    0.00945573    0.00202056       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000240471    0.0366147      1.01106     0.00187418     0.122141        100    
[art]:TriggerResults:TriggerResultInserter    2.0568e-05    3.16826e-05   0.000131635   2.66235e-05   1.45583e-05      100    
end_path:pduneana:PDSPAnalyzer                2.0391e-05      19.6843       63.7103     0.000287257     20.9774        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.16687       23.0417       68.6977       18.8601       21.0969        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000793481    0.138862       2.8588      0.00490624     0.351011        100    
reco:pandoracali:CalibrationdEdXPDSP           0.432682       1.68567       12.0703      0.856855       2.15293        100    
reco:pandoracalipid:Chi2ParticleID            0.00300396    0.00490316     0.0116171    0.00455344     0.0013665       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.447297       1.48133       12.3518      0.923079       2.00441        100    
reco:pandora2calipid:Chi2ParticleID           0.00526557    0.00952714     0.0160253    0.00945573    0.00202056       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000240471    0.0366147      1.01106     0.00187418     0.122141        100    
[art]:TriggerResults:TriggerResultInserter    2.0568e-05    3.16826e-05   0.000131635   2.66235e-05   1.45583e-05      100    
end_path:pduneana:PDSPAnalyzer                2.0391e-05      19.6843       63.7103     0.000287257     20.9774        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4898.62 MB
  Peak resident set size usage (VmHWM): 3712.49 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4898.62 MB
  Peak resident set size usage (VmHWM): 3712.49 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         49         49          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3039.277319 Real = 4030.192777

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4898.62 VmHWM = 3712.49

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END