Jobsub ID 94175.108@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94175.108@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:36:01
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:51:36
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:38:21
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_46_20210415T094916Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42936861_27_20210415T112108Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42920571_28_20210415T072253Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_38_20210415T121831Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_79_20210415T131805Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_100_20210415T071930Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43492941_23_20210415T131542Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18803601_2_20210415T172709Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18803601_14_20210415T173228Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_0_20210415T164225Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:51:36
Saved logs	justin-logs:94175.108-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 13.819950 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 527.245 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.9472e+01  1.0014e+00  1.0059e+00  2.1464e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 19.493122 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.0712e-04  3.3733e-04  1.7676e+00  2.2325e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002640 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:49:15 GMT  run: 18828981 subRun: 124 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 9
evttime: 376837787
run: 18828981 ; subrun: 124 ; event: 9
evttime: 376837787
########## EvtNo.9
33
2
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 8.95713 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.2981e+00  1.2124e+00  1.6375e+00  8.4482e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 7.306509 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.8926e-05  8.0615e-06  1.6363e+00  2.3637e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002403 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 8.0823 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.1305e+00  1.1460e+00  1.4600e+00  4.5246e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3.135070 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 4833.69 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.1934e+01  1.0003e+00  1.0051e+00  2.3929e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 21.958407 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:50:16 GMT  run: 18828981 subRun: 124 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 18828981 ; subrun: 124 ; event: 10
evttime: 376837790
run: 18828981 ; subrun: 124 ; event: 10
evttime: 376837790
########## EvtNo.10
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
1
211
Weights; 12
11-Nov-2024 07:51:02 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/47/5e/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_0_20210415T164225Z.root"
11-Nov-2024 07:51:02 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/47/5e/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_0_20210415T164225Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.757675       28.1795       62.666        36.4918       21.2266        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000669392    0.106753       4.16298     0.00491068      0.43281        100    
reco:pandoracali:CalibrationdEdXPDSP           0.389778       1.41027       8.74967      0.838313       1.63241        100    
reco:pandoracalipid:Chi2ParticleID            0.00348953    0.00472336    0.00676029    0.00456017    0.000748041      100    
reco:pandora2cali:CalibrationdEdXPDSP          0.314977       1.10898       5.21065      0.753306      0.886155        100    
reco:pandora2calipid:Chi2ParticleID           0.00587189    0.00913629     0.0144478    0.00881427    0.00177778       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000272751    0.0282724     0.376038     0.000514805    0.0699373       100    
[art]:TriggerResults:TriggerResultInserter     2.138e-05    2.86692e-05   9.6147e-05    2.56165e-05   1.03682e-05      100    
end_path:pduneana:PDSPAnalyzer                2.2856e-05      25.5108       60.5319       32.4964       20.9621        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.757675       28.1795       62.666        36.4918       21.2266        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000669392    0.106753       4.16298     0.00491068      0.43281        100    
reco:pandoracali:CalibrationdEdXPDSP           0.389778       1.41027       8.74967      0.838313       1.63241        100    
reco:pandoracalipid:Chi2ParticleID            0.00348953    0.00472336    0.00676029    0.00456017    0.000748041      100    
reco:pandora2cali:CalibrationdEdXPDSP          0.314977       1.10898       5.21065      0.753306      0.886155        100    
reco:pandora2calipid:Chi2ParticleID           0.00587189    0.00913629     0.0144478    0.00881427    0.00177778       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000272751    0.0282724     0.376038     0.000514805    0.0699373       100    
[art]:TriggerResults:TriggerResultInserter     2.138e-05    2.86692e-05   9.6147e-05    2.56165e-05   1.03682e-05      100    
end_path:pduneana:PDSPAnalyzer                2.2856e-05      25.5108       60.5319       32.4964       20.9621        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4843.03 MB
  Peak resident set size usage (VmHWM): 3712.58 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4843.03 MB
  Peak resident set size usage (VmHWM): 3712.58 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         62         62          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3578.615943 Real = 4287.635597

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4843.03 VmHWM = 3712.58

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END