Jobsub ID 94191.173@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94191.173@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:39:27
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:45:27
From worker node	Hostname	r21-n18.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:41:45
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_116_20210629T224805Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21634531_27_20210629T195140Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_50_20210630T021300Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45717331_76_20210629T221848Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45717331_13_20210629T220909Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_27_20210630T020658Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45717331_69_20210630T014243Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_45_20210629T224006Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_109_20210630T052302Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_75_20210630T022411Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:45:27
Saved logs	justin-logs:94191.173-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

MATRIX FORCED POS-DEF BY ADDING 0.002478 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.68518 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.3939e+00  1.1974e+00  1.5595e+00  5.6371e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.399565 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.0639e-06  9.5567e-08  1.5222e+00  2.4778e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002485 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:43:35 GMT  run: 45718611 subRun: 366 event: 757
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718611 ; subrun: 366 ; event: 757
evttime: 378354547
run: 45718611 ; subrun: 366 ; event: 757
evttime: 378354547
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:43:48 GMT  run: 45718611 subRun: 366 event: 758
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718611 ; subrun: 366 ; event: 758
evttime: 378354549
run: 45718611 ; subrun: 366 ; event: 758
evttime: 378354549
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:44:01 GMT  run: 45718611 subRun: 366 event: 759
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718611 ; subrun: 366 ; event: 759
evttime: 378354550
run: 45718611 ; subrun: 366 ; event: 759
evttime: 378354550
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:44:12 GMT  run: 45718611 subRun: 366 event: 760
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718611 ; subrun: 366 ; event: 760
evttime: 378354553
run: 45718611 ; subrun: 366 ; event: 760
evttime: 378354553
########## EvtNo.760
176
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5218e-01  7.9821e-03  1.8239e+00  2.3203e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.154501 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 27.1965 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6365e+00  9.4413e-01  1.0812e+00  3.6112e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.640124 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002215 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.8348e-01  1.1473e-02  1.7803e+00  2.6917e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.486168 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 32314 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1425e+02  1.0000e+00  1.0002e+00  1.1624e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 114.361380 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
211
Weights; 12
11-Nov-2024 08:44:51 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/80/c4/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_75_20210630T022411Z.root"
11-Nov-2024 08:44:51 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/80/c4/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_75_20210630T022411Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.851294       19.0329       52.3123       8.95547       17.8192        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000633888    0.149087       4.47998     0.00407357     0.498082        100    
reco:pandoracali:CalibrationdEdXPDSP           0.327437       1.80356       8.65692      0.973497       1.90117        100    
reco:pandoracalipid:Chi2ParticleID            0.00266934    0.00490989     0.0108884    0.00421828    0.00168743       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.35433       1.45463       11.0166       1.04822       1.40687        100    
reco:pandora2calipid:Chi2ParticleID           0.00469298    0.00847322     0.018219     0.00829817    0.00208736       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000200997    0.0321798     0.935562     0.00320488     0.107858        100    
[art]:TriggerResults:TriggerResultInserter    1.7869e-05    2.80159e-05   0.00010339    2.2024e-05    1.37762e-05      100    
end_path:pduneana:PDSPAnalyzer                1.7586e-05      15.5795       46.6067     3.8114e-05      17.6273        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.851294       19.0329       52.3123       8.95547       17.8192        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000633888    0.149087       4.47998     0.00407357     0.498082        100    
reco:pandoracali:CalibrationdEdXPDSP           0.327437       1.80356       8.65692      0.973497       1.90117        100    
reco:pandoracalipid:Chi2ParticleID            0.00266934    0.00490989     0.0108884    0.00421828    0.00168743       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.35433       1.45463       11.0166       1.04822       1.40687        100    
reco:pandora2calipid:Chi2ParticleID           0.00469298    0.00847322     0.018219     0.00829817    0.00208736       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000200997    0.0321798     0.935562     0.00320488     0.107858        100    
[art]:TriggerResults:TriggerResultInserter    1.7869e-05    2.80159e-05   0.00010339    2.2024e-05    1.37762e-05      100    
end_path:pduneana:PDSPAnalyzer                1.7586e-05      15.5795       46.6067     3.8114e-05      17.6273        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4625.58 MB
  Peak resident set size usage (VmHWM): 3405.47 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4625.58 MB
  Peak resident set size usage (VmHWM): 3405.47 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         46         46          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2432.001251 Real = 3719.865146

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4625.58 VmHWM = 3405.47

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END