Jobsub ID 94196.163@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94196.163@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:54:14
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 09:06:55
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:56:26
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46263891_77_20210628T204642Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_94_20210629T013900Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710411_78_20210629T004939Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_41_20210629T004834Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_96_20210629T013855Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710411_26_20210629T042535Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21631871_26_20210629T102702Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_115_20210629T084649Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_65_20210629T013853Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_51_20210629T082543Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 09:06:55
Saved logs	justin-logs:94196.163-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

ent 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.92353 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6239e+00  1.0952e+00  1.2835e+00  3.2452e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.627142 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.3782e-07  4.1746e-06  1.6097e+00  2.3903e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002391 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:04:20 GMT  run: 46266471 subRun: 227 event: 517
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46266471 ; subrun: 227 ; event: 517
evttime: 378340253
run: 46266471 ; subrun: 227 ; event: 517
evttime: 378340253
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:04:33 GMT  run: 46266471 subRun: 227 event: 518
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46266471 ; subrun: 227 ; event: 518
evttime: 378340254
run: 46266471 ; subrun: 227 ; event: 518
evttime: 378340254
########## EvtNo.518
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:05:25 GMT  run: 46266471 subRun: 227 event: 519
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46266471 ; subrun: 227 ; event: 519
evttime: 378340255
run: 46266471 ; subrun: 227 ; event: 519
evttime: 378340255
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:05:42 GMT  run: 46266471 subRun: 227 event: 520
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46266471 ; subrun: 227 ; event: 520
evttime: 378340256
run: 46266471 ; subrun: 227 ; event: 520
evttime: 378340256
########## EvtNo.520
81
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.9539e-02  4.4875e-03  1.7877e+00  2.3073e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.101847 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.3757e-06  1.1795e-06  1.5304e+00  2.4696e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002475 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.90933 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.4289e+00  1.2656e+00  1.7121e+00  5.4512e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.434377 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 10790.7 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.7135e+01  1.0001e+00  1.0008e+00  4.9134e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 47.184044 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
11-Nov-2024 09:06:21 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/00/5f/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_51_20210629T082543Z.root"
11-Nov-2024 09:06:21 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/00/5f/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_51_20210629T082543Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.15601       25.0469       61.6224       33.4076       20.6915        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000791966    0.149516       7.17489     0.00428346     0.724309        100    
reco:pandoracali:CalibrationdEdXPDSP           0.381754       1.61149       9.28796      0.910558       1.99076        100    
reco:pandoracalipid:Chi2ParticleID            0.00328013    0.00488104    0.00788882    0.00478166    0.00101523       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.402588       1.25206       9.84664      0.851684       1.34049        100    
reco:pandora2calipid:Chi2ParticleID           0.00554926    0.00967883     0.0171396    0.00972708    0.00202562       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000212882    0.0326644     0.525246     0.00320524     0.0774505       100    
[art]:TriggerResults:TriggerResultInserter    2.0005e-05    3.02257e-05   0.000148214   2.4897e-05    1.64504e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9042e-05      21.9861       56.9626       30.8969       20.743         100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.15601       25.0469       61.6224       33.4076       20.6915        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000791966    0.149516       7.17489     0.00428346     0.724309        100    
reco:pandoracali:CalibrationdEdXPDSP           0.381754       1.61149       9.28796      0.910558       1.99076        100    
reco:pandoracalipid:Chi2ParticleID            0.00328013    0.00488104    0.00788882    0.00478166    0.00101523       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.402588       1.25206       9.84664      0.851684       1.34049        100    
reco:pandora2calipid:Chi2ParticleID           0.00554926    0.00967883     0.0171396    0.00972708    0.00202562       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000212882    0.0326644     0.525246     0.00320524     0.0774505       100    
[art]:TriggerResults:TriggerResultInserter    2.0005e-05    3.02257e-05   0.000148214   2.4897e-05    1.64504e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9042e-05      21.9861       56.9626       30.8969       20.743         100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4775.94 MB
  Peak resident set size usage (VmHWM): 3526.14 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4775.94 MB
  Peak resident set size usage (VmHWM): 3526.14 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         55         55          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3265.616557 Real = 4122.810224

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4775.94 VmHWM = 3526.14

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END