Jobsub ID 286902.44@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286902.44@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:34:32
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:38:02
From worker node	Hostname	r22-n15.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:35:48
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42982071_95_20210424T154136Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_19183454_0_20210427T185140Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710191_1_20210628T231059Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710191_44_20210628T231448Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_30_20210629T004827Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46263891_105_20210628T222712Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_86_20210629T013854Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_132_20210629T092213Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21634531_21_20210629T163950Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46268961_124_20210629T143115Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:38:02
Saved logs	justin-logs:286902.44-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 759.331 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.7954e+01  1.0150e+00  1.0539e+00  2.9885e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 27.983433 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.52809e+14 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.8247e+06  1.0001e+00  1.0048e+00  3.8247e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3828478.213216 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.1570e-07  9.4206e-07  1.6098e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.2178e-07  9.5529e-07  1.6067e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.31763e+12 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.2958e+06  1.0014e+00  1.0113e+00  1.2958e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1297064.319641 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.7179e-07  1.5371e-07  8.6136e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.3272e-07  1.2294e-07  8.5174e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
3
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:36:35 GMT  run: 46268961 subRun: 237 event: 1248
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46268961 ; subrun: 237 ; event: 1248
evttime: 378344537
run: 46268961 ; subrun: 237 ; event: 1248
evttime: 378344537
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:36:59 GMT  run: 46268961 subRun: 237 event: 1249
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46268961 ; subrun: 237 ; event: 1249
evttime: 378344540
run: 46268961 ; subrun: 237 ; event: 1249
evttime: 378344540
########## EvtNo.1249
No G4 version of the good particle was found, returning null pointer
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 07:37:03 GMT run: 46268961 subRun: 237 event: 1249 PDSPAnalyzer_module.cc:1384
No true beam particle
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:37:22 GMT  run: 46268961 subRun: 237 event: 1250
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46268961 ; subrun: 237 ; event: 1250
evttime: 378344542
run: 46268961 ; subrun: 237 ; event: 1250
evttime: 378344542
11-Nov-2024 07:37:25 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/8a/c9/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46268961_124_20210629T143115Z.root"
11-Nov-2024 07:37:25 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/8a/c9/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46268961_124_20210629T143115Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.13143       19.6449       74.4821       5.26364       20.3717        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000649701    0.131395       4.08171      0.0042409     0.439995        100    
reco:pandoracali:CalibrationdEdXPDSP           0.310613       1.82723       9.96183       1.01497       2.01908        100    
reco:pandoracalipid:Chi2ParticleID            0.00294419    0.00507471     0.0103407    0.00474214    0.00140318       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.3751        1.18119       6.43415      0.875657      0.838107        100    
reco:pandora2calipid:Chi2ParticleID           0.00535844    0.00951573     0.0163102    0.00916832    0.00214758       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000226876    0.0630356      2.25622     0.00289664     0.252338        100    
[art]:TriggerResults:TriggerResultInserter    1.9989e-05    2.84376e-05   0.000103257   2.3353e-05    1.28635e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9237e-05      16.4269       61.346      5.15205e-05     20.0684        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.13143       19.6449       74.4821       5.26364       20.3717        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000649701    0.131395       4.08171      0.0042409     0.439995        100    
reco:pandoracali:CalibrationdEdXPDSP           0.310613       1.82723       9.96183       1.01497       2.01908        100    
reco:pandoracalipid:Chi2ParticleID            0.00294419    0.00507471     0.0103407    0.00474214    0.00140318       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.3751        1.18119       6.43415      0.875657      0.838107        100    
reco:pandora2calipid:Chi2ParticleID           0.00535844    0.00951573     0.0163102    0.00916832    0.00214758       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000226876    0.0630356      2.25622     0.00289664     0.252338        100    
[art]:TriggerResults:TriggerResultInserter    1.9989e-05    2.84376e-05   0.000103257   2.3353e-05    1.28635e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9237e-05      16.4269       61.346      5.15205e-05     20.0684        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4856.32 MB
  Peak resident set size usage (VmHWM): 3670.56 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4856.32 MB
  Peak resident set size usage (VmHWM): 3670.56 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         46         46          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2683.020813 Real = 3627.210940

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4856.32 VmHWM = 3670.56

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END