Jobsub ID 286937.107@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286937.107@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 08:53:11
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:07:53
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 08:54:50
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42929541_122_20210415T082056Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_1_20210629T075446Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270331_50_20210629T193136Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_93_20210630T025316Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_6_20210630T023715Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_2_20210630T013734Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21640501_121_20210630T051632Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_3_20210630T144818Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_113_20210630T145802Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_27_20210630T144907Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:07:53
Saved logs	justin-logs:286937.107-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

0  8.1855e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 6.201789 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:03:43 GMT  run: 46273851 subRun: 296 event: 275
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 275
evttime: 378365211
run: 46273851 ; subrun: 296 ; event: 275
evttime: 378365211
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:04:05 GMT  run: 46273851 subRun: 296 event: 276
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 276
evttime: 378365212
run: 46273851 ; subrun: 296 ; event: 276
evttime: 378365212
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:04:57 GMT  run: 46273851 subRun: 296 event: 277
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 277
evttime: 378365214
run: 46273851 ; subrun: 296 ; event: 277
evttime: 378365214
########## EvtNo.277
127
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5223e-01  5.2285e-03  1.7669e+00  2.3801e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.154609 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 24.9245 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7873e+00  9.4484e-01  1.0722e+00  3.7703e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.791069 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.1683e-07  6.4201e-06  1.5649e+00  2.4351e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002435 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.1703 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9842e+00  1.1814e+00  1.4913e+00  4.3115e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.988561 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.4925e-07  1.2244e-06  1.5645e+00  2.4355e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002436 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:05:51 GMT  run: 46273851 subRun: 296 event: 278
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 278
evttime: 378365216
run: 46273851 ; subrun: 296 ; event: 278
evttime: 378365216
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:06:19 GMT  run: 46273851 subRun: 296 event: 279
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 279
evttime: 378365217
run: 46273851 ; subrun: 296 ; event: 279
evttime: 378365217
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:07:18 GMT  run: 46273851 subRun: 296 event: 280
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273851 ; subrun: 296 ; event: 280
evttime: 378365221
run: 46273851 ; subrun: 296 ; event: 280
evttime: 378365221
11-Nov-2024 10:07:21 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/38/2a/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_27_20210630T144907Z.root"
11-Nov-2024 10:07:21 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/38/2a/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_27_20210630T144907Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.929241        23.77        70.2661       30.6214       21.7271        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000670401     0.14775       4.13522     0.00437182     0.464105        100    
reco:pandoracali:CalibrationdEdXPDSP           0.389509       1.93253       37.7169      0.842501       4.06151        100    
reco:pandoracalipid:Chi2ParticleID            0.00283593    0.00491385    0.00844015     0.0047933    0.00107856       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.36677       1.47483       13.3429      0.893516       2.06017        100    
reco:pandora2calipid:Chi2ParticleID           0.00639686    0.00965846     0.017324     0.00937322    0.00219092       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000255848    0.0232505     0.276389     0.000870414    0.0592402       100    
[art]:TriggerResults:TriggerResultInserter    1.9642e-05    2.78914e-05    0.0001244    2.4073e-05    1.23048e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9832e-05      20.1765       63.6944       14.3715       21.0214        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.929241        23.77        70.2661       30.6214       21.7271        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000670401     0.14775       4.13522     0.00437182     0.464105        100    
reco:pandoracali:CalibrationdEdXPDSP           0.389509       1.93253       37.7169      0.842501       4.06151        100    
reco:pandoracalipid:Chi2ParticleID            0.00283593    0.00491385    0.00844015     0.0047933    0.00107856       100    
reco:pandora2cali:CalibrationdEdXPDSP           0.36677       1.47483       13.3429      0.893516       2.06017        100    
reco:pandora2calipid:Chi2ParticleID           0.00639686    0.00965846     0.017324     0.00937322    0.00219092       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000255848    0.0232505     0.276389     0.000870414    0.0592402       100    
[art]:TriggerResults:TriggerResultInserter    1.9642e-05    2.78914e-05    0.0001244    2.4073e-05    1.23048e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9832e-05      20.1765       63.6944       14.3715       21.0214        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4741.67 MB
  Peak resident set size usage (VmHWM): 3537.69 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4741.67 MB
  Peak resident set size usage (VmHWM): 3537.69 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         51         51          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3032.092468 Real = 4281.037852

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4741.67 VmHWM = 3537.69

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END