Jobsub ID 94232.36@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94232.36@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 10:01:03
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 11:21:47
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 10:15:27
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_142_20210415T122158Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_22_20210415T110420Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42936861_31_20210415T112140Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_44_20210415T204152Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_118_20210415T231829Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42911561_41_20210415T222632Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_68_20210415T204129Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_18828981_39_20210415T204300Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_84_20210416T015325Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_54_20210421T011634Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 11:21:47
Saved logs	justin-logs:94232.36-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

======== MATRIX FORCED POS-DEF BY ADDING 0.308401 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.8861e-07  3.5621e-06  1.7585e+00  2.2415e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002242 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002109 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.47571 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7908e+00  1.1336e+00  1.3561e+00  3.3011e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.794062 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1804e-07  2.2764e-06  1.7582e+00  2.2418e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002242 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
-13
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 11:20:37 GMT  run: 42981851 subRun: 171 event: 550
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981851 ; subrun: 171 ; event: 550
evttime: 376945368
run: 42981851 ; subrun: 171 ; event: 550
evttime: 376945368
########## EvtNo.550
67
2
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 8.99247 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.7497e+00  1.8096e+00  2.9480e+00  8.9920e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 9.758662 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.2073e-06  8.9176e-06  1.4534e+00  2.5466e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002548 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.2377 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9538e+00  1.4376e+00  1.7847e+00  3.7314e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.957552 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 12735.1 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.3569e+02  1.0005e+00  1.0015e+00  3.3769e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 336.025825 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6212e-05  7.6079e-06  1.2538e+00  2.7462e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002762 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
11-Nov-2024 11:21:12 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/ec/ff/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_54_20210421T011634Z.root"
11-Nov-2024 11:21:12 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/ec/ff/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981851_54_20210421T011634Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.988839       24.6592       171.074       31.5693       25.3341        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000789579    0.0676242     0.453241      0.0053322     0.105449        100    
reco:pandoracali:CalibrationdEdXPDSP           0.389958       1.56159       9.8685       0.873291       2.03419        100    
reco:pandoracalipid:Chi2ParticleID            0.00313754    0.00462131    0.00969124     0.0044553     0.0011181       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.358753      0.929483       2.80748      0.832449      0.407132        100    
reco:pandora2calipid:Chi2ParticleID           0.00586988    0.00918324     0.0154365     0.0091266    0.00176894       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000192788    0.0282865     0.293128     0.00284922     0.0647344       100    
[art]:TriggerResults:TriggerResultInserter    2.0462e-05    2.88696e-05   0.000140454    2.508e-05    1.41114e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9169e-05      22.0579       169.117       29.9807       24.9065        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.988839       24.6592       171.074       31.5693       25.3341        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000789579    0.0676242     0.453241      0.0053322     0.105449        100    
reco:pandoracali:CalibrationdEdXPDSP           0.389958       1.56159       9.8685       0.873291       2.03419        100    
reco:pandoracalipid:Chi2ParticleID            0.00313754    0.00462131    0.00969124     0.0044553     0.0011181       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.358753      0.929483       2.80748      0.832449      0.407132        100    
reco:pandora2calipid:Chi2ParticleID           0.00586988    0.00918324     0.0154365     0.0091266    0.00176894       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000192788    0.0282865     0.293128     0.00284922     0.0647344       100    
[art]:TriggerResults:TriggerResultInserter    2.0462e-05    2.88696e-05   0.000140454    2.508e-05    1.41114e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9169e-05      22.0579       169.117       29.9807       24.9065        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4712.06 MB
  Peak resident set size usage (VmHWM): 3474.87 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4712.06 MB
  Peak resident set size usage (VmHWM): 3474.87 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         53         53          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3277.071085 Real = 3869.942029

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4712.06 VmHWM = 3474.87

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END