Jobsub ID 286919.69@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286919.69@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:46:48
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:53:28
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:47:57
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_84_20210702T014618Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_50_20210702T014423Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_13_20210702T035927Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46312891_101_20210702T041256Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46319221_33_20210702T065909Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46330711_85_20210702T112639Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46341141_125_20210702T130545Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_17_20210702T182305Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45714361_86_20210702T211246Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_62_20210702T200323Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:53:28
Saved logs	justin-logs:286919.69-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

BTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46306331 ; subrun: 378 ; event: 628
evttime: 378409804
run: 46306331 ; subrun: 378 ; event: 628
evttime: 378409804
########## EvtNo.628
72
2
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1.01147 added to diagonal of error matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.66577e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.8156e+07  1.0746e+00  1.2220e+00  2.8156e+07
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 28184406.096066 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.5199e-07  1.1138e-07  1.4407e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9963e-07  1.0103e-07  1.3985e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6006e-04  4.8231e-05  1.9837e+00  2.0165e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002277 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1.00988 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.2927e-01  3.7586e-04  1.7734e+00  2.6555e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 0.431923 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1238e-05  3.0564e-04  2.1554e-03  3.9976e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004009 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.5697e-04  4.8352e-05  1.9821e+00  2.0182e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002375 TO DIAGONAL.
Getting reco beam calo
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 08:51:38 GMT run: 46306331 subRun: 378 event: 628
Invalid track range -999 return -1
%MSG
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 08:51:38 GMT run: 46306331 subRun: 378 event: 628
Invalid track range -999 return -1
%MSG
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 08:51:38 GMT run: 46306331 subRun: 378 event: 628
Invalid track range -999 return -1
%MSG
%MSG-e TrackMomentumCalculator:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 08:51:38 GMT run: 46306331 subRun: 378 event: 628
Invalid track range -999 return -1
%MSG
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:52:36 GMT  run: 46306331 subRun: 378 event: 629
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46306331 ; subrun: 378 ; event: 629
evttime: 378409805
run: 46306331 ; subrun: 378 ; event: 629
evttime: 378409805
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:52:53 GMT  run: 46306331 subRun: 378 event: 630
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46306331 ; subrun: 378 ; event: 630
evttime: 378409807
run: 46306331 ; subrun: 378 ; event: 630
evttime: 378409807
11-Nov-2024 08:52:55 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/77/f4/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_62_20210702T200323Z.root"
11-Nov-2024 08:52:55 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/77/f4/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_62_20210702T200323Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.07799       22.3918       123.486       9.23615       23.454         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000840768    0.0971277      2.28744     0.00590885     0.251508        100    
reco:pandoracali:CalibrationdEdXPDSP           0.435921       1.54023       10.1567      0.857492       1.88188        100    
reco:pandoracalipid:Chi2ParticleID            0.00322721    0.00508193    0.00896181    0.00471956    0.00120799       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.355295       1.12113       3.73055      0.823986      0.733338        100    
reco:pandora2calipid:Chi2ParticleID           0.00595162    0.00911228     0.0159234    0.00873364    0.00185056       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000264809    0.0379606     0.580144     0.00340426     0.0911536       100    
[art]:TriggerResults:TriggerResultInserter    2.1605e-05    3.33112e-05   0.000125128   2.7994e-05    1.4763e-05       100    
end_path:pduneana:PDSPAnalyzer                2.4943e-05      19.5805       112.117     6.57835e-05     23.2834        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.07799       22.3918       123.486       9.23615       23.454         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000840768    0.0971277      2.28744     0.00590885     0.251508        100    
reco:pandoracali:CalibrationdEdXPDSP           0.435921       1.54023       10.1567      0.857492       1.88188        100    
reco:pandoracalipid:Chi2ParticleID            0.00322721    0.00508193    0.00896181    0.00471956    0.00120799       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.355295       1.12113       3.73055      0.823986      0.733338        100    
reco:pandora2calipid:Chi2ParticleID           0.00595162    0.00911228     0.0159234    0.00873364    0.00185056       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000264809    0.0379606     0.580144     0.00340426     0.0911536       100    
[art]:TriggerResults:TriggerResultInserter    2.1605e-05    3.33112e-05   0.000125128   2.7994e-05    1.4763e-05       100    
end_path:pduneana:PDSPAnalyzer                2.4943e-05      19.5805       112.117     6.57835e-05     23.2834        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4659.31 MB
  Peak resident set size usage (VmHWM): 3436.22 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4659.31 MB
  Peak resident set size usage (VmHWM): 3436.22 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         46         46          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3049.398716 Real = 3826.860009

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4659.31 VmHWM = 3436.22

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END