Jobsub ID 94196.148@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94196.148@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:54:14
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 09:03:10
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:56:17
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_135_20210416T164731Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42961101_18_20210416T094134Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_53_20210416T163930Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42961101_13_20210416T093919Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_110_20210416T145612Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_51_20210416T145603Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586751_11_20210416T154129Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_119_20210416T145612Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586281_80_20210416T145609Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586751_23_20210416T154136Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 09:03:10
Saved logs	justin-logs:94196.148-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
0
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:00:24 GMT  run: 43586751 subRun: 193 event: 236
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586751 ; subrun: 193 ; event: 236
evttime: 376856849
run: 43586751 ; subrun: 193 ; event: 236
evttime: 376856849
########## EvtNo.236
133
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.2979e-01  4.1756e-03  1.8125e+00  2.3131e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.132104 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 16625.7 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.6064e+01  1.0001e+00  1.0003e+00  7.8063e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 76.141572 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.3413 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.5163e+00  1.2068e+00  1.5434e+00  4.7662e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3.521105 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7211.69 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.9402e+01  1.0001e+00  1.0003e+00  5.1402e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 49.453720 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:01:23 GMT  run: 43586751 subRun: 193 event: 237
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586751 ; subrun: 193 ; event: 237
evttime: 376856851
run: 43586751 ; subrun: 193 ; event: 237
evttime: 376856851
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:01:43 GMT  run: 43586751 subRun: 193 event: 238
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586751 ; subrun: 193 ; event: 238
evttime: 376856852
run: 43586751 ; subrun: 193 ; event: 238
evttime: 376856852
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:02:09 GMT  run: 43586751 subRun: 193 event: 239
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586751 ; subrun: 193 ; event: 239
evttime: 376856854
run: 43586751 ; subrun: 193 ; event: 239
evttime: 376856854
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:02:34 GMT  run: 43586751 subRun: 193 event: 240
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43586751 ; subrun: 193 ; event: 240
evttime: 376856856
run: 43586751 ; subrun: 193 ; event: 240
evttime: 376856856
11-Nov-2024 09:02:37 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/f3/da/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586751_23_20210416T154136Z.root"
11-Nov-2024 09:02:37 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/f3/da/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43586751_23_20210416T154136Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.8007        24.0931       60.5967       31.6306       20.7269        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000969519    0.0993663      1.94002     0.00436505     0.249892        100    
reco:pandoracali:CalibrationdEdXPDSP           0.333968       1.5389        12.7792      0.831004       2.09175        100    
reco:pandoracalipid:Chi2ParticleID            0.00265758    0.00462397    0.00832137    0.00437463    0.00113455       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.386437       1.04849       3.5723       0.862731      0.676853        100    
reco:pandora2calipid:Chi2ParticleID           0.00533094    0.00904286     0.017153     0.00878504    0.00206851       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00020959     0.0340474     0.408235     0.00312996     0.0760972       100    
[art]:TriggerResults:TriggerResultInserter    1.9581e-05    2.88974e-05   9.8676e-05    2.35825e-05   1.35631e-05      100    
end_path:pduneana:PDSPAnalyzer                1.7974e-05      21.3582       59.2713       29.9289       20.5133        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.8007        24.0931       60.5967       31.6306       20.7269        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000969519    0.0993663      1.94002     0.00436505     0.249892        100    
reco:pandoracali:CalibrationdEdXPDSP           0.333968       1.5389        12.7792      0.831004       2.09175        100    
reco:pandoracalipid:Chi2ParticleID            0.00265758    0.00462397    0.00832137    0.00437463    0.00113455       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.386437       1.04849       3.5723       0.862731      0.676853        100    
reco:pandora2calipid:Chi2ParticleID           0.00533094    0.00904286     0.017153     0.00878504    0.00206851       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00020959     0.0340474     0.408235     0.00312996     0.0760972       100    
[art]:TriggerResults:TriggerResultInserter    1.9581e-05    2.88974e-05   9.8676e-05    2.35825e-05   1.35631e-05      100    
end_path:pduneana:PDSPAnalyzer                1.7974e-05      21.3582       59.2713       29.9289       20.5133        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4389.39 MB
  Peak resident set size usage (VmHWM): 3120.56 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4389.39 MB
  Peak resident set size usage (VmHWM): 3120.56 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         55         55          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3150.129505 Real = 3909.957572

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4389.39 VmHWM = 3120.56

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END