Jobsub ID 94193.144@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94193.144@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:45:18
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:54:18
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:47:32
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_13_20210630T023828Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_93_20210630T025316Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_13_20210630T010004Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46287381_4_20210701T080402Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_144_20210701T063512Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_86_20210701T062514Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46297271_89_20210701T122938Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46297271_131_20210701T130200Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_1_20210701T200140Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_98_20210701T200154Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:54:18
Saved logs	justin-logs:94193.144-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

0 event: 987
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46304361 ; subrun: 360 ; event: 987
evttime: 378389413
run: 46304361 ; subrun: 360 ; event: 987
evttime: 378389413
########## EvtNo.987
61
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4124e-01  1.6245e-03  1.8652e+00  2.2745e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.143510 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 22.2623 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4900e+00  9.3350e-01  1.0675e+00  3.4890e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.493484 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.2836e-05 -1.3028e-07  1.7765e+00  2.2236e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002276 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1225e-01  1.2902e-03  1.8535e+00  2.2574e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.114505 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2905.22 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0004e+01  1.0000e+00  1.0014e+00  1.2002e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 10.015946 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 16877.1 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.2138e+01  1.0001e+00  1.0014e+00  3.4136e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 32.171814 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.7747e-05  2.1575e-04  1.4913e+00  2.5086e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002606 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:52:23 GMT  run: 46304361 subRun: 360 event: 988
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46304361 ; subrun: 360 ; event: 988
evttime: 378389414
run: 46304361 ; subrun: 360 ; event: 988
evttime: 378389414
########## EvtNo.988
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
4
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:53:21 GMT  run: 46304361 subRun: 360 event: 989
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46304361 ; subrun: 360 ; event: 989
evttime: 378389415
run: 46304361 ; subrun: 360 ; event: 989
evttime: 378389415
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:53:40 GMT  run: 46304361 subRun: 360 event: 990
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46304361 ; subrun: 360 ; event: 990
evttime: 378389416
run: 46304361 ; subrun: 360 ; event: 990
evttime: 378389416
11-Nov-2024 08:53:43 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/ac/67/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_98_20210701T200154Z.root"
11-Nov-2024 08:53:43 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/ac/67/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_98_20210701T200154Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.960171       21.8154       81.7959       6.9678        21.7206        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000863437    0.155348       4.4305      0.00587753     0.536061        100    
reco:pandoracali:CalibrationdEdXPDSP           0.360578       1.48154       21.0398      0.856804       2.48309        100    
reco:pandoracalipid:Chi2ParticleID            0.00303694    0.00484993    0.00881371    0.00469136    0.00109037       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.401218       1.0466        5.14635      0.845714      0.707544        100    
reco:pandora2calipid:Chi2ParticleID           0.00596837     0.0094613     0.013506     0.00945528    0.00169651       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00020376     0.0370757      1.25848      0.0030351     0.138378        100    
[art]:TriggerResults:TriggerResultInserter    2.0363e-05    2.86337e-05   9.9164e-05    2.5301e-05     1.079e-05       100    
end_path:pduneana:PDSPAnalyzer                1.8762e-05       19.08        69.5012     4.2564e-05      21.0413        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.960171       21.8154       81.7959       6.9678        21.7206        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000863437    0.155348       4.4305      0.00587753     0.536061        100    
reco:pandoracali:CalibrationdEdXPDSP           0.360578       1.48154       21.0398      0.856804       2.48309        100    
reco:pandoracalipid:Chi2ParticleID            0.00303694    0.00484993    0.00881371    0.00469136    0.00109037       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.401218       1.0466        5.14635      0.845714      0.707544        100    
reco:pandora2calipid:Chi2ParticleID           0.00596837     0.0094613     0.013506     0.00945528    0.00169651       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00020376     0.0370757      1.25848      0.0030351     0.138378        100    
[art]:TriggerResults:TriggerResultInserter    2.0363e-05    2.86337e-05   9.9164e-05    2.5301e-05     1.079e-05       100    
end_path:pduneana:PDSPAnalyzer                1.8762e-05       19.08        69.5012     4.2564e-05      21.0413        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4633.34 MB
  Peak resident set size usage (VmHWM): 3503.53 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4633.34 MB
  Peak resident set size usage (VmHWM): 3503.53 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         47         47          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2984.137554 Real = 3898.903013

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4633.34 VmHWM = 3503.53

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END