Jobsub ID 286937.61@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286937.61@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 08:53:11
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:07:18
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 08:54:50
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_82_20210415T140050Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21631871_94_20210629T104220Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21634531_20_20210629T195039Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45714361_18_20210629T165311Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_111_20210630T022721Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_18_20210630T024011Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_75_20210630T111546Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_96_20210630T111718Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_82_20210630T145217Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_138_20210630T112504Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:07:18
Saved logs	justin-logs:286937.61-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

:
        -1.2658e-01  2.4782e-03  1.7351e+00  2.3890e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.128968 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.6693e-01  5.4329e-03  1.7957e+00  2.5658e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.369494 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.6693e-01  5.4329e-03  1.7957e+00  2.5658e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.369494 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:03:52 GMT  run: 46273041 subRun: 277 event: 1385
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 1385
evttime: 378362080
run: 46273041 ; subrun: 277 ; event: 1385
evttime: 378362080
########## EvtNo.1385
125
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1651e-01  4.0392e-03  1.7045e+00  2.4080e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.118921 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.5475e-07  5.7160e-06  1.5520e+00  2.4480e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002449 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.54707 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.3804e+00  1.2351e+00  1.5856e+00  4.5597e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3.384953 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.8594e-06  4.8760e-07  1.5515e+00  2.4485e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002451 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
3
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:04:38 GMT  run: 46273041 subRun: 277 event: 1386
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 1386
evttime: 378362081
run: 46273041 ; subrun: 277 ; event: 1386
evttime: 378362081
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:05:22 GMT  run: 46273041 subRun: 277 event: 1387
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 1387
evttime: 378362083
run: 46273041 ; subrun: 277 ; event: 1387
evttime: 378362083
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:05:45 GMT  run: 46273041 subRun: 277 event: 1388
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 1388
evttime: 378362084
run: 46273041 ; subrun: 277 ; event: 1388
evttime: 378362084
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:06:02 GMT  run: 46273041 subRun: 277 event: 1389
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 1389
evttime: 378362086
run: 46273041 ; subrun: 277 ; event: 1389
evttime: 378362086
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:06:41 GMT  run: 46273041 subRun: 277 event: 1390
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 1390
evttime: 378362088
run: 46273041 ; subrun: 277 ; event: 1390
evttime: 378362088
11-Nov-2024 10:06:44 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/0b/f9/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_138_20210630T112504Z.root"
11-Nov-2024 10:06:44 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/0b/f9/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_138_20210630T112504Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.993275       23.6544       69.2422       29.3056       21.3155        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000790997    0.155533       2.90811     0.00420882     0.427957        100    
reco:pandoracali:CalibrationdEdXPDSP           0.371172       1.88402       14.564       0.882541       2.7011         100    
reco:pandoracalipid:Chi2ParticleID            0.00275009    0.00466646    0.00858695    0.00452375    0.00103278       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.304667       1.32583       11.5733      0.762845       1.68315        100    
reco:pandora2calipid:Chi2ParticleID           0.00474932     0.008965      0.0134909    0.00885945    0.00178053       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000180091    0.0235555     0.346348     0.00315612     0.0582916       100    
[art]:TriggerResults:TriggerResultInserter    1.9858e-05    2.63838e-05   9.1862e-05    2.3472e-05    9.21717e-06      100    
end_path:pduneana:PDSPAnalyzer                1.8814e-05      20.2513       61.3372       27.5718       20.882         100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.993275       23.6544       69.2422       29.3056       21.3155        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000790997    0.155533       2.90811     0.00420882     0.427957        100    
reco:pandoracali:CalibrationdEdXPDSP           0.371172       1.88402       14.564       0.882541       2.7011         100    
reco:pandoracalipid:Chi2ParticleID            0.00275009    0.00466646    0.00858695    0.00452375    0.00103278       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.304667       1.32583       11.5733      0.762845       1.68315        100    
reco:pandora2calipid:Chi2ParticleID           0.00474932     0.008965      0.0134909    0.00885945    0.00178053       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000180091    0.0235555     0.346348     0.00315612     0.0582916       100    
[art]:TriggerResults:TriggerResultInserter    1.9858e-05    2.63838e-05   9.1862e-05    2.3472e-05    9.21717e-06      100    
end_path:pduneana:PDSPAnalyzer                1.8814e-05      20.2513       61.3372       27.5718       20.882         100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4526.67 MB
  Peak resident set size usage (VmHWM): 3435.86 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4526.67 MB
  Peak resident set size usage (VmHWM): 3435.86 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         51         51          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3010.738529 Real = 4240.631401

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4526.67 VmHWM = 3435.86

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END