Jobsub ID 94193.135@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94193.135@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:45:18
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:54:01
From worker node	Hostname	r22-n15.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:47:18
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42929541_44_20210415T080806Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42920571_31_20210415T044826Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510331_117_20210415T080453Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510331_27_20210415T073345Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42929541_65_20210415T081224Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_50_20210415T071045Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42929541_122_20210415T082056Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42936861_17_20210415T112103Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_82_20210415T140050Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_118_20210415T094927Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:54:01
Saved logs	justin-logs:94193.135-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

IT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3852602.956716 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9228e-07  6.4654e-08  1.4000e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.8016e-07  6.5900e-08  1.3841e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1.15798e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0071e+06  1.0001e+00  1.0010e+00  1.0071e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1008121.158723 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.2163e-07  1.0061e-06  1.4793e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.0401e-07  9.7438e-07  1.4880e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:52:13 GMT  run: 43511441 subRun: 100 event: 1188
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43511441 ; subrun: 100 ; event: 1188
evttime: 376831843
run: 43511441 ; subrun: 100 ; event: 1188
evttime: 376831843
########## EvtNo.1188
21
2
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 94.2717 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.8325e+01  1.5017e+00  2.1801e+00  1.8643e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 18.343559 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 75.7887 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4573e+01  1.3256e+00  1.8249e+00  1.5422e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 14.587992 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:53:05 GMT  run: 43511441 subRun: 100 event: 1189
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43511441 ; subrun: 100 ; event: 1189
evttime: 376831845
run: 43511441 ; subrun: 100 ; event: 1189
evttime: 376831845
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:53:24 GMT  run: 43511441 subRun: 100 event: 1190
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 43511441 ; subrun: 100 ; event: 1190
evttime: 376831847
run: 43511441 ; subrun: 100 ; event: 1190
evttime: 376831847
11-Nov-2024 08:53:26 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/f8/77/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_118_20210415T094927Z.root"
11-Nov-2024 08:53:26 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/f8/77/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43511441_118_20210415T094927Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.809236       23.2508       68.8164       30.1157       20.9842        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000916891    0.0545066     0.400797     0.00497501     0.0812106       100    
reco:pandoracali:CalibrationdEdXPDSP            0.39052       1.57638       10.5818      0.872274       1.99921        100    
reco:pandoracalipid:Chi2ParticleID            0.00278082    0.00492235    0.00876724    0.00464962    0.00113365       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.371039       1.09325       4.55084      0.820661      0.741446        100    
reco:pandora2calipid:Chi2ParticleID            0.0056372    0.00945108     0.016405     0.00928841    0.00210042       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000235686    0.0284303     0.294932     0.00357737     0.0663325       100    
[art]:TriggerResults:TriggerResultInserter    1.9768e-05    2.92775e-05   0.000127881    2.423e-05    1.42929e-05      100    
end_path:pduneana:PDSPAnalyzer                2.0042e-05      20.4833       60.177        28.5793       20.459         100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.809236       23.2508       68.8164       30.1157       20.9842        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000916891    0.0545066     0.400797     0.00497501     0.0812106       100    
reco:pandoracali:CalibrationdEdXPDSP            0.39052       1.57638       10.5818      0.872274       1.99921        100    
reco:pandoracalipid:Chi2ParticleID            0.00278082    0.00492235    0.00876724    0.00464962    0.00113365       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.371039       1.09325       4.55084      0.820661      0.741446        100    
reco:pandora2calipid:Chi2ParticleID            0.0056372    0.00945108     0.016405     0.00928841    0.00210042       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000235686    0.0284303     0.294932     0.00357737     0.0663325       100    
[art]:TriggerResults:TriggerResultInserter    1.9768e-05    2.92775e-05   0.000127881    2.423e-05    1.42929e-05      100    
end_path:pduneana:PDSPAnalyzer                2.0042e-05      20.4833       60.177        28.5793       20.459         100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4626.4 MB
  Peak resident set size usage (VmHWM): 3336.09 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4626.4 MB
  Peak resident set size usage (VmHWM): 3336.09 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         55         55          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3073.464334 Real = 3894.333480

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4626.4 VmHWM = 3336.09

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END