Jobsub ID 94217.62@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94217.62@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 09:02:32
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:22:21
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 09:05:04
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710411_74_20210629T004937Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_37_20210630T113259Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46283101_82_20210701T023910Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46277861_126_20210701T011703Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46282661_119_20210701T023611Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46282661_30_20210701T022234Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_131_20210701T063245Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46286411_19_20210701T074804Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46285521_106_20210701T072711Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_105_20210701T062906Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:22:21
Saved logs	justin-logs:94217.62-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

.3682e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002369 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.85409 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.8037e+00  1.1507e+00  1.3712e+00  3.2818e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.806994 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9410e-07  1.7287e-06  1.6314e+00  2.3686e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002369 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:20:26 GMT  run: 45745741 subRun: 453 event: 1058
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45745741 ; subrun: 453 ; event: 1058
evttime: 378379262
run: 45745741 ; subrun: 453 ; event: 1058
evttime: 378379262
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:20:42 GMT  run: 45745741 subRun: 453 event: 1059
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45745741 ; subrun: 453 ; event: 1059
evttime: 378379264
run: 45745741 ; subrun: 453 ; event: 1059
evttime: 378379264
########## EvtNo.1059
42
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7064e-01  6.5111e-03  1.7709e+00  2.3932e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.173030 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.4660e-05  6.2497e-06  1.6440e+00  2.3561e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002451 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.6055 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.2849e+00  1.1603e+00  1.4207e+00  3.7039e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.288651 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1750.09 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5914e+01  1.0003e+00  1.0031e+00  1.7911e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 15.932154 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.0579e-04  2.1306e-05  1.4846e+00  2.5156e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002721 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:21:41 GMT  run: 45745741 subRun: 453 event: 1060
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45745741 ; subrun: 453 ; event: 1060
evttime: 378379265
run: 45745741 ; subrun: 453 ; event: 1060
evttime: 378379265
11-Nov-2024 10:21:44 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/86/1b/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_105_20210701T062906Z.root"
11-Nov-2024 10:21:44 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/86/1b/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_105_20210701T062906Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.904038       22.7765       81.4698       8.90198       22.4969        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000772413     0.1933        3.79824      0.004367      0.580796        100    
reco:pandoracali:CalibrationdEdXPDSP           0.434448       2.19591       17.7809       1.02567       3.0144         100    
reco:pandoracalipid:Chi2ParticleID            0.00309375    0.00460222    0.00737748    0.00456639    0.000793673      100    
reco:pandora2cali:CalibrationdEdXPDSP          0.374059       1.62968       15.4767      0.942278       2.06913        100    
reco:pandora2calipid:Chi2ParticleID           0.00612115    0.00916613     0.0141639    0.00878441    0.00173753       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000196459    0.0451074     0.852369     0.00296821     0.125157        100    
[art]:TriggerResults:TriggerResultInserter    2.0478e-05    3.0536e-05    8.8284e-05    2.46555e-05   1.36763e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9141e-05      18.6982       59.7815     4.01755e-05     21.8372        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.904038       22.7765       81.4698       8.90198       22.4969        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000772413     0.1933        3.79824      0.004367      0.580796        100    
reco:pandoracali:CalibrationdEdXPDSP           0.434448       2.19591       17.7809       1.02567       3.0144         100    
reco:pandoracalipid:Chi2ParticleID            0.00309375    0.00460222    0.00737748    0.00456639    0.000793673      100    
reco:pandora2cali:CalibrationdEdXPDSP          0.374059       1.62968       15.4767      0.942278       2.06913        100    
reco:pandora2calipid:Chi2ParticleID           0.00612115    0.00916613     0.0141639    0.00878441    0.00173753       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000196459    0.0451074     0.852369     0.00296821     0.125157        100    
[art]:TriggerResults:TriggerResultInserter    2.0478e-05    3.0536e-05    8.8284e-05    2.46555e-05   1.36763e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9141e-05      18.6982       59.7815     4.01755e-05     21.8372        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4617.14 MB
  Peak resident set size usage (VmHWM): 3520.54 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4617.14 MB
  Peak resident set size usage (VmHWM): 3520.54 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         44         44          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2849.198858 Real = 4528.629039

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4617.14 VmHWM = 3520.54

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END