Jobsub ID 286898.109@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286898.109@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:23:47
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:34:21
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:24:53
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_57_20210629T224104Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21640501_56_20210630T051036Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_121_20210630T114736Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_90_20210630T145412Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46285521_27_20210701T071336Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_105_20210701T062906Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45745741_37_20210701T061052Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46287381_54_20210701T080746Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_68_20210701T200154Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_102_20210702T014633Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:34:21
Saved logs	justin-logs:286898.109-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

AL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 3277.37 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.9448e+01  1.0002e+00  1.0025e+00  2.1445e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 19.469498 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:32:53 GMT  run: 46306331 subRun: 378 event: 1029
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46306331 ; subrun: 378 ; event: 1029
evttime: 378396059
run: 46306331 ; subrun: 378 ; event: 1029
evttime: 378396059
########## EvtNo.1029
142
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1868e-01  5.4179e-03  1.7346e+00  2.3787e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.121062 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.3089e-07  7.4468e-06  1.5975e+00  2.4025e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002403 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 4.6691e+06 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1148e+02  1.0000e+00  1.0002e+00  1.1348e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 111.590290 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.50389 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.0251e+00  1.2601e+00  1.7090e+00  5.0560e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.030159 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.9380e-07  5.9157e-06  1.5973e+00  2.4027e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002403 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 607046 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.8895e+01  1.0000e+00  1.0002e+00  4.0894e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 38.935522 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
3
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:33:41 GMT  run: 46306331 subRun: 378 event: 1030
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46306331 ; subrun: 378 ; event: 1030
evttime: 378396060
run: 46306331 ; subrun: 378 ; event: 1030
evttime: 378396060
11-Nov-2024 07:33:44 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/1e/6b/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_102_20210702T014633Z.root"
11-Nov-2024 07:33:44 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/1e/6b/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_102_20210702T014633Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.876218       25.7705       65.0126       34.3519       21.6797        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000746517    0.0919467      1.04593     0.00473798     0.174437        100    
reco:pandoracali:CalibrationdEdXPDSP           0.413976       1.59226       9.98514      0.801605       2.08814        100    
reco:pandoracalipid:Chi2ParticleID            0.00289535    0.00472235    0.00932534    0.00456574    0.00106516       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.315849       1.19068       5.61049      0.835165      0.994463        100    
reco:pandora2calipid:Chi2ParticleID           0.00571953    0.00915172     0.0150756    0.00896733    0.00177241       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00022575     0.0308653     0.537147     0.00306089     0.0849181       100    
[art]:TriggerResults:TriggerResultInserter    2.0136e-05    2.90603e-05   0.000123741   2.41385e-05   1.3484e-05       100    
end_path:pduneana:PDSPAnalyzer                2.0181e-05      22.8503       62.7323       30.7387       21.2646        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.876218       25.7705       65.0126       34.3519       21.6797        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000746517    0.0919467      1.04593     0.00473798     0.174437        100    
reco:pandoracali:CalibrationdEdXPDSP           0.413976       1.59226       9.98514      0.801605       2.08814        100    
reco:pandoracalipid:Chi2ParticleID            0.00289535    0.00472235    0.00932534    0.00456574    0.00106516       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.315849       1.19068       5.61049      0.835165      0.994463        100    
reco:pandora2calipid:Chi2ParticleID           0.00571953    0.00915172     0.0150756    0.00896733    0.00177241       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.00022575     0.0308653     0.537147     0.00306089     0.0849181       100    
[art]:TriggerResults:TriggerResultInserter    2.0136e-05    2.90603e-05   0.000123741   2.41385e-05   1.3484e-05       100    
end_path:pduneana:PDSPAnalyzer                2.0181e-05      22.8503       62.7323       30.7387       21.2646        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4809.38 MB
  Peak resident set size usage (VmHWM): 3659.24 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4809.38 MB
  Peak resident set size usage (VmHWM): 3659.24 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         56         56          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3296.536075 Real = 4051.983386

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4809.38 VmHWM = 3659.24

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END