Jobsub ID 286906.14@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286906.14@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:49:51
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:03:54
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:51:11
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_91_20210630T111706Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_111_20210630T115502Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_108_20210630T145455Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_68_20210630T111435Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_107_20210630T111833Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_14_20210630T144840Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_30_20210630T144910Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_69_20210630T145218Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_123_20210630T170820Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_111_20210630T170831Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:03:54
Saved logs	justin-logs:286906.14-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

OND-DERIVATIVE MATRIX:
        -2.3512e+00  1.0708e+00  1.2149e+00  4.0655e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.355225 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.68321e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.2935e+05  1.0000e+00  1.0000e+00  5.2935e+05
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 529875.865210 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:02:17 GMT  run: 45738751 subRun: 418 event: 1119
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45738751 ; subrun: 418 ; event: 1119
evttime: 378366828
run: 45738751 ; subrun: 418 ; event: 1119
evttime: 378366828
########## EvtNo.1119
59
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7612e-01  6.0456e-03  1.7783e+00  2.3918e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.178513 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.44304 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5524e+00  8.0020e-01  1.2020e+00  3.5502e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.555992 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.5440e-05  2.3532e-06  1.6338e+00  2.3662e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002402 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.14281 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.0431e+00  1.1405e+00  1.3752e+00  3.5275e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.046636 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5590.84 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9296e+01  1.0002e+00  1.0016e+00  3.1294e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 29.327494 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:03:19 GMT  run: 45738751 subRun: 418 event: 1120
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45738751 ; subrun: 418 ; event: 1120
evttime: 378366830
run: 45738751 ; subrun: 418 ; event: 1120
evttime: 378366830
11-Nov-2024 08:03:21 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/a0/dc/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_111_20210630T170831Z.root"
11-Nov-2024 08:03:21 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/a0/dc/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_111_20210630T170831Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.05742       26.575        63.3399       34.1458       21.9987        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000744032    0.127319       2.05744      0.0047927     0.312444        100    
reco:pandoracali:CalibrationdEdXPDSP           0.407022       1.60999        12.61        1.01267       1.81743        100    
reco:pandoracalipid:Chi2ParticleID            0.00321508    0.00486232    0.00989967    0.00476927    0.00108586       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.356849       1.31329       4.62798      0.969142      0.922793        100    
reco:pandora2calipid:Chi2ParticleID           0.00640285    0.00978994     0.0167516    0.00947181     0.0020684       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000248472    0.0423423     0.382099     0.00366869     0.0779934       100    
[art]:TriggerResults:TriggerResultInserter    2.0547e-05    3.20935e-05   0.000128971   2.6824e-05    1.48758e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9569e-05      23.4669       61.0244       31.3699       21.8857        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.05742       26.575        63.3399       34.1458       21.9987        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000744032    0.127319       2.05744      0.0047927     0.312444        100    
reco:pandoracali:CalibrationdEdXPDSP           0.407022       1.60999        12.61        1.01267       1.81743        100    
reco:pandoracalipid:Chi2ParticleID            0.00321508    0.00486232    0.00989967    0.00476927    0.00108586       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.356849       1.31329       4.62798      0.969142      0.922793        100    
reco:pandora2calipid:Chi2ParticleID           0.00640285    0.00978994     0.0167516    0.00947181     0.0020684       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000248472    0.0423423     0.382099     0.00366869     0.0779934       100    
[art]:TriggerResults:TriggerResultInserter    2.0547e-05    3.20935e-05   0.000128971   2.6824e-05    1.48758e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9569e-05      23.4669       61.0244       31.3699       21.8857        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4620.95 MB
  Peak resident set size usage (VmHWM): 3523.88 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4620.95 MB
  Peak resident set size usage (VmHWM): 3523.88 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         57         57          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3370.522514 Real = 4260.943296

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4620.95 VmHWM = 3523.88

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END