Jobsub ID 94188.53@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94188.53@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:25:56
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:33:58
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:27:38
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_41_20210630T144942Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_69_20210630T111434Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_115_20210630T114732Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_135_20210630T115000Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21641521_4_20210630T112644Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_123_20210630T145918Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_10_20210630T144839Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_0_20210630T164335Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_106_20210630T170801Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_35_20210630T165824Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:33:58
Saved logs	justin-logs:94188.53-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

egative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.98611e+14 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1092e+06  1.0000e+00  1.0003e+00  1.1092e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1110320.278142 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:32:08 GMT  run: 45738751 subRun: 418 event: 359
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45738751 ; subrun: 418 ; event: 359
evttime: 378366713
run: 45738751 ; subrun: 418 ; event: 359
evttime: 378366713
########## EvtNo.359
74
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0593e-01  3.0581e-03  1.7385e+00  2.3643e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.108294 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.5067e-07  1.1034e-05  1.5030e+00  2.4970e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002498 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.25537 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.3654e+00  1.2077e+00  1.6155e+00  6.5422e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 5.371916 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0606e-06  4.2778e-06  1.5029e+00  2.4971e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002498 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:32:54 GMT  run: 45738751 subRun: 418 event: 360
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45738751 ; subrun: 418 ; event: 360
evttime: 378366715
run: 45738751 ; subrun: 418 ; event: 360
evttime: 378366715
########## EvtNo.360
59
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.2889e-02  2.1295e-03  1.6892e+00  2.3715e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.065260 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.4576e-06  2.7609e-06  1.4818e+00  2.5182e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002523 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.41704 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.5584e+00  1.2594e+00  1.7818e+00  5.5172e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 4.563909 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.2232e-05 -2.3144e-07  1.4816e+00  2.5184e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002541 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
11-Nov-2024 08:33:24 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/62/28/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_35_20210630T165824Z.root"
11-Nov-2024 08:33:24 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/62/28/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_35_20210630T165824Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.800723       21.9452       66.1314       8.46181       21.2587        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000889285    0.174267       5.09635     0.00458622     0.558005        100    
reco:pandoracali:CalibrationdEdXPDSP           0.379686       1.6276        13.711       0.874918       2.04955        100    
reco:pandoracalipid:Chi2ParticleID            0.00312066    0.00471158     0.011287     0.00441661    0.00122655       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.401319       1.16817       6.99078      0.822914      0.979697        100    
reco:pandora2calipid:Chi2ParticleID           0.00547964    0.00924856     0.0156676    0.00893491    0.00192172       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000218211    0.0320775     0.405595      0.0032313     0.0728932       100    
[art]:TriggerResults:TriggerResultInserter    2.0534e-05    3.0245e-05    9.3602e-05    2.55105e-05   1.28941e-05      100    
end_path:pduneana:PDSPAnalyzer                 1.977e-05      18.9286       64.5319     5.2852e-05      21.0074        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.800723       21.9452       66.1314       8.46181       21.2587        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000889285    0.174267       5.09635     0.00458622     0.558005        100    
reco:pandoracali:CalibrationdEdXPDSP           0.379686       1.6276        13.711       0.874918       2.04955        100    
reco:pandoracalipid:Chi2ParticleID            0.00312066    0.00471158     0.011287     0.00441661    0.00122655       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.401319       1.16817       6.99078      0.822914      0.979697        100    
reco:pandora2calipid:Chi2ParticleID           0.00547964    0.00924856     0.0156676    0.00893491    0.00192172       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000218211    0.0320775     0.405595      0.0032313     0.0728932       100    
[art]:TriggerResults:TriggerResultInserter    2.0534e-05    3.0245e-05    9.3602e-05    2.55105e-05   1.28941e-05      100    
end_path:pduneana:PDSPAnalyzer                 1.977e-05      18.9286       64.5319     5.2852e-05      21.0074        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4638.4 MB
  Peak resident set size usage (VmHWM): 3556.57 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4638.4 MB
  Peak resident set size usage (VmHWM): 3556.57 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         47         47          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2908.986263 Real = 3873.615825

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4638.4 VmHWM = 3556.57

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END