Jobsub ID 94188.104@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94188.104@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:25:56
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:33:16
From worker node	Hostname	r22-n15.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:27:29
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_62_20210629T224222Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_119_20210629T225055Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_34_20210630T024931Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_139_20210630T040947Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_34_20210630T020713Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_72_20210630T024949Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718611_74_20210630T022400Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_86_20210630T111704Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_90_20210630T111705Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_10_20210630T110010Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:33:16
Saved logs	justin-logs:94188.104-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

ENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0460e+00  8.2897e-01  1.1896e+00  3.0274e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.048995 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002428 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.56316 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6650e+00  1.1635e+00  1.4364e+00  4.0651e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.669024 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6568e-06  8.7107e-06  1.1809e+00  2.8191e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002822 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:30:52 GMT  run: 46273041 subRun: 277 event: 107
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 107
evttime: 378363694
run: 46273041 ; subrun: 277 ; event: 107
evttime: 378363694
########## EvtNo.107
123
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.1381e-01  4.2719e-03  1.7280e+00  2.3815e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.116188 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -8.6548e-07  3.4255e-06  1.5530e+00  2.4470e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002448 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.10823 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.5742e+00  1.2509e+00  1.6275e+00  4.6958e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3.578939 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.8453e-06  2.6569e-07  1.5523e+00  2.4477e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002452 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:31:33 GMT  run: 46273041 subRun: 277 event: 108
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 108
evttime: 378363695
run: 46273041 ; subrun: 277 ; event: 108
evttime: 378363695
########## EvtNo.108
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
1
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:32:20 GMT  run: 46273041 subRun: 277 event: 109
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 109
evttime: 378363696
run: 46273041 ; subrun: 277 ; event: 109
evttime: 378363696
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:32:37 GMT  run: 46273041 subRun: 277 event: 110
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46273041 ; subrun: 277 ; event: 110
evttime: 378363697
run: 46273041 ; subrun: 277 ; event: 110
evttime: 378363697
11-Nov-2024 08:32:42 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/b5/dd/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_10_20210630T110010Z.root"
11-Nov-2024 08:32:42 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/b5/dd/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_10_20210630T110010Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.03112       21.4457       63.632        10.1076       20.4455        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000786618    0.325863       19.4822     0.00575589      1.96025        100    
reco:pandoracali:CalibrationdEdXPDSP           0.374924       1.68116       15.2343      0.897999       2.26807        100    
reco:pandoracalipid:Chi2ParticleID            0.00310892     0.004696     0.00934933    0.00443547    0.00109129       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.403058       1.53534       19.5864      0.875061       2.11939        100    
reco:pandora2calipid:Chi2ParticleID           0.00589258     0.009217      0.0188181    0.00890428    0.00187433       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000236447     0.03372      0.409065     0.00290816     0.0736552       100    
[art]:TriggerResults:TriggerResultInserter     2.016e-05    2.86233e-05   9.9389e-05    2.3482e-05    1.29266e-05      100    
end_path:pduneana:PDSPAnalyzer                2.1209e-05      17.8551       62.1248     5.87675e-05     20.4859        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.03112       21.4457       63.632        10.1076       20.4455        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000786618    0.325863       19.4822     0.00575589      1.96025        100    
reco:pandoracali:CalibrationdEdXPDSP           0.374924       1.68116       15.2343      0.897999       2.26807        100    
reco:pandoracalipid:Chi2ParticleID            0.00310892     0.004696     0.00934933    0.00443547    0.00109129       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.403058       1.53534       19.5864      0.875061       2.11939        100    
reco:pandora2calipid:Chi2ParticleID           0.00589258     0.009217      0.0188181    0.00890428    0.00187433       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000236447     0.03372      0.409065     0.00290816     0.0736552       100    
[art]:TriggerResults:TriggerResultInserter     2.016e-05    2.86233e-05   9.9389e-05    2.3482e-05    1.29266e-05      100    
end_path:pduneana:PDSPAnalyzer                2.1209e-05      17.8551       62.1248     5.87675e-05     20.4859        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4876.1 MB
  Peak resident set size usage (VmHWM): 3688.3 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4876.1 MB
  Peak resident set size usage (VmHWM): 3688.3 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         45         45          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2840.409297 Real = 3844.217495

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4876.1 VmHWM = 3688.3

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END