Jobsub ID 94172.128@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94172.128@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 06:22:13
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 07:29:54
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 06:24:43
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_120_20210415T231829Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_25_20210416T013753Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_121_20210415T231830Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_81_20210415T230801Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_35_20210416T014721Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42934371_85_20210415T230936Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42939311_62_20210416T061612Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42961101_117_20210416T095351Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42961101_116_20210416T095315Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_113_20210416T164426Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 07:29:54
Saved logs	justin-logs:94172.128-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

ckTrackerService:  ProcessEvent 11-Nov-2024 07:26:51 GMT  run: 42981491 subRun: 166 event: 1137
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981491 ; subrun: 166 ; event: 1137
evttime: 376858333
run: 42981491 ; subrun: 166 ; event: 1137
evttime: 376858333
########## EvtNo.1137
48
2
 MINUIT WARNING IN HESSE   
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN HESSE   
 ============== 1.03146 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.0992e-05  1.5786e-07  1.6607e+00  2.3393e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002360 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.00298 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -8.7402e-01  1.0612e+00  1.2142e+00  2.5986e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 0.876618 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.2845e-04  8.2642e-07  1.6613e+00  2.3388e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002467 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:27:54 GMT  run: 42981491 subRun: 166 event: 1138
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981491 ; subrun: 166 ; event: 1138
evttime: 376858334
run: 42981491 ; subrun: 166 ; event: 1138
evttime: 376858334
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:28:17 GMT  run: 42981491 subRun: 166 event: 1139
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981491 ; subrun: 166 ; event: 1139
evttime: 376858335
run: 42981491 ; subrun: 166 ; event: 1139
evttime: 376858335
########## EvtNo.1139
5
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.5627e-02  2.4897e-07  1.7756e+00  2.2600e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.037887 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.4856e-03 -2.1231e-06  1.7580e+00  2.2485e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.008734 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 359.888 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.2088e+00  9.8849e-01  1.0119e+00  7.2084e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 5.215993 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.7075e-02  1.8007e-06  1.7755e+00  2.2615e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.039337 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7021e-02 -8.4798e-07  1.7641e+00  2.2529e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.019274 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 07:29:14 GMT  run: 42981491 subRun: 166 event: 1140
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 42981491 ; subrun: 166 ; event: 1140
evttime: 376858336
run: 42981491 ; subrun: 166 ; event: 1140
evttime: 376858336
11-Nov-2024 07:29:17 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/7a/4a/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_113_20210416T164426Z.root"
11-Nov-2024 07:29:17 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/7a/4a/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_42981491_113_20210416T164426Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.07774       20.8701       57.8643       4.89591       21.1772        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00108628     0.0932627      1.69603     0.00663058     0.199285        100    
reco:pandoracali:CalibrationdEdXPDSP           0.354177       1.71092       7.78502       1.03417       1.76569        100    
reco:pandoracalipid:Chi2ParticleID            0.00320037     0.0048768    0.00980793    0.00460452    0.00111506       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.421695       1.13437       4.62246      0.997983      0.675376        100    
reco:pandora2calipid:Chi2ParticleID           0.00576963    0.00918997     0.0129502    0.00901896    0.00159272       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000319597    0.0426754     0.371376     0.00380544     0.0808719       100    
[art]:TriggerResults:TriggerResultInserter    2.0626e-05    3.49459e-05   0.000160323    2.834e-05    1.82774e-05      100    
end_path:pduneana:PDSPAnalyzer                2.5172e-05      17.8742       55.8459     5.90655e-05     21.1828        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.07774       20.8701       57.8643       4.89591       21.1772        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00108628     0.0932627      1.69603     0.00663058     0.199285        100    
reco:pandoracali:CalibrationdEdXPDSP           0.354177       1.71092       7.78502       1.03417       1.76569        100    
reco:pandoracalipid:Chi2ParticleID            0.00320037     0.0048768    0.00980793    0.00460452    0.00111506       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.421695       1.13437       4.62246      0.997983      0.675376        100    
reco:pandora2calipid:Chi2ParticleID           0.00576963    0.00918997     0.0129502    0.00901896    0.00159272       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000319597    0.0426754     0.371376     0.00380544     0.0808719       100    
[art]:TriggerResults:TriggerResultInserter    2.0626e-05    3.49459e-05   0.000160323    2.834e-05    1.82774e-05      100    
end_path:pduneana:PDSPAnalyzer                2.5172e-05      17.8742       55.8459     5.90655e-05     21.1828        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4829.65 MB
  Peak resident set size usage (VmHWM): 3738.14 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4829.65 MB
  Peak resident set size usage (VmHWM): 3738.14 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         43         43          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2927.502104 Real = 3778.571673

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4829.65 VmHWM = 3738.14

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END