Jobsub ID 94196.177@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94196.177@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 07:54:14
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 08:58:40
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 07:56:38
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45714361_0_20210629T165137Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_100_20210629T224655Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270331_62_20210629T193242Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_109_20210629T224657Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45714361_42_20210629T165524Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45717331_121_20210629T224441Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45717331_117_20210629T224008Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_104_20210630T025651Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45717331_115_20210629T224007Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_47_20210630T043603Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 08:58:40
Saved logs	justin-logs:94196.177-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

1.7158e+00  2.3981e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.120438 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.9569e-07  5.7511e-06  1.5638e+00  2.4362e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002437 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.2921 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.4383e+00  1.2356e+00  1.5861e+00  4.6165e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3.442908 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6945e-06  1.8100e-06  1.5634e+00  2.4366e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002438 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:56:58 GMT  run: 45718791 subRun: 369 event: 477
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718791 ; subrun: 369 ; event: 477
evttime: 378356421
run: 45718791 ; subrun: 369 ; event: 477
evttime: 378356421
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:57:09 GMT  run: 45718791 subRun: 369 event: 478
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718791 ; subrun: 369 ; event: 478
evttime: 378356422
run: 45718791 ; subrun: 369 ; event: 478
evttime: 378356422
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:57:24 GMT  run: 45718791 subRun: 369 event: 479
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718791 ; subrun: 369 ; event: 479
evttime: 378356423
run: 45718791 ; subrun: 369 ; event: 479
evttime: 378356423
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 08:57:35 GMT  run: 45718791 subRun: 369 event: 480
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 45718791 ; subrun: 369 ; event: 480
evttime: 378356424
run: 45718791 ; subrun: 369 ; event: 480
evttime: 378356424
########## EvtNo.480
113
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.6246e-01  1.5170e-02  1.8355e+00  2.6118e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.465071 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.5202e-07  8.7156e-06  1.5572e+00  2.4428e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002443 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002015 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.75712 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4212e+00  1.1125e+00  1.3089e+00  2.9998e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.424169 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5876e-07  4.4879e-06  1.5569e+00  2.4431e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002443 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
11-Nov-2024 08:58:07 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/08/b2/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_47_20210630T043603Z.root"
11-Nov-2024 08:58:07 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/08/b2/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45718791_47_20210630T043603Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.780167       21.8908       61.457        21.7968       20.6275        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00071837     0.131946       3.46504     0.00461802     0.428708        100    
reco:pandoracali:CalibrationdEdXPDSP           0.304038       1.4875        11.4178      0.688953       2.13542        100    
reco:pandoracalipid:Chi2ParticleID            0.00253768    0.00453249    0.00958462    0.00432786    0.00112128       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.328634       1.16404       8.41222      0.709953       1.37163        100    
reco:pandora2calipid:Chi2ParticleID           0.00527423    0.00856226     0.0125848    0.00838755    0.00175262       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000192228    0.0321547     0.438041     0.00294666     0.0820085       100    
[art]:TriggerResults:TriggerResultInserter     2.013e-05    2.80573e-05   0.000107163    2.389e-05    1.15385e-05      100    
end_path:pduneana:PDSPAnalyzer                 1.993e-05      19.0616       60.4188       13.7694       19.8297        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.780167       21.8908       61.457        21.7968       20.6275        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00071837     0.131946       3.46504     0.00461802     0.428708        100    
reco:pandoracali:CalibrationdEdXPDSP           0.304038       1.4875        11.4178      0.688953       2.13542        100    
reco:pandoracalipid:Chi2ParticleID            0.00253768    0.00453249    0.00958462    0.00432786    0.00112128       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.328634       1.16404       8.41222      0.709953       1.37163        100    
reco:pandora2calipid:Chi2ParticleID           0.00527423    0.00856226     0.0125848    0.00838755    0.00175262       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000192228    0.0321547     0.438041     0.00294666     0.0820085       100    
[art]:TriggerResults:TriggerResultInserter     2.013e-05    2.80573e-05   0.000107163    2.389e-05    1.15385e-05      100    
end_path:pduneana:PDSPAnalyzer                 1.993e-05      19.0616       60.4188       13.7694       19.8297        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4598.14 MB
  Peak resident set size usage (VmHWM): 3465.42 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4598.14 MB
  Peak resident set size usage (VmHWM): 3465.42 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2904.138993 Real = 3619.396240

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4598.14 VmHWM = 3465.42

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END