Jobsub ID 94219.73@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94219.73@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 09:06:58
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:34:11
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 09:09:09
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_106_20210415T072216Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_42_20210415T121834Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510331_113_20210415T080431Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43510071_77_20210415T131801Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_43352885_0_20210427T162830Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710191_78_20210628T232443Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21627691_76_20210629T013853Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45710411_81_20210629T004942Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_114_20210629T084648Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_53_20210629T082904Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:34:11
Saved logs	justin-logs:94219.73-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.26699e+12 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.7957e+06  1.0559e+00  1.1859e+00  2.7957e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2798457.652290 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6981e-07  5.2673e-07  9.8196e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.8481e-07  5.1570e-07  9.8414e-07  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.003999 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 3.54283e+12 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.0860e+06  1.0458e+00  1.1544e+00  2.0860e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2088037.945982 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.4534e-07  5.2732e-07  1.0072e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7803e-07  8.7273e-08  1.4049e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 4.00158e+12 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.1898e+06  1.0263e+00  1.0911e+00  2.1898e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2191983.209729 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:32:22 GMT  run: 46266471 subRun: 227 event: 539
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46266471 ; subrun: 227 ; event: 539
evttime: 378339952
run: 46266471 ; subrun: 227 ; event: 539
evttime: 378339952
########## EvtNo.539
No G4 version of the good particle was found, returning null pointer
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  11-Nov-2024 10:32:24 GMT run: 46266471 subRun: 227 event: 539 PDSPAnalyzer_module.cc:1384
No true beam particle
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:32:38 GMT  run: 46266471 subRun: 227 event: 540
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46266471 ; subrun: 227 ; event: 540
evttime: 378339954
run: 46266471 ; subrun: 227 ; event: 540
evttime: 378339954
########## EvtNo.540
21
2
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
11-Nov-2024 10:33:37 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/9c/dd/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_53_20210629T082904Z.root"
11-Nov-2024 10:33:37 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/9c/dd/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46266471_53_20210629T082904Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.09769       28.7143       75.7353       31.9956       22.9396        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00103878     0.170941       3.12246     0.00537506     0.523822        100    
reco:pandoracali:CalibrationdEdXPDSP           0.388619       3.00128       56.0192       0.85393       7.54388        100    
reco:pandoracalipid:Chi2ParticleID            0.00291952    0.00473346     0.0103322    0.00450755    0.00111871       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.364215       2.55491       25.4276      0.899517       4.76304        100    
reco:pandora2calipid:Chi2ParticleID           0.00538241    0.00922072     0.0176141    0.00922427     0.002189        100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000263704    0.0303137     0.302543     0.00382274     0.0606717       100    
[art]:TriggerResults:TriggerResultInserter    2.1104e-05    3.22051e-05   9.9296e-05    2.7264e-05    1.36806e-05      100    
end_path:pduneana:PDSPAnalyzer                2.3978e-05      22.9423       70.9846       29.6601       21.6559        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.09769       28.7143       75.7353       31.9956       22.9396        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00103878     0.170941       3.12246     0.00537506     0.523822        100    
reco:pandoracali:CalibrationdEdXPDSP           0.388619       3.00128       56.0192       0.85393       7.54388        100    
reco:pandoracalipid:Chi2ParticleID            0.00291952    0.00473346     0.0103322    0.00450755    0.00111871       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.364215       2.55491       25.4276      0.899517       4.76304        100    
reco:pandora2calipid:Chi2ParticleID           0.00538241    0.00922072     0.0176141    0.00922427     0.002189        100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000263704    0.0303137     0.302543     0.00382274     0.0606717       100    
[art]:TriggerResults:TriggerResultInserter    2.1104e-05    3.22051e-05   9.9296e-05    2.7264e-05    1.36806e-05      100    
end_path:pduneana:PDSPAnalyzer                2.3978e-05      22.9423       70.9846       29.6601       21.6559        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4531.35 MB
  Peak resident set size usage (VmHWM): 3301.96 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4531.35 MB
  Peak resident set size usage (VmHWM): 3301.96 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         58         58          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 3316.237773 Real = 4997.017327

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4531.35 VmHWM = 3301.96

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END