Jobsub ID 94210.30@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94210.30@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 08:39:37
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 09:51:11
From worker node	Hostname	r23-n13.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 08:40:47
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_31_20210630T144910Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_122_20210630T145920Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273851_67_20210630T145216Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46270741_120_20210630T170045Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_136_20210630T171237Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46273041_49_20210630T111201Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_3_20210630T164413Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_85_20210630T170806Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_45738751_82_20210630T170814Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21655491_101_20210701T021553Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 09:51:11
Saved logs	justin-logs:94210.30-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

nal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 3.25388e+16 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5877e+26  1.0000e+00  1.0029e+00  1.5877e+26
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 158926915084435810326413312.000000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.1236e-07  1.2340e-06  1.8244e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9539e-07  1.1997e-06  1.8368e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.87465e+14 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6836e+06  1.0000e+00  1.0026e+00  1.6836e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1685254.691788 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.52305e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.4062e+05  1.0000e+00  1.0029e+00  4.4062e+05
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 441056.993611 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.3761e-07  1.1867e-06  1.7416e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.1922e-07  1.1546e-06  1.7606e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.53911e+14 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.3570e+06  1.0000e+00  1.0024e+00  1.3570e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1358379.218238 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.8998e-07  9.7695e-07  1.6311e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.0018e-07  9.8924e-07  1.6229e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 09:50:04 GMT  run: 21655491 subRun: 438 event: 1020
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 21655491 ; subrun: 438 ; event: 1020
evttime: 378375084
run: 21655491 ; subrun: 438 ; event: 1020
evttime: 378375084
########## EvtNo.1020
We found no beam particles for this event... moving on
Getting abscex weigths
Making trajs 
0
211
Weights; 12
11-Nov-2024 09:50:34 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/e2/8e/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21655491_101_20210701T021553Z.root"
11-Nov-2024 09:50:34 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/e2/8e/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_21655491_101_20210701T021553Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.798187       22.3399       66.4388       16.8366       20.347         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000810122     0.11926       2.47291     0.00497987     0.322534        100    
reco:pandoracali:CalibrationdEdXPDSP           0.387067       1.75445       13.2106      0.873138       2.3647         100    
reco:pandoracalipid:Chi2ParticleID            0.00285842    0.00479361    0.00757415    0.00462702    0.00100095       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.318406        1.241        10.1205      0.817805       1.36228        100    
reco:pandora2calipid:Chi2ParticleID           0.00467961    0.00923438     0.0156927    0.00908798    0.00195129       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000206848    0.0353706     0.469705     0.00318121     0.0870327       100    
[art]:TriggerResults:TriggerResultInserter    2.0542e-05    3.24503e-05   0.000130832   2.5889e-05    1.64892e-05      100    
end_path:pduneana:PDSPAnalyzer                2.1711e-05      19.1752       63.5091      0.208401       20.4378        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.798187       22.3399       66.4388       16.8366       20.347         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000810122     0.11926       2.47291     0.00497987     0.322534        100    
reco:pandoracali:CalibrationdEdXPDSP           0.387067       1.75445       13.2106      0.873138       2.3647         100    
reco:pandoracalipid:Chi2ParticleID            0.00285842    0.00479361    0.00757415    0.00462702    0.00100095       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.318406        1.241        10.1205      0.817805       1.36228        100    
reco:pandora2calipid:Chi2ParticleID           0.00467961    0.00923438     0.0156927    0.00908798    0.00195129       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000206848    0.0353706     0.469705     0.00318121     0.0870327       100    
[art]:TriggerResults:TriggerResultInserter    2.0542e-05    3.24503e-05   0.000130832   2.5889e-05    1.64892e-05      100    
end_path:pduneana:PDSPAnalyzer                2.1711e-05      19.1752       63.5091      0.208401       20.4378        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5011.26 MB
  Peak resident set size usage (VmHWM): 3861.85 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5011.26 MB
  Peak resident set size usage (VmHWM): 3861.85 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         51         51          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2977.179102 Real = 4116.647180

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 5011.26 VmHWM = 3861.85

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END