Jobsub ID 286947.76@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	286947.76@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 09:46:36
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:56:41
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 09:48:10
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46286411_40_20210701T115014Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_141_20210702T014819Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_20_20210702T010738Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_105_20210702T012235Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_110_20210702T012501Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46306331_105_20210702T014628Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_106_20210702T012457Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_51_20210702T011539Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_77_20210702T011835Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_73_20210702T011846Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:56:41
Saved logs	justin-logs:286947.76-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

INUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.94327 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7791e+00  1.1408e+00  1.3647e+00  3.2736e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.782350 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.7898e-07  3.5049e-06  1.6867e+00  2.3132e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002314 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
2212
Weights; 2
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:55:07 GMT  run: 46305731 subRun: 376 event: 739
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46305731 ; subrun: 376 ; event: 739
evttime: 378395102
run: 46305731 ; subrun: 376 ; event: 739
evttime: 378395102
########## EvtNo.739
273
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.6876e-01  1.3954e-02  1.9211e+00  2.7337e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.671490 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.5047e-01  4.0628e-01  1.5594e+00  2.5848e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 0.553056 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.5200e-07  2.9454e-06  1.7573e+00  2.2427e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002243 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 152097 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7373e+01  1.0000e+00  1.0001e+00  1.9373e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 17.392060 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.49047 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.4050e+00  1.0714e+00  1.2967e+00  4.0370e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.409087 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 70074.6 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6319e+01  1.0000e+00  1.0001e+00  2.8319e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 26.347033 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
2
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:56:06 GMT  run: 46305731 subRun: 376 event: 740
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46305731 ; subrun: 376 ; event: 740
evttime: 378395102
run: 46305731 ; subrun: 376 ; event: 740
evttime: 378395102
11-Nov-2024 10:56:07 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/7b/6e/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_73_20210702T011846Z.root"
11-Nov-2024 10:56:07 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/7b/6e/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_73_20210702T011846Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.660976       22.4232       88.4756       8.68155       22.2076        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000914871    0.100422       1.63966     0.00497536     0.229529        100    
reco:pandoracali:CalibrationdEdXPDSP           0.335577       2.1128        49.0644      0.936234       5.33506        100    
reco:pandoracalipid:Chi2ParticleID            0.00258522    0.00468903    0.00965554    0.00439272    0.00118004       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.315135       1.21953       9.57779      0.930967       1.26948        100    
reco:pandora2calipid:Chi2ParticleID           0.00444667    0.00931496     0.0192572    0.00906168    0.00209228       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000238521    0.0414426     0.418485     0.00428074     0.0894997       100    
[art]:TriggerResults:TriggerResultInserter    2.0037e-05    2.79032e-05   9.6827e-05    2.4177e-05    1.10068e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9742e-05      18.9345       58.5755     4.01085e-05     21.2665        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.660976       22.4232       88.4756       8.68155       22.2076        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000914871    0.100422       1.63966     0.00497536     0.229529        100    
reco:pandoracali:CalibrationdEdXPDSP           0.335577       2.1128        49.0644      0.936234       5.33506        100    
reco:pandoracalipid:Chi2ParticleID            0.00258522    0.00468903    0.00965554    0.00439272    0.00118004       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.315135       1.21953       9.57779      0.930967       1.26948        100    
reco:pandora2calipid:Chi2ParticleID           0.00444667    0.00931496     0.0192572    0.00906168    0.00209228       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000238521    0.0414426     0.418485     0.00428074     0.0894997       100    
[art]:TriggerResults:TriggerResultInserter    2.0037e-05    2.79032e-05   9.6827e-05    2.4177e-05    1.10068e-05      100    
end_path:pduneana:PDSPAnalyzer                1.9742e-05      18.9345       58.5755     4.01085e-05     21.2665        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4876.27 MB
  Peak resident set size usage (VmHWM): 3663.64 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4876.27 MB
  Peak resident set size usage (VmHWM): 3663.64 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         46         46          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2906.177432 Real = 4006.429599

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4876.27 VmHWM = 3663.64

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END