Jobsub ID 94217.87@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	94217.87@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4002
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-11 09:02:32
Site	UK_Liverpool
Entry	DUNE_UK_Liverpool_hepgrid5
Last heartbeat	2024-11-11 10:15:13
From worker node	Hostname	r22-n02.ph.liv.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	259200 (72 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-11 09:05:12
Input files	pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46287381_93_20210701T081240Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46297271_69_20210701T122928Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46297271_16_20210701T122339Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46297271_66_20210701T122926Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_137_20210701T201405Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_71_20210701T200157Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_80_20210701T200200Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46297271_17_20210701T122340Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46304361_109_20210701T200336Z.root pdsp_mc_reco:PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_24_20210702T011004Z.root
Jobscript	Exit code	134
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-11 10:15:05
Saved logs	justin-logs:94217.87-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.3196e+01  1.0000e+00  1.0000e+00  2.5196e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 23.221592 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:13:09 GMT  run: 46305731 subRun: 376 event: 248
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46305731 ; subrun: 376 ; event: 248
evttime: 378395569
run: 46305731 ; subrun: 376 ; event: 248
evttime: 378395569
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:13:28 GMT  run: 46305731 subRun: 376 event: 249
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46305731 ; subrun: 376 ; event: 249
evttime: 378395570
run: 46305731 ; subrun: 376 ; event: 249
evttime: 378395570
########## EvtNo.249
194
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.7627e-01  3.6421e-03  1.7985e+00  2.3741e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.178641 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 7.41345 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.6378e-01  8.4904e-01  1.1721e+00  2.9427e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 0.966723 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.8868e-07  4.5613e-06  1.6633e+00  2.3367e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002337 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 474281 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -5.9143e+00  1.0000e+00  1.0000e+00  7.9143e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 5.922264 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 10.2114 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -3.9574e+00  1.0723e+00  1.4269e+00  5.4583e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 3.962890 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9411e-07  1.6460e-06  1.6627e+00  2.3373e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002338 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 128874 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.1695e+00  1.0000e+00  1.0000e+00  4.1695e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2.173648 TO DIAGONAL.
Getting reco beam calo
Getting abscex weigths
Making trajs 
1
211
Weights; 12
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2024 10:14:31 GMT  run: 46305731 subRun: 376 event: 250
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
run: 46305731 ; subrun: 376 ; event: 250
evttime: 378395572
run: 46305731 ; subrun: 376 ; event: 250
evttime: 378395572
11-Nov-2024 10:14:32 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/3b/06/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_24_20210702T011004Z.root"
11-Nov-2024 10:14:32 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/pdsp_mc_reco/3b/06/PDSPProd4a_protoDUNE_sp_reco_stage1_p1GeV_35ms_sce_datadriven_46305731_24_20210702T011004Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.839184       18.9064       68.1237       6.14894       20.0141        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000660029    0.174856       4.86649     0.00477818      0.58772        100    
reco:pandoracali:CalibrationdEdXPDSP           0.420296        2.047        15.7795      0.938435       2.74763        100    
reco:pandoracalipid:Chi2ParticleID            0.00262919    0.00467076    0.00884566    0.00436824    0.00112707       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.293268       1.52757       16.8872      0.809419       2.13988        100    
reco:pandora2calipid:Chi2ParticleID           0.00447232     0.0091167     0.0191876    0.00903948    0.00203843       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000194532    0.0400291      0.71875     0.00299372     0.112496        100    
[art]:TriggerResults:TriggerResultInserter     1.989e-05    2.73953e-05   9.2176e-05    2.42375e-05   9.55323e-06      100    
end_path:pduneana:PDSPAnalyzer                1.7829e-05      15.1026       59.4546     3.4383e-05      19.9376        100    
================================================================================================================================

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.839184       18.9064       68.1237       6.14894       20.0141        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000660029    0.174856       4.86649     0.00477818      0.58772        100    
reco:pandoracali:CalibrationdEdXPDSP           0.420296        2.047        15.7795      0.938435       2.74763        100    
reco:pandoracalipid:Chi2ParticleID            0.00262919    0.00467076    0.00884566    0.00436824    0.00112707       100    
reco:pandora2cali:CalibrationdEdXPDSP          0.293268       1.52757       16.8872      0.809419       2.13988        100    
reco:pandora2calipid:Chi2ParticleID           0.00447232     0.0091167     0.0191876    0.00903948    0.00203843       100    
fpath:fTruth:ProtoDUNETruthBeamFilter         0.000194532    0.0400291      0.71875     0.00299372     0.112496        100    
[art]:TriggerResults:TriggerResultInserter     1.989e-05    2.73953e-05   9.2176e-05    2.42375e-05   9.55323e-06      100    
end_path:pduneana:PDSPAnalyzer                1.7829e-05      15.1026       59.4546     3.4383e-05      19.9376        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4763.65 MB
  Peak resident set size usage (VmHWM): 3656.58 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4763.65 MB
  Peak resident set size usage (VmHWM): 3656.58 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         38         38          0 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 2545.017297 Real = 4088.459267

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4763.65 VmHWM = 3656.58

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Socket error: connection reset by peer
  ROOT severity: 3000
---- FatalRootError END