Jobsub ID 105145.73@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105145.73@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4171
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-20 16:54:59
Site	CH_UNIBE-LHEP
Entry	OSG_CH_UNIBE_LHEP_ce02
Last heartbeat	2024-11-20 17:16:59
From worker node	Hostname	wn-1-12.local
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 17:08:44
Input files	hd-protodune:np04hd_raw_run028008_0213_dataflow6_datawriter_0_20240713T171441.hdf5 hd-protodune:np04hd_raw_run028008_0212_dataflow3_datawriter_0_20240713T171358.hdf5 hd-protodune:np04hd_raw_run028008_0212_dataflow0_datawriter_0_20240713T171358.hdf5 hd-protodune:np04hd_raw_run028008_0214_dataflow0_datawriter_0_20240713T171534.hdf5 hd-protodune:np04hd_raw_run028008_0217_dataflow6_datawriter_0_20240713T171733.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-20 17:16:59
Saved logs	justin-logs:105145.73-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

amEvent_module.cc:1625
71 33 Breaking 2A Gen -4.99183e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -1.69347e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
72 34 Breaking 2A Gen -1.56503e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
73 34 Breaking 2A Gen -1.39568e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -397.807

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
74 35 Breaking 2A Gen -7.95002e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
75 35 Breaking 2A Gen -4.22439e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
76 35 Breaking 2A Gen -3.20694e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -8.73665e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
77 36 Breaking 2A Gen -8.73666e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -396.188

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
78 37 Breaking 2A Gen -1.01049e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
79 37 Breaking 2A Gen -6.99134e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
80 37 Breaking 2A Gen -6.08099e+07

%MSG
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:15:51 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 1
TOF, P: 95.419 -1
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 95.1621 0.993726
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: -1 -1

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         1.17653       5.91287       179.876       1.34002       25.3871        65     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.6301e-05    3.11868e-05   0.000136162   2.8732e-05    1.64514e-05      65     
produce:triggerrawdecoder:PDHDTriggerReader3      0.865832      0.994448       1.42725       0.9631       0.0968226       65     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.156751      0.187551      0.270488       0.18944      0.0243106       65     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0313439     0.0333981     0.0602512     0.0319338    0.00470915       65     
produce:beamevent:BeamEvent                      0.000246241     4.55988       178.514     0.00151767      25.3717        65     
[art]:TriggerResults:TriggerResultInserter       1.6529e-05    3.31878e-05   0.000148626   2.8992e-05    1.94454e-05      65     
end_path:out1:RootOutput                          3.218e-06    6.5644e-06    3.8152e-05     5.671e-06    4.52669e-06      65     
end_path:out1:RootOutput(write)                   0.0696863     0.139217      0.254865      0.138737      0.0348762       64     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         1.17653       5.91287       179.876       1.34002       25.3871        65     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.6301e-05    3.11868e-05   0.000136162   2.8732e-05    1.64514e-05      65     
produce:triggerrawdecoder:PDHDTriggerReader3      0.865832      0.994448       1.42725       0.9631       0.0968226       65     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.156751      0.187551      0.270488       0.18944      0.0243106       65     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0313439     0.0333981     0.0602512     0.0319338    0.00470915       65     
produce:beamevent:BeamEvent                      0.000246241     4.55988       178.514     0.00151767      25.3717        65     
[art]:TriggerResults:TriggerResultInserter       1.6529e-05    3.31878e-05   0.000148626   2.8992e-05    1.94454e-05      65     
end_path:out1:RootOutput                          3.218e-06    6.5644e-06    3.8152e-05     5.671e-06    4.52669e-06      65     
end_path:out1:RootOutput(write)                   0.0696863     0.139217      0.254865      0.138737      0.0348762       64     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6032.29 MB
  Peak resident set size usage (VmHWM): 4943.6 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6032.29 MB
  Peak resident set size usage (VmHWM): 4943.6 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 65 passed = 65 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         65         65          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 38.309117 Real = 438.560826

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6032.29 VmHWM = 4943.6

%MSG-s ArtException:  PostEndJob 20-Nov-2024 18:16:38 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 64 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 65 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
%MSG-s ArtException:  PostEndJob 20-Nov-2024 18:16:38 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 64 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 65 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.