Jobsub ID 105279.97@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105279.97@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4182
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-20 18:34:09
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat	2024-11-20 19:44:01
From worker node	Hostname	iut2-c369.iu.edu
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 18:42:27
Input files	hd-protodune:np04hd_raw_run028059_0096_dataflow0_datawriter_0_20240718T154357.hdf5 hd-protodune:np04hd_raw_run028059_0098_dataflow0_datawriter_0_20240718T154553.hdf5 hd-protodune:np04hd_raw_run028059_0099_dataflow6_datawriter_0_20240718T154710.hdf5 hd-protodune:np04hd_raw_run028059_0099_dataflow4_datawriter_0_20240718T154710.hdf5 hd-protodune:np04hd_raw_run028059_0099_dataflow5_datawriter_0_20240718T154710.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-20 19:44:01
Saved logs	justin-logs:105279.97-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

G
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1641
Breaking 2A 1A -3.58335e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
139 72 Breaking 2A Gen -1.76446e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
140 72 Breaking 2A Gen -1.70247e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
141 72 Breaking 2A Gen -1.49016e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
142 72 Breaking 2A Gen -1.40612e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
143 72 Breaking 2A Gen -1.35494e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
144 72 Breaking 2A Gen -7.0389e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
145 72 Breaking 2A Gen -6.67539e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
146 72 Breaking 2A Gen -1.07517e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1641
Breaking 2A 1A -1.3164e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
147 73 Breaking 2A Gen -2.56563e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
148 73 Breaking 2A Gen -1.24923e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
149 73 Breaking 2A Gen -1.04857e+08

%MSG
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 14:43:08 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 96.4198 1.04308
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.2461 1.02564
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: -1 -1

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         0.37975       37.0567       1148.58       1.1965        175.604        96     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        1.628e-05    2.58285e-05   0.000109596   2.2537e-05    1.26423e-05      96     
produce:triggerrawdecoder:PDHDTriggerReader3      0.267866      0.985321       2.49199      0.858863      0.531358        96     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0296441     0.173139      0.505707      0.137795       0.1094         96     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00572407     0.0288066     0.0705172     0.021744      0.017747        96     
produce:beamevent:BeamEvent                      0.000236154     35.7735       1146.64      0.0010991      175.599        96     
[art]:TriggerResults:TriggerResultInserter       1.2554e-05    1.74419e-05   6.0664e-05     1.595e-05    5.95697e-06      96     
end_path:out1:RootOutput                          2.715e-06    4.32416e-06   2.2612e-05    4.0425e-06    2.07436e-06      96     
end_path:out1:RootOutput(write)                   0.051385      0.0965481     0.154982      0.0981773     0.0206712       95     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         0.37975       37.0567       1148.58       1.1965        175.604        96     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        1.628e-05    2.58285e-05   0.000109596   2.2537e-05    1.26423e-05      96     
produce:triggerrawdecoder:PDHDTriggerReader3      0.267866      0.985321       2.49199      0.858863      0.531358        96     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0296441     0.173139      0.505707      0.137795       0.1094         96     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00572407     0.0288066     0.0705172     0.021744      0.017747        96     
produce:beamevent:BeamEvent                      0.000236154     35.7735       1146.64      0.0010991      175.599        96     
[art]:TriggerResults:TriggerResultInserter       1.2554e-05    1.74419e-05   6.0664e-05     1.595e-05    5.95697e-06      96     
end_path:out1:RootOutput                          2.715e-06    4.32416e-06   2.2612e-05    4.0425e-06    2.07436e-06      96     
end_path:out1:RootOutput(write)                   0.051385      0.0965481     0.154982      0.0981773     0.0206712       95     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5461.01 MB
  Peak resident set size usage (VmHWM): 4546.45 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5461.01 MB
  Peak resident set size usage (VmHWM): 4546.45 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 96 passed = 96 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         96         96          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 32.370923 Real = 3634.296106

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 5461.01 VmHWM = 4546.45

%MSG-s ArtException:  PostEndJob 20-Nov-2024 14:43:46 EST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 95 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 96 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
%MSG-s ArtException:  PostEndJob 20-Nov-2024 14:43:46 EST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 95 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 96 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.