Jobsub ID 105319.140@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105319.140@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4182
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-20 20:07:21
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat	2024-11-20 20:22:51
From worker node	Hostname	cit2-c036.mwt2.org
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 20:12:39
Input files	hd-protodune:np04hd_raw_run028059_0099_dataflow4_datawriter_0_20240718T154710.hdf5 hd-protodune:np04hd_raw_run028072_0043_dataflow2_datawriter_0_20240719T190311.hdf5 hd-protodune:np04hd_raw_run028072_0119_dataflow0_datawriter_0_20240719T193722.hdf5 hd-protodune:np04hd_raw_run028072_0042_dataflow7_datawriter_0_20240719T190249.hdf5 hd-protodune:np04hd_raw_run028072_0631_dataflow5_datawriter_0_20240720T005240.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-20 20:22:51
Saved logs	justin-logs:105319.140-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
140 72 Breaking 2A Gen -1.70247e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
141 72 Breaking 2A Gen -1.49016e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
142 72 Breaking 2A Gen -1.40612e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
143 72 Breaking 2A Gen -1.35494e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
144 72 Breaking 2A Gen -7.0389e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
145 72 Breaking 2A Gen -6.67539e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
146 72 Breaking 2A Gen -1.07517e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1641
Breaking 2A 1A -1.3164e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
147 73 Breaking 2A Gen -2.56563e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
148 73 Breaking 2A Gen -1.24923e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
149 73 Breaking 2A Gen -1.04857e+08

%MSG
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 15:21:21 EST  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 96.4198 1.04308
20-Nov-2024 15:21:21 EST  Opened output file with pattern "%ifb_beam.root"
20-Nov-2024 15:21:21 EST  Opened output file with pattern "%ifb_beam.root"
%MSG-w FastCloning:  PostProcessEvent 20-Nov-2024 15:21:22 EST  run: 28059 subRun: 1 event: 24548
Fast cloning has been deactivated for the following reasons:
 - Input source does not read art/ROOT files.
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.2461 1.02564
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: -1 -1

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         2.83383       174.132       505.675       13.8858       234.48          3     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.2958e-05    9.5719e-05    0.000238855   2.5344e-05    0.000101217       3     
produce:triggerrawdecoder:PDHDTriggerReader3       2.23071       2.43201       2.81105       2.25427      0.268193         3     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.223226      0.331072      0.413852       0.35614      0.0798158        3     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0373404     0.0579002     0.0725701     0.0637902     0.0149733        3     
produce:beamevent:BeamEvent                      0.00690528      171.204       502.211       11.3937       234.104         3     
[art]:TriggerResults:TriggerResultInserter        2.65e-05     7.17877e-05   0.000118421   7.0442e-05    3.75387e-05       3     
end_path:out1:RootOutput                          4.272e-06    1.79213e-05   3.4526e-05    1.4966e-05    1.25267e-05       3     
end_path:out1:RootOutput(write)                   0.152181      0.158834      0.165487      0.158834      0.006653         2     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         2.83383       174.132       505.675       13.8858       234.48          3     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.2958e-05    9.5719e-05    0.000238855   2.5344e-05    0.000101217       3     
produce:triggerrawdecoder:PDHDTriggerReader3       2.23071       2.43201       2.81105       2.25427      0.268193         3     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.223226      0.331072      0.413852       0.35614      0.0798158        3     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0373404     0.0579002     0.0725701     0.0637902     0.0149733        3     
produce:beamevent:BeamEvent                      0.00690528      171.204       502.211       11.3937       234.104         3     
[art]:TriggerResults:TriggerResultInserter        2.65e-05     7.17877e-05   0.000118421   7.0442e-05    3.75387e-05       3     
end_path:out1:RootOutput                          4.272e-06    1.79213e-05   3.4526e-05    1.4966e-05    1.25267e-05       3     
end_path:out1:RootOutput(write)                   0.152181      0.158834      0.165487      0.158834      0.006653         2     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6551 MB
  Peak resident set size usage (VmHWM): 5638.63 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6551 MB
  Peak resident set size usage (VmHWM): 5638.63 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 3 passed = 3 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          3          3          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 29.894075 Real = 546.232055

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6551 VmHWM = 5638.63

%MSG-s ArtException:  PostEndJob 20-Nov-2024 15:22:27 EST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 3 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
%MSG-s ArtException:  PostEndJob 20-Nov-2024 15:22:27 EST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 3 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.