Jobsub ID 105327.92@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105327.92@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4182
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-20 20:16:24
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2024-11-20 20:23:33
From worker node	Hostname	dunegli-3999751-0-fnpc23112.fnal.gov
	cpuinfo	AMD EPYC 9354 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 20:22:30
Input files	hd-protodune:np04hd_raw_run028059_0099_dataflow4_datawriter_0_20240718T154710.hdf5 hd-protodune:np04hd_raw_run028059_2064_dataflow6_datawriter_0_20240719T105533.hdf5 hd-protodune:np04hd_raw_run028059_2064_dataflow1_datawriter_0_20240719T105533.hdf5 hd-protodune:np04hd_raw_run028059_2066_dataflow2_datawriter_0_20240719T110301.hdf5 hd-protodune:np04hd_raw_run028059_2065_dataflow5_datawriter_0_20240719T110004.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-20 20:23:33
Saved logs	justin-logs:105327.92-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

SG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
140 72 Breaking 2A Gen -1.70247e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
141 72 Breaking 2A Gen -1.49016e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
142 72 Breaking 2A Gen -1.40612e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
143 72 Breaking 2A Gen -1.35494e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
144 72 Breaking 2A Gen -7.0389e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
145 72 Breaking 2A Gen -6.67539e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
146 72 Breaking 2A Gen -1.07517e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1641
Breaking 2A 1A -1.3164e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
147 73 Breaking 2A Gen -2.56563e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
148 73 Breaking 2A Gen -1.24923e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1625
149 73 Breaking 2A Gen -1.04857e+08

%MSG
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 96.4198 1.04308
20-Nov-2024 20:22:52 UTC  Opened output file with pattern "%ifb_beam.root"
20-Nov-2024 20:22:52 UTC  Opened output file with pattern "%ifb_beam.root"
%MSG-w FastCloning:  PostProcessEvent 20-Nov-2024 20:22:52 UTC  run: 28059 subRun: 1 event: 24548
Fast cloning has been deactivated for the following reasons:
 - Input source does not read art/ROOT files.
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.2461 1.02564
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: -1 -1

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.294643       6.12145       12.5637       5.50601       5.02769         3     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.0511e-05    5.33377e-05   0.000114444   2.5058e-05    4.32486e-05       3     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0585099     0.122527      0.169479      0.139592      0.0468825        3     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00155685    0.00637445     0.0155424    0.00202405    0.00648556        3     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000275319   0.000482611   0.000669639   0.000502875   0.000161617       3     
produce:beamevent:BeamEvent                      0.00296732      5.91876       12.3892       5.36407       5.07186         3     
[art]:TriggerResults:TriggerResultInserter       1.6956e-05    4.29387e-05   6.2074e-05    4.9786e-05    1.90451e-05       3     
end_path:out1:RootOutput                          4.477e-06    1.0005e-05    1.8147e-05     7.391e-06    5.87889e-06       3     
end_path:out1:RootOutput(write)                   0.105586      0.109139      0.112692      0.109139     0.00355319        2     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.294643       6.12145       12.5637       5.50601       5.02769         3     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.0511e-05    5.33377e-05   0.000114444   2.5058e-05    4.32486e-05       3     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0585099     0.122527      0.169479      0.139592      0.0468825        3     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00155685    0.00637445     0.0155424    0.00202405    0.00648556        3     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000275319   0.000482611   0.000669639   0.000502875   0.000161617       3     
produce:beamevent:BeamEvent                      0.00296732      5.91876       12.3892       5.36407       5.07186         3     
[art]:TriggerResults:TriggerResultInserter       1.6956e-05    4.29387e-05   6.2074e-05    4.9786e-05    1.90451e-05       3     
end_path:out1:RootOutput                          4.477e-06    1.0005e-05    1.8147e-05     7.391e-06    5.87889e-06       3     
end_path:out1:RootOutput(write)                   0.105586      0.109139      0.112692      0.109139     0.00355319        2     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6559.12 MB
  Peak resident set size usage (VmHWM): 5643.33 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6559.12 MB
  Peak resident set size usage (VmHWM): 5643.33 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 3 passed = 3 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          3          3          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 14.234660 Real = 27.284216

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6559.12 VmHWM = 5643.33

%MSG-s ArtException:  PostEndJob 20-Nov-2024 20:23:23 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 3 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
%MSG-s ArtException:  PostEndJob 20-Nov-2024 20:23:23 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 3 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.