Jobsub ID 297952.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297952.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4171
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-20 17:06:48
Site	ES_CIEMAT
Entry	DUNE_T2_ES_CIEMAT_condorce1
Last heartbeat	2024-11-20 18:00:51
From worker node	Hostname	gaew0209.ciemat.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	232200 (64 hours)
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 17:17:07
Input files	hd-protodune:np04hd_raw_run028008_0213_dataflow6_datawriter_0_20240713T171441.hdf5 hd-protodune:np04hd_raw_run028008_0212_dataflow3_datawriter_0_20240713T171358.hdf5 hd-protodune:np04hd_raw_run028008_0212_dataflow0_datawriter_0_20240713T171358.hdf5 hd-protodune:np04hd_raw_run028008_0214_dataflow0_datawriter_0_20240713T171534.hdf5 hd-protodune:np04hd_raw_run028008_0217_dataflow6_datawriter_0_20240713T171733.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-20 18:00:51
Saved logs	justin-logs:297952.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

ent_module.cc:1625
71 33 Breaking 2A Gen -4.99183e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -1.69347e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
72 34 Breaking 2A Gen -1.56503e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
73 34 Breaking 2A Gen -1.39568e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -397.807

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
74 35 Breaking 2A Gen -7.95002e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
75 35 Breaking 2A Gen -4.22439e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
76 35 Breaking 2A Gen -3.20694e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -8.73665e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
77 36 Breaking 2A Gen -8.73666e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1641
Breaking 2A 1A -396.188

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
78 37 Breaking 2A Gen -1.01049e+08

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
79 37 Breaking 2A Gen -6.99134e+07

%MSG
%MSG-i BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1625
80 37 Breaking 2A Gen -6.08099e+07

%MSG
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 20-Nov-2024 18:59:29 CET  run: 28008 subRun: 1 event: 52648 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 1
TOF, P: 95.419 -1
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 95.1621 0.993726
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: -1 -1

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         0.39313       38.9612       1588.9       0.444809       223.277        65     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.0239e-05    2.58336e-05   0.00011806    2.2786e-05    1.24376e-05      65     
produce:triggerrawdecoder:PDHDTriggerReader3      0.241067       0.25301      0.325681      0.251058      0.013458        65     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0433383     0.0499904     0.0743034     0.0526747    0.00530576       65     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00874403    0.00901486    0.00998802    0.00897722    0.000228954      65     
produce:beamevent:BeamEvent                      0.000236044     38.5255       1588.46     0.00105207      223.269        65     
[art]:TriggerResults:TriggerResultInserter       1.7915e-05    2.43925e-05   0.000125252   2.1553e-05    1.33697e-05      65     
end_path:out1:RootOutput                          3.475e-06    5.0626e-06    4.2847e-05     4.22e-06     4.87038e-06      65     
end_path:out1:RootOutput(write)                   0.0860164     0.125237      0.197865      0.121346      0.0250806       64     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         0.39313       38.9612       1588.9       0.444809       223.277        65     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.0239e-05    2.58336e-05   0.00011806    2.2786e-05    1.24376e-05      65     
produce:triggerrawdecoder:PDHDTriggerReader3      0.241067       0.25301      0.325681      0.251058      0.013458        65     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0433383     0.0499904     0.0743034     0.0526747    0.00530576       65     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00874403    0.00901486    0.00998802    0.00897722    0.000228954      65     
produce:beamevent:BeamEvent                      0.000236044     38.5255       1588.46     0.00105207      223.269        65     
[art]:TriggerResults:TriggerResultInserter       1.7915e-05    2.43925e-05   0.000125252   2.1553e-05    1.33697e-05      65     
end_path:out1:RootOutput                          3.475e-06    5.0626e-06    4.2847e-05     4.22e-06     4.87038e-06      65     
end_path:out1:RootOutput(write)                   0.0860164     0.125237      0.197865      0.121346      0.0250806       64     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6031.65 MB
  Peak resident set size usage (VmHWM): 5047.08 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6031.65 MB
  Peak resident set size usage (VmHWM): 5047.08 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 65 passed = 65 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         65         65          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 40.856517 Real = 2560.862724

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6031.65 VmHWM = 5047.08

%MSG-s ArtException:  PostEndJob 20-Nov-2024 19:00:29 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 64 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 65 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
%MSG-s ArtException:  PostEndJob 20-Nov-2024 19:00:29 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 64 entries while dunedaq::trgdataformats::TriggerActivityDatas_triggerrawdecoder_daqinTCs_BeamEvent. has 65 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.