Jobsub ID 301212.134@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301212.134@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 4201 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| RSS bytes | 2096103424 (1999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2024-11-22 16:54:27 | |
| Site | CA_SFU | |
| Entry | DUNE_CA_SFU_lcg-ce3 | |
| Last heartbeat | 2024-11-22 20:31:43 | |
| From worker node | Hostname | cdr1062.int.cedar.computecanada.ca |
| cpuinfo | Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 2096103424 (1999 MiB) | |
| Wall seconds limit | 84598 (23 hours) | |
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2024-11-22 17:12:59 | |
| Input files | hd-protodune:np04hd_raw_run028023_0202_dataflow5_datawriter_0_20240716T185948.hdf5 hd-protodune:np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5 hd-protodune:np04hd_raw_run028023_0200_dataflow0_datawriter_0_20240716T185718.hdf5 hd-protodune:np04hd_raw_run028023_0200_dataflow4_datawriter_0_20240716T185716.hdf5 hd-protodune:np04hd_raw_run028023_0201_dataflow3_datawriter_0_20240716T185835.hdf5 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Outputting started | ||
| Output files | ||
| Finished | 2024-11-22 20:31:43 | |
| Saved logs | justin-logs:301212.134-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
'
[Fri Nov 22 12:23:00 2024] perform_with_timeout: rc=7, try=10, delay=220, t0=1732306494, dt=486 timeout=1200
%MSG-w BeamEvent: BeamEvent:beamevent@BeginModule 22-Nov-2024 12:30:24 PST run: 28023 subRun: 1 event: 53485 BeamEvent_module.cc:1846
Could not get XCET2 info
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 94.8827 0.968501
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 94.9199 1.06781
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.2441 0.986764
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 95.7265 0.98509
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 92.9788 0.984782
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 95.6465 1.01616
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 96.8652 -1
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 95.0948 1.09288
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 103.034 -1
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 147.655 1.06947
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.8086 1.07112
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 98.4209 0.972334
Timing trigger: 12
Matched: 1
CKovs: 1 1
TOF, P: 96.2315 0.947384
Timing trigger: 8
Matched: 0
CKovs: 0 0
TOF, P: 0 -1
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
major: Virtual Object Layer
minor: Iteration failed
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
major: Plugin for dynamically loaded library
minor: Iteration failed
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
major: Plugin for dynamically loaded library
minor: Can't open directory or file
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can't open object
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/35/dc/np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5', tent_flags = 0
major: File accessibility
minor: Unable to open file
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/35/dc/np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5', errno = 52, error message = 'Invalid exchange', flags = 0, o_flags = 0
major: File accessibility
minor: Unable to open file
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 2.76552 358.559 6361.02 2.9887 1405.06 33
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 1.7639e-05 2.16321e-05 9.5523e-05 1.8771e-05 1.32635e-05 33
produce:triggerrawdecoder:PDHDTriggerReader3 2.0256 2.34708 2.85845 2.34348 0.166437 33
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.360927 0.445353 0.564074 0.43344 0.0577304 33
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0722757 0.0724369 0.0730773 0.0723982 0.000152233 33
produce:beamevent:BeamEvent 0.000179769 355.57 6358.11 0.00119553 1405.09 33
[art]:TriggerResults:TriggerResultInserter 1.352e-05 1.63833e-05 5.828e-05 1.4955e-05 7.47875e-06 33
end_path:out1:RootOutput 2.828e-06 3.89561e-06 1.8917e-05 3.269e-06 2.77175e-06 33
end_path:out1:RootOutput(write) 0.0788022 0.124813 0.159326 0.124431 0.0213814 33
===================================================================================================================================
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 2.76552 358.559 6361.02 2.9887 1405.06 33
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 1.7639e-05 2.16321e-05 9.5523e-05 1.8771e-05 1.32635e-05 33
produce:triggerrawdecoder:PDHDTriggerReader3 2.0256 2.34708 2.85845 2.34348 0.166437 33
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.360927 0.445353 0.564074 0.43344 0.0577304 33
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0722757 0.0724369 0.0730773 0.0723982 0.000152233 33
produce:beamevent:BeamEvent 0.000179769 355.57 6358.11 0.00119553 1405.09 33
[art]:TriggerResults:TriggerResultInserter 1.352e-05 1.63833e-05 5.828e-05 1.4955e-05 7.47875e-06 33
end_path:out1:RootOutput 2.828e-06 3.89561e-06 1.8917e-05 3.269e-06 2.77175e-06 33
end_path:out1:RootOutput(write) 0.0788022 0.124813 0.159326 0.124431 0.0213814 33
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1346.88 MB
Peak resident set size usage (VmHWM): 454.689 MB
====================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1346.88 MB
Peak resident set size usage (VmHWM): 454.689 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 33 passed = 33 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 33 33 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 6.913296 Real = 11898.450755
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 1346.88 VmHWM = 454.689
%MSG-s ArtException: PostEndJob 22-Nov-2024 12:31:28 PST ModuleEndJob
---- HDF5RawDataFile BEGIN
File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/35/dc/np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/35/dc/np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
%MSG-s ArtException: PostEndJob 22-Nov-2024 12:31:28 PST ModuleEndJob
---- HDF5RawDataFile BEGIN
File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/35/dc/np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/35/dc/np04hd_raw_run028023_0169_dataflow5_datawriter_0_20240716T181851.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.