Jobsub ID 105523.124@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 105523.124@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 4191 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| RSS bytes | 2096103424 (1999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2024-11-21 02:15:16 | |
| Site | UK_Lancaster | |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 | |
| Last heartbeat | 2024-11-21 05:49:45 | |
| From worker node | Hostname | comp08-05 |
| cpuinfo | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 2096103424 (1999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2024-11-21 02:16:39 | |
| Input files | hd-protodune:np04hd_raw_run028758_0732_dataflow4_datawriter_0_20240817T222443.hdf5 hd-protodune:np04hd_raw_run028758_0730_dataflow0_datawriter_0_20240817T222318.hdf5 hd-protodune:np04hd_raw_run028758_0731_dataflow3_datawriter_0_20240817T222402.hdf5 hd-protodune:np04hd_raw_run028758_0730_dataflow1_datawriter_0_20240817T222318.hdf5 hd-protodune:np04hd_raw_run028758_0732_dataflow1_datawriter_0_20240817T222443.hdf5 | |
| Jobscript | Exit code | 139 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Outputting started | ||
| Output files | ||
| Finished | 2024-11-21 05:49:45 | |
| Saved logs | justin-logs:105523.124-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
easy_perform() failed: 'Server returned nothing (no headers, no data)'
[Thu Nov 21 05:20:59 2024] perform_with_timeout: URL='https://ifb-data.fnal.gov:8104/ifbeam/data/data?b=DUNE_CERN_SEP2018_TIMBER&t0=1723933500.000&t1=1723933680.000&f=csv'
[Thu Nov 21 05:20:59 2024] perform_with_timeout: rc=52, try=7, delay=58, t0=1732165769, dt=614 timeout=1200
%MSG-w BeamEvent: BeamEvent:beamevent@BeginModule 21-Nov-2024 05:29:52 GMT run: 28758 subRun: 1 event: 175913 BeamEvent_module.cc:1828
Could not get XCET1 info
%MSG
[Thu Nov 21 05:35:00 2024] perform_with_timeout: curl_easy_perform() failed: 'Server returned nothing (no headers, no data)'
[Thu Nov 21 05:35:00 2024] perform_with_timeout: URL='https://ifb-data.fnal.gov:8104/ifbeam/data/data?b=DUNE_CERN_SEP2018_TIMBER&t0=1723933500.000&t1=1723933680.000&f=csv'
[Thu Nov 21 05:35:00 2024] perform_with_timeout: rc=52, try=4, delay=8, t0=1732166992, dt=308 timeout=1200
[Thu Nov 21 05:39:26 2024] perform_with_timeout: curl_easy_perform() failed: 'SSL connect error'
[Thu Nov 21 05:39:26 2024] perform_with_timeout: URL='https://ifb-data.fnal.gov:8104/ifbeam/data/data?b=DUNE_CERN_SEP2018_TIMBER&t0=1723933500.000&t1=1723933680.000&f=csv'
[Thu Nov 21 05:39:26 2024] perform_with_timeout: rc=35, try=7, delay=61, t0=1732166992, dt=573 timeout=1200
[Thu Nov 21 05:40:44 2024] perform_with_timeout: curl_easy_perform() failed: 'Server returned nothing (no headers, no data)'
[Thu Nov 21 05:40:44 2024] perform_with_timeout: URL='https://ifb-data.fnal.gov:8104/ifbeam/data/data?b=DUNE_CERN_SEP2018_TIMBER&t0=1723933500.000&t1=1723933680.000&f=csv'
[Thu Nov 21 05:40:44 2024] perform_with_timeout: rc=52, try=8, delay=78, t0=1732166992, dt=652 timeout=1200
%MSG-w BeamEvent: BeamEvent:beamevent@BeginModule 21-Nov-2024 05:48:22 GMT run: 28758 subRun: 1 event: 175913 BeamEvent_module.cc:1846
Could not get XCET2 info
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 101.479 5.10478
21-Nov-2024 05:48:23 GMT Closed output file "np04hd_raw_run028758_0732_dataflow1_datawriter_0_20240817T222443_beam.root"
21-Nov-2024 05:48:23 GMT Closed output file "np04hd_raw_run028758_0732_dataflow1_datawriter_0_20240817T222443_beam.root"
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5I.c line 456 in H5Idec_ref(): can't decrement ID ref count
major: Object atom
minor: Unable to decrement reference count
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Iint.c line 1018 in H5I_dec_app_ref(): can't decrement ID ref count
major: Object atom
minor: Unable to decrement reference count
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 251 in H5F__close_cb(): unable to close file
major: File accessibility
minor: Unable to close file
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3983 in H5VL_file_close(): file close failed
major: Virtual Object Layer
minor: Unable to close file
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3952 in H5VL__file_close(): file close failed
major: Virtual Object Layer
minor: Unable to close file
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 838 in H5VL__native_file_close(): can't close file
major: File accessibility
minor: Unable to decrement reference count
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 2349 in H5F__close(): can't close file
major: File accessibility
minor: Unable to close file
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 2522 in H5F_try_close(): problems closing file
major: File accessibility
minor: Unable to close file
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1605 in H5F__dest(): unable to close file
major: File accessibility
minor: Unable to close file
#009: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 830 in H5FD_close(): close failed
major: Virtual File Layer
minor: Unable to close file
#010: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 456 in H5FD__sec2_close(): unable to close file, errno = 110, error message = 'Connection timed out'
major: Low-level I/O
minor: Unable to close file
HighFive::~Object: reference counter decrease failure
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 0.882016 83.6798 4910.2 1.01113 527.302 150
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 1.2135e-05 1.74171e-05 0.000104233 1.61175e-05 8.09856e-06 150
produce:triggerrawdecoder:PDHDTriggerReader3 0.650745 0.753724 1.15777 0.731706 0.0823829 150
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.114279 0.132003 0.236913 0.121262 0.018936 150
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0228334 0.0236131 0.0242198 0.0239068 0.000444108 150
produce:beamevent:BeamEvent 0.000169987 82.6691 4909.2 0.0155802 527.295 150
[art]:TriggerResults:TriggerResultInserter 1.1378e-05 1.75057e-05 5.2566e-05 1.72065e-05 4.77013e-06 150
end_path:out1:RootOutput 2.559e-06 3.54007e-06 1.7139e-05 3.149e-06 1.69296e-06 150
end_path:out1:RootOutput(write) 0.0618121 0.10102 0.153294 0.101009 0.0193279 150
===================================================================================================================================
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 0.882016 83.6798 4910.2 1.01113 527.302 150
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 1.2135e-05 1.74171e-05 0.000104233 1.61175e-05 8.09856e-06 150
produce:triggerrawdecoder:PDHDTriggerReader3 0.650745 0.753724 1.15777 0.731706 0.0823829 150
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.114279 0.132003 0.236913 0.121262 0.018936 150
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0228334 0.0236131 0.0242198 0.0239068 0.000444108 150
produce:beamevent:BeamEvent 0.000169987 82.6691 4909.2 0.0155802 527.295 150
[art]:TriggerResults:TriggerResultInserter 1.1378e-05 1.75057e-05 5.2566e-05 1.72065e-05 4.77013e-06 150
end_path:out1:RootOutput 2.559e-06 3.54007e-06 1.7139e-05 3.149e-06 1.69296e-06 150
end_path:out1:RootOutput(write) 0.0618121 0.10102 0.153294 0.101009 0.0193279 150
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1659.01 MB
Peak resident set size usage (VmHWM): 812.937 MB
====================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1659.01 MB
Peak resident set size usage (VmHWM): 812.937 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 150 passed = 150 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 150 150 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 31.005901 Real = 12766.431613
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 1659.01 VmHWM = 812.937