Jobsub ID 393390.176@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 393390.176@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 6684 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-05-07 14:28:02 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc01 |
| Last heartbeat | 2025-05-08 02:48:07 |
| From worker node | Hostname | wn-la-15.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2025-05-07 15:50:06 |
| Input files | hd-protodune:np04hd_raw_run032177_2133_dataflow1_datawriter_0_20241027T201458.hdf5
|
| Jobscript | Exit code | 245 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-05-08 02:48:07 |
| Saved logs | justin-logs:393390.176-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
er: no samples within desired window for channel 9754
wclsFrameSaver: no samples within desired window for channel 9754
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9755
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9756
wclsFrameSaver: no samples within desired window for channel 9757
wclsFrameSaver: no samples within desired window for channel 9757
wclsFrameSaver: no samples within desired window for channel 9757
wclsFrameSaver: no samples within desired window for channel 9757
wclsFrameSaver: no samples within desired window for channel 9757
wclsFrameSaver: no samples within desired window for channel 9758
wclsFrameSaver: no samples within desired window for channel 9758
wclsFrameSaver: no samples within desired window for channel 9758
wclsFrameSaver: no samples within desired window for channel 9840
wclsFrameSaver: no samples within desired window for channel 9846
wclsFrameSaver: no samples within desired window for channel 9960
wclsFrameSaver: no samples within desired window for channel 9961
wclsFrameSaver: no samples within desired window for channel 9961
wclsFrameSaver: no samples within desired window for channel 9962
wclsFrameSaver: no samples within desired window for channel 9962
wclsFrameSaver: no samples within desired window for channel 9963
wclsFrameSaver: no samples within desired window for channel 9975
wclsFrameSaver: no samples within desired window for channel 9976
wclsFrameSaver: no samples within desired window for channel 9976
wclsFrameSaver: no samples within desired window for channel 10012
wclsFrameSaver: no samples within desired window for channel 10238
FrameSaver: q=1.46154e+07 n=2022283 tag=wiener
08-May-2025 04:46:14 CEST Closed output file "proc_bsmtrigger_protodunehd_1_393390_176_1746633024_reco.root"
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5I.c line 456 in H5Idec_ref(): can't decrement ID ref count
major: Object atom
minor: Unable to decrement reference count
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Iint.c line 1018 in H5I_dec_app_ref(): can't decrement ID ref count
major: Object atom
minor: Unable to decrement reference count
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 251 in H5F__close_cb(): unable to close file
major: File accessibility
minor: Unable to close file
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3983 in H5VL_file_close(): file close failed
major: Virtual Object Layer
minor: Unable to close file
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3952 in H5VL__file_close(): file close failed
major: Virtual Object Layer
minor: Unable to close file
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 838 in H5VL__native_file_close(): can't close file
major: File accessibility
minor: Unable to decrement reference count
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 2349 in H5F__close(): can't close file
major: File accessibility
minor: Unable to close file
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 2522 in H5F_try_close(): problems closing file
major: File accessibility
minor: Unable to close file
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1605 in H5F__dest(): unable to close file
major: File accessibility
minor: Unable to close file
#009: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 830 in H5FD_close(): close failed
major: Virtual File Layer
minor: Unable to close file
#010: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 456 in H5FD__sec2_close(): unable to close file, errno = 110, error message = 'Connection timed out'
major: Low-level I/O
minor: Unable to close file
HighFive::~Object: reference counter decrease failure
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 1.33574 1960.34 25714.4 479.617 5509.57 20
--------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.2359e-05 0.000123382 0.000310185 0.000111637 4.54328e-05 20
produce:spillflag:PDHDSPSSpillFlagProducer 6.04e-05 9.98549e-05 0.000415442 8.4907e-05 7.38817e-05 20
produce:triggerrawdecoder:PDHDTriggerReader3 1.10747 1.19581 1.42551 1.17179 0.0749347 20
produce:triggertypefilter:PDHDTriggerTypeFilter 6.0881e-05 9.51597e-05 0.000342203 7.8652e-05 5.8408e-05 20
produce:tpcrawdecoder:PDHDTPCReader 11.9379 25.9177 70.0352 21.5904 13.0048 19
produce:timingrawdecoder:PDHDTimingRawDecoder 0.018685 0.0202564 0.0351857 0.0193785 0.00356878 19
produce:fembfilter:PDHDFEMBFilter 0.000112689 0.000164792 0.000525516 0.000136564 9.047e-05 19
produce:wclsdatahd:WireCellToolkit 163.638 216.896 263.422 227.214 24.5155 19
produce:gaushit:GausHitFinder 0.751676 1.25268 2.41461 1.22584 0.377753 19
produce:reco3d:SpacePointSolver 3.17606 39.4292 389.888 12.2276 86.9179 19
produce:hitpdune:DisambigFromSpacePoints 0.714159 3.19819 13.7958 1.99302 3.53962 19
produce:pandora:StandardPandora 32.9522 1702.48 24952 159.12 5529.32 19
produce:pandoraWriter:StandardPandora 0.255143 0.381414 0.674583 0.351123 0.114257 19
produce:pandoraTrack:LArPandoraTrackCreation 4.4457 15.3559 44.4014 10.9987 10.838 19
produce:pandoraShower:LArPandoraShowerCreation 5.68525 13.7499 36.8581 11.0412 7.91424 19
produce:pandoracalo:Calorimetry 1.45514 7.04524 21.6767 5.23332 5.41281 19
produce:pandoracalonosce:Calorimetry 1.46985 6.9279 22.143 5.30908 5.45007 19
produce:pandorapid:Chi2ParticleID 0.00218198 0.00422095 0.00748701 0.00391861 0.00143623 19
produce:pandoraShowercalo:ShowerCalorimetry 3.95728 11.6571 34.6168 9.62808 7.50492 19
produce:pandoraShowercalonosce:ShowerCalorimetry 3.62612 11.2328 32.5868 8.81359 6.98672 19
[art]:TriggerResults:TriggerResultInserter 3.6544e-05 6.67449e-05 0.000154761 6.5353e-05 2.37574e-05 20
end_path:out1:RootOutput 3.0656e-05 6.30789e-05 0.000420388 4.4382e-05 8.22913e-05 20
end_path:out1:RootOutput(write) 3.9109e-05 5.66428 8.11631 6.15257 1.59624 20
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5659.33 MB
Peak resident set size usage (VmHWM): 3806.18 MB
Details saved in: 'mem.db'
====================================================================================================
Error running. Exiting with 245 