21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 214563.1@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 214563.1@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7321 | |
| Stage ID | 1 | |
| User name | sergeym@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 4 |
| GPU | No | |
| RSS bytes | 15728640000 (15000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-04 22:02:50 | |
| Site | UK_Lancaster | |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 | |
| Last heartbeat | 2025-06-05 20:01:22 | |
| From worker node | Hostname | comp09-27 |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 4 | |
| RSS bytes | 15728640000 (15000 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-04 22:13:54 | |
| Input files | ||
| Jobscript | Exit code | 0 |
| Real time | 21h (78429s) | |
| CPU time | 21h (77249s = 98%) | |
| Max RSS bytes | 10305585152 (9828 MiB) | |
| Outputting started | 2025-06-05 20:01:04 | |
| Output files | ||
| Finished | 2025-06-05 20:01:22 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
, 1598 indices spanning [1599,3196] "loose_lf" [21:00:22.207] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 load plane index: 1, ntraces=2560, input bad regions: 0 [21:00:25.175] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=52742516.16154893 Qloss=0, 3198 indices spanning [0,4795] "tight_lf" [21:00:25.857] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=117564114.0195353 Qloss=0, 3196 indices spanning [1599,6393] "loose_lf" [21:00:27.441] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 load plane index: 2, ntraces=2560, input bad regions: 0 [21:00:30.537] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=136538124.6667983 Qloss=0, 5119 indices spanning [0,8314] "tight_lf" [21:00:30.631] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=136538124.6667983 Qloss=0, 5117 indices spanning [1599,10235] "loose_lf" [21:00:34.085] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 0, Qtot=134524345 Qloss=-3661979, 5674 indices spanning [90179,95852] "wiener" [21:00:34.637] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 0, Qtot=-0.02057290426455438 Qloss=0, 800 indices spanning [95853,96652] "decon" [21:00:34.747] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 0, Qtot=126541651 Qloss=-2719626, 5618 indices spanning [96653,102270] "gauss" [21:00:35.854] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=147459068 Qloss=-7546491, 9358 indices spanning [123196,132553] "wiener" [21:00:36.374] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=0.21140664559789002 Qloss=0, 1600 indices spanning [95853,133353] "decon" [21:00:36.484] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=136688570 Qloss=-3564291, 8000 indices spanning [133354,141353] "gauss" [21:00:37.481] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=140863219 Qloss=-4659860, 4539 indices spanning [149934,154472] "wiener" [21:00:38.407] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=135734058.1623144 Qloss=0, 3518 indices spanning [95853,156390] "decon" [21:00:38.547] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=137283653 Qloss=-1067871, 3221 indices spanning [156391,159611] "gauss" [21:00:38.552] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 produce 159612 traces: 19571 wiener0, 3518 decon_charge0, 16839 gauss0, frame tag: sigproc [21:00:38.552] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 output frame: frame: ident=149 time=-250000 tick=500 with 159612 traces. frame tags:[ "sigproc" ] 12 tagged trace sets:[ "break_roi_1st0":11318 [0] "break_roi_2nd0":11340 [0] "cleanup_roi0":10681 [0] "decon_charge0":3518 [0] "extend_roi0":12126 [0] "gauss0":16839 [0] "loose_lf0":5117 [0] "mp2_roi0":47369 [0] "mp3_roi0":5278 [0] "shrink_roi0":11336 [0] "tight_lf0":5119 [0] "wiener0":19571 [19571] ] cmm:[ ] [21:00:38.737] D [ hio ] <HDF5FrameTap:hio_sp0> HDF5FrameTap: see frame #149 with 159612 traces with frame tags: "sigproc" and trace tags: "break_roi_1st0" "break_roi_2nd0" "cleanup_roi0" "decon_charge0" "extend_roi0" "gauss0" "loose_lf0" "mp2_roi0" "mp3_roi0" "shrink_roi0" "tight_lf0" "wiener0" looking for tags: "loose_lf0" "cleanup_roi0" "mp3_roi0" "mp2_roi0" [21:00:38.992] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:00:39.265] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_loose_lf0 with 2560 channels 6000 ticks @t=-0.25 ms qtot=344497020 [21:00:39.304] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:00:39.553] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_cleanup_roi0 with 2082 channels 5999 ticks @t=-0.25 ms qtot=440484930 [21:00:39.582] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:00:39.714] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_mp3_roi0 with 1418 channels 6000 ticks @t=-0.25 ms qtot=534352000 [21:00:39.749] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:00:39.891] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_mp2_roi0 with 1600 channels 6000 ticks @t=-0.25 ms qtot=3229524000 [21:00:39.893] D [ glue ] sink frame: #149 @-0.25 with 159612 traces, frame tags:[sigproc], trace tags:[break_roi_1st0, break_roi_2nd0, cleanup_roi0, decon_charge0, extend_roi0, gauss0, loose_lf0, mp2_roi0, mp3_roi0, shrink_roi0, tight_lf0, wiener0] [21:00:39.943] D [ gen ] <DepoTransform:depotransform-apa0> EOS at call=99 [21:00:39.943] D [ gen ] <Reframer:reframer-apa0> EOS at call=99 [21:00:39.943] D [ gen ] <IncoherentAddNoise:addnoise-empericalnoise-apa0> EOS at call=99 [21:00:39.943] D [ gen ] <Digitizer:digitizer-apa0> see EOS at call=99 [21:00:39.943] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=99 anode=0 [21:00:39.943] D [ hio ] <HDF5FrameTap:hio_sp0> EOS at call=99 [21:00:39.943] D [ glue ] frame sink sees EOS [21:00:39.943] D [ gen ] <DepoBagger:0> send bag #49 with 545896 depos followed by EOS [21:00:41.419] D [ gen ] <DepoFluxSplat:0> splat 49 ndepos=97185/545896/[0] ntraces=1378364 [21:00:41.846] D [ hio ] <HDF5FrameTap:hio_truth0> HDF5FrameTap: see frame #49 with 1378364 traces with frame tags: "deposplat0" and trace tags: looking for tags: "deposplat0" [21:00:42.282] D [ hio ] <HDF5FrameTap:hio_truth0> gzip:0 chunks:[0,0] [21:00:42.529] D [ hio ] <HDF5FrameTap:hio_truth0> saved /49/frame_deposplat0 with 2068 channels 6007 ticks @t=-0.3125 ms qtot=410844960 [21:00:42.575] D [ glue ] sink frame: #49 @-0.3125 with 1378364 traces, frame tags:[deposplat0], trace tags:[] [21:00:42.985] D [ gen ] <DepoFluxSplat:0> EOS at 99 [21:00:42.985] D [ hio ] <HDF5FrameTap:hio_truth0> EOS at call=99 [21:00:42.985] D [ glue ] frame sink sees EOS [21:00:42.985] D [ pgraph ] <Pgrapher:> graph execution complete [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DepoTransform : 55.59 sec [21:00:42.989] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 26.87 sec [21:00:42.989] I [ timer ] Timer: WireCell::Hio::HDF5FrameTap : 6.69 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DepoFluxSplat : 6.37 sec [21:00:42.989] I [ timer ] Timer: WireCell::Hio::HDF5FrameTap : 6.14 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DumpFrames : 5.82 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::IncoherentAddNoise : 5.66 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::Reframer : 5.5 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::Digitizer : 5.46 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::Retagger : 5.43 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DumpFrames : 5.37 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DepoBagger : 5.33 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::Drifter : 5.31 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DepoBagger : 5.23 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DepoFanout : 4.11 sec [21:00:42.989] I [ timer ] Timer: WireCell::Gen::DepoFanout : 3.06 sec [21:00:42.989] I [ timer ] Timer: wcls::SimDepoSource : 1.36 sec [21:00:42.989] I [ timer ] Timer: Total node execution : 159.30000054836273 sec 05-Jun-2025 21:00:55 BST Closed output file "g4_deposplat.root" 05-Jun-2025 21:00:55 BST Closed input file "g4.root" ================================================================================================================================ TimeTracker printout (sec) Min Avg Max Median RMS nEvts ================================================================================================================================ Full event 151.082 245.715 351.039 246.809 36.3451 50 -------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00120353 0.0210392 0.738505 0.00395639 0.103195 50 reco:rns:RandomNumberSaver 5.863e-05 0.0267994 1.33595 6.21735e-05 0.187022 50 reco:wirecell:WireCellToolkit 151.073 245.51 348.956 246.804 36.415 50 [art]:TriggerResults:TriggerResultInserter 8.0748e-05 0.000117517 0.000617584 8.72575e-05 0.000112943 50 end_path:out1:RootOutput 7.18e-06 1.19265e-05 0.000158503 8.6165e-06 2.11149e-05 50 end_path:out1:RootOutput(write) 0.000308783 0.155717 7.44134 0.000486163 1.04171 50 ================================================================================================================================ ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 5804.84 MB Peak resident set size usage (VmHWM): 3286.97 MB Details saved in: 'mem.db' ==================================================================================================== [21:00:56.813] D [ main ] finalizing component: "HDF5FrameTap":"hio_truth0" [21:00:56.986] D [ main ] finalizing component: "HDF5FrameTap":"hio_sp0" ToolBasedRawDigitPrepService:dtor: Event count: 0 ToolBasedRawDigitPrepService:dtor: Call count: 0 ToolBasedRawDigitPrepService:dtor: Time report for 4 tools. ToolBasedRawDigitPrepService:dtor: digitReader: 0.00 sec ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.00 sec ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.00 sec ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 0.00 sec Art has completed and will exit with status 0.