21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 214563.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 214563.0@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7321 | |
| Stage ID | 1 | |
| User name | sergeym@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 4 |
| GPU | No | |
| RSS bytes | 15728640000 (15000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-04 22:02:50 | |
| Site | UK_Brunel | |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_22 | |
| Last heartbeat | 2025-06-05 20:18:00 | |
| From worker node | Hostname | wn-b8-26-01.brunel.ac.uk |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 4 | |
| RSS bytes | 15728640000 (15000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-04 22:05:20 | |
| Input files | monte-carlo-007321-000001 | |
| Jobscript | Exit code | 0 |
| Real time | 22h (79937s) | |
| CPU time | 21h (77347s = 96%) | |
| Max RSS bytes | 10726084608 (10229 MiB) | |
| Outputting started | 2025-06-05 20:17:38 | |
| Output files | ||
| Finished | 2025-06-05 20:18:00 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
7 indices spanning [1600,3196] "loose_lf" [21:17:15.341] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 load plane index: 1, ntraces=2560, input bad regions: 0 [21:17:17.060] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=50035035.74619831 Qloss=0, 3198 indices spanning [0,4794] "tight_lf" [21:17:17.469] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=136529294.68140584 Qloss=0, 3194 indices spanning [1600,6391] "loose_lf" [21:17:18.504] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 load plane index: 2, ntraces=2560, input bad regions: 0 [21:17:20.097] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=173042766.54035208 Qloss=0, 5118 indices spanning [0,8311] "tight_lf" [21:17:20.142] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=173042766.54035208 Qloss=0, 5114 indices spanning [1600,10231] "loose_lf" [21:17:24.289] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 0, Qtot=186702425 Qloss=-8151869, 9033 indices spanning [97834,106866] "wiener" [21:17:24.549] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 0, Qtot=0.15052469517104328 Qloss=0, 800 indices spanning [106867,107666] "decon" [21:17:24.650] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 0, Qtot=174058348 Qloss=-2965686, 7488 indices spanning [107667,115154] "gauss" [21:17:25.530] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=181941745 Qloss=-9537193, 9900 indices spanning [137239,147138] "wiener" [21:17:25.819] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=0.20090660359710455 Qloss=0, 1600 indices spanning [106867,147938] "decon" [21:17:25.919] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 1, Qtot=168093170 Qloss=-3679860, 7775 indices spanning [147939,155713] "gauss" [21:17:26.398] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=179997297 Qloss=-9050579, 5065 indices spanning [169166,174230] "wiener" [21:17:26.824] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=169737990.84267762 Qloss=0, 3519 indices spanning [106867,176149] "decon" [21:17:26.893] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 save plane index: 2, Qtot=172043191 Qloss=-833991, 4050 indices spanning [176150,180199] "gauss" [21:17:26.894] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 produce 180200 traces: 23998 wiener0, 3519 decon_charge0, 19313 gauss0, frame tag: sigproc [21:17:26.894] D [sigproc ] <OmnibusSigProc:apa0sigproc0> call=98 output frame: frame: ident=149 time=-250000 tick=500 with 180200 traces. frame tags:[ "sigproc" ] 12 tagged trace sets:[ "break_roi_1st0":13865 [0] "break_roi_2nd0":13972 [0] "cleanup_roi0":12717 [0] "decon_charge0":3519 [0] "extend_roi0":14802 [0] "gauss0":19313 [0] "loose_lf0":5114 [0] "mp2_roi0":46683 [0] "mp3_roi0":7141 [0] "shrink_roi0":13958 [0] "tight_lf0":5118 [0] "wiener0":23998 [23998] ] cmm:[ ] [21:17:27.147] D [ hio ] <HDF5FrameTap:hio_sp0> HDF5FrameTap: see frame #149 with 180200 traces with frame tags: "sigproc" and trace tags: "break_roi_1st0" "break_roi_2nd0" "cleanup_roi0" "decon_charge0" "extend_roi0" "gauss0" "loose_lf0" "mp2_roi0" "mp3_roi0" "shrink_roi0" "tight_lf0" "wiener0" looking for tags: "loose_lf0" "cleanup_roi0" "mp3_roi0" "mp2_roi0" [21:17:27.328] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:17:28.236] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_loose_lf0 with 2560 channels 6000 ticks @t=-0.25 ms qtot=451747650 [21:17:28.253] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:17:28.979] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_cleanup_roi0 with 2085 channels 5999 ticks @t=-0.25 ms qtot=564930000 [21:17:28.993] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:17:29.482] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_mp3_roi0 with 1600 channels 5993 ticks @t=-0.25 ms qtot=610756000 [21:17:29.499] D [ hio ] <HDF5FrameTap:hio_sp0> gzip:0 chunks:[0,0] [21:17:30.129] D [ hio ] <HDF5FrameTap:hio_sp0> saved /149/frame_mp2_roi0 with 1600 channels 5997 ticks @t=-0.25 ms qtot=3147212000 [21:17:30.131] D [ glue ] sink frame: #149 @-0.25 with 180200 traces, frame tags:[sigproc], trace tags:[break_roi_1st0, break_roi_2nd0, cleanup_roi0, decon_charge0, extend_roi0, gauss0, loose_lf0, mp2_roi0, mp3_roi0, shrink_roi0, tight_lf0, wiener0] [21:17:30.182] D [ gen ] <DepoTransform:depotransform-apa0> EOS at call=99 [21:17:30.182] D [ gen ] <Reframer:reframer-apa0> EOS at call=99 [21:17:30.182] D [ gen ] <IncoherentAddNoise:addnoise-empericalnoise-apa0> EOS at call=99 [21:17:30.182] D [ gen ] <Digitizer:digitizer-apa0> see EOS at call=99 [21:17:30.182] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=99 anode=0 [21:17:30.182] D [ hio ] <HDF5FrameTap:hio_sp0> EOS at call=99 [21:17:30.182] D [ glue ] frame sink sees EOS [21:17:30.183] D [ gen ] <DepoBagger:0> send bag #49 with 422443 depos followed by EOS [21:17:32.525] D [ gen ] <DepoFluxSplat:0> splat 49 ndepos=120252/422443/[0] ntraces=1732344 [21:17:32.872] D [ hio ] <HDF5FrameTap:hio_truth0> HDF5FrameTap: see frame #49 with 1732344 traces with frame tags: "deposplat0" and trace tags: looking for tags: "deposplat0" [21:17:33.358] D [ hio ] <HDF5FrameTap:hio_truth0> gzip:0 chunks:[0,0] [21:17:34.181] D [ hio ] <HDF5FrameTap:hio_truth0> saved /49/frame_deposplat0 with 2080 channels 6008 ticks @t=-0.3125 ms qtot=515037150 [21:17:34.223] D [ glue ] sink frame: #49 @-0.3125 with 1732344 traces, frame tags:[deposplat0], trace tags:[] [21:17:34.558] D [ gen ] <DepoFluxSplat:0> EOS at 99 [21:17:34.558] D [ hio ] <HDF5FrameTap:hio_truth0> EOS at call=99 [21:17:34.558] D [ glue ] frame sink sees EOS [21:17:34.558] D [ pgraph ] <Pgrapher:> graph execution complete [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DepoTransform : 55.25 sec [21:17:34.562] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 20.32 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DepoFluxSplat : 8.18 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::Digitizer : 6.46 sec [21:17:34.562] I [ timer ] Timer: WireCell::Hio::HDF5FrameTap : 6.3 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DumpFrames : 6.13 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DepoBagger : 5.96 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::IncoherentAddNoise : 5.92 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DumpFrames : 5.92 sec [21:17:34.562] I [ timer ] Timer: WireCell::Hio::HDF5FrameTap : 5.89 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::Retagger : 5.85 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::Reframer : 5.76 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DepoBagger : 5.24 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::Drifter : 4.9 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DepoFanout : 4.68 sec [21:17:34.562] I [ timer ] Timer: WireCell::Gen::DepoFanout : 3.4 sec [21:17:34.562] I [ timer ] Timer: wcls::SimDepoSource : 1.4 sec [21:17:34.562] I [ timer ] Timer: Total node execution : 157.5600003004074 sec 05-Jun-2025 21:17:36 BST Closed output file "g4_deposplat.root" 05-Jun-2025 21:17:36 BST Closed input file "g4.root" ================================================================================================================================ TimeTracker printout (sec) Min Avg Max Median RMS nEvts ================================================================================================================================ Full event 166.948 272.328 432.862 267.697 61.0433 50 -------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00120905 0.00603777 0.0248774 0.00428344 0.0057812 50 reco:rns:RandomNumberSaver 4.8272e-05 8.07934e-05 0.000641881 6.32485e-05 8.54659e-05 50 reco:wirecell:WireCellToolkit 166.943 272.282 432.85 267.692 61.0302 50 [art]:TriggerResults:TriggerResultInserter 4.5127e-05 0.000133934 0.00165102 6.1121e-05 0.000246611 50 end_path:out1:RootOutput 6.904e-06 1.32419e-05 0.000143289 1.03225e-05 1.88871e-05 50 end_path:out1:RootOutput(write) 0.000174053 0.000696329 0.00608852 0.000300502 0.00129566 50 ================================================================================================================================ ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 6533.82 MB Peak resident set size usage (VmHWM): 4332.96 MB Details saved in: 'mem.db' ==================================================================================================== [21:17:37.363] D [ main ] finalizing component: "HDF5FrameTap":"hio_truth0" [21:17:37.390] D [ main ] finalizing component: "HDF5FrameTap":"hio_sp0" ToolBasedRawDigitPrepService:dtor: Event count: 0 ToolBasedRawDigitPrepService:dtor: Call count: 0 ToolBasedRawDigitPrepService:dtor: Time report for 4 tools. ToolBasedRawDigitPrepService:dtor: digitReader: 0.00 sec ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.00 sec ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.00 sec ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 0.00 sec Art has completed and will exit with status 0.