justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 421154.3@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID421154.3@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7876
Stage ID1
User namedrivera@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-24 16:24:57
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-06-25 02:03:18
From worker nodeHostnamecomp14-31
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-24 20:20:51
Input fileshd-protodune:np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453.hdf5
JobscriptExit code0
Real time5h (20526s)
CPU time5h (19809s = 96%)
Max RSS bytes2693447680 (2568 MiB)
Outputting started2025-06-25 02:02:58
Output files
Finished2025-06-25 02:03:18
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

trace set: wiener with 117101 traces, 117101 summary
Retagger: tagging trace set: gauss with 102284 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 102284 traces tagged "gauss"
FrameSaver: q=5.95335e+06 n=1867070 tag=gauss
wclsFrameSaver: saving 117101 traces tagged "wiener"
FrameSaver: q=6.67863e+06 n=1792708 tag=wiener
24-Jun-2025 22:08:04 BST  Opened output file with pattern "np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453_1_421154_3_1750796464_deco_reco.root"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 98175 traces, 98175 summary
Retagger: tagging trace set: gauss with 86407 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 86407 traces tagged "gauss"
FrameSaver: q=3.84081e+06 n=1461589 tag=gauss
wclsFrameSaver: saving 98175 traces tagged "wiener"
FrameSaver: q=4.34588e+06 n=1404765 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 118130 traces, 118130 summary
Retagger: tagging trace set: gauss with 103006 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 103006 traces tagged "gauss"
FrameSaver: q=1.09358e+07 n=2074275 tag=gauss
wclsFrameSaver: saving 118130 traces tagged "wiener"
FrameSaver: q=1.23302e+07 n=1999814 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 88174 traces, 88174 summary
Retagger: tagging trace set: gauss with 80150 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 80150 traces tagged "gauss"
FrameSaver: q=7.43513e+06 n=1585966 tag=gauss
wclsFrameSaver: saving 88174 traces tagged "wiener"
FrameSaver: q=8.37046e+06 n=1542934 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 102236 traces, 102236 summary
Retagger: tagging trace set: gauss with 92125 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 92125 traces tagged "gauss"
FrameSaver: q=1.24747e+07 n=1928685 tag=gauss
wclsFrameSaver: saving 102236 traces tagged "wiener"
FrameSaver: q=1.37966e+07 n=1882365 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 118051 traces, 118051 summary
Retagger: tagging trace set: gauss with 102298 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 102298 traces tagged "gauss"
FrameSaver: q=1.95019e+06 n=1520692 tag=gauss
wclsFrameSaver: saving 118051 traces tagged "wiener"
FrameSaver: q=2.26138e+06 n=1434949 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 86057 traces, 86057 summary
Retagger: tagging trace set: gauss with 78901 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 78901 traces tagged "gauss"
FrameSaver: q=9.55503e+06 n=1607342 tag=gauss
wclsFrameSaver: saving 86057 traces tagged "wiener"
FrameSaver: q=1.09429e+07 n=1572038 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 125376 traces, 125376 summary
Retagger: tagging trace set: gauss with 111410 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 111410 traces tagged "gauss"
FrameSaver: q=9.63592e+06 n=2265788 tag=gauss
wclsFrameSaver: saving 125376 traces tagged "wiener"
FrameSaver: q=1.09665e+07 n=2174060 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 134577 traces, 134577 summary
Retagger: tagging trace set: gauss with 114887 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 114887 traces tagged "gauss"
FrameSaver: q=7.2979e+06 n=2116650 tag=gauss
wclsFrameSaver: saving 134577 traces tagged "wiener"
FrameSaver: q=8.30688e+06 n=2013510 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 125426 traces, 125426 summary
Retagger: tagging trace set: gauss with 108860 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 108860 traces tagged "gauss"
FrameSaver: q=6.95688e+06 n=1955607 tag=gauss
wclsFrameSaver: saving 125426 traces tagged "wiener"
FrameSaver: q=7.89604e+06 n=1877784 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 89434 traces, 89434 summary
Retagger: tagging trace set: gauss with 81558 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 81558 traces tagged "gauss"
FrameSaver: q=2.99249e+06 n=1290493 tag=gauss
wclsFrameSaver: saving 89434 traces tagged "wiener"
FrameSaver: q=3.51597e+06 n=1255342 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 97919 traces, 97919 summary
Retagger: tagging trace set: gauss with 86046 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 86046 traces tagged "gauss"
FrameSaver: q=2.23122e+06 n=1255114 tag=gauss
wclsFrameSaver: saving 97919 traces tagged "wiener"
FrameSaver: q=2.6454e+06 n=1202633 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 92833 traces, 92833 summary
Retagger: tagging trace set: gauss with 84353 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 84353 traces tagged "gauss"
FrameSaver: q=6.11236e+06 n=1565444 tag=gauss
wclsFrameSaver: saving 92833 traces tagged "wiener"
FrameSaver: q=7.00443e+06 n=1512770 tag=wiener
25-Jun-2025 03:02:56 BST  Closed output file "np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453_1_421154_3_1750796464_deco_reco.root"
25-Jun-2025 03:02:56 BST  Closed input file "np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453_1_421154_3_1750796464_deco.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          290.133       1533.2        4360.31       1082.55       1178.33        13     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00052132    0.00259802    0.00889149    0.000719102   0.00329873       13     
produce:fembfilter:PDHDFEMBFilter                  0.672051       1.09039       1.41549       1.15779      0.211596        13     
produce:wclsdatahd:WireCellToolkit                  60.8452       82.8067       88.6053       84.6169       6.95204        13     
produce:gaushit:GausHitFinder                       1.41031       1.94489       2.54762       1.87583      0.372811        13     
produce:nhitsfilter:NumberOfHitsFilter            0.00095024    0.00129072    0.00158845    0.00127523    0.000212974      13     
produce:reco3d:SpacePointSolver                     58.1795       149.502       292.402       143.066       79.3656        13     
produce:hitpdune:DisambigFromSpacePoints            2.50768       8.13146       15.3131       8.23087       3.94113        13     
produce:pandora:StandardPandora                     117.749       1248.39       3908.52       791.369       1100.3         13     
produce:pandoraWriter:StandardPandora              0.355122      0.530295      0.657498      0.562859      0.103633        13     
produce:pandoraTrack:LArPandoraTrackCreation        3.27506       13.9253       27.8501       12.7568       6.38138        13     
produce:pandoraShower:LArPandoraShowerCreation      3.8283        12.8141       22.0177       13.6244       5.05959        13     
produce:pandoracalo:Calorimetry                     2.3064        6.03736       12.2678       5.49747       2.79643        13     
produce:pandoracalonosce:Calorimetry                2.61146       5.77286       11.6972       4.90036       2.57922        13     
[art]:TriggerResults:TriggerResultInserter        2.0479e-05    6.78164e-05   9.3711e-05    6.8663e-05    1.72175e-05      13     
end_path:out1:RootOutput                           9.95e-06     2.50912e-05   3.7157e-05    2.4836e-05    6.38286e-06      13     
end_path:out1:RootOutput(write)                     1.33251       1.92763       2.51575       1.92336      0.402094        13     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4474.3 MB
  Peak resident set size usage (VmHWM): 2693.45 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453_1_421154_3_1750796464_deco_reco.root', 'start_time': 1750796990.224443, 'end_time': 1750816977.7199366}
input flag: -i root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/hd-protodune/b6/d1/np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453.hdf5 --parent hd-protodune:np04hd_raw_run032793_0002_dataflow6_datawriter_0_20241120T203453.hdf5
justIN time: 2025-08-14 16:37:09 UTC       justIN version: 01.03.02