justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 418934.12@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID418934.12@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7811
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-20 16:39:59
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-06-20 23:26:31
From worker nodeHostnamewn1208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 20:11:18
Input fileshd-protodune:np04hd_raw_run027087_0350_dataflow1_datawriter_0_20240614T175306.hdf5
JobscriptExit code0
Real time3h (11657s)
CPU time3h (11027s = 94%)
Max RSS bytes2676117504 (2552 MiB)
Outputting started2025-06-20 23:25:36
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/calcuttj/new_data_proc_v10_04_06d00/07811/1/002/pdhd_processing_1_418934_12_1750450294_stage1_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/calcuttj/new_data_proc_v10_04_06d00/07811/1/002/pdhd_processing_1_418934_12_1750450294_stage1_stage2_tfile.root
Finished2025-06-20 23:26:31
Saved logsjustin-logs:418934.12-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

q=2.62841e+06 n=1105153 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 72662 traces, 72662 summary
Retagger: tagging trace set: gauss with 58046 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58046 traces tagged "gauss"
FrameSaver: q=1.51617e+06 n=901309 tag=gauss
wclsFrameSaver: saving 72662 traces tagged "wiener"
FrameSaver: q=1.71405e+06 n=844353 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 71722 traces, 71722 summary
Retagger: tagging trace set: gauss with 57852 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 57852 traces tagged "gauss"
FrameSaver: q=1.49379e+06 n=870649 tag=gauss
wclsFrameSaver: saving 71722 traces tagged "wiener"
FrameSaver: q=1.71356e+06 n=823182 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 84618 traces, 84618 summary
Retagger: tagging trace set: gauss with 64861 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 64861 traces tagged "gauss"
FrameSaver: q=2.10224e+06 n=1060190 tag=gauss
wclsFrameSaver: saving 84618 traces tagged "wiener"
FrameSaver: q=2.37724e+06 n=1013639 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 95316 traces, 95316 summary
Retagger: tagging trace set: gauss with 75842 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 75842 traces tagged "gauss"
FrameSaver: q=2.30081e+06 n=1216097 tag=gauss
wclsFrameSaver: saving 95316 traces tagged "wiener"
FrameSaver: q=2.67868e+06 n=1175609 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 65207 traces, 65207 summary
Retagger: tagging trace set: gauss with 53299 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53299 traces tagged "gauss"
FrameSaver: q=1.18909e+06 n=693603 tag=gauss
wclsFrameSaver: saving 65207 traces tagged "wiener"
FrameSaver: q=1.48485e+06 n=663488 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 77778 traces, 77778 summary
Retagger: tagging trace set: gauss with 60624 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60624 traces tagged "gauss"
FrameSaver: q=1.61722e+06 n=873095 tag=gauss
wclsFrameSaver: saving 77778 traces tagged "wiener"
FrameSaver: q=1.91125e+06 n=830989 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 79826 traces, 79826 summary
Retagger: tagging trace set: gauss with 62498 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 62498 traces tagged "gauss"
FrameSaver: q=1.7044e+06 n=902168 tag=gauss
wclsFrameSaver: saving 79826 traces tagged "wiener"
FrameSaver: q=2.05127e+06 n=888116 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 73399 traces, 73399 summary
Retagger: tagging trace set: gauss with 58382 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58382 traces tagged "gauss"
FrameSaver: q=1.42779e+06 n=815167 tag=gauss
wclsFrameSaver: saving 73399 traces tagged "wiener"
FrameSaver: q=1.71554e+06 n=778865 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 90565 traces, 90565 summary
Retagger: tagging trace set: gauss with 69857 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69857 traces tagged "gauss"
FrameSaver: q=2.05092e+06 n=1066122 tag=gauss
wclsFrameSaver: saving 90565 traces tagged "wiener"
FrameSaver: q=2.41449e+06 n=1022605 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 82848 traces, 82848 summary
Retagger: tagging trace set: gauss with 65392 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 65392 traces tagged "gauss"
FrameSaver: q=1.85193e+06 n=1036205 tag=gauss
wclsFrameSaver: saving 82848 traces tagged "wiener"
FrameSaver: q=2.09586e+06 n=965334 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 63811 traces, 63811 summary
Retagger: tagging trace set: gauss with 53524 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53524 traces tagged "gauss"
FrameSaver: q=1.21139e+06 n=766623 tag=gauss
wclsFrameSaver: saving 63811 traces tagged "wiener"
FrameSaver: q=1.44236e+06 n=751587 tag=wiener
21-Jun-2025 00:25:35 BST  Closed output file "pdhd_processing_1_418934_12_1750450294_stage1_stage2.root"
21-Jun-2025 00:25:35 BST  Closed input file "pdhd_processing_1_418934_12_1750450294_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              164.56        247.118       392.299       226.509       64.6119        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00156334     0.0054663     0.0192587    0.00493746    0.00302439       40     
produce:fembfilter:PDHDFEMBFilter                       1.05963       1.17242       1.42173       1.1451       0.0787076       40     
produce:wclsdatahd:WireCellToolkit                      64.3561       78.8462       87.6409       79.9701       5.78745        40     
produce:gaushit:GausHitFinder                          0.808205       1.44217       2.32083       1.3881       0.329944        40     
produce:nhitsfilter:NumberOfHitsFilter                0.000528597   0.000882734   0.00159045    0.000831499   0.000202145      40     
produce:reco3d:SpacePointSolver                        0.401787       1.94222       8.36125       1.35479       1.79639        40     
produce:hitpdune:DisambigFromSpacePoints               0.338054       1.03568       2.24699      0.942631      0.474781        40     
produce:pandora:StandardPandora                         17.4773       43.6368       101.586       35.4299       23.5863        40     
produce:pandoraWriter:StandardPandora                  0.232334      0.386226      0.588511      0.381021      0.0800774       40     
produce:pandoraTrack:LArPandoraTrackCreation            1.98414       4.71642       10.6078       4.3626        1.9192         40     
produce:pandoraShower:LArPandoraShowerCreation          2.34745       5.77046       16.0759       5.34415       3.03741        40     
produce:pandoracalo:Calorimetry                        0.848606       2.20044       6.4496        2.00218       1.07382        40     
produce:pandoracalonosce:Calorimetry                   0.774234       2.00567       5.70431       1.81521      0.990833        40     
produce:pandorapid:Chi2ParticleID                      0.001918     0.00412181    0.00646949    0.00403159    0.000886999      40     
produce:pandoraShowercalo:ShowerCalorimetry             1.53617       4.90289       15.6943       4.07726       2.75746        40     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.34732       4.63295       13.2676       4.3527        2.44123        40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.148101      0.283035      0.463824      0.286186      0.0614209       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0482658     0.0993685     0.157052      0.0982527     0.0231307       40     
produce:emtrkmichelid:EmTrackMichelId                   36.7806       61.2099       97.7596       57.0593       13.9767        40     
[art]:TriggerResults:TriggerResultInserter            4.4544e-05    7.75664e-05   0.000123614   8.03115e-05   2.05854e-05      40     
end_path:out1:RootOutput                               6.021e-06    1.39366e-05   3.9805e-05    1.41665e-05   6.76406e-06      40     
end_path:michelremoving:michelremoving                 0.0323044     0.125669      0.428489      0.0742478     0.107003        40     
end_path:michelremovinggnocchi:michelremoving          0.0296562     0.126002      0.447607      0.0718176      0.11316        40     
end_path:calibanagnocchi:CalibAnaTree                   12.2054       30.6167       91.1072       22.3562       19.1489        40     
end_path:out1:RootOutput(write)                         1.10361       1.6141        2.64174       1.58137      0.315861        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5079.32 MB
  Peak resident set size usage (VmHWM): 2676.12 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_418934_12_1750450294_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_418934_12_1750450294_stage1_stage2_tfile.root', 'start_time': 1750451966.9319558, 'end_time': 1750461936.4305232}
justIN time: 2025-08-15 08:15:02 UTC       justIN version: 01.03.02