justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358072.13@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358072.13@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:05
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2025-03-27 12:41:15
From worker nodeHostnamedune001-4917530.0-lcg2715.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:26:33
Input filesmonte-carlo-005747-000062
JobscriptExit code0
Real time14m (863s)
CPU time14m (847s = 98%)
Max RSS bytes1714962432 (1635 MiB)
Outputting started2025-03-27 12:40:56
Output files
Finished2025-03-27 12:41:15
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ow with regularization...
Begin: 2.11835e+07
0 2.11826e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:36:02 UTC  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358072_13_1743078401_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:36:02 UTC  Opened input file "prod_muon_5GeV_protodunevd_1_358072_13_1743078401_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3832 traces, 3832 summary
Retagger: tagging trace set: gauss with 2388 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2388 traces tagged "gauss"
FrameSaver: q=392058 n=36490 tag=gauss
wclsFrameSaver: saving 3832 traces tagged "wiener"
FrameSaver: q=431981 n=29886 tag=wiener
27-Mar-2025 12:36:30 UTC  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358072_13_1743078401_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4092 traces, 4092 summary
Retagger: tagging trace set: gauss with 2590 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2590 traces tagged "gauss"
FrameSaver: q=458712 n=40269 tag=gauss
wclsFrameSaver: saving 4092 traces tagged "wiener"
FrameSaver: q=500902 n=33183 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4441 traces, 4441 summary
Retagger: tagging trace set: gauss with 2761 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2761 traces tagged "gauss"
FrameSaver: q=552007 n=44774 tag=gauss
wclsFrameSaver: saving 4441 traces tagged "wiener"
FrameSaver: q=595939 n=36948 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3902 traces, 3902 summary
Retagger: tagging trace set: gauss with 2430 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2430 traces tagged "gauss"
FrameSaver: q=407662 n=37151 tag=gauss
wclsFrameSaver: saving 3902 traces tagged "wiener"
FrameSaver: q=447846 n=30546 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3965 traces, 3965 summary
Retagger: tagging trace set: gauss with 2509 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2509 traces tagged "gauss"
FrameSaver: q=430137 n=38020 tag=gauss
wclsFrameSaver: saving 3965 traces tagged "wiener"
FrameSaver: q=471093 n=31322 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4783 traces, 4783 summary
Retagger: tagging trace set: gauss with 2973 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2973 traces tagged "gauss"
FrameSaver: q=615205 n=50113 tag=gauss
wclsFrameSaver: saving 4783 traces tagged "wiener"
FrameSaver: q=662088 n=41433 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4465 traces, 4465 summary
Retagger: tagging trace set: gauss with 2758 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2758 traces tagged "gauss"
FrameSaver: q=550544 n=45939 tag=gauss
wclsFrameSaver: saving 4465 traces tagged "wiener"
FrameSaver: q=593670 n=38144 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4022 traces, 4022 summary
Retagger: tagging trace set: gauss with 2535 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2535 traces tagged "gauss"
FrameSaver: q=440466 n=39083 tag=gauss
wclsFrameSaver: saving 4022 traces tagged "wiener"
FrameSaver: q=481291 n=32284 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4213 traces, 4213 summary
Retagger: tagging trace set: gauss with 2690 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2690 traces tagged "gauss"
FrameSaver: q=495245 n=41984 tag=gauss
wclsFrameSaver: saving 4213 traces tagged "wiener"
FrameSaver: q=539613 n=34911 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3795 traces, 3795 summary
Retagger: tagging trace set: gauss with 2409 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2409 traces tagged "gauss"
FrameSaver: q=415860 n=36775 tag=gauss
wclsFrameSaver: saving 3795 traces tagged "wiener"
FrameSaver: q=454337 n=30211 tag=wiener
27-Mar-2025 12:40:56 UTC  Closed output file "prod_muon_5GeV_protodunevd_1_358072_13_1743078401_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:40:56 UTC  Closed input file "prod_muon_5GeV_protodunevd_1_358072_13_1743078401_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       25.6386       29.0783       33.0764       28.3285       2.57013        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000417317   0.000742334    0.0011525    0.000708029   0.000245819      10     
reco:rns:RandomNumberSaver                     4.8943e-05    9.30897e-05   0.000375244   6.13115e-05   9.4576e-05       10     
reco:wclsdatavd:WireCellToolkit                  25.2529       28.5241       32.5792       27.9013       2.60273        10     
reco:gaushit:GausHitFinder                      0.0194828     0.0271363     0.0400882     0.0267162    0.00636698       10     
reco:reco3d:SpacePointSolver                    0.186029      0.222137      0.289954      0.214182      0.032515        10     
reco:hitpdune:DisambigFromSpacePoints          0.00220494    0.00312488    0.00416612    0.00307167    0.000598509      10     
reco:pandora:StandardPandora                    0.106539       0.24031       1.0984       0.143531      0.287369        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00308927    0.00483896    0.00648716    0.00493479    0.00102917       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000801754   0.00188063    0.00287973    0.00197064    0.000637424      10     
reco:pandoraStdcalo:Calorimetry                0.00399433    0.00508853    0.00652421    0.00507715    0.00077372       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00159571    0.00202824    0.00295974    0.00181099    0.000469802      10     
[art]:TriggerResults:TriggerResultInserter     1.1707e-05    1.75102e-05   4.2654e-05    1.4206e-05    8.80818e-06      10     
end_path:out1:RootOutput                        6.72e-06     9.4561e-06    1.9629e-05    7.8315e-06    3.65256e-06      10     
end_path:out1:RootOutput(write)                 0.0333142     0.0423423     0.0570644     0.0403896    0.00783808       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3701.93 MB
  Peak resident set size usage (VmHWM): 1604.7 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358072_13_1743078401_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078931.4201686, 'end_time': 1743079256.6736374}
justIN time: 2025-04-03 08:25:27 UTC       justIN version: 01.03.00