justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358072.42@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358072.42@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:05
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2025-03-27 12:37:16
From worker nodeHostnamedune001-5180343.0-lcg2709.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:26:42
Input filesmonte-carlo-005747-000082
JobscriptExit code0
Real time10m (618s)
CPU time10m (609s = 98%)
Max RSS bytes1730142208 (1649 MiB)
Outputting started2025-03-27 12:37:01
Output files
Finished2025-03-27 12:37:16
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Now with regularization...
Begin: 3.6699e+07
0 3.66913e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:33:02 UTC  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358072_42_1743078408_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:33:02 UTC  Opened input file "prod_muon_5GeV_protodunevd_1_358072_42_1743078408_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4356 traces, 4356 summary
Retagger: tagging trace set: gauss with 2750 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2750 traces tagged "gauss"
FrameSaver: q=539347 n=44336 tag=gauss
wclsFrameSaver: saving 4356 traces tagged "wiener"
FrameSaver: q=585975 n=36808 tag=wiener
27-Mar-2025 12:33:25 UTC  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358072_42_1743078408_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3873 traces, 3873 summary
Retagger: tagging trace set: gauss with 2488 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2488 traces tagged "gauss"
FrameSaver: q=408854 n=37327 tag=gauss
wclsFrameSaver: saving 3873 traces tagged "wiener"
FrameSaver: q=448018 n=30782 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3968 traces, 3968 summary
Retagger: tagging trace set: gauss with 2442 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2442 traces tagged "gauss"
FrameSaver: q=447415 n=37950 tag=gauss
wclsFrameSaver: saving 3968 traces tagged "wiener"
FrameSaver: q=490046 n=31281 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3908 traces, 3908 summary
Retagger: tagging trace set: gauss with 2421 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2421 traces tagged "gauss"
FrameSaver: q=424370 n=37127 tag=gauss
wclsFrameSaver: saving 3908 traces tagged "wiener"
FrameSaver: q=465581 n=30375 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4256 traces, 4256 summary
Retagger: tagging trace set: gauss with 2658 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2658 traces tagged "gauss"
FrameSaver: q=493525 n=41954 tag=gauss
wclsFrameSaver: saving 4256 traces tagged "wiener"
FrameSaver: q=537844 n=34728 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4249 traces, 4249 summary
Retagger: tagging trace set: gauss with 2605 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2605 traces tagged "gauss"
FrameSaver: q=524198 n=42816 tag=gauss
wclsFrameSaver: saving 4249 traces tagged "wiener"
FrameSaver: q=567347 n=35261 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3996 traces, 3996 summary
Retagger: tagging trace set: gauss with 2523 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2523 traces tagged "gauss"
FrameSaver: q=450911 n=39125 tag=gauss
wclsFrameSaver: saving 3996 traces tagged "wiener"
FrameSaver: q=492011 n=32212 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3976 traces, 3976 summary
Retagger: tagging trace set: gauss with 2426 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2426 traces tagged "gauss"
FrameSaver: q=418627 n=37729 tag=gauss
wclsFrameSaver: saving 3976 traces tagged "wiener"
FrameSaver: q=459942 n=30763 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3926 traces, 3926 summary
Retagger: tagging trace set: gauss with 2428 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2428 traces tagged "gauss"
FrameSaver: q=431314 n=37603 tag=gauss
wclsFrameSaver: saving 3926 traces tagged "wiener"
FrameSaver: q=471713 n=30966 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3907 traces, 3907 summary
Retagger: tagging trace set: gauss with 2432 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2432 traces tagged "gauss"
FrameSaver: q=428187 n=38012 tag=gauss
wclsFrameSaver: saving 3907 traces tagged "wiener"
FrameSaver: q=468179 n=31187 tag=wiener
27-Mar-2025 12:37:01 UTC  Closed output file "prod_muon_5GeV_protodunevd_1_358072_42_1743078408_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:37:01 UTC  Closed input file "prod_muon_5GeV_protodunevd_1_358072_42_1743078408_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       22.8267       23.6282        26.96        23.1434       1.16988        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000308265   0.000495299   0.000732726   0.000486449   0.00012976       10     
reco:rns:RandomNumberSaver                     3.7286e-05    6.28661e-05   0.00022486    4.3566e-05    5.43966e-05      10     
reco:wclsdatavd:WireCellToolkit                  21.5659       23.1923       26.6496       22.8152       1.2683         10     
reco:gaushit:GausHitFinder                      0.0183278     0.0205031     0.0262006     0.0190521    0.00246041       10     
reco:reco3d:SpacePointSolver                    0.153126      0.175187      0.192836      0.172445      0.0139085       10     
reco:hitpdune:DisambigFromSpacePoints          0.00203759    0.00247971    0.00428957    0.00230031    0.000627777      10     
reco:pandora:StandardPandora                    0.0905421     0.188684      0.959098      0.0995372     0.257015        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00266745     0.0035557    0.00626437    0.00315019    0.00108247       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000604733   0.00144812    0.00346206    0.00122824    0.00081012       10     
reco:pandoraStdcalo:Calorimetry                 0.0034113    0.00408632    0.00628537     0.0038538    0.000778767      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00140361    0.00167101    0.00328501    0.00151614    0.000541077      10     
[art]:TriggerResults:TriggerResultInserter     1.0857e-05    1.80994e-05    5.281e-05    1.2619e-05    1.24843e-05      10     
end_path:out1:RootOutput                        5.479e-06    8.0944e-06    1.9009e-05     6.62e-06     3.86736e-06      10     
end_path:out1:RootOutput(write)                 0.0301367     0.0340445     0.0546347     0.031291     0.00698819       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3725.59 MB
  Peak resident set size usage (VmHWM): 1583.56 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358072_42_1743078408_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078763.4277117, 'end_time': 1743079021.626812}
justIN time: 2025-04-03 08:25:42 UTC       justIN version: 01.03.00