justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358067.156@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358067.156@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:50
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 11:59:16
From worker nodeHostnamet2wn168.physics.ox.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:37:27
Input filesmonte-carlo-005747-000129
JobscriptExit code0
Real time21m (1280s)
CPU time20m (1233s = 96%)
Max RSS bytes1739583488 (1658 MiB)
Outputting started2025-03-27 11:58:47
Output files
Finished2025-03-27 11:59:16
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

h regularization...
Begin: 2.28078e+07
0 2.28036e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:51:31 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358067_156_1743075462_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:51:31 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358067_156_1743075462_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3975 traces, 3975 summary
Retagger: tagging trace set: gauss with 2463 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2463 traces tagged "gauss"
FrameSaver: q=435328 n=38130 tag=gauss
wclsFrameSaver: saving 3975 traces tagged "wiener"
FrameSaver: q=475661 n=31261 tag=wiener
27-Mar-2025 11:52:14 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358067_156_1743075462_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3913 traces, 3913 summary
Retagger: tagging trace set: gauss with 2392 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2392 traces tagged "gauss"
FrameSaver: q=447776 n=37813 tag=gauss
wclsFrameSaver: saving 3913 traces tagged "wiener"
FrameSaver: q=489178 n=31055 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3923 traces, 3923 summary
Retagger: tagging trace set: gauss with 2440 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2440 traces tagged "gauss"
FrameSaver: q=421402 n=37407 tag=gauss
wclsFrameSaver: saving 3923 traces tagged "wiener"
FrameSaver: q=461750 n=30862 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3938 traces, 3938 summary
Retagger: tagging trace set: gauss with 2457 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2457 traces tagged "gauss"
FrameSaver: q=417071 n=37666 tag=gauss
wclsFrameSaver: saving 3938 traces tagged "wiener"
FrameSaver: q=457295 n=30946 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3987 traces, 3987 summary
Retagger: tagging trace set: gauss with 2529 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2529 traces tagged "gauss"
FrameSaver: q=436554 n=38245 tag=gauss
wclsFrameSaver: saving 3987 traces tagged "wiener"
FrameSaver: q=478085 n=31560 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4065 traces, 4065 summary
Retagger: tagging trace set: gauss with 2572 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2572 traces tagged "gauss"
FrameSaver: q=438471 n=39755 tag=gauss
wclsFrameSaver: saving 4065 traces tagged "wiener"
FrameSaver: q=478429 n=33037 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3866 traces, 3866 summary
Retagger: tagging trace set: gauss with 2438 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2438 traces tagged "gauss"
FrameSaver: q=425503 n=37144 tag=gauss
wclsFrameSaver: saving 3866 traces tagged "wiener"
FrameSaver: q=465219 n=30615 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3986 traces, 3986 summary
Retagger: tagging trace set: gauss with 2482 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2482 traces tagged "gauss"
FrameSaver: q=442497 n=38598 tag=gauss
wclsFrameSaver: saving 3986 traces tagged "wiener"
FrameSaver: q=484004 n=31701 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4556 traces, 4556 summary
Retagger: tagging trace set: gauss with 2766 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2766 traces tagged "gauss"
FrameSaver: q=551574 n=46363 tag=gauss
wclsFrameSaver: saving 4556 traces tagged "wiener"
FrameSaver: q=597853 n=38262 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3971 traces, 3971 summary
Retagger: tagging trace set: gauss with 2492 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2492 traces tagged "gauss"
FrameSaver: q=444950 n=38760 tag=gauss
wclsFrameSaver: saving 3971 traces tagged "wiener"
FrameSaver: q=485984 n=32129 tag=wiener
27-Mar-2025 11:58:47 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358067_156_1743075462_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 11:58:47 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358067_156_1743075462_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.5114       42.8341       47.5025       42.2433       2.26338        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000673511   0.000795918   0.00120655     0.0007818    0.000146889      10     
reco:rns:RandomNumberSaver                     4.8331e-05    9.27933e-05   0.000406707   5.87965e-05   0.000104905      10     
reco:wclsdatavd:WireCellToolkit                  39.1497       41.9207       46.6832       41.107        2.40637        10     
reco:gaushit:GausHitFinder                      0.0275419     0.0416218     0.0514025     0.0434414    0.00801621       10     
reco:reco3d:SpacePointSolver                    0.235752      0.326505      0.402153      0.333429      0.0496714       10     
reco:hitpdune:DisambigFromSpacePoints          0.00326053    0.00546949     0.0127188     0.0048711    0.00254379       10     
reco:pandora:StandardPandora                    0.176824       0.44303       2.29576      0.250215      0.618231        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00365562    0.00677312     0.0137835    0.00632793    0.00259762       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00142429    0.00312004    0.00630661    0.00309261    0.00128675       10     
reco:pandoraStdcalo:Calorimetry                0.00544426    0.00863167     0.0149776    0.00812364    0.00248851       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00265792    0.00406964    0.00926969    0.00364327    0.00183574       10     
[art]:TriggerResults:TriggerResultInserter     1.4457e-05    3.64157e-05   0.000127792   2.52925e-05   3.22232e-05      10     
end_path:out1:RootOutput                        6.572e-06    1.26841e-05   4.0646e-05    9.5735e-06    9.48529e-06      10     
end_path:out1:RootOutput(write)                 0.0554047     0.0678809     0.0765036     0.0691005    0.00666626       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3422.46 MB
  Peak resident set size usage (VmHWM): 1586.2 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358067_156_1743075462_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076252.1937063, 'end_time': 1743076727.7572896}
justIN time: 2025-04-03 08:30:35 UTC       justIN version: 01.03.00