justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.27@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.27@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:33:03
From worker nodeHostnamet2wn065.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:10:43
Input filesmonte-carlo-005746-000158
JobscriptExit code0
Real time20m (1227s)
CPU time16m (968s = 78%)
Max RSS bytes1661624320 (1584 MiB)
Outputting started2025-03-27 12:31:11
Output files
Finished2025-03-27 12:33:03
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

no regularization...
Begin: 4.5324e+07
0 4.38135e+07
1 4.37968e+07
Now with regularization...
Begin: 4.2887e+07
0 4.2886e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:24:05 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_27_1743077482_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:24:07 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166657_27_1743077482_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1677 traces, 1677 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=142842 n=17209 tag=gauss
wclsFrameSaver: saving 1677 traces tagged "wiener"
FrameSaver: q=161122 n=14892 tag=wiener
27-Mar-2025 12:24:50 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_27_1743077482_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1728 traces, 1728 summary
Retagger: tagging trace set: gauss with 1258 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1258 traces tagged "gauss"
FrameSaver: q=141797 n=17023 tag=gauss
wclsFrameSaver: saving 1728 traces tagged "wiener"
FrameSaver: q=161437 n=14809 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1575 traces, 1575 summary
Retagger: tagging trace set: gauss with 1101 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1101 traces tagged "gauss"
FrameSaver: q=137014 n=15656 tag=gauss
wclsFrameSaver: saving 1575 traces tagged "wiener"
FrameSaver: q=156038 n=13366 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1745 traces, 1745 summary
Retagger: tagging trace set: gauss with 1229 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1229 traces tagged "gauss"
FrameSaver: q=154875 n=17481 tag=gauss
wclsFrameSaver: saving 1745 traces tagged "wiener"
FrameSaver: q=175838 n=15163 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1727 traces, 1727 summary
Retagger: tagging trace set: gauss with 1297 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1297 traces tagged "gauss"
FrameSaver: q=144469 n=17503 tag=gauss
wclsFrameSaver: saving 1727 traces tagged "wiener"
FrameSaver: q=163174 n=15320 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1706 traces, 1706 summary
Retagger: tagging trace set: gauss with 1263 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1263 traces tagged "gauss"
FrameSaver: q=140547 n=17105 tag=gauss
wclsFrameSaver: saving 1706 traces tagged "wiener"
FrameSaver: q=159417 n=14933 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1659 traces, 1659 summary
Retagger: tagging trace set: gauss with 1218 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1218 traces tagged "gauss"
FrameSaver: q=138601 n=16680 tag=gauss
wclsFrameSaver: saving 1659 traces tagged "wiener"
FrameSaver: q=156868 n=14597 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1162 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1162 traces tagged "gauss"
FrameSaver: q=141467 n=16242 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=161029 n=14030 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1189 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1189 traces tagged "gauss"
FrameSaver: q=142563 n=16764 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=158455 n=14631 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1621 traces, 1621 summary
Retagger: tagging trace set: gauss with 1184 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1184 traces tagged "gauss"
FrameSaver: q=145649 n=16699 tag=gauss
wclsFrameSaver: saving 1621 traces tagged "wiener"
FrameSaver: q=162846 n=14472 tag=wiener
27-Mar-2025 12:31:10 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166657_27_1743077482_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:31:10 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166657_27_1743077482_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       39.9347       41.4335       43.5325       41.4518       1.04036        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000401871   0.00160387    0.00634928    0.000587733   0.00185651       10     
reco:rns:RandomNumberSaver                     4.4423e-05    9.39346e-05   0.000489304   4.9668e-05    0.000131881      10     
reco:wclsdatavd:WireCellToolkit                  37.998        40.8746       43.1378        41.04        1.36333        10     
reco:gaushit:GausHitFinder                      0.0190792     0.022721      0.0257051     0.0228135    0.00196396       10     
reco:reco3d:SpacePointSolver                    0.0552996     0.072454      0.0868822     0.0735998    0.00953876       10     
reco:hitpdune:DisambigFromSpacePoints          0.00169993    0.00257941    0.00486087    0.00240582    0.000828238      10     
reco:pandora:StandardPandora                    0.145807      0.405968       2.11624      0.223257      0.570793        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00151916    0.00277574    0.00666878    0.00246867    0.00134409       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000195746    0.0005143    0.00220516    0.000254585   0.000601599      10     
reco:pandoraStdcalo:Calorimetry                0.00150543    0.00208545    0.00576341    0.00165007    0.00123295       10     
reco:pandoraGnocalo:GnocchiCalorimetry          0.0010072    0.00260463     0.0161289    0.00110041    0.00450867       10     
[art]:TriggerResults:TriggerResultInserter      1.572e-05    2.86116e-05   0.000108503   1.9632e-05    2.67827e-05      10     
end_path:out1:RootOutput                        4.338e-06    8.6131e-06    4.1909e-05    4.9995e-06    1.1104e-05       10     
end_path:out1:RootOutput(write)                 0.0363472     0.0410657     0.0451939     0.0415535    0.00316727       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3387.36 MB
  Peak resident set size usage (VmHWM): 1484.55 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_27_1743077482_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078155.2783265, 'end_time': 1743078671.4254508}
justIN time: 2025-04-03 08:22:36 UTC       justIN version: 01.03.00