justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.121@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.121@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:42
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-03-27 12:13:50
From worker nodeHostnamewn-d20-004.beowulf.cluster
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:39:45
Input filesmonte-carlo-005746-000103
JobscriptExit code0
Real time20m (1246s)
CPU time17m (1020s = 81%)
Max RSS bytes1693667328 (1615 MiB)
Outputting started2025-03-27 12:00:32
Output files
Finished2025-03-27 12:13:50
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

0551e+06
0 5.1921e+06
1 5.18062e+06
2 5.17187e+06
3 5.16416e+06
4 5.1578e+06
5 5.15194e+06
6 5.14654e+06
7 5.14214e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:52:42 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_121_1743075628_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:52:43 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166655_121_1743075628_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1654 traces, 1654 summary
Retagger: tagging trace set: gauss with 1218 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1218 traces tagged "gauss"
FrameSaver: q=141109 n=16436 tag=gauss
wclsFrameSaver: saving 1654 traces tagged "wiener"
FrameSaver: q=160312 n=14398 tag=wiener
27-Mar-2025 11:53:32 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_121_1743075628_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1630 traces, 1630 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=140163 n=16641 tag=gauss
wclsFrameSaver: saving 1630 traces tagged "wiener"
FrameSaver: q=158509 n=14515 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1561 traces, 1561 summary
Retagger: tagging trace set: gauss with 1102 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1102 traces tagged "gauss"
FrameSaver: q=137422 n=15531 tag=gauss
wclsFrameSaver: saving 1561 traces tagged "wiener"
FrameSaver: q=156047 n=13422 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1638 traces, 1638 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=156055 n=17352 tag=gauss
wclsFrameSaver: saving 1638 traces tagged "wiener"
FrameSaver: q=173950 n=15246 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1561 traces, 1561 summary
Retagger: tagging trace set: gauss with 1106 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1106 traces tagged "gauss"
FrameSaver: q=138046 n=15620 tag=gauss
wclsFrameSaver: saving 1561 traces tagged "wiener"
FrameSaver: q=156679 n=13450 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1146 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1146 traces tagged "gauss"
FrameSaver: q=136925 n=15781 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=155006 n=13809 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1632 traces, 1632 summary
Retagger: tagging trace set: gauss with 1215 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1215 traces tagged "gauss"
FrameSaver: q=139360 n=16771 tag=gauss
wclsFrameSaver: saving 1632 traces tagged "wiener"
FrameSaver: q=157144 n=14708 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1693 traces, 1693 summary
Retagger: tagging trace set: gauss with 1202 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1202 traces tagged "gauss"
FrameSaver: q=150361 n=16990 tag=gauss
wclsFrameSaver: saving 1693 traces tagged "wiener"
FrameSaver: q=170040 n=14625 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1781 traces, 1781 summary
Retagger: tagging trace set: gauss with 1242 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1242 traces tagged "gauss"
FrameSaver: q=145866 n=17937 tag=gauss
wclsFrameSaver: saving 1781 traces tagged "wiener"
FrameSaver: q=166831 n=15609 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1532 traces, 1532 summary
Retagger: tagging trace set: gauss with 1098 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1098 traces tagged "gauss"
FrameSaver: q=135937 n=15428 tag=gauss
wclsFrameSaver: saving 1532 traces tagged "wiener"
FrameSaver: q=154857 n=13351 tag=wiener
27-Mar-2025 12:00:31 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166655_121_1743075628_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:00:31 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166655_121_1743075628_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.5862       46.1361       48.8152       46.059        1.7099         10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000539517   0.00123146    0.00531871    0.000694299   0.00138357       10     
reco:rns:RandomNumberSaver                     3.4925e-05    8.64047e-05   0.000418699   4.7995e-05    0.000111601      10     
reco:wclsdatavd:WireCellToolkit                  42.1871       45.5454       47.3398       45.6722       1.50193        10     
reco:gaushit:GausHitFinder                      0.0194187     0.0217721     0.0245204     0.0214997    0.00143542       10     
reco:reco3d:SpacePointSolver                    0.0611457     0.0777567     0.120161      0.074359      0.0164836       10     
reco:hitpdune:DisambigFromSpacePoints          0.00199723    0.00264371     0.0046625    0.00242635    0.000740305      10     
reco:pandora:StandardPandora                    0.225986      0.438109       2.16438      0.250955      0.575546        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00138725    0.00246218    0.00504842    0.00231754    0.00093974       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000192643   0.000454446   0.00180897    0.000251073   0.000477768      10     
reco:pandoraStdcalo:Calorimetry                0.00142918    0.00185885    0.00407601    0.00152085    0.000760647      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000931566    0.0012611    0.00276076     0.0011114    0.000504771      10     
[art]:TriggerResults:TriggerResultInserter     1.5288e-05    2.07291e-05   5.0004e-05    1.7373e-05    9.95163e-06      10     
end_path:out1:RootOutput                        3.867e-06    6.8888e-06    2.1049e-05    5.0295e-06    4.85233e-06      10     
end_path:out1:RootOutput(write)                 0.0335646     0.0393334     0.0451814     0.0394006     0.0028526       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3452.82 MB
  Peak resident set size usage (VmHWM): 1637.56 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_121_1743075628_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076315.5124795, 'end_time': 1743076831.8815346}
justIN time: 2025-04-03 08:27:26 UTC       justIN version: 01.03.00