justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.161@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.161@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2025-03-27 12:40:50
From worker nodeHostnamedune001-5180171.0-lcg2614.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:23:56
Input filesmonte-carlo-005746-000451
JobscriptExit code0
Real time16m (992s)
CPU time16m (972s = 97%)
Max RSS bytes1699393536 (1620 MiB)
Outputting started2025-03-27 12:40:29
Output files
Finished2025-03-27 12:40:50
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

754e+06
2 6.31516e+06
3 6.30447e+06
4 6.29424e+06
5 6.28501e+06
6 6.27681e+06
7 6.26962e+06
8 6.26317e+06
9 6.25721e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:32:57 UTC  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_161_1743078247_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:32:58 UTC  Opened input file "prod_muon_400MeV_protodunevd_1_358071_161_1743078247_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1695 traces, 1695 summary
Retagger: tagging trace set: gauss with 1198 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1198 traces tagged "gauss"
FrameSaver: q=152011 n=17112 tag=gauss
wclsFrameSaver: saving 1695 traces tagged "wiener"
FrameSaver: q=172544 n=14892 tag=wiener
27-Mar-2025 12:33:47 UTC  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_161_1743078247_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1645 traces, 1645 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=149775 n=17125 tag=gauss
wclsFrameSaver: saving 1645 traces tagged "wiener"
FrameSaver: q=167593 n=14984 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1556 traces, 1556 summary
Retagger: tagging trace set: gauss with 1117 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1117 traces tagged "gauss"
FrameSaver: q=139094 n=15445 tag=gauss
wclsFrameSaver: saving 1556 traces tagged "wiener"
FrameSaver: q=158305 n=13375 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1622 traces, 1622 summary
Retagger: tagging trace set: gauss with 1168 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1168 traces tagged "gauss"
FrameSaver: q=138969 n=16375 tag=gauss
wclsFrameSaver: saving 1622 traces tagged "wiener"
FrameSaver: q=157728 n=14079 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1729 traces, 1729 summary
Retagger: tagging trace set: gauss with 1191 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1191 traces tagged "gauss"
FrameSaver: q=141945 n=17094 tag=gauss
wclsFrameSaver: saving 1729 traces tagged "wiener"
FrameSaver: q=161924 n=14698 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1592 traces, 1592 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=136745 n=15806 tag=gauss
wclsFrameSaver: saving 1592 traces tagged "wiener"
FrameSaver: q=155430 n=13661 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1748 traces, 1748 summary
Retagger: tagging trace set: gauss with 1210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1210 traces tagged "gauss"
FrameSaver: q=143453 n=17478 tag=gauss
wclsFrameSaver: saving 1748 traces tagged "wiener"
FrameSaver: q=162788 n=15190 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1501 traces, 1501 summary
Retagger: tagging trace set: gauss with 1159 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1159 traces tagged "gauss"
FrameSaver: q=139166 n=16094 tag=gauss
wclsFrameSaver: saving 1501 traces tagged "wiener"
FrameSaver: q=154560 n=14108 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1596 traces, 1596 summary
Retagger: tagging trace set: gauss with 1218 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1218 traces tagged "gauss"
FrameSaver: q=142517 n=16679 tag=gauss
wclsFrameSaver: saving 1596 traces tagged "wiener"
FrameSaver: q=160894 n=14694 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1602 traces, 1602 summary
Retagger: tagging trace set: gauss with 1112 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1112 traces tagged "gauss"
FrameSaver: q=153207 n=16412 tag=gauss
wclsFrameSaver: saving 1602 traces tagged "wiener"
FrameSaver: q=171453 n=14205 tag=wiener
27-Mar-2025 12:40:28 UTC  Closed output file "prod_muon_400MeV_protodunevd_1_358071_161_1743078247_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:40:28 UTC  Closed input file "prod_muon_400MeV_protodunevd_1_358071_161_1743078247_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       36.2817       44.6075       51.3326       45.1956       5.05041        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000433031    0.0008622    0.00137619    0.000790895   0.00029758       10     
reco:rns:RandomNumberSaver                     4.3171e-05    0.00011047    0.000375669   7.6677e-05    9.26466e-05      10     
reco:wclsdatavd:WireCellToolkit                  35.9804       43.8476       50.9706       44.3039       4.7973         10     
reco:gaushit:GausHitFinder                      0.0136934     0.0241686     0.0463924     0.023204     0.00924067       10     
reco:reco3d:SpacePointSolver                    0.0512322     0.120134      0.305862      0.0841884     0.0755599       10     
reco:hitpdune:DisambigFromSpacePoints           0.0017593    0.00388446     0.0110185    0.00233047    0.00278577       10     
reco:pandora:StandardPandora                    0.144811      0.552394       3.39094      0.215093      0.948987        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00218765     0.0059295     0.0224002    0.00338637    0.00582055       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000175774   0.00108976    0.00518006    0.000291853   0.00161846       10     
reco:pandoraStdcalo:Calorimetry                0.00129186    0.00351608     0.0145033    0.00236898    0.00376355       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000696767   0.00199656    0.00894216    0.00116067    0.00234629       10     
[art]:TriggerResults:TriggerResultInserter     1.4421e-05    4.72142e-05   0.000254546   2.52255e-05   6.94164e-05      10     
end_path:out1:RootOutput                        7.58e-06     1.98005e-05   6.3661e-05    1.1245e-05    1.6616e-05       10     
end_path:out1:RootOutput(write)                 0.0230806     0.0362606     0.0604959     0.0353885     0.0104291       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3614.36 MB
  Peak resident set size usage (VmHWM): 1563.11 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_161_1743078247_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078740.6055505, 'end_time': 1743079228.853823}
justIN time: 2025-04-03 08:31:23 UTC       justIN version: 01.03.00