justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.162@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.162@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2025-03-27 12:38:31
From worker nodeHostnamedune001-4917532.0-lcg2651.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:23:52
Input filesmonte-carlo-005746-000450
JobscriptExit code0
Real time14m (853s)
CPU time13m (834s = 97%)
Max RSS bytes1705201664 (1626 MiB)
Outputting started2025-03-27 12:38:05
Output files
Finished2025-03-27 12:38:31
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

arization...
Begin: 1.54168e+07
0 1.51544e+07
1 1.51524e+07
Now with regularization...
Begin: 1.41848e+07
0 1.41847e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:31:50 UTC  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_162_1743078245_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:31:51 UTC  Opened input file "prod_muon_400MeV_protodunevd_1_358071_162_1743078245_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1672 traces, 1672 summary
Retagger: tagging trace set: gauss with 1164 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1164 traces tagged "gauss"
FrameSaver: q=141347 n=16619 tag=gauss
wclsFrameSaver: saving 1672 traces tagged "wiener"
FrameSaver: q=161101 n=14254 tag=wiener
27-Mar-2025 12:32:40 UTC  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_162_1743078245_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1598 traces, 1598 summary
Retagger: tagging trace set: gauss with 1190 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1190 traces tagged "gauss"
FrameSaver: q=137640 n=16275 tag=gauss
wclsFrameSaver: saving 1598 traces tagged "wiener"
FrameSaver: q=155584 n=14304 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1585 traces, 1585 summary
Retagger: tagging trace set: gauss with 1215 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1215 traces tagged "gauss"
FrameSaver: q=137129 n=15935 tag=gauss
wclsFrameSaver: saving 1585 traces tagged "wiener"
FrameSaver: q=155195 n=14250 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1666 traces, 1666 summary
Retagger: tagging trace set: gauss with 1158 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1158 traces tagged "gauss"
FrameSaver: q=146418 n=16751 tag=gauss
wclsFrameSaver: saving 1666 traces tagged "wiener"
FrameSaver: q=166294 n=14330 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1656 traces, 1656 summary
Retagger: tagging trace set: gauss with 1204 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1204 traces tagged "gauss"
FrameSaver: q=151337 n=16849 tag=gauss
wclsFrameSaver: saving 1656 traces tagged "wiener"
FrameSaver: q=169783 n=14618 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1515 traces, 1515 summary
Retagger: tagging trace set: gauss with 1094 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1094 traces tagged "gauss"
FrameSaver: q=138958 n=15693 tag=gauss
wclsFrameSaver: saving 1515 traces tagged "wiener"
FrameSaver: q=156188 n=13535 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1721 traces, 1721 summary
Retagger: tagging trace set: gauss with 1271 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1271 traces tagged "gauss"
FrameSaver: q=154885 n=17857 tag=gauss
wclsFrameSaver: saving 1721 traces tagged "wiener"
FrameSaver: q=174070 n=15717 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1562 traces, 1562 summary
Retagger: tagging trace set: gauss with 1138 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1138 traces tagged "gauss"
FrameSaver: q=137678 n=15860 tag=gauss
wclsFrameSaver: saving 1562 traces tagged "wiener"
FrameSaver: q=155388 n=13778 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1704 traces, 1704 summary
Retagger: tagging trace set: gauss with 1253 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1253 traces tagged "gauss"
FrameSaver: q=142286 n=16963 tag=gauss
wclsFrameSaver: saving 1704 traces tagged "wiener"
FrameSaver: q=161107 n=14840 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1563 traces, 1563 summary
Retagger: tagging trace set: gauss with 1160 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1160 traces tagged "gauss"
FrameSaver: q=143670 n=15822 tag=gauss
wclsFrameSaver: saving 1563 traces tagged "wiener"
FrameSaver: q=160682 n=13818 tag=wiener
27-Mar-2025 12:38:04 UTC  Closed output file "prod_muon_400MeV_protodunevd_1_358071_162_1743078245_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:38:04 UTC  Closed input file "prod_muon_400MeV_protodunevd_1_358071_162_1743078245_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       31.571        36.7342       48.4948       35.1092       4.89361        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000441729   0.00105124    0.00198872    0.00104708    0.000510833      10     
reco:rns:RandomNumberSaver                     4.4754e-05    0.000116117   0.000509717   6.1956e-05    0.000133348      10     
reco:wclsdatavd:WireCellToolkit                  31.2763       36.2699       46.4756       34.8296       4.47325        10     
reco:gaushit:GausHitFinder                      0.0135001     0.0186577     0.0318563     0.0154384    0.00595588       10     
reco:reco3d:SpacePointSolver                    0.039392      0.0654133     0.0965607     0.0639763     0.0184024       10     
reco:hitpdune:DisambigFromSpacePoints          0.00138222    0.00314242     0.0116766    0.00198559    0.00291863       10     
reco:pandora:StandardPandora                    0.121846      0.335801       1.8193       0.162174      0.496209        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00148371    0.00308648    0.00777516     0.002528     0.00170367       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000169638   0.000518613   0.00272393    0.000267296   0.000738987      10     
reco:pandoraStdcalo:Calorimetry                0.00136434    0.00225664    0.00624078    0.00184323    0.00135763       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000724831    0.0013494    0.00487509    0.000926885   0.00118346       10     
[art]:TriggerResults:TriggerResultInserter     1.3817e-05    3.06639e-05   9.1121e-05    2.5323e-05    2.13042e-05      10     
end_path:out1:RootOutput                        7.654e-06    1.62314e-05   4.3632e-05    1.4697e-05    9.58621e-06      10     
end_path:out1:RootOutput(write)                 0.0200845     0.0274685     0.0472832     0.0250146    0.00796594       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3589.48 MB
  Peak resident set size usage (VmHWM): 1552.5 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_162_1743078245_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078671.1692016, 'end_time': 1743079085.4646816}
justIN time: 2025-04-03 08:31:22 UTC       justIN version: 01.03.00