justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.93@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.93@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:39:28
From worker nodeHostnamet2wn004.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:15:51
Input filesmonte-carlo-005746-000285
JobscriptExit code0
Real time22m (1349s)
CPU time19m (1171s = 86%)
Max RSS bytes1685729280 (1607 MiB)
Outputting started2025-03-27 12:38:20
Output files
Finished2025-03-27 12:39:28
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

20292e+06
3 8.17405e+06
4 8.15654e+06
5 8.14556e+06
6 8.13859e+06
Now with regularization...
Begin: 6.96981e+06
0 6.96548e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:30:22 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_93_1743077800_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:30:22 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166657_93_1743077800_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1646 traces, 1646 summary
Retagger: tagging trace set: gauss with 1210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1210 traces tagged "gauss"
FrameSaver: q=157550 n=17203 tag=gauss
wclsFrameSaver: saving 1646 traces tagged "wiener"
FrameSaver: q=174529 n=15022 tag=wiener
27-Mar-2025 12:31:14 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_93_1743077800_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1605 traces, 1605 summary
Retagger: tagging trace set: gauss with 1149 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1149 traces tagged "gauss"
FrameSaver: q=144525 n=15861 tag=gauss
wclsFrameSaver: saving 1605 traces tagged "wiener"
FrameSaver: q=163311 n=13886 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1688 traces, 1688 summary
Retagger: tagging trace set: gauss with 1246 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1246 traces tagged "gauss"
FrameSaver: q=153121 n=17370 tag=gauss
wclsFrameSaver: saving 1688 traces tagged "wiener"
FrameSaver: q=171828 n=15113 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1572 traces, 1572 summary
Retagger: tagging trace set: gauss with 1108 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1108 traces tagged "gauss"
FrameSaver: q=144724 n=15900 tag=gauss
wclsFrameSaver: saving 1572 traces tagged "wiener"
FrameSaver: q=163154 n=13598 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1581 traces, 1581 summary
Retagger: tagging trace set: gauss with 1229 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1229 traces tagged "gauss"
FrameSaver: q=138420 n=16232 tag=gauss
wclsFrameSaver: saving 1581 traces tagged "wiener"
FrameSaver: q=155936 n=14404 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1668 traces, 1668 summary
Retagger: tagging trace set: gauss with 1177 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1177 traces tagged "gauss"
FrameSaver: q=143664 n=16384 tag=gauss
wclsFrameSaver: saving 1668 traces tagged "wiener"
FrameSaver: q=163653 n=14264 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1586 traces, 1586 summary
Retagger: tagging trace set: gauss with 1155 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1155 traces tagged "gauss"
FrameSaver: q=154095 n=16474 tag=gauss
wclsFrameSaver: saving 1586 traces tagged "wiener"
FrameSaver: q=172333 n=14198 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1599 traces, 1599 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=136160 n=15932 tag=gauss
wclsFrameSaver: saving 1599 traces tagged "wiener"
FrameSaver: q=154940 n=13903 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1728 traces, 1728 summary
Retagger: tagging trace set: gauss with 1246 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1246 traces tagged "gauss"
FrameSaver: q=143832 n=17295 tag=gauss
wclsFrameSaver: saving 1728 traces tagged "wiener"
FrameSaver: q=163490 n=15052 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1649 traces, 1649 summary
Retagger: tagging trace set: gauss with 1223 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1223 traces tagged "gauss"
FrameSaver: q=154231 n=17071 tag=gauss
wclsFrameSaver: saving 1649 traces tagged "wiener"
FrameSaver: q=172374 n=15114 tag=wiener
27-Mar-2025 12:38:19 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166657_93_1743077800_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:38:19 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166657_93_1743077800_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       43.9633       47.1445       50.7981       47.4869       2.00965        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000329134   0.000616571   0.00126489    0.000556924   0.000256092      10     
reco:rns:RandomNumberSaver                     3.4226e-05    8.01407e-05   0.000353963   4.95805e-05   9.21225e-05      10     
reco:wclsdatavd:WireCellToolkit                  43.5028       46.4965       48.2791       47.1693       1.69902        10     
reco:gaushit:GausHitFinder                      0.0229311     0.0254912     0.0319999     0.0249403    0.00247252       10     
reco:reco3d:SpacePointSolver                    0.0509817     0.0849557      0.11477      0.0848561     0.0169629       10     
reco:hitpdune:DisambigFromSpacePoints          0.00162469     0.0029263    0.00592397    0.00271895    0.00114684       10     
reco:pandora:StandardPandora                    0.098482      0.474637       2.63097       0.24488       0.72042        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0019637    0.00385979     0.0121507    0.00291955    0.00285426       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000237128   0.000402431   0.00118301    0.000280897   0.000284482      10     
reco:pandoraStdcalo:Calorimetry                0.00133914    0.00236177    0.00636353     0.0020125    0.00135378       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000784483   0.00158103    0.00416556     0.0013514    0.000883139      10     
[art]:TriggerResults:TriggerResultInserter     1.5159e-05    2.61993e-05   7.9473e-05    2.0705e-05    1.79582e-05      10     
end_path:out1:RootOutput                        4.098e-06    7.1589e-06    2.8054e-05     4.92e-06     6.98666e-06      10     
end_path:out1:RootOutput(write)                 0.0383915     0.0456111     0.051207      0.0457468    0.00401768       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3385.34 MB
  Peak resident set size usage (VmHWM): 1557.19 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_93_1743077800_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078537.9637144, 'end_time': 1743079100.435787}
justIN time: 2025-04-03 08:21:31 UTC       justIN version: 01.03.00