justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.21@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.21@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:36:17
From worker nodeHostnamee4066
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:18:42
Input filesmonte-carlo-005746-000037
JobscriptExit code0
Real time17m (1036s)
CPU time16m (990s = 95%)
Max RSS bytes1729916928 (1649 MiB)
Outputting started2025-03-27 12:36:00
Output files
Finished2025-03-27 12:36:17
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

1.3445e+07
0 8.7008e+06
1 8.22189e+06
2 8.21348e+06
3 8.2115e+06
Now with regularization...
Begin: 6.96827e+06
0 6.9642e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:27:43 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_21_1743077934_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:27:43 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358071_21_1743077934_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1738 traces, 1738 summary
Retagger: tagging trace set: gauss with 1256 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1256 traces tagged "gauss"
FrameSaver: q=145000 n=17511 tag=gauss
wclsFrameSaver: saving 1738 traces tagged "wiener"
FrameSaver: q=165271 n=15202 tag=wiener
27-Mar-2025 07:28:41 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_21_1743077934_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1561 traces, 1561 summary
Retagger: tagging trace set: gauss with 1131 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1131 traces tagged "gauss"
FrameSaver: q=136745 n=15649 tag=gauss
wclsFrameSaver: saving 1561 traces tagged "wiener"
FrameSaver: q=154950 n=13609 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1608 traces, 1608 summary
Retagger: tagging trace set: gauss with 1146 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1146 traces tagged "gauss"
FrameSaver: q=143335 n=16133 tag=gauss
wclsFrameSaver: saving 1608 traces tagged "wiener"
FrameSaver: q=162912 n=13982 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1533 traces, 1533 summary
Retagger: tagging trace set: gauss with 1110 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1110 traces tagged "gauss"
FrameSaver: q=138502 n=15798 tag=gauss
wclsFrameSaver: saving 1533 traces tagged "wiener"
FrameSaver: q=154656 n=13698 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1654 traces, 1654 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=153585 n=16780 tag=gauss
wclsFrameSaver: saving 1654 traces tagged "wiener"
FrameSaver: q=173086 n=14571 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1565 traces, 1565 summary
Retagger: tagging trace set: gauss with 1219 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1219 traces tagged "gauss"
FrameSaver: q=136884 n=15930 tag=gauss
wclsFrameSaver: saving 1565 traces tagged "wiener"
FrameSaver: q=154355 n=14007 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1143 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1143 traces tagged "gauss"
FrameSaver: q=139135 n=16132 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=157119 n=14060 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1689 traces, 1689 summary
Retagger: tagging trace set: gauss with 1222 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1222 traces tagged "gauss"
FrameSaver: q=139765 n=16817 tag=gauss
wclsFrameSaver: saving 1689 traces tagged "wiener"
FrameSaver: q=158416 n=14516 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1623 traces, 1623 summary
Retagger: tagging trace set: gauss with 1131 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1131 traces tagged "gauss"
FrameSaver: q=137298 n=16105 tag=gauss
wclsFrameSaver: saving 1623 traces tagged "wiener"
FrameSaver: q=156841 n=13880 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1679 traces, 1679 summary
Retagger: tagging trace set: gauss with 1228 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1228 traces tagged "gauss"
FrameSaver: q=152901 n=17107 tag=gauss
wclsFrameSaver: saving 1679 traces tagged "wiener"
FrameSaver: q=171487 n=14929 tag=wiener
27-Mar-2025 07:35:58 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358071_21_1743077934_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:35:58 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358071_21_1743077934_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.9114       48.3154       57.064        48.6255       3.82358        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000819143   0.00112304    0.00191937    0.000997353   0.000356971      10     
reco:rns:RandomNumberSaver                     3.8332e-05    9.68424e-05   0.000449361   5.66615e-05   0.000118493      10     
reco:wclsdatavd:WireCellToolkit                  42.5863       47.7291       54.4527       48.2276       3.32215        10     
reco:gaushit:GausHitFinder                      0.0180144     0.0229182     0.0276936     0.0231022    0.00307223       10     
reco:reco3d:SpacePointSolver                    0.054362      0.0792228     0.109456      0.076237      0.0160084       10     
reco:hitpdune:DisambigFromSpacePoints          0.00152625    0.00292139    0.00803986    0.00243443    0.00173758       10     
reco:pandora:StandardPandora                    0.175399      0.416594       2.39941      0.198979      0.661136        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00214438    0.00289644    0.00559882    0.00253548    0.000963872      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000203491   0.000319339   0.000917118   0.000234449   0.000208165      10     
reco:pandoraStdcalo:Calorimetry                0.00137431    0.00215237    0.00512135    0.00172638    0.00104919       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000834122   0.00129227    0.00294114    0.00113879    0.000571452      10     
[art]:TriggerResults:TriggerResultInserter     1.5239e-05    2.55669e-05   6.1755e-05    1.9887e-05    1.43286e-05      10     
end_path:out1:RootOutput                        7.393e-06    1.26986e-05   3.0877e-05    8.3855e-06     8.971e-06       10     
end_path:out1:RootOutput(write)                 0.035436      0.0517316     0.0652807     0.0513717    0.00844684       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3414.27 MB
  Peak resident set size usage (VmHWM): 1614.94 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_21_1743077934_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078425.1606555, 'end_time': 1743078959.2036138}
justIN time: 2025-04-03 08:23:33 UTC       justIN version: 01.03.00