justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.8@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.8@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:23:29
From worker nodeHostnamee4095
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:06:04
Input filesmonte-carlo-005746-000017
JobscriptExit code0
Real time17m (1027s)
CPU time16m (978s = 95%)
Max RSS bytes1730662400 (1650 MiB)
Outputting started2025-03-27 12:23:13
Output files
Finished2025-03-27 12:23:29
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

th no regularization...
Begin: 1.29038e+08
0 1.24666e+08
1 1.24578e+08
Now with regularization...
Begin: 1.23789e+08
0 1.23785e+08
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:15:01 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_8_1743077174_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:15:02 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_8_1743077174_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1605 traces, 1605 summary
Retagger: tagging trace set: gauss with 1175 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1175 traces tagged "gauss"
FrameSaver: q=142057 n=16150 tag=gauss
wclsFrameSaver: saving 1605 traces tagged "wiener"
FrameSaver: q=160252 n=14015 tag=wiener
27-Mar-2025 07:15:50 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_8_1743077174_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1618 traces, 1618 summary
Retagger: tagging trace set: gauss with 1136 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1136 traces tagged "gauss"
FrameSaver: q=140679 n=16150 tag=gauss
wclsFrameSaver: saving 1618 traces tagged "wiener"
FrameSaver: q=159559 n=13859 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1678 traces, 1678 summary
Retagger: tagging trace set: gauss with 1148 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1148 traces tagged "gauss"
FrameSaver: q=143453 n=16591 tag=gauss
wclsFrameSaver: saving 1678 traces tagged "wiener"
FrameSaver: q=162355 n=14217 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1516 traces, 1516 summary
Retagger: tagging trace set: gauss with 1122 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1122 traces tagged "gauss"
FrameSaver: q=138297 n=15545 tag=gauss
wclsFrameSaver: saving 1516 traces tagged "wiener"
FrameSaver: q=154680 n=13591 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1549 traces, 1549 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=137094 n=15678 tag=gauss
wclsFrameSaver: saving 1549 traces tagged "wiener"
FrameSaver: q=154443 n=13882 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1173 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1173 traces tagged "gauss"
FrameSaver: q=139322 n=16138 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=158205 n=13961 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1628 traces, 1628 summary
Retagger: tagging trace set: gauss with 1139 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1139 traces tagged "gauss"
FrameSaver: q=136462 n=15963 tag=gauss
wclsFrameSaver: saving 1628 traces tagged "wiener"
FrameSaver: q=155717 n=13893 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1658 traces, 1658 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=148618 n=16853 tag=gauss
wclsFrameSaver: saving 1658 traces tagged "wiener"
FrameSaver: q=166804 n=14593 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1712 traces, 1712 summary
Retagger: tagging trace set: gauss with 1300 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1300 traces tagged "gauss"
FrameSaver: q=145045 n=17713 tag=gauss
wclsFrameSaver: saving 1712 traces tagged "wiener"
FrameSaver: q=163985 n=15496 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1657 traces, 1657 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=147861 n=16932 tag=gauss
wclsFrameSaver: saving 1657 traces tagged "wiener"
FrameSaver: q=165817 n=14624 tag=wiener
27-Mar-2025 07:23:11 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_8_1743077174_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:23:11 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_8_1743077174_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.8657       47.8114       49.8813       48.5995       2.13186        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000549862   0.00108512    0.00198677    0.00100061    0.000420249      10     
reco:rns:RandomNumberSaver                     5.0795e-05    0.00012103    0.000593753   6.99305e-05   0.000158102      10     
reco:wclsdatavd:WireCellToolkit                  42.4141       47.1803       49.3473       48.2291       2.23755        10     
reco:gaushit:GausHitFinder                      0.0205547     0.0248389     0.029805      0.0253799    0.00278983       10     
reco:reco3d:SpacePointSolver                    0.0668801     0.0924847     0.129312      0.0878678     0.0205029       10     
reco:hitpdune:DisambigFromSpacePoints          0.00213014    0.00296964    0.00571901     0.0025913    0.00100317       10     
reco:pandora:StandardPandora                    0.186337      0.442343       2.21674      0.243074      0.592966        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0015689    0.00426414    0.00898924    0.00342267    0.00221989       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00026657    0.000425333   0.00138876    0.000329619   0.000323347      10     
reco:pandoraStdcalo:Calorimetry                0.00173838    0.00256728    0.00676865    0.00213285    0.00141469       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00118294    0.00162376     0.0044612    0.00132194    0.000950214      10     
[art]:TriggerResults:TriggerResultInserter     1.6481e-05    3.60259e-05   0.000123352   2.55635e-05   2.99296e-05      10     
end_path:out1:RootOutput                        8.696e-06    1.42617e-05   2.7131e-05    1.13365e-05   6.54258e-06      10     
end_path:out1:RootOutput(write)                 0.0379193     0.0505536     0.0667032     0.0502814    0.00970225       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3395.76 MB
  Peak resident set size usage (VmHWM): 1596.96 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_8_1743077174_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077661.4833133, 'end_time': 1743078192.2253652}
justIN time: 2025-04-03 08:15:20 UTC       justIN version: 01.03.00