justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.179@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.179@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:24:13
From worker nodeHostnamee2555
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:05:09
Input filesmonte-carlo-005747-000068
JobscriptExit code0
Real time18m (1121s)
CPU time17m (1072s = 95%)
Max RSS bytes1755430912 (1674 MiB)
Outputting started2025-03-27 12:23:51
Output files
Finished2025-03-27 12:24:13
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

th regularization...
Begin: 2.09827e+07
0 2.09818e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:16:07 CDT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_179_1743077121_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:16:08 CDT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_179_1743077121_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4134 traces, 4134 summary
Retagger: tagging trace set: gauss with 2562 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2562 traces tagged "gauss"
FrameSaver: q=497675 n=41360 tag=gauss
wclsFrameSaver: saving 4134 traces tagged "wiener"
FrameSaver: q=539879 n=33976 tag=wiener
27-Mar-2025 07:16:56 CDT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_179_1743077121_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3941 traces, 3941 summary
Retagger: tagging trace set: gauss with 2474 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2474 traces tagged "gauss"
FrameSaver: q=422776 n=37687 tag=gauss
wclsFrameSaver: saving 3941 traces tagged "wiener"
FrameSaver: q=463843 n=31157 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3944 traces, 3944 summary
Retagger: tagging trace set: gauss with 2446 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2446 traces tagged "gauss"
FrameSaver: q=398368 n=36928 tag=gauss
wclsFrameSaver: saving 3944 traces tagged "wiener"
FrameSaver: q=439003 n=30156 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4008 traces, 4008 summary
Retagger: tagging trace set: gauss with 2494 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2494 traces tagged "gauss"
FrameSaver: q=453423 n=38897 tag=gauss
wclsFrameSaver: saving 4008 traces tagged "wiener"
FrameSaver: q=494519 n=32207 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3984 traces, 3984 summary
Retagger: tagging trace set: gauss with 2477 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2477 traces tagged "gauss"
FrameSaver: q=427350 n=38432 tag=gauss
wclsFrameSaver: saving 3984 traces tagged "wiener"
FrameSaver: q=469165 n=31771 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3936 traces, 3936 summary
Retagger: tagging trace set: gauss with 2472 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2472 traces tagged "gauss"
FrameSaver: q=423005 n=38067 tag=gauss
wclsFrameSaver: saving 3936 traces tagged "wiener"
FrameSaver: q=462424 n=31293 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4274 traces, 4274 summary
Retagger: tagging trace set: gauss with 2596 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2596 traces tagged "gauss"
FrameSaver: q=523680 n=42616 tag=gauss
wclsFrameSaver: saving 4274 traces tagged "wiener"
FrameSaver: q=567490 n=35302 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4093 traces, 4093 summary
Retagger: tagging trace set: gauss with 2602 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2602 traces tagged "gauss"
FrameSaver: q=463612 n=40425 tag=gauss
wclsFrameSaver: saving 4093 traces tagged "wiener"
FrameSaver: q=505111 n=33569 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4477 traces, 4477 summary
Retagger: tagging trace set: gauss with 2801 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2801 traces tagged "gauss"
FrameSaver: q=518064 n=44845 tag=gauss
wclsFrameSaver: saving 4477 traces tagged "wiener"
FrameSaver: q=562860 n=37063 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3878 traces, 3878 summary
Retagger: tagging trace set: gauss with 2475 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2475 traces tagged "gauss"
FrameSaver: q=422448 n=37644 tag=gauss
wclsFrameSaver: saving 3878 traces tagged "wiener"
FrameSaver: q=461655 n=31054 tag=wiener
27-Mar-2025 07:23:50 CDT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_179_1743077121_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:23:50 CDT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_179_1743077121_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       44.1731       45.3377       47.6934       45.1869      0.895001        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000806624   0.00109638    0.00143882    0.00108484    0.000175906      10     
reco:rns:RandomNumberSaver                     4.8792e-05    0.000114892   0.000476925   7.1639e-05    0.000121916      10     
reco:wclsdatavd:WireCellToolkit                  43.5637       44.4737       45.1498       44.493       0.464601        10     
reco:gaushit:GausHitFinder                      0.038168      0.0442532     0.0514136     0.0435293    0.00372515       10     
reco:reco3d:SpacePointSolver                    0.292067      0.323857      0.350689      0.320923      0.0168522       10     
reco:hitpdune:DisambigFromSpacePoints          0.00385452     0.0050593     0.0083536     0.0047391    0.00115728       10     
reco:pandora:StandardPandora                    0.181419      0.394682       2.02862      0.219858      0.544881        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.004915     0.00620663     0.0110869    0.00582359    0.00167929       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00123367    0.00284121    0.00611445    0.00247566    0.00129476       10     
reco:pandoraStdcalo:Calorimetry                0.00707889    0.00828846     0.0142963    0.00766872    0.00204551       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00312276    0.00389608    0.00774888    0.00358477    0.00130622       10     
[art]:TriggerResults:TriggerResultInserter     2.2663e-05    3.3531e-05    9.1251e-05    2.75225e-05   1.94582e-05      10     
end_path:out1:RootOutput                        9.669e-06    1.77343e-05   3.8973e-05    1.2108e-05    1.09024e-05      10     
end_path:out1:RootOutput(write)                 0.0563285     0.0672496     0.0824313     0.0654352    0.00802802       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3431.69 MB
  Peak resident set size usage (VmHWM): 1648.66 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_179_1743077121_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077731.610056, 'end_time': 1743078230.952134}
justIN time: 2025-04-03 08:30:41 UTC       justIN version: 01.03.00