justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.45@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.45@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:16:39
From worker nodeHostnamee4057
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:57:15
Input filesmonte-carlo-005747-000170
JobscriptExit code0
Real time19m (1145s)
CPU time18m (1102s = 96%)
Max RSS bytes1727873024 (1647 MiB)
Outputting started2025-03-27 12:16:21
Output files
Finished2025-03-27 12:16:39
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

w with regularization...
Begin: 2.62131e+07
0 2.62101e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:07:55 CDT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_45_1743076645_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:07:55 CDT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_45_1743076645_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4027 traces, 4027 summary
Retagger: tagging trace set: gauss with 2501 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2501 traces tagged "gauss"
FrameSaver: q=513035 n=40333 tag=gauss
wclsFrameSaver: saving 4027 traces tagged "wiener"
FrameSaver: q=555104 n=33454 tag=wiener
27-Mar-2025 07:08:45 CDT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_45_1743076645_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3828 traces, 3828 summary
Retagger: tagging trace set: gauss with 2414 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2414 traces tagged "gauss"
FrameSaver: q=419465 n=36759 tag=gauss
wclsFrameSaver: saving 3828 traces tagged "wiener"
FrameSaver: q=459162 n=30295 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3980 traces, 3980 summary
Retagger: tagging trace set: gauss with 2455 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2455 traces tagged "gauss"
FrameSaver: q=464954 n=38993 tag=gauss
wclsFrameSaver: saving 3980 traces tagged "wiener"
FrameSaver: q=507334 n=32005 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4198 traces, 4198 summary
Retagger: tagging trace set: gauss with 2571 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2571 traces tagged "gauss"
FrameSaver: q=509463 n=42531 tag=gauss
wclsFrameSaver: saving 4198 traces tagged "wiener"
FrameSaver: q=551006 n=35255 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4039 traces, 4039 summary
Retagger: tagging trace set: gauss with 2498 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2498 traces tagged "gauss"
FrameSaver: q=465936 n=39950 tag=gauss
wclsFrameSaver: saving 4039 traces tagged "wiener"
FrameSaver: q=507647 n=32772 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3875 traces, 3875 summary
Retagger: tagging trace set: gauss with 2443 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2443 traces tagged "gauss"
FrameSaver: q=413940 n=36980 tag=gauss
wclsFrameSaver: saving 3875 traces tagged "wiener"
FrameSaver: q=454075 n=30399 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4046 traces, 4046 summary
Retagger: tagging trace set: gauss with 2515 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2515 traces tagged "gauss"
FrameSaver: q=465682 n=39753 tag=gauss
wclsFrameSaver: saving 4046 traces tagged "wiener"
FrameSaver: q=508429 n=32610 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3879 traces, 3879 summary
Retagger: tagging trace set: gauss with 2381 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2381 traces tagged "gauss"
FrameSaver: q=413628 n=36901 tag=gauss
wclsFrameSaver: saving 3879 traces tagged "wiener"
FrameSaver: q=452758 n=30104 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4130 traces, 4130 summary
Retagger: tagging trace set: gauss with 2604 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2604 traces tagged "gauss"
FrameSaver: q=462071 n=40452 tag=gauss
wclsFrameSaver: saving 4130 traces tagged "wiener"
FrameSaver: q=504327 n=33387 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3905 traces, 3905 summary
Retagger: tagging trace set: gauss with 2439 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2439 traces tagged "gauss"
FrameSaver: q=451430 n=37801 tag=gauss
wclsFrameSaver: saving 3905 traces tagged "wiener"
FrameSaver: q=492320 n=31246 tag=wiener
27-Mar-2025 07:16:19 CDT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_45_1743076645_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:16:19 CDT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_45_1743076645_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       47.0147       49.1032       51.3964       48.9578       1.30601        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00094547     0.0015652    0.00195722    0.00155799    0.000274161      10     
reco:rns:RandomNumberSaver                     7.8156e-05    0.000137795   0.000485338   0.000100308   0.00011705       10     
reco:wclsdatavd:WireCellToolkit                  45.4494       48.0364       50.5757       48.056        1.54955        10     
reco:gaushit:GausHitFinder                      0.040347      0.046621      0.0550658     0.0469634    0.00433676       10     
reco:reco3d:SpacePointSolver                     0.3633       0.412975       0.43413      0.417468      0.0187667       10     
reco:hitpdune:DisambigFromSpacePoints          0.00460695    0.00579255     0.0101524    0.00553564    0.00149537       10     
reco:pandora:StandardPandora                    0.196319       0.47063       2.62315      0.238589      0.717789        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00565956     0.0115074     0.0208155    0.00805005    0.00534353       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00176212    0.00342204    0.00773435    0.00305375    0.00169261       10     
reco:pandoraStdcalo:Calorimetry                0.00689658    0.00925921     0.0149874    0.00885449    0.00213773       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00333659     0.0042905    0.00801914    0.00386351    0.00129329       10     
[art]:TriggerResults:TriggerResultInserter     2.0409e-05    4.49052e-05   0.000119554   3.30215e-05   3.08372e-05      10     
end_path:out1:RootOutput                        5.139e-06    9.9115e-06    3.9935e-05     6.592e-06    1.00611e-05      10     
end_path:out1:RootOutput(write)                 0.0675926     0.0861861     0.127995      0.0818207     0.017126        10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3442.65 MB
  Peak resident set size usage (VmHWM): 1607.88 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_45_1743076645_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077237.0732512, 'end_time': 1743077780.6204205}
justIN time: 2025-04-06 12:24:20 UTC       justIN version: 01.03.00