justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.10@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.10@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:01:18
From worker nodeHostnamee4046
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:43:52
Input filesmonte-carlo-005746-000152
JobscriptExit code0
Real time17m (1028s)
CPU time15m (910s = 88%)
Max RSS bytes1729736704 (1649 MiB)
Outputting started2025-03-27 12:01:01
Output files
Finished2025-03-27 12:01:18
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ularization...
Begin: 7.8646e+06
0 7.8568e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 06:53:37 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_10_1743075873_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 06:53:38 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_10_1743075873_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1743 traces, 1743 summary
Retagger: tagging trace set: gauss with 1214 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1214 traces tagged "gauss"
FrameSaver: q=143280 n=17072 tag=gauss
wclsFrameSaver: saving 1743 traces tagged "wiener"
FrameSaver: q=162590 n=14726 tag=wiener
27-Mar-2025 06:54:23 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_10_1743075873_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1614 traces, 1614 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=144791 n=16718 tag=gauss
wclsFrameSaver: saving 1614 traces tagged "wiener"
FrameSaver: q=166274 n=14544 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1657 traces, 1657 summary
Retagger: tagging trace set: gauss with 1187 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1187 traces tagged "gauss"
FrameSaver: q=147542 n=16835 tag=gauss
wclsFrameSaver: saving 1657 traces tagged "wiener"
FrameSaver: q=167393 n=14657 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1533 traces, 1533 summary
Retagger: tagging trace set: gauss with 1133 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1133 traces tagged "gauss"
FrameSaver: q=138465 n=15618 tag=gauss
wclsFrameSaver: saving 1533 traces tagged "wiener"
FrameSaver: q=155857 n=13637 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1687 traces, 1687 summary
Retagger: tagging trace set: gauss with 1276 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1276 traces tagged "gauss"
FrameSaver: q=139292 n=17008 tag=gauss
wclsFrameSaver: saving 1687 traces tagged "wiener"
FrameSaver: q=158080 n=14983 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1624 traces, 1624 summary
Retagger: tagging trace set: gauss with 1123 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1123 traces tagged "gauss"
FrameSaver: q=136496 n=15931 tag=gauss
wclsFrameSaver: saving 1624 traces tagged "wiener"
FrameSaver: q=156288 n=13926 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1551 traces, 1551 summary
Retagger: tagging trace set: gauss with 1082 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1082 traces tagged "gauss"
FrameSaver: q=137519 n=15603 tag=gauss
wclsFrameSaver: saving 1551 traces tagged "wiener"
FrameSaver: q=155483 n=13335 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1600 traces, 1600 summary
Retagger: tagging trace set: gauss with 1123 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1123 traces tagged "gauss"
FrameSaver: q=135830 n=15608 tag=gauss
wclsFrameSaver: saving 1600 traces tagged "wiener"
FrameSaver: q=155201 n=13474 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1209 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1209 traces tagged "gauss"
FrameSaver: q=152631 n=17177 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=170570 n=14881 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1132 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1132 traces tagged "gauss"
FrameSaver: q=149704 n=16321 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=168520 n=14190 tag=wiener
27-Mar-2025 07:01:00 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_10_1743075873_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:01:00 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_10_1743075873_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.4218       43.1759       45.1729       43.4832       1.72173        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000646317   0.000856338   0.00118473    0.000790166   0.00018808       10     
reco:rns:RandomNumberSaver                      5.337e-05    0.000117271    0.000423     7.28565e-05   0.000107817      10     
reco:wclsdatavd:WireCellToolkit                  40.0691       42.4585       44.7954       42.3951       1.65721        10     
reco:gaushit:GausHitFinder                      0.0170611     0.0224107     0.0278835     0.0229123    0.00271715       10     
reco:reco3d:SpacePointSolver                    0.0611999     0.0751606     0.0937083     0.0753267    0.00768747       10     
reco:hitpdune:DisambigFromSpacePoints          0.00210452    0.00265321    0.00596617    0.00222908    0.00112245       10     
reco:pandora:StandardPandora                    0.174294      0.557564       3.65713      0.211926       1.03349        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00167166    0.00280317    0.00635977    0.00255944    0.00123664       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000238265   0.000565268   0.00232124    0.000262786   0.000626946      10     
reco:pandoraStdcalo:Calorimetry                0.00135303    0.00216004    0.00558621    0.00175765    0.00117014       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000686583   0.00138826    0.00334663    0.00118029    0.000687496      10     
[art]:TriggerResults:TriggerResultInserter     1.4657e-05    4.02392e-05   7.8186e-05    3.55565e-05   1.95221e-05      10     
end_path:out1:RootOutput                        6.773e-06    1.96968e-05   7.0713e-05    1.0705e-05    1.84027e-05      10     
end_path:out1:RootOutput(write)                 0.0330064     0.046704      0.0666495     0.0412202     0.0114501       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3497.84 MB
  Peak resident set size usage (VmHWM): 1682.44 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_10_1743075873_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076377.0171518, 'end_time': 1743076861.3587136}
justIN time: 2025-04-03 08:22:34 UTC       justIN version: 01.03.00