justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358067.178@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358067.178@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:50
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-03-27 12:15:58
From worker nodeHostnamewn-d22-007.beowulf.cluster
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:43:20
Input filesmonte-carlo-005747-000082
JobscriptExit code0
Real time19m (1150s)
CPU time18m (1086s = 94%)
Max RSS bytes1726746624 (1646 MiB)
Outputting started2025-03-27 12:02:31
Output files
Finished2025-03-27 12:15:58
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

h regularization...
Begin: 1.67866e+07
0 1.67833e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:55:43 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358067_178_1743075811_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:55:44 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358067_178_1743075811_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3924 traces, 3924 summary
Retagger: tagging trace set: gauss with 2452 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2452 traces tagged "gauss"
FrameSaver: q=446167 n=37982 tag=gauss
wclsFrameSaver: saving 3924 traces tagged "wiener"
FrameSaver: q=485472 n=31328 tag=wiener
27-Mar-2025 11:56:23 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358067_178_1743075811_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4489 traces, 4489 summary
Retagger: tagging trace set: gauss with 2827 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2827 traces tagged "gauss"
FrameSaver: q=562765 n=45547 tag=gauss
wclsFrameSaver: saving 4489 traces tagged "wiener"
FrameSaver: q=607283 n=38164 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4451 traces, 4451 summary
Retagger: tagging trace set: gauss with 2806 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2806 traces tagged "gauss"
FrameSaver: q=543863 n=45361 tag=gauss
wclsFrameSaver: saving 4451 traces tagged "wiener"
FrameSaver: q=589762 n=37718 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3974 traces, 3974 summary
Retagger: tagging trace set: gauss with 2433 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2433 traces tagged "gauss"
FrameSaver: q=424271 n=37743 tag=gauss
wclsFrameSaver: saving 3974 traces tagged "wiener"
FrameSaver: q=466130 n=30903 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3884 traces, 3884 summary
Retagger: tagging trace set: gauss with 2417 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2417 traces tagged "gauss"
FrameSaver: q=423548 n=37019 tag=gauss
wclsFrameSaver: saving 3884 traces tagged "wiener"
FrameSaver: q=464041 n=30429 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4050 traces, 4050 summary
Retagger: tagging trace set: gauss with 2579 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2579 traces tagged "gauss"
FrameSaver: q=471982 n=39988 tag=gauss
wclsFrameSaver: saving 4050 traces tagged "wiener"
FrameSaver: q=513437 n=33248 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4334 traces, 4334 summary
Retagger: tagging trace set: gauss with 2665 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2665 traces tagged "gauss"
FrameSaver: q=527815 n=43981 tag=gauss
wclsFrameSaver: saving 4334 traces tagged "wiener"
FrameSaver: q=572008 n=36397 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3903 traces, 3903 summary
Retagger: tagging trace set: gauss with 2447 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2447 traces tagged "gauss"
FrameSaver: q=419653 n=37540 tag=gauss
wclsFrameSaver: saving 3903 traces tagged "wiener"
FrameSaver: q=458996 n=30997 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3987 traces, 3987 summary
Retagger: tagging trace set: gauss with 2534 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2534 traces tagged "gauss"
FrameSaver: q=446714 n=39264 tag=gauss
wclsFrameSaver: saving 3987 traces tagged "wiener"
FrameSaver: q=485922 n=32487 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3906 traces, 3906 summary
Retagger: tagging trace set: gauss with 2400 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2400 traces tagged "gauss"
FrameSaver: q=420436 n=37071 tag=gauss
wclsFrameSaver: saving 3906 traces tagged "wiener"
FrameSaver: q=461585 n=30353 tag=wiener
27-Mar-2025 12:02:30 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358067_178_1743075811_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:02:30 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358067_178_1743075811_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       39.1152       40.0598       40.9509       40.1327       0.50918        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00059421    0.000849754   0.00125692    0.000780822   0.000218418      10     
reco:rns:RandomNumberSaver                     4.3001e-05    7.80004e-05    0.0003285     5.022e-05    8.35756e-05      10     
reco:wclsdatavd:WireCellToolkit                  36.5844        39.19        40.2751       39.4246      0.972353        10     
reco:gaushit:GausHitFinder                      0.0381339     0.0470333     0.061355      0.0453435    0.00742587       10     
reco:reco3d:SpacePointSolver                    0.285823      0.309963       0.34663      0.308589      0.0159546       10     
reco:hitpdune:DisambigFromSpacePoints           0.0044451    0.00521402    0.00647849    0.00494159    0.000655093      10     
reco:pandora:StandardPandora                    0.148578      0.412845       2.08295      0.237381      0.558646        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00490473    0.00685875     0.0109863    0.00586437    0.00201631       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00168552    0.00307081    0.00495449    0.00316643    0.00104365       10     
reco:pandoraStdcalo:Calorimetry                0.00644219    0.00799121    0.00984228    0.00757318    0.00109848       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00342837    0.00387313    0.00507847    0.00382775    0.000445984      10     
[art]:TriggerResults:TriggerResultInserter     1.7282e-05    2.31439e-05   4.6238e-05    2.0539e-05    8.14332e-06      10     
end_path:out1:RootOutput                        7.584e-06    1.02736e-05   2.1691e-05     8.847e-06    3.9389e-06       10     
end_path:out1:RootOutput(write)                 0.0549173     0.0678685     0.0806147     0.0666594    0.00655083       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3426.25 MB
  Peak resident set size usage (VmHWM): 1627.82 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358067_178_1743075811_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076502.6135223, 'end_time': 1743076951.338286}
justIN time: 2025-04-03 08:27:28 UTC       justIN version: 01.03.00