justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.76@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.76@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-03-27 12:19:58
From worker nodeHostnameheplnc163.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:59:48
Input filesmonte-carlo-005747-000001
JobscriptExit code0
Real time19m (1188s)
CPU time18m (1136s = 95%)
Max RSS bytes1713295360 (1633 MiB)
Outputting started2025-03-27 12:19:37
Output files
Finished2025-03-27 12:19:58
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

w with regularization...
Begin: 2.14923e+07
0 2.14899e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:12:36 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_76_1743076800_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:12:36 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_76_1743076800_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3942 traces, 3942 summary
Retagger: tagging trace set: gauss with 2449 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2449 traces tagged "gauss"
FrameSaver: q=439449 n=38259 tag=gauss
wclsFrameSaver: saving 3942 traces tagged "wiener"
FrameSaver: q=479671 n=31387 tag=wiener
27-Mar-2025 12:13:17 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_76_1743076800_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4031 traces, 4031 summary
Retagger: tagging trace set: gauss with 2500 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2500 traces tagged "gauss"
FrameSaver: q=435674 n=38655 tag=gauss
wclsFrameSaver: saving 4031 traces tagged "wiener"
FrameSaver: q=477253 n=31913 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3888 traces, 3888 summary
Retagger: tagging trace set: gauss with 2441 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2441 traces tagged "gauss"
FrameSaver: q=415937 n=37227 tag=gauss
wclsFrameSaver: saving 3888 traces tagged "wiener"
FrameSaver: q=456602 n=30743 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3888 traces, 3888 summary
Retagger: tagging trace set: gauss with 2385 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2385 traces tagged "gauss"
FrameSaver: q=418796 n=37158 tag=gauss
wclsFrameSaver: saving 3888 traces tagged "wiener"
FrameSaver: q=458525 n=30621 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3985 traces, 3985 summary
Retagger: tagging trace set: gauss with 2512 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2512 traces tagged "gauss"
FrameSaver: q=432796 n=38628 tag=gauss
wclsFrameSaver: saving 3985 traces tagged "wiener"
FrameSaver: q=472920 n=31724 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3971 traces, 3971 summary
Retagger: tagging trace set: gauss with 2467 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2467 traces tagged "gauss"
FrameSaver: q=434319 n=38080 tag=gauss
wclsFrameSaver: saving 3971 traces tagged "wiener"
FrameSaver: q=476156 n=31265 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3981 traces, 3981 summary
Retagger: tagging trace set: gauss with 2541 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2541 traces tagged "gauss"
FrameSaver: q=428068 n=38412 tag=gauss
wclsFrameSaver: saving 3981 traces tagged "wiener"
FrameSaver: q=468794 n=31747 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3916 traces, 3916 summary
Retagger: tagging trace set: gauss with 2420 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2420 traces tagged "gauss"
FrameSaver: q=443813 n=38255 tag=gauss
wclsFrameSaver: saving 3916 traces tagged "wiener"
FrameSaver: q=484810 n=31337 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4720 traces, 4720 summary
Retagger: tagging trace set: gauss with 2935 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2935 traces tagged "gauss"
FrameSaver: q=709191 n=51575 tag=gauss
wclsFrameSaver: saving 4720 traces tagged "wiener"
FrameSaver: q=755294 n=43170 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4040 traces, 4040 summary
Retagger: tagging trace set: gauss with 2529 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2529 traces tagged "gauss"
FrameSaver: q=458339 n=39673 tag=gauss
wclsFrameSaver: saving 4040 traces tagged "wiener"
FrameSaver: q=500394 n=32724 tag=wiener
27-Mar-2025 12:19:36 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_76_1743076800_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:19:36 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_76_1743076800_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       39.0039       41.3865       42.593        41.592       0.959874        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000412115   0.000756532   0.00124161    0.000721226   0.000218327      10     
reco:rns:RandomNumberSaver                      5.319e-05    9.95763e-05   0.000468631   5.79145e-05   0.000123061      10     
reco:wclsdatavd:WireCellToolkit                  38.0261       40.4873       41.9084       40.8997       1.23075        10     
reco:gaushit:GausHitFinder                      0.0290135     0.0476009     0.0769533     0.0463007     0.0112442       10     
reco:reco3d:SpacePointSolver                    0.297244      0.324874      0.398779      0.321612      0.0264871       10     
reco:hitpdune:DisambigFromSpacePoints          0.00464165     0.0053554    0.00757108    0.00493686    0.00103302       10     
reco:pandora:StandardPandora                    0.165561      0.420537       2.21429      0.218204      0.599351        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0046193    0.00664219     0.0107072    0.00595974    0.00211917       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00160311    0.00300003    0.00768907    0.00221266    0.00190189       10     
reco:pandoraStdcalo:Calorimetry                0.00567233    0.00843405     0.0122324    0.00801335    0.00173137       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00218393    0.00396091    0.00651293    0.00382705    0.00100431       10     
[art]:TriggerResults:TriggerResultInserter      1.603e-05    2.53808e-05   7.1444e-05    2.1085e-05    1.55763e-05      10     
end_path:out1:RootOutput                        5.841e-06    1.02402e-05   2.7351e-05     8.967e-06    5.8601e-06       10     
end_path:out1:RootOutput(write)                 0.0434303     0.073014      0.0916332     0.0787369     0.015393        10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3443.14 MB
  Peak resident set size usage (VmHWM): 1595.37 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_76_1743076800_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077516.9243016, 'end_time': 1743077977.1950126}
justIN time: 2025-04-03 08:25:31 UTC       justIN version: 01.03.00