justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.154@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.154@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-03-27 12:25:57
From worker nodeHostnameheplnc148.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:04:26
Input filesmonte-carlo-005747-000329
JobscriptExit code0
Real time21m (1270s)
CPU time20m (1206s = 94%)
Max RSS bytes1738067968 (1657 MiB)
Outputting started2025-03-27 12:25:37
Output files
Finished2025-03-27 12:25:57
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

h regularization...
Begin: 3.16967e+07
0 3.16953e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:18:03 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_154_1743077077_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:18:03 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_154_1743077077_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3807 traces, 3807 summary
Retagger: tagging trace set: gauss with 2355 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2355 traces tagged "gauss"
FrameSaver: q=413964 n=36606 tag=gauss
wclsFrameSaver: saving 3807 traces tagged "wiener"
FrameSaver: q=455169 n=29819 tag=wiener
27-Mar-2025 12:18:45 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_154_1743077077_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3931 traces, 3931 summary
Retagger: tagging trace set: gauss with 2417 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2417 traces tagged "gauss"
FrameSaver: q=417184 n=37256 tag=gauss
wclsFrameSaver: saving 3931 traces tagged "wiener"
FrameSaver: q=457791 n=30519 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3916 traces, 3916 summary
Retagger: tagging trace set: gauss with 2474 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2474 traces tagged "gauss"
FrameSaver: q=431844 n=38140 tag=gauss
wclsFrameSaver: saving 3916 traces tagged "wiener"
FrameSaver: q=472164 n=31558 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4159 traces, 4159 summary
Retagger: tagging trace set: gauss with 2571 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2571 traces tagged "gauss"
FrameSaver: q=485631 n=41130 tag=gauss
wclsFrameSaver: saving 4159 traces tagged "wiener"
FrameSaver: q=528383 n=33965 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4171 traces, 4171 summary
Retagger: tagging trace set: gauss with 2628 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2628 traces tagged "gauss"
FrameSaver: q=463335 n=40619 tag=gauss
wclsFrameSaver: saving 4171 traces tagged "wiener"
FrameSaver: q=504888 n=33488 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4077 traces, 4077 summary
Retagger: tagging trace set: gauss with 2535 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2535 traces tagged "gauss"
FrameSaver: q=513547 n=40811 tag=gauss
wclsFrameSaver: saving 4077 traces tagged "wiener"
FrameSaver: q=555631 n=33599 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4001 traces, 4001 summary
Retagger: tagging trace set: gauss with 2488 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2488 traces tagged "gauss"
FrameSaver: q=424536 n=38285 tag=gauss
wclsFrameSaver: saving 4001 traces tagged "wiener"
FrameSaver: q=465910 n=31535 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3878 traces, 3878 summary
Retagger: tagging trace set: gauss with 2435 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2435 traces tagged "gauss"
FrameSaver: q=427370 n=37215 tag=gauss
wclsFrameSaver: saving 3878 traces tagged "wiener"
FrameSaver: q=467270 n=30711 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4263 traces, 4263 summary
Retagger: tagging trace set: gauss with 2628 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2628 traces tagged "gauss"
FrameSaver: q=473876 n=41507 tag=gauss
wclsFrameSaver: saving 4263 traces tagged "wiener"
FrameSaver: q=518947 n=34202 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4056 traces, 4056 summary
Retagger: tagging trace set: gauss with 2537 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2537 traces tagged "gauss"
FrameSaver: q=471763 n=40003 tag=gauss
wclsFrameSaver: saving 4056 traces tagged "wiener"
FrameSaver: q=512936 n=33270 tag=wiener
27-Mar-2025 12:25:36 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_154_1743077077_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:25:36 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_154_1743077077_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       41.6706       44.5961       45.8482       44.9914       1.31085        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00060432    0.00091405    0.00181211    0.000839463   0.000327208      10     
reco:rns:RandomNumberSaver                     5.6277e-05    8.9402e-05    0.00035514    5.87765e-05   8.86483e-05      10     
reco:wclsdatavd:WireCellToolkit                  38.8372       43.6327       45.099        44.1916       1.82632        10     
reco:gaushit:GausHitFinder                      0.029624      0.0452467     0.0526753     0.0458074    0.00613508       10     
reco:reco3d:SpacePointSolver                    0.245863      0.342239      0.368262      0.352514      0.0337852       10     
reco:hitpdune:DisambigFromSpacePoints          0.00347935    0.00560922    0.00898206    0.00551768    0.00130815       10     
reco:pandora:StandardPandora                     0.19447      0.458187       2.28875      0.246358      0.610934        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00494488    0.00715833     0.0131012    0.00633833    0.00233453       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00184691    0.00302968    0.00602117     0.0028039    0.00112726       10     
reco:pandoraStdcalo:Calorimetry                0.00716067    0.00918104     0.0142178    0.00855086    0.00190177       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00354383    0.00448066     0.0080423    0.00409467    0.00121969       10     
[art]:TriggerResults:TriggerResultInserter     1.9857e-05    3.50821e-05   0.00012174    2.59695e-05   2.90563e-05      10     
end_path:out1:RootOutput                        6.291e-06    1.15167e-05    3.698e-05     8.696e-06    8.53719e-06      10     
end_path:out1:RootOutput(write)                 0.0768588     0.081771      0.0878272     0.0817156     0.0029323       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3425.46 MB
  Peak resident set size usage (VmHWM): 1629.13 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_154_1743077077_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077847.5026243, 'end_time': 1743078336.922121}
justIN time: 2025-04-03 08:31:32 UTC       justIN version: 01.03.00