justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166656.142@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166656.142@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:45
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2025-03-27 12:20:19
From worker nodeHostnamen226.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:02:10
Input filesmonte-carlo-005747-000272
JobscriptExit code0
Real time17m (1076s)
CPU time16m (1019s = 94%)
Max RSS bytes1754873856 (1673 MiB)
Outputting started2025-03-27 12:20:07
Output files
Finished2025-03-27 12:20:19
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ith regularization...
Begin: 2.03117e+07
0 2.031e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:11:57 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_166656_142_1743076943_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:11:57 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_166656_142_1743076943_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3928 traces, 3928 summary
Retagger: tagging trace set: gauss with 2421 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2421 traces tagged "gauss"
FrameSaver: q=421609 n=37778 tag=gauss
wclsFrameSaver: saving 3928 traces tagged "wiener"
FrameSaver: q=463501 n=30803 tag=wiener
27-Mar-2025 12:13:04 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_166656_142_1743076943_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4272 traces, 4272 summary
Retagger: tagging trace set: gauss with 2749 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2749 traces tagged "gauss"
FrameSaver: q=519149 n=43203 tag=gauss
wclsFrameSaver: saving 4272 traces tagged "wiener"
FrameSaver: q=562398 n=36269 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3906 traces, 3906 summary
Retagger: tagging trace set: gauss with 2463 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2463 traces tagged "gauss"
FrameSaver: q=431201 n=37532 tag=gauss
wclsFrameSaver: saving 3906 traces tagged "wiener"
FrameSaver: q=471477 n=30977 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3986 traces, 3986 summary
Retagger: tagging trace set: gauss with 2486 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2486 traces tagged "gauss"
FrameSaver: q=430056 n=38137 tag=gauss
wclsFrameSaver: saving 3986 traces tagged "wiener"
FrameSaver: q=471801 n=31261 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4016 traces, 4016 summary
Retagger: tagging trace set: gauss with 2494 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2494 traces tagged "gauss"
FrameSaver: q=446455 n=38750 tag=gauss
wclsFrameSaver: saving 4016 traces tagged "wiener"
FrameSaver: q=487563 n=31823 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3841 traces, 3841 summary
Retagger: tagging trace set: gauss with 2410 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2410 traces tagged "gauss"
FrameSaver: q=412114 n=36511 tag=gauss
wclsFrameSaver: saving 3841 traces tagged "wiener"
FrameSaver: q=452405 n=30047 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3837 traces, 3837 summary
Retagger: tagging trace set: gauss with 2387 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2387 traces tagged "gauss"
FrameSaver: q=411586 n=36271 tag=gauss
wclsFrameSaver: saving 3837 traces tagged "wiener"
FrameSaver: q=450953 n=29892 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4007 traces, 4007 summary
Retagger: tagging trace set: gauss with 2470 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2470 traces tagged "gauss"
FrameSaver: q=433032 n=38728 tag=gauss
wclsFrameSaver: saving 4007 traces tagged "wiener"
FrameSaver: q=473829 n=31808 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4207 traces, 4207 summary
Retagger: tagging trace set: gauss with 2624 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2624 traces tagged "gauss"
FrameSaver: q=502715 n=41850 tag=gauss
wclsFrameSaver: saving 4207 traces tagged "wiener"
FrameSaver: q=544316 n=34501 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4018 traces, 4018 summary
Retagger: tagging trace set: gauss with 2500 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2500 traces tagged "gauss"
FrameSaver: q=444534 n=38872 tag=gauss
wclsFrameSaver: saving 4018 traces tagged "wiener"
FrameSaver: q=485996 n=32109 tag=wiener
27-Mar-2025 12:20:07 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_166656_142_1743076943_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:20:07 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_166656_142_1743076943_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       44.0216       46.4476       48.6341       46.3477       1.26506        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000405661   0.000512601   0.00074122    0.000497127   9.22121e-05      10     
reco:rns:RandomNumberSaver                     3.3485e-05    6.2993e-05    0.000295059   3.63135e-05   7.74519e-05      10     
reco:wclsdatavd:WireCellToolkit                  40.2892       45.4912       47.9985       45.7289       1.99828        10     
reco:gaushit:GausHitFinder                      0.0318519     0.0353832     0.0432641     0.0344315    0.00332289       10     
reco:reco3d:SpacePointSolver                    0.268967      0.282422      0.296545      0.284071     0.00962037       10     
reco:hitpdune:DisambigFromSpacePoints          0.00402694    0.00467814    0.00615982    0.00443784    0.000597713      10     
reco:pandora:StandardPandora                    0.220303      0.555749       3.32361       0.24812      0.922869        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00388131    0.00522731    0.00702342    0.00512104    0.000808576      10     
reco:pandoraShower:LArPandoraShowerCreation    0.00132773    0.00229195    0.00375006    0.00206903    0.00080042       10     
reco:pandoraStdcalo:Calorimetry                0.00649429    0.00709735    0.00883184    0.00693146    0.000618974      10     
reco:pandoraGnocalo:GnocchiCalorimetry          0.0026149    0.00304917    0.00435221    0.00276203    0.000624768      10     
[art]:TriggerResults:TriggerResultInserter     1.4148e-05    1.91631e-05   4.5034e-05    1.50425e-05   8.99504e-06      10     
end_path:out1:RootOutput                        2.69e-06     5.2449e-06    1.4849e-05     4.513e-06    3.37012e-06      10     
end_path:out1:RootOutput(write)                 0.0509159     0.0561211     0.083162      0.0526515    0.00921122       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3412.25 MB
  Peak resident set size usage (VmHWM): 1654.65 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_166656_142_1743076943_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077486.680074, 'end_time': 1743078007.4187036}
justIN time: 2025-04-03 08:29:39 UTC       justIN version: 01.03.00