justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.86@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.86@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2025-03-27 12:20:15
From worker nodeHostnamen154.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:59:56
Input filesmonte-carlo-005747-000077
JobscriptExit code0
Real time20m (1205s)
CPU time17m (1040s = 86%)
Max RSS bytes1760759808 (1679 MiB)
Outputting started2025-03-27 12:20:01
Output files
Finished2025-03-27 12:20:15
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

w with regularization...
Begin: 2.07594e+07
0 2.07506e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:11:48 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_86_1743076863_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:11:48 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_86_1743076863_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4206 traces, 4206 summary
Retagger: tagging trace set: gauss with 2573 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2573 traces tagged "gauss"
FrameSaver: q=486226 n=41947 tag=gauss
wclsFrameSaver: saving 4206 traces tagged "wiener"
FrameSaver: q=529188 n=34465 tag=wiener
27-Mar-2025 12:12:45 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_86_1743076863_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4505 traces, 4505 summary
Retagger: tagging trace set: gauss with 2780 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2780 traces tagged "gauss"
FrameSaver: q=521100 n=45048 tag=gauss
wclsFrameSaver: saving 4505 traces tagged "wiener"
FrameSaver: q=568328 n=37240 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3953 traces, 3953 summary
Retagger: tagging trace set: gauss with 2480 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2480 traces tagged "gauss"
FrameSaver: q=431149 n=38351 tag=gauss
wclsFrameSaver: saving 3953 traces tagged "wiener"
FrameSaver: q=471677 n=31583 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4140 traces, 4140 summary
Retagger: tagging trace set: gauss with 2555 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2555 traces tagged "gauss"
FrameSaver: q=558219 n=42103 tag=gauss
wclsFrameSaver: saving 4140 traces tagged "wiener"
FrameSaver: q=600149 n=34831 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3909 traces, 3909 summary
Retagger: tagging trace set: gauss with 2496 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2496 traces tagged "gauss"
FrameSaver: q=435337 n=37825 tag=gauss
wclsFrameSaver: saving 3909 traces tagged "wiener"
FrameSaver: q=476101 n=31249 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4011 traces, 4011 summary
Retagger: tagging trace set: gauss with 2501 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2501 traces tagged "gauss"
FrameSaver: q=416713 n=38683 tag=gauss
wclsFrameSaver: saving 4011 traces tagged "wiener"
FrameSaver: q=456234 n=31829 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3942 traces, 3942 summary
Retagger: tagging trace set: gauss with 2464 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2464 traces tagged "gauss"
FrameSaver: q=429641 n=37721 tag=gauss
wclsFrameSaver: saving 3942 traces tagged "wiener"
FrameSaver: q=470129 n=31062 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3994 traces, 3994 summary
Retagger: tagging trace set: gauss with 2527 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2527 traces tagged "gauss"
FrameSaver: q=465745 n=39336 tag=gauss
wclsFrameSaver: saving 3994 traces tagged "wiener"
FrameSaver: q=506572 n=32496 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4189 traces, 4189 summary
Retagger: tagging trace set: gauss with 2607 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2607 traces tagged "gauss"
FrameSaver: q=465079 n=40545 tag=gauss
wclsFrameSaver: saving 4189 traces tagged "wiener"
FrameSaver: q=507663 n=33463 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4224 traces, 4224 summary
Retagger: tagging trace set: gauss with 2644 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2644 traces tagged "gauss"
FrameSaver: q=481961 n=41804 tag=gauss
wclsFrameSaver: saving 4224 traces tagged "wiener"
FrameSaver: q=525129 n=34356 tag=wiener
27-Mar-2025 12:20:01 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_86_1743076863_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:20:01 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_86_1743076863_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.8833       47.3313       55.0386       47.0623       3.08733        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000431397   0.00056134    0.000799808   0.000530559   0.000123562      10     
reco:rns:RandomNumberSaver                     3.6886e-05    5.86522e-05   0.000233234   3.8177e-05     5.827e-05       10     
reco:wclsdatavd:WireCellToolkit                  39.1622       46.3205       53.9988        46.43        3.54036        10     
reco:gaushit:GausHitFinder                      0.0332473     0.0399826     0.0571347     0.0373991    0.00703766       10     
reco:reco3d:SpacePointSolver                    0.278501      0.312647      0.456021      0.299068      0.0496395       10     
reco:hitpdune:DisambigFromSpacePoints          0.00422787    0.00525042    0.00803907    0.00485011    0.00117643       10     
reco:pandora:StandardPandora                    0.225226      0.567796       3.26291      0.258177      0.899669        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00478247     0.0068121     0.0114458    0.00635013    0.00183262       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00174156    0.00303139    0.00551978    0.00233755    0.00133284       10     
reco:pandoraStdcalo:Calorimetry                 0.0067109     0.0079669     0.010804     0.00750545    0.00134144       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00270291    0.00324092    0.00537073    0.00282066    0.000878312      10     
[art]:TriggerResults:TriggerResultInserter     1.3244e-05    2.02422e-05   5.1369e-05    1.56045e-05   1.08822e-05      10     
end_path:out1:RootOutput                        5.635e-06    7.5058e-06    1.6539e-05     6.096e-06    3.20476e-06      10     
end_path:out1:RootOutput(write)                 0.0510975     0.0580226     0.0886071     0.0544596     0.0105669       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3449.29 MB
  Peak resident set size usage (VmHWM): 1687.15 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_86_1743076863_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077479.9281898, 'end_time': 1743078001.5343382}
justIN time: 2025-04-03 08:21:39 UTC       justIN version: 01.03.00