justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.29@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.29@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-03-27 12:32:30
From worker nodeHostnamemff2203.farm.particle.cz
cpuinfoAMD EPYC 7352 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:19:26
Input filesmonte-carlo-005746-000370
JobscriptExit code0
Real time12m (770s)
CPU time12m (752s = 97%)
Max RSS bytes1735434240 (1655 MiB)
Outputting started2025-03-27 12:32:16
Output files
Finished2025-03-27 12:32:30
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

n...
Begin: 2.40145e+07
0 2.02357e+07
1 2.01446e+07
2 2.01248e+07
Now with regularization...
Begin: 1.91425e+07
0 1.91345e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:26:30 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_29_1743077973_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:26:30 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_29_1743077973_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1677 traces, 1677 summary
Retagger: tagging trace set: gauss with 1230 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1230 traces tagged "gauss"
FrameSaver: q=155262 n=17412 tag=gauss
wclsFrameSaver: saving 1677 traces tagged "wiener"
FrameSaver: q=173868 n=15298 tag=wiener
27-Mar-2025 13:27:00 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_29_1743077973_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1628 traces, 1628 summary
Retagger: tagging trace set: gauss with 1164 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1164 traces tagged "gauss"
FrameSaver: q=136957 n=16199 tag=gauss
wclsFrameSaver: saving 1628 traces tagged "wiener"
FrameSaver: q=156809 n=13999 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1632 traces, 1632 summary
Retagger: tagging trace set: gauss with 1118 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1118 traces tagged "gauss"
FrameSaver: q=144348 n=16248 tag=gauss
wclsFrameSaver: saving 1632 traces tagged "wiener"
FrameSaver: q=163167 n=14068 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1516 traces, 1516 summary
Retagger: tagging trace set: gauss with 1087 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1087 traces tagged "gauss"
FrameSaver: q=137465 n=15418 tag=gauss
wclsFrameSaver: saving 1516 traces tagged "wiener"
FrameSaver: q=155204 n=13252 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1700 traces, 1700 summary
Retagger: tagging trace set: gauss with 1209 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1209 traces tagged "gauss"
FrameSaver: q=139212 n=16754 tag=gauss
wclsFrameSaver: saving 1700 traces tagged "wiener"
FrameSaver: q=159481 n=14504 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1218 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1218 traces tagged "gauss"
FrameSaver: q=152260 n=16931 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=168548 n=14882 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1723 traces, 1723 summary
Retagger: tagging trace set: gauss with 1317 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1317 traces tagged "gauss"
FrameSaver: q=153182 n=17555 tag=gauss
wclsFrameSaver: saving 1723 traces tagged "wiener"
FrameSaver: q=171582 n=15437 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1660 traces, 1660 summary
Retagger: tagging trace set: gauss with 1173 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1173 traces tagged "gauss"
FrameSaver: q=147892 n=16444 tag=gauss
wclsFrameSaver: saving 1660 traces tagged "wiener"
FrameSaver: q=168286 n=14342 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1656 traces, 1656 summary
Retagger: tagging trace set: gauss with 1206 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1206 traces tagged "gauss"
FrameSaver: q=148458 n=16788 tag=gauss
wclsFrameSaver: saving 1656 traces tagged "wiener"
FrameSaver: q=166380 n=14639 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1558 traces, 1558 summary
Retagger: tagging trace set: gauss with 1116 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1116 traces tagged "gauss"
FrameSaver: q=138184 n=15849 tag=gauss
wclsFrameSaver: saving 1558 traces tagged "wiener"
FrameSaver: q=157157 n=13690 tag=wiener
27-Mar-2025 13:32:15 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_29_1743077973_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:32:15 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_29_1743077973_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       29.6425       34.2107       35.6924       34.7649       1.71924        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000258354   0.000361967   0.00052838    0.000324295   9.87186e-05      10     
reco:rns:RandomNumberSaver                     2.3645e-05    5.35163e-05   0.000249617   2.98625e-05   6.60122e-05      10     
reco:wclsdatavd:WireCellToolkit                  27.9743       33.7847       35.3911       34.4661       2.09383        10     
reco:gaushit:GausHitFinder                      0.0126486     0.0137019     0.0158633     0.0134259    0.000915879      10     
reco:reco3d:SpacePointSolver                    0.0452972     0.0516806     0.0715173     0.0502815    0.00697574       10     
reco:hitpdune:DisambigFromSpacePoints          0.00136759    0.00177829    0.00329362    0.00165347    0.000541822      10     
reco:pandora:StandardPandora                    0.168092      0.323575       1.55214       0.18938      0.409611        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00123238    0.00186148    0.00429349    0.00163664    0.000835768      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000146299   0.000354383   0.00162981     0.000167     0.000445466      10     
reco:pandoraStdcalo:Calorimetry                0.000956247   0.00145422    0.00415453     0.0010945    0.000916502      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000614615   0.00091407    0.00226424    0.000789249   0.000454803      10     
[art]:TriggerResults:TriggerResultInserter      1.082e-05    1.53472e-05   4.4645e-05    1.21685e-05   9.80301e-06      10     
end_path:out1:RootOutput                        2.996e-06    6.2769e-06    1.8475e-05    5.6305e-06    4.26817e-06      10     
end_path:out1:RootOutput(write)                 0.0261643     0.0277869     0.0303113     0.0273912    0.00128455       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3465.22 MB
  Peak resident set size usage (VmHWM): 1653.65 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_29_1743077973_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078366.023549, 'end_time': 1743078736.3142772}
justIN time: 2025-04-03 08:21:24 UTC       justIN version: 01.03.00