justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.26@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.26@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-03-27 12:31:52
From worker nodeHostnamemff2203.farm.particle.cz
cpuinfoAMD EPYC 7352 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:19:26
Input filesmonte-carlo-005746-000371
JobscriptExit code0
Real time12m (732s)
CPU time11m (715s = 97%)
Max RSS bytes1735008256 (1654 MiB)
Outputting started2025-03-27 12:31:38
Output files
Finished2025-03-27 12:31:52
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ating with no regularization...
Begin: 1.57996e+08
0 1.57857e+08
Now with regularization...
Begin: 1.57403e+08
0 1.57388e+08
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:26:27 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_26_1743077973_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:26:27 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_26_1743077973_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1644 traces, 1644 summary
Retagger: tagging trace set: gauss with 1174 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1174 traces tagged "gauss"
FrameSaver: q=140492 n=16553 tag=gauss
wclsFrameSaver: saving 1644 traces tagged "wiener"
FrameSaver: q=158856 n=14342 tag=wiener
27-Mar-2025 13:26:57 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_26_1743077973_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1584 traces, 1584 summary
Retagger: tagging trace set: gauss with 1204 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1204 traces tagged "gauss"
FrameSaver: q=138725 n=16236 tag=gauss
wclsFrameSaver: saving 1584 traces tagged "wiener"
FrameSaver: q=155167 n=14418 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1657 traces, 1657 summary
Retagger: tagging trace set: gauss with 1182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1182 traces tagged "gauss"
FrameSaver: q=145552 n=16641 tag=gauss
wclsFrameSaver: saving 1657 traces tagged "wiener"
FrameSaver: q=164895 n=14592 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1615 traces, 1615 summary
Retagger: tagging trace set: gauss with 1182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1182 traces tagged "gauss"
FrameSaver: q=137339 n=16007 tag=gauss
wclsFrameSaver: saving 1615 traces tagged "wiener"
FrameSaver: q=155651 n=13984 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1527 traces, 1527 summary
Retagger: tagging trace set: gauss with 1129 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1129 traces tagged "gauss"
FrameSaver: q=143620 n=16127 tag=gauss
wclsFrameSaver: saving 1527 traces tagged "wiener"
FrameSaver: q=162970 n=13974 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1694 traces, 1694 summary
Retagger: tagging trace set: gauss with 1205 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1205 traces tagged "gauss"
FrameSaver: q=140645 n=16963 tag=gauss
wclsFrameSaver: saving 1694 traces tagged "wiener"
FrameSaver: q=159515 n=14695 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1641 traces, 1641 summary
Retagger: tagging trace set: gauss with 1230 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1230 traces tagged "gauss"
FrameSaver: q=137494 n=16479 tag=gauss
wclsFrameSaver: saving 1641 traces tagged "wiener"
FrameSaver: q=155549 n=14493 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1649 traces, 1649 summary
Retagger: tagging trace set: gauss with 1202 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1202 traces tagged "gauss"
FrameSaver: q=141084 n=16575 tag=gauss
wclsFrameSaver: saving 1649 traces tagged "wiener"
FrameSaver: q=160426 n=14568 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1604 traces, 1604 summary
Retagger: tagging trace set: gauss with 1138 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1138 traces tagged "gauss"
FrameSaver: q=137693 n=15800 tag=gauss
wclsFrameSaver: saving 1604 traces tagged "wiener"
FrameSaver: q=156513 n=13530 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1529 traces, 1529 summary
Retagger: tagging trace set: gauss with 1128 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1128 traces tagged "gauss"
FrameSaver: q=140863 n=15694 tag=gauss
wclsFrameSaver: saving 1529 traces tagged "wiener"
FrameSaver: q=156420 n=13729 tag=wiener
27-Mar-2025 13:31:38 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_26_1743077973_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:31:38 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_26_1743077973_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       29.0693       30.7545       33.123        30.6244       1.2283         10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000284864   0.000396293   0.000695249   0.00036617    0.000109964      10     
reco:rns:RandomNumberSaver                     2.8344e-05    6.45395e-05   0.000283893   3.59485e-05   7.38721e-05      10     
reco:wclsdatavd:WireCellToolkit                  27.8178       30.3508       32.8605       30.3348       1.42157        10     
reco:gaushit:GausHitFinder                      0.0125158     0.0133708     0.0144195     0.0133635    0.000650453      10     
reco:reco3d:SpacePointSolver                    0.0412839     0.0467822     0.0596243     0.044263     0.00531101       10     
reco:hitpdune:DisambigFromSpacePoints          0.00121865     0.0016493    0.00306279    0.00149467    0.000494062      10     
reco:pandora:StandardPandora                    0.161916      0.306661       1.47317      0.175194      0.389009        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.000982066   0.00196409    0.00461816    0.00175687    0.000925407      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000144877   0.000207786   0.000653679   0.000152947   0.000149452      10     
reco:pandoraStdcalo:Calorimetry                0.000886804   0.00132687    0.00381222    0.00105182    0.00083339       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000679729   0.000907262   0.00234981    0.000762272   0.000482491      10     
[art]:TriggerResults:TriggerResultInserter     1.1412e-05    1.71556e-05    5.246e-05    1.33205e-05   1.18461e-05      10     
end_path:out1:RootOutput                        3.436e-06    6.3892e-06    2.1591e-05     4.469e-06    5.17822e-06      10     
end_path:out1:RootOutput(write)                 0.0244174     0.0277737     0.0304003     0.0278272    0.00170405       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3417.18 MB
  Peak resident set size usage (VmHWM): 1613.09 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_26_1743077973_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078363.7992854, 'end_time': 1743078698.6753356}
justIN time: 2025-04-03 08:21:32 UTC       justIN version: 01.03.00