justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.148@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.148@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-03-27 12:35:19
From worker nodeHostnamemff2304.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:24:09
Input filesmonte-carlo-005746-000081
JobscriptExit code0
Real time10m (657s)
CPU time10m (645s = 98%)
Max RSS bytes1735065600 (1654 MiB)
Outputting started2025-03-27 12:35:07
Output files
Finished2025-03-27 12:35:19
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

gin: 6.29279e+06
0 5.81502e+06
1 5.78302e+06
2 5.78125e+06
Now with regularization...
Begin: 4.70815e+06
0 4.70802e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:30:39 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_148_1743078254_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:30:40 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_148_1743078254_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1619 traces, 1619 summary
Retagger: tagging trace set: gauss with 1121 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1121 traces tagged "gauss"
FrameSaver: q=136676 n=16071 tag=gauss
wclsFrameSaver: saving 1619 traces tagged "wiener"
FrameSaver: q=156132 n=13935 tag=wiener
27-Mar-2025 13:31:08 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_148_1743078254_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1649 traces, 1649 summary
Retagger: tagging trace set: gauss with 1135 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1135 traces tagged "gauss"
FrameSaver: q=136766 n=15937 tag=gauss
wclsFrameSaver: saving 1649 traces tagged "wiener"
FrameSaver: q=155798 n=13806 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1730 traces, 1730 summary
Retagger: tagging trace set: gauss with 1272 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1272 traces tagged "gauss"
FrameSaver: q=141233 n=17492 tag=gauss
wclsFrameSaver: saving 1730 traces tagged "wiener"
FrameSaver: q=159831 n=15349 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1588 traces, 1588 summary
Retagger: tagging trace set: gauss with 1284 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1284 traces tagged "gauss"
FrameSaver: q=139098 n=16890 tag=gauss
wclsFrameSaver: saving 1588 traces tagged "wiener"
FrameSaver: q=154885 n=15109 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1615 traces, 1615 summary
Retagger: tagging trace set: gauss with 1188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1188 traces tagged "gauss"
FrameSaver: q=139985 n=16294 tag=gauss
wclsFrameSaver: saving 1615 traces tagged "wiener"
FrameSaver: q=156982 n=14110 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1539 traces, 1539 summary
Retagger: tagging trace set: gauss with 1140 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1140 traces tagged "gauss"
FrameSaver: q=138887 n=16088 tag=gauss
wclsFrameSaver: saving 1539 traces tagged "wiener"
FrameSaver: q=155332 n=13957 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1566 traces, 1566 summary
Retagger: tagging trace set: gauss with 1121 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1121 traces tagged "gauss"
FrameSaver: q=136922 n=15732 tag=gauss
wclsFrameSaver: saving 1566 traces tagged "wiener"
FrameSaver: q=155354 n=13621 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1203 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1203 traces tagged "gauss"
FrameSaver: q=140754 n=16529 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=158236 n=14485 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1115 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1115 traces tagged "gauss"
FrameSaver: q=138925 n=15701 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=157535 n=13618 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1625 traces, 1625 summary
Retagger: tagging trace set: gauss with 1201 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1201 traces tagged "gauss"
FrameSaver: q=137582 n=16133 tag=gauss
wclsFrameSaver: saving 1625 traces tagged "wiener"
FrameSaver: q=155826 n=14098 tag=wiener
27-Mar-2025 13:35:06 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_148_1743078254_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:35:06 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_148_1743078254_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       25.653        26.422        28.2562       26.2391       0.74429        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000185497   0.000225217   0.000340201   0.000211512   4.0632e-05       10     
reco:rns:RandomNumberSaver                      1.913e-05    3.69754e-05   0.000162716   2.2196e-05    4.20994e-05      10     
reco:wclsdatavd:WireCellToolkit                  25.4266       26.0546       26.8448       26.0046       0.48241        10     
reco:gaushit:GausHitFinder                      0.0117981     0.012467      0.0132447     0.012559     0.000497324      10     
reco:reco3d:SpacePointSolver                    0.0368238     0.0446527     0.0679478     0.0419199    0.00885268       10     
reco:hitpdune:DisambigFromSpacePoints          0.00106085    0.00144103    0.00268908    0.00128444    0.000453221      10     
reco:pandora:StandardPandora                     0.14127      0.278737       1.35074      0.156938      0.357608        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00111973    0.00153734    0.00282894    0.00145579    0.000471976      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000125655   0.000288574   0.00139599    0.000136784   0.000377494      10     
reco:pandoraStdcalo:Calorimetry                0.000827169   0.00108047    0.00232955    0.000918186    0.0004317       10     
reco:pandoraGnocalo:GnocchiCalorimetry          0.0005856    0.000794149   0.00169557    0.000689059   0.000305905      10     
[art]:TriggerResults:TriggerResultInserter      8.93e-06     1.26643e-05   3.1641e-05     1.007e-05    6.4744e-06       10     
end_path:out1:RootOutput                        2.36e-06     3.5343e-06     9.42e-06     2.8655e-06    1.98261e-06      10     
end_path:out1:RootOutput(write)                 0.0224303     0.0238548     0.0271454     0.0232171    0.00143131       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3345.21 MB
  Peak resident set size usage (VmHWM): 1535.87 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_148_1743078254_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078617.9695182, 'end_time': 1743078906.9699085}
justIN time: 2025-04-03 08:31:38 UTC       justIN version: 01.03.00