justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.75@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.75@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-03-27 12:36:22
From worker nodeHostnamemilan04.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:20:32
Input filesmonte-carlo-005746-000392
JobscriptExit code0
Real time15m (925s)
CPU time14m (847s = 91%)
Max RSS bytes1701093376 (1622 MiB)
Outputting started2025-03-27 12:35:57
Output files
Finished2025-03-27 12:36:22
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

3017e+06
1 8.20074e+06
2 8.17864e+06
3 8.16785e+06
4 8.16135e+06
Now with regularization...
Begin: 7.39688e+06
0 7.39057e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:29:39 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_75_1743078041_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:29:40 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_75_1743078041_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1625 traces, 1625 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=139806 n=16610 tag=gauss
wclsFrameSaver: saving 1625 traces tagged "wiener"
FrameSaver: q=158047 n=14432 tag=wiener
27-Mar-2025 13:30:16 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_75_1743078041_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1663 traces, 1663 summary
Retagger: tagging trace set: gauss with 1180 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1180 traces tagged "gauss"
FrameSaver: q=147395 n=16638 tag=gauss
wclsFrameSaver: saving 1663 traces tagged "wiener"
FrameSaver: q=167444 n=14328 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1623 traces, 1623 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=139776 n=16390 tag=gauss
wclsFrameSaver: saving 1623 traces tagged "wiener"
FrameSaver: q=158104 n=14263 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1648 traces, 1648 summary
Retagger: tagging trace set: gauss with 1223 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1223 traces tagged "gauss"
FrameSaver: q=155725 n=17057 tag=gauss
wclsFrameSaver: saving 1648 traces tagged "wiener"
FrameSaver: q=174247 n=14933 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1518 traces, 1518 summary
Retagger: tagging trace set: gauss with 1219 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1219 traces tagged "gauss"
FrameSaver: q=141589 n=16545 tag=gauss
wclsFrameSaver: saving 1518 traces tagged "wiener"
FrameSaver: q=156535 n=14711 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1719 traces, 1719 summary
Retagger: tagging trace set: gauss with 1251 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1251 traces tagged "gauss"
FrameSaver: q=150930 n=17675 tag=gauss
wclsFrameSaver: saving 1719 traces tagged "wiener"
FrameSaver: q=171150 n=15279 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1620 traces, 1620 summary
Retagger: tagging trace set: gauss with 1235 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1235 traces tagged "gauss"
FrameSaver: q=140672 n=16883 tag=gauss
wclsFrameSaver: saving 1620 traces tagged "wiener"
FrameSaver: q=158024 n=14956 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=139674 n=16337 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=157395 n=14211 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1564 traces, 1564 summary
Retagger: tagging trace set: gauss with 1091 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1091 traces tagged "gauss"
FrameSaver: q=137256 n=15760 tag=gauss
wclsFrameSaver: saving 1564 traces tagged "wiener"
FrameSaver: q=156590 n=13497 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1513 traces, 1513 summary
Retagger: tagging trace set: gauss with 1094 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1094 traces tagged "gauss"
FrameSaver: q=135860 n=15364 tag=gauss
wclsFrameSaver: saving 1513 traces tagged "wiener"
FrameSaver: q=152902 n=13311 tag=wiener
27-Mar-2025 13:35:56 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_75_1743078041_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:35:56 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_75_1743078041_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       35.7932       36.8932       38.3934       36.656       0.906186        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000352549   0.00195152     0.0114615    0.000634537   0.00324162       10     
reco:rns:RandomNumberSaver                      3.917e-05    7.98719e-05   0.000287199    5.053e-05    7.2769e-05       10     
reco:wclsdatavd:WireCellToolkit                  33.9551       36.4017       38.0678       36.338        1.16743        10     
reco:gaushit:GausHitFinder                      0.0134412     0.0185927     0.0214966     0.0195017    0.00256583       10     
reco:reco3d:SpacePointSolver                    0.0412092     0.0623945     0.0831052     0.060022      0.0113931       10     
reco:hitpdune:DisambigFromSpacePoints          0.00160324    0.00224482    0.00402989    0.00210684    0.000624776      10     
reco:pandora:StandardPandora                    0.162465      0.360942       1.81677      0.203799      0.485631        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00148213    0.00236272    0.00409896    0.00225681    0.000664454      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000186199   0.000252067   0.000570957   0.000204114   0.000110873      10     
reco:pandoraStdcalo:Calorimetry                0.00123662    0.00168817    0.00324955     0.0015438    0.000534261      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000901076   0.00116945    0.00230539    0.00106249    0.000385262      10     
[art]:TriggerResults:TriggerResultInserter      1.538e-05    2.1005e-05     4.752e-05     1.75e-05     9.20731e-06      10     
end_path:out1:RootOutput                        3.93e-06      8.937e-06     1.799e-05     7.25e-06     4.34774e-06      10     
end_path:out1:RootOutput(write)                 0.0316121     0.0362485     0.0431913     0.0359251    0.00332866       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3369.54 MB
  Peak resident set size usage (VmHWM): 1583.43 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_75_1743078041_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078523.5868714, 'end_time': 1743078957.1313772}
justIN time: 2025-04-03 08:24:30 UTC       justIN version: 01.03.00