justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.145@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.145@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-03-27 12:35:51
From worker nodeHostnamemff2203.farm.particle.cz
cpuinfoAMD EPYC 7352 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:22:58
Input filesmonte-carlo-005746-000084
JobscriptExit code0
Real time12m (759s)
CPU time12m (740s = 97%)
Max RSS bytes1729204224 (1649 MiB)
Outputting started2025-03-27 12:35:37
Output files
Finished2025-03-27 12:35:51
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

egin: 1.01721e+07
0 9.90499e+06
1 9.86251e+06
2 9.85755e+06
Now with regularization...
Begin: 9.0789e+06
0 9.07571e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:30:04 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_145_1743078184_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:30:04 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_145_1743078184_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1642 traces, 1642 summary
Retagger: tagging trace set: gauss with 1200 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1200 traces tagged "gauss"
FrameSaver: q=139518 n=16637 tag=gauss
wclsFrameSaver: saving 1642 traces tagged "wiener"
FrameSaver: q=158200 n=14401 tag=wiener
27-Mar-2025 13:30:34 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_145_1743078184_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1627 traces, 1627 summary
Retagger: tagging trace set: gauss with 1175 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1175 traces tagged "gauss"
FrameSaver: q=145535 n=16437 tag=gauss
wclsFrameSaver: saving 1627 traces tagged "wiener"
FrameSaver: q=164854 n=14227 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1680 traces, 1680 summary
Retagger: tagging trace set: gauss with 1213 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1213 traces tagged "gauss"
FrameSaver: q=143578 n=16896 tag=gauss
wclsFrameSaver: saving 1680 traces tagged "wiener"
FrameSaver: q=163298 n=14523 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1735 traces, 1735 summary
Retagger: tagging trace set: gauss with 1254 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1254 traces tagged "gauss"
FrameSaver: q=146249 n=17637 tag=gauss
wclsFrameSaver: saving 1735 traces tagged "wiener"
FrameSaver: q=165321 n=15433 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1569 traces, 1569 summary
Retagger: tagging trace set: gauss with 1108 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1108 traces tagged "gauss"
FrameSaver: q=139076 n=15896 tag=gauss
wclsFrameSaver: saving 1569 traces tagged "wiener"
FrameSaver: q=158016 n=13600 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1538 traces, 1538 summary
Retagger: tagging trace set: gauss with 1110 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1110 traces tagged "gauss"
FrameSaver: q=136314 n=15533 tag=gauss
wclsFrameSaver: saving 1538 traces tagged "wiener"
FrameSaver: q=154935 n=13468 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1561 traces, 1561 summary
Retagger: tagging trace set: gauss with 1142 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1142 traces tagged "gauss"
FrameSaver: q=136920 n=15397 tag=gauss
wclsFrameSaver: saving 1561 traces tagged "wiener"
FrameSaver: q=154044 n=13419 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1561 traces, 1561 summary
Retagger: tagging trace set: gauss with 1121 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1121 traces tagged "gauss"
FrameSaver: q=138387 n=15814 tag=gauss
wclsFrameSaver: saving 1561 traces tagged "wiener"
FrameSaver: q=156131 n=13619 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1863 traces, 1863 summary
Retagger: tagging trace set: gauss with 1321 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1321 traces tagged "gauss"
FrameSaver: q=149623 n=18718 tag=gauss
wclsFrameSaver: saving 1863 traces tagged "wiener"
FrameSaver: q=170022 n=16163 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1727 traces, 1727 summary
Retagger: tagging trace set: gauss with 1322 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1322 traces tagged "gauss"
FrameSaver: q=144626 n=17705 tag=gauss
wclsFrameSaver: saving 1727 traces tagged "wiener"
FrameSaver: q=162411 n=15524 tag=wiener
27-Mar-2025 13:35:36 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_145_1743078184_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:35:36 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_145_1743078184_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       29.7903       32.9143       34.9558       33.2814       1.44045        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000286899   0.00037464    0.000607451   0.000341924   9.19992e-05      10     
reco:rns:RandomNumberSaver                     2.8654e-05    5.73466e-05   0.000238095   3.7036e-05    6.04262e-05      10     
reco:wclsdatavd:WireCellToolkit                  28.1813       32.501        34.6027       32.9926       1.74053        10     
reco:gaushit:GausHitFinder                      0.0122186     0.0136627     0.0161837     0.013655     0.00112146       10     
reco:reco3d:SpacePointSolver                    0.0400097     0.0497287     0.0620386     0.0483987    0.00588331       10     
reco:hitpdune:DisambigFromSpacePoints          0.00137455    0.00180085    0.00357851    0.00161983    0.000620392      10     
reco:pandora:StandardPandora                    0.129171      0.311945       1.49078      0.187346      0.393807        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00142682    0.00192272    0.00365435     0.001768     0.000605335      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000143684   0.000338699   0.00155609    0.000176061   0.000417338      10     
reco:pandoraStdcalo:Calorimetry                 0.0009522    0.00129127    0.00292352    0.00108221    0.000567771      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000631687   0.000873929   0.00192128    0.000774575   0.00035251       10     
[art]:TriggerResults:TriggerResultInserter     1.1081e-05    1.74222e-05   6.0906e-05    1.2835e-05    1.4518e-05       10     
end_path:out1:RootOutput                        3.627e-06    5.3974e-06    1.7053e-05    4.1375e-06    3.89919e-06      10     
end_path:out1:RootOutput(write)                 0.0266337     0.0287312     0.0346631     0.0280514    0.00241087       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3404.21 MB
  Peak resident set size usage (VmHWM): 1583.94 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_145_1743078184_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078579.7089818, 'end_time': 1743078937.4194782}
justIN time: 2025-04-03 08:29:43 UTC       justIN version: 01.03.00