justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.113@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.113@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-03-27 12:37:50
From worker nodeHostnamemilan04.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:22:20
Input filesmonte-carlo-005746-000444
JobscriptExit code0
Real time15m (902s)
CPU time14m (845s = 93%)
Max RSS bytes1705701376 (1626 MiB)
Outputting started2025-03-27 12:37:22
Output files
Finished2025-03-27 12:37:50
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

egin: 2.99272e+07
0 2.86906e+07
1 2.85981e+07
2 2.85898e+07
Now with regularization...
Begin: 2.77497e+07
0 2.77485e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:31:08 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_113_1743078153_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:31:08 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_113_1743078153_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1650 traces, 1650 summary
Retagger: tagging trace set: gauss with 1230 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1230 traces tagged "gauss"
FrameSaver: q=138779 n=16860 tag=gauss
wclsFrameSaver: saving 1650 traces tagged "wiener"
FrameSaver: q=156556 n=14747 tag=wiener
27-Mar-2025 13:31:45 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_113_1743078153_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1605 traces, 1605 summary
Retagger: tagging trace set: gauss with 1149 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1149 traces tagged "gauss"
FrameSaver: q=139233 n=16188 tag=gauss
wclsFrameSaver: saving 1605 traces tagged "wiener"
FrameSaver: q=159110 n=13987 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1127 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1127 traces tagged "gauss"
FrameSaver: q=140187 n=15839 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=158025 n=13637 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1722 traces, 1722 summary
Retagger: tagging trace set: gauss with 1273 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1273 traces tagged "gauss"
FrameSaver: q=158432 n=18013 tag=gauss
wclsFrameSaver: saving 1722 traces tagged "wiener"
FrameSaver: q=177016 n=15827 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1555 traces, 1555 summary
Retagger: tagging trace set: gauss with 1193 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1193 traces tagged "gauss"
FrameSaver: q=139233 n=16261 tag=gauss
wclsFrameSaver: saving 1555 traces tagged "wiener"
FrameSaver: q=155716 n=14338 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1612 traces, 1612 summary
Retagger: tagging trace set: gauss with 1223 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1223 traces tagged "gauss"
FrameSaver: q=139625 n=16435 tag=gauss
wclsFrameSaver: saving 1612 traces tagged "wiener"
FrameSaver: q=157078 n=14483 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1579 traces, 1579 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=138598 n=16157 tag=gauss
wclsFrameSaver: saving 1579 traces tagged "wiener"
FrameSaver: q=156642 n=14044 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1580 traces, 1580 summary
Retagger: tagging trace set: gauss with 1123 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1123 traces tagged "gauss"
FrameSaver: q=144984 n=15976 tag=gauss
wclsFrameSaver: saving 1580 traces tagged "wiener"
FrameSaver: q=163926 n=13777 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1647 traces, 1647 summary
Retagger: tagging trace set: gauss with 1208 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1208 traces tagged "gauss"
FrameSaver: q=142860 n=16649 tag=gauss
wclsFrameSaver: saving 1647 traces tagged "wiener"
FrameSaver: q=160660 n=14469 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1536 traces, 1536 summary
Retagger: tagging trace set: gauss with 1129 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1129 traces tagged "gauss"
FrameSaver: q=137066 n=15562 tag=gauss
wclsFrameSaver: saving 1536 traces tagged "wiener"
FrameSaver: q=154942 n=13523 tag=wiener
27-Mar-2025 13:37:22 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_113_1743078153_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:37:22 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_113_1743078153_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       35.4155       36.6999       38.102        36.6729      0.849306        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000291459   0.000495849   0.00121276    0.000391398   0.000265106      10     
reco:rns:RandomNumberSaver                     2.5289e-05    7.29717e-05   0.000416248    3.357e-05    0.000114645      10     
reco:wclsdatavd:WireCellToolkit                  33.8744       36.2178       37.7687       36.3633       1.11464        10     
reco:gaushit:GausHitFinder                      0.0179447     0.0197078     0.0264793     0.0190797    0.00236075       10     
reco:reco3d:SpacePointSolver                    0.0503678     0.057931      0.066623      0.058729     0.00516191       10     
reco:hitpdune:DisambigFromSpacePoints          0.00176329     0.0021639    0.00404484    0.00199712    0.000632933      10     
reco:pandora:StandardPandora                    0.139297      0.355706       1.87286      0.193199       0.50609        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00139404    0.00233458    0.00461338    0.00215832    0.000807321      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000193169   0.000461771   0.00176533    0.000224984   0.00046941       10     
reco:pandoraStdcalo:Calorimetry                 0.0010634    0.00164363    0.00356466    0.00148906    0.000661172      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000841477   0.00114331    0.00244713    0.00101819    0.000445841      10     
[art]:TriggerResults:TriggerResultInserter      1.366e-05    2.05589e-05    5.107e-05     1.821e-05    1.03502e-05      10     
end_path:out1:RootOutput                        3.42e-06      7.235e-06     2.251e-05     5.73e-06     5.32644e-06      10     
end_path:out1:RootOutput(write)                 0.0305435     0.0368746     0.0470192     0.0355945    0.00544571       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3395.28 MB
  Peak resident set size usage (VmHWM): 1607.41 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_113_1743078153_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078637.080484, 'end_time': 1743079042.7832425}
justIN time: 2025-04-03 08:30:26 UTC       justIN version: 01.03.00