justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.15@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.15@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-03-27 12:30:02
From worker nodeHostnamemff2304.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:18:16
Input filesmonte-carlo-005746-000332
JobscriptExit code0
Real time11m (693s)
CPU time11m (680s = 98%)
Max RSS bytes1734975488 (1654 MiB)
Outputting started2025-03-27 12:29:50
Output files
Finished2025-03-27 12:30:02
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

2841e+06
0 6.21583e+06
1 6.13851e+06
2 6.13091e+06
3 6.12884e+06
Now with regularization...
Begin: 5.21411e+06
0 5.21294e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:24:44 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_15_1743077902_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:24:44 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_15_1743077902_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1715 traces, 1715 summary
Retagger: tagging trace set: gauss with 1254 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1254 traces tagged "gauss"
FrameSaver: q=149524 n=17235 tag=gauss
wclsFrameSaver: saving 1715 traces tagged "wiener"
FrameSaver: q=169134 n=15193 tag=wiener
27-Mar-2025 13:25:11 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_15_1743077902_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1619 traces, 1619 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=152154 n=16554 tag=gauss
wclsFrameSaver: saving 1619 traces tagged "wiener"
FrameSaver: q=170964 n=14339 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1646 traces, 1646 summary
Retagger: tagging trace set: gauss with 1262 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1262 traces tagged "gauss"
FrameSaver: q=143154 n=17079 tag=gauss
wclsFrameSaver: saving 1646 traces tagged "wiener"
FrameSaver: q=160921 n=15031 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1675 traces, 1675 summary
Retagger: tagging trace set: gauss with 1176 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1176 traces tagged "gauss"
FrameSaver: q=142450 n=16794 tag=gauss
wclsFrameSaver: saving 1675 traces tagged "wiener"
FrameSaver: q=161566 n=14401 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1531 traces, 1531 summary
Retagger: tagging trace set: gauss with 1048 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1048 traces tagged "gauss"
FrameSaver: q=137689 n=15071 tag=gauss
wclsFrameSaver: saving 1531 traces tagged "wiener"
FrameSaver: q=157052 n=12922 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1535 traces, 1535 summary
Retagger: tagging trace set: gauss with 1108 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1108 traces tagged "gauss"
FrameSaver: q=136573 n=15603 tag=gauss
wclsFrameSaver: saving 1535 traces tagged "wiener"
FrameSaver: q=155055 n=13397 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1670 traces, 1670 summary
Retagger: tagging trace set: gauss with 1164 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1164 traces tagged "gauss"
FrameSaver: q=140565 n=16446 tag=gauss
wclsFrameSaver: saving 1670 traces tagged "wiener"
FrameSaver: q=159890 n=14156 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1587 traces, 1587 summary
Retagger: tagging trace set: gauss with 1191 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1191 traces tagged "gauss"
FrameSaver: q=137947 n=16087 tag=gauss
wclsFrameSaver: saving 1587 traces tagged "wiener"
FrameSaver: q=155919 n=14156 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1193 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1193 traces tagged "gauss"
FrameSaver: q=142177 n=16908 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=160321 n=14661 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1634 traces, 1634 summary
Retagger: tagging trace set: gauss with 1207 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1207 traces tagged "gauss"
FrameSaver: q=135078 n=16448 tag=gauss
wclsFrameSaver: saving 1634 traces tagged "wiener"
FrameSaver: q=153624 n=14336 tag=wiener
27-Mar-2025 13:29:49 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_15_1743077902_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:29:49 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_15_1743077902_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       27.3916       30.2974       32.9546       30.3207       1.69973        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000208037   0.000257872   0.000561129   0.000227218   0.000101571      10     
reco:rns:RandomNumberSaver                      2.042e-05    3.98304e-05   0.000170306   2.3316e-05    4.38968e-05      10     
reco:wclsdatavd:WireCellToolkit                  25.9374       29.9121       32.6437       30.0487       1.91933        10     
reco:gaushit:GausHitFinder                      0.0114795     0.0125446     0.0142083     0.0124168    0.000733668      10     
reco:reco3d:SpacePointSolver                    0.0360017     0.0528679     0.0937534     0.0468073     0.0164341       10     
reco:hitpdune:DisambigFromSpacePoints          0.00120843    0.00161195    0.00253333    0.00148071    0.000368143      10     
reco:pandora:StandardPandora                    0.142729      0.287901       1.3576       0.167908      0.356876        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.000915181    0.0015313    0.00281251    0.00152397    0.00048762       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000123424   0.000235944   0.00087156    0.000134789   0.000226006      10     
reco:pandoraStdcalo:Calorimetry                0.000834329   0.00106511    0.00227023    0.000953863    0.0004062       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000628301   0.000779148   0.00163593    0.000699629   0.000287103      10     
[art]:TriggerResults:TriggerResultInserter      9.68e-06     1.22593e-05   3.0471e-05    1.01955e-05   6.08766e-06      10     
end_path:out1:RootOutput                        2.53e-06     3.5761e-06     9.75e-06      2.85e-06     2.07162e-06      10     
end_path:out1:RootOutput(write)                 0.0230893     0.0243571     0.0268092     0.0241708    0.00101254       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3433.42 MB
  Peak resident set size usage (VmHWM): 1626.64 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_15_1743077902_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078262.2872996, 'end_time': 1743078589.9793935}
justIN time: 2025-04-03 08:24:25 UTC       justIN version: 01.03.00