justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.18@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.18@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-03-27 12:30:36
From worker nodeHostnamemff2304.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:18:47
Input filesmonte-carlo-005746-000344
JobscriptExit code0
Real time11m (696s)
CPU time11m (683s = 98%)
Max RSS bytes1741656064 (1660 MiB)
Outputting started2025-03-27 12:30:24
Output files
Finished2025-03-27 12:30:36
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

regularization...
Begin: 2.69403e+07
0 2.60564e+07
1 2.60337e+07
Now with regularization...
Begin: 2.53013e+07
0 2.52998e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:25:17 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_18_1743077933_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:25:17 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_18_1743077933_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1646 traces, 1646 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=138434 n=16416 tag=gauss
wclsFrameSaver: saving 1646 traces tagged "wiener"
FrameSaver: q=158658 n=14186 tag=wiener
27-Mar-2025 13:25:45 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_18_1743077933_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1194 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1194 traces tagged "gauss"
FrameSaver: q=145526 n=16871 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=166495 n=14692 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1160 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1160 traces tagged "gauss"
FrameSaver: q=137441 n=16282 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=157169 n=14050 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1693 traces, 1693 summary
Retagger: tagging trace set: gauss with 1188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1188 traces tagged "gauss"
FrameSaver: q=137888 n=16695 tag=gauss
wclsFrameSaver: saving 1693 traces tagged "wiener"
FrameSaver: q=157912 n=14508 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1674 traces, 1674 summary
Retagger: tagging trace set: gauss with 1180 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1180 traces tagged "gauss"
FrameSaver: q=152028 n=17131 tag=gauss
wclsFrameSaver: saving 1674 traces tagged "wiener"
FrameSaver: q=172594 n=14957 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1591 traces, 1591 summary
Retagger: tagging trace set: gauss with 1117 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1117 traces tagged "gauss"
FrameSaver: q=137537 n=15867 tag=gauss
wclsFrameSaver: saving 1591 traces tagged "wiener"
FrameSaver: q=155919 n=13759 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1705 traces, 1705 summary
Retagger: tagging trace set: gauss with 1240 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1240 traces tagged "gauss"
FrameSaver: q=155936 n=17547 tag=gauss
wclsFrameSaver: saving 1705 traces tagged "wiener"
FrameSaver: q=174757 n=15172 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1725 traces, 1725 summary
Retagger: tagging trace set: gauss with 1210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1210 traces tagged "gauss"
FrameSaver: q=152760 n=17153 tag=gauss
wclsFrameSaver: saving 1725 traces tagged "wiener"
FrameSaver: q=172959 n=14785 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1676 traces, 1676 summary
Retagger: tagging trace set: gauss with 1280 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1280 traces tagged "gauss"
FrameSaver: q=144439 n=17059 tag=gauss
wclsFrameSaver: saving 1676 traces tagged "wiener"
FrameSaver: q=161901 n=15092 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1725 traces, 1725 summary
Retagger: tagging trace set: gauss with 1277 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1277 traces tagged "gauss"
FrameSaver: q=147682 n=18029 tag=gauss
wclsFrameSaver: saving 1725 traces tagged "wiener"
FrameSaver: q=165050 n=15623 tag=wiener
27-Mar-2025 13:30:23 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_18_1743077933_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:30:23 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_18_1743077933_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       27.3523       30.3689       31.5567       30.6485       1.24509        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000182216   0.000285495   0.000461906   0.000276169   8.59344e-05      10     
reco:rns:RandomNumberSaver                      1.832e-05    4.03632e-05   0.000160045   2.4486e-05    4.07443e-05      10     
reco:wclsdatavd:WireCellToolkit                  25.8577       29.9815       31.2903       30.4285       1.55673        10     
reco:gaushit:GausHitFinder                      0.0119317     0.0129072     0.0138006     0.012957     0.00050768       10     
reco:reco3d:SpacePointSolver                    0.0407572     0.0489039     0.0697043     0.0464446    0.00772586       10     
reco:hitpdune:DisambigFromSpacePoints          0.00133217    0.00168728    0.00305956     0.0015096    0.00049079       10     
reco:pandora:StandardPandora                    0.104859      0.292426       1.39232      0.175966      0.367627        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.000976783    0.001565     0.00308181    0.00149319    0.000574607      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000128925   0.000189394   0.000432645   0.00014048    0.000102856      10     
reco:pandoraStdcalo:Calorimetry                0.000920691   0.00118375    0.00253119    0.00102916    0.000456432      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000683453   0.000828582   0.00175408    0.000726555   0.000309873      10     
[art]:TriggerResults:TriggerResultInserter      9.571e-06    1.26974e-05   3.3572e-05    1.0205e-05    6.97772e-06      10     
end_path:out1:RootOutput                        2.67e-06     3.9522e-06    1.1491e-05     3.16e-06     2.53732e-06      10     
end_path:out1:RootOutput(write)                 0.0237616     0.025338      0.0303588     0.024583     0.00187434       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3487.48 MB
  Peak resident set size usage (VmHWM): 1664.72 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_18_1743077933_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078295.7780666, 'end_time': 1743078624.143386}
justIN time: 2025-04-03 08:22:34 UTC       justIN version: 01.03.00