justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.33@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.33@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-03-27 12:31:10
From worker nodeHostnamemff2304.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:19:25
Input filesmonte-carlo-005746-000368
JobscriptExit code0
Real time11m (692s)
CPU time11m (680s = 98%)
Max RSS bytes1738706944 (1658 MiB)
Outputting started2025-03-27 12:30:58
Output files
Finished2025-03-27 12:31:10
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

53836e+06
2 5.51939e+06
3 5.5103e+06
4 5.50544e+06
Now with regularization...
Begin: 4.51266e+06
0 4.50656e+06
1 4.50262e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:25:52 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_33_1743077971_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:25:53 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_33_1743077971_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1602 traces, 1602 summary
Retagger: tagging trace set: gauss with 1125 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1125 traces tagged "gauss"
FrameSaver: q=138230 n=16195 tag=gauss
wclsFrameSaver: saving 1602 traces tagged "wiener"
FrameSaver: q=157745 n=13894 tag=wiener
27-Mar-2025 13:26:20 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_33_1743077971_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1574 traces, 1574 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=136921 n=15605 tag=gauss
wclsFrameSaver: saving 1574 traces tagged "wiener"
FrameSaver: q=155808 n=13532 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1707 traces, 1707 summary
Retagger: tagging trace set: gauss with 1236 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1236 traces tagged "gauss"
FrameSaver: q=157755 n=17284 tag=gauss
wclsFrameSaver: saving 1707 traces tagged "wiener"
FrameSaver: q=176982 n=14921 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1636 traces, 1636 summary
Retagger: tagging trace set: gauss with 1176 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1176 traces tagged "gauss"
FrameSaver: q=138176 n=16104 tag=gauss
wclsFrameSaver: saving 1636 traces tagged "wiener"
FrameSaver: q=157513 n=14093 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1207 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1207 traces tagged "gauss"
FrameSaver: q=153917 n=17308 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=172129 n=15255 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1581 traces, 1581 summary
Retagger: tagging trace set: gauss with 1119 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1119 traces tagged "gauss"
FrameSaver: q=137359 n=15762 tag=gauss
wclsFrameSaver: saving 1581 traces tagged "wiener"
FrameSaver: q=155741 n=13645 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1631 traces, 1631 summary
Retagger: tagging trace set: gauss with 1201 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1201 traces tagged "gauss"
FrameSaver: q=139584 n=16267 tag=gauss
wclsFrameSaver: saving 1631 traces tagged "wiener"
FrameSaver: q=156546 n=14069 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1557 traces, 1557 summary
Retagger: tagging trace set: gauss with 1231 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1231 traces tagged "gauss"
FrameSaver: q=137934 n=16135 tag=gauss
wclsFrameSaver: saving 1557 traces tagged "wiener"
FrameSaver: q=154479 n=14325 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1497 traces, 1497 summary
Retagger: tagging trace set: gauss with 1171 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1171 traces tagged "gauss"
FrameSaver: q=136227 n=15497 tag=gauss
wclsFrameSaver: saving 1497 traces tagged "wiener"
FrameSaver: q=152162 n=13793 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1701 traces, 1701 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=137562 n=16801 tag=gauss
wclsFrameSaver: saving 1701 traces tagged "wiener"
FrameSaver: q=158819 n=14704 tag=wiener
27-Mar-2025 13:30:57 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_33_1743077971_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:30:57 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_33_1743077971_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       27.6221       30.2567       33.7161       30.0187       1.58562        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000198616   0.000271114   0.000389514   0.000239458   6.91085e-05      10     
reco:rns:RandomNumberSaver                     1.8841e-05    3.61693e-05   0.000152695   2.24555e-05   3.91467e-05      10     
reco:wclsdatavd:WireCellToolkit                  26.1145       29.8814       33.4831       29.7502       1.82667        10     
reco:gaushit:GausHitFinder                      0.0114605     0.0122934     0.0132685     0.0120938    0.000620742      10     
reco:reco3d:SpacePointSolver                    0.0346127     0.045202      0.0711297     0.0440431    0.00977705       10     
reco:hitpdune:DisambigFromSpacePoints          0.00122335    0.00151944     0.0029387    0.00139481    0.000483173      10     
reco:pandora:StandardPandora                    0.151132      0.286298       1.38494      0.164616      0.366381        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00105932    0.00157604    0.00272322    0.00146773    0.000431149      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000128655   0.000165355   0.000394013   0.000137285   7.71009e-05      10     
reco:pandoraStdcalo:Calorimetry                0.000898561   0.00109559     0.0022678    0.000982819   0.000393444      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000636852   0.000792924   0.00154794    0.000714029   0.000254954      10     
[art]:TriggerResults:TriggerResultInserter      9.29e-06     1.24475e-05   3.2831e-05    1.02505e-05   6.80527e-06      10     
end_path:out1:RootOutput                        2.63e-06     3.7881e-06    1.0131e-05     3.175e-06    2.14242e-06      10     
end_path:out1:RootOutput(write)                 0.0227543     0.0239888     0.0257874     0.0236342    0.000865712      10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3445.92 MB
  Peak resident set size usage (VmHWM): 1636.36 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_33_1743077971_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078330.9393299, 'end_time': 1743078658.1876752}
justIN time: 2025-04-03 08:23:36 UTC       justIN version: 01.03.00