justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.92@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.92@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:39:28
From worker nodeHostnamet2wn004.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:16:02
Input filesmonte-carlo-005746-000283
JobscriptExit code0
Real time22m (1345s)
CPU time19m (1185s = 88%)
Max RSS bytes1698074624 (1619 MiB)
Outputting started2025-03-27 12:38:27
Output files
Finished2025-03-27 12:39:28
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

...
Begin: 8.95937e+07
0 6.83153e+07
1 6.76254e+07
2 6.75897e+07
Now with regularization...
Begin: 6.65572e+07
0 6.65435e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:30:22 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_92_1743077800_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:30:22 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166657_92_1743077800_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1675 traces, 1675 summary
Retagger: tagging trace set: gauss with 1198 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1198 traces tagged "gauss"
FrameSaver: q=140096 n=16566 tag=gauss
wclsFrameSaver: saving 1675 traces tagged "wiener"
FrameSaver: q=159441 n=14339 tag=wiener
27-Mar-2025 12:31:14 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_92_1743077800_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1666 traces, 1666 summary
Retagger: tagging trace set: gauss with 1205 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1205 traces tagged "gauss"
FrameSaver: q=142801 n=16753 tag=gauss
wclsFrameSaver: saving 1666 traces tagged "wiener"
FrameSaver: q=161601 n=14443 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1667 traces, 1667 summary
Retagger: tagging trace set: gauss with 1191 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1191 traces tagged "gauss"
FrameSaver: q=137876 n=16533 tag=gauss
wclsFrameSaver: saving 1667 traces tagged "wiener"
FrameSaver: q=156951 n=14288 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1724 traces, 1724 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=141013 n=17025 tag=gauss
wclsFrameSaver: saving 1724 traces tagged "wiener"
FrameSaver: q=161616 n=14887 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1609 traces, 1609 summary
Retagger: tagging trace set: gauss with 1183 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1183 traces tagged "gauss"
FrameSaver: q=137678 n=16109 tag=gauss
wclsFrameSaver: saving 1609 traces tagged "wiener"
FrameSaver: q=155593 n=14172 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1261 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1261 traces tagged "gauss"
FrameSaver: q=141559 n=16991 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=156903 n=15120 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1253 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1253 traces tagged "gauss"
FrameSaver: q=141572 n=17008 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=158030 n=14956 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=135943 n=16119 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=155284 n=13995 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1713 traces, 1713 summary
Retagger: tagging trace set: gauss with 1284 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1284 traces tagged "gauss"
FrameSaver: q=151693 n=17693 tag=gauss
wclsFrameSaver: saving 1713 traces tagged "wiener"
FrameSaver: q=170255 n=15616 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1418 traces, 1418 summary
Retagger: tagging trace set: gauss with 1021 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1021 traces tagged "gauss"
FrameSaver: q=135159 n=14442 tag=gauss
wclsFrameSaver: saving 1418 traces tagged "wiener"
FrameSaver: q=155039 n=12396 tag=wiener
27-Mar-2025 12:38:27 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166657_92_1743077800_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:38:27 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166657_92_1743077800_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       43.472        47.8915       50.8838       48.1172       2.22079        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000364783   0.000615424   0.00172632    0.000450278   0.000386335      10     
reco:rns:RandomNumberSaver                     4.0198e-05    8.29109e-05   0.000358952   5.29715e-05   9.22992e-05      10     
reco:wclsdatavd:WireCellToolkit                  43.2087       47.236        50.394        47.717        1.96179        10     
reco:gaushit:GausHitFinder                      0.0144319     0.0239486     0.0276116     0.0255241    0.00403597       10     
reco:reco3d:SpacePointSolver                    0.0501867     0.0811882     0.0958087     0.0832857     0.0127386       10     
reco:hitpdune:DisambigFromSpacePoints           0.0017791    0.00282196    0.00550007    0.00270278    0.000985575      10     
reco:pandora:StandardPandora                    0.156362      0.487464       2.62575      0.255889      0.713685        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00212661    0.00341211    0.00690973     0.0028909    0.00140191       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000222709   0.000493325   0.00161987    0.000296617   0.00043996       10     
reco:pandoraStdcalo:Calorimetry                0.00133734    0.00231386    0.00553986    0.00205391    0.00111798       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00076267    0.00162585     0.004242     0.00139769    0.000903404      10     
[art]:TriggerResults:TriggerResultInserter     1.4658e-05    2.62988e-05   8.1086e-05    1.99185e-05   1.85464e-05      10     
end_path:out1:RootOutput                        3.607e-06     6.71e-06     2.4056e-05    4.7145e-06    5.86208e-06      10     
end_path:out1:RootOutput(write)                 0.0299726     0.0456168     0.0580168     0.0459461     0.0068073       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3399.68 MB
  Peak resident set size usage (VmHWM): 1566.06 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_92_1743077800_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078536.5186794, 'end_time': 1743079107.6347938}
justIN time: 2025-04-03 08:24:39 UTC       justIN version: 01.03.00