justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.146@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.146@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:35:50
From worker nodeHostnamet2wn011.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:18:23
Input filesmonte-carlo-005746-000349
JobscriptExit code0
Real time16m (1001s)
CPU time14m (897s = 89%)
Max RSS bytes1719111680 (1639 MiB)
Outputting started2025-03-27 12:35:05
Output files
Finished2025-03-27 12:35:50
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

larization...
Begin: 2.71065e+07
0 2.59092e+07
1 2.58935e+07
Now with regularization...
Begin: 2.51763e+07
0 2.51758e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:27:42 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_146_1743077940_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:27:43 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166657_146_1743077940_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=149347 n=16712 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=168803 n=14432 tag=wiener
27-Mar-2025 12:28:27 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_146_1743077940_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1556 traces, 1556 summary
Retagger: tagging trace set: gauss with 1146 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1146 traces tagged "gauss"
FrameSaver: q=137096 n=15657 tag=gauss
wclsFrameSaver: saving 1556 traces tagged "wiener"
FrameSaver: q=154473 n=13601 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1691 traces, 1691 summary
Retagger: tagging trace set: gauss with 1260 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1260 traces tagged "gauss"
FrameSaver: q=143248 n=17213 tag=gauss
wclsFrameSaver: saving 1691 traces tagged "wiener"
FrameSaver: q=161569 n=15115 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1574 traces, 1574 summary
Retagger: tagging trace set: gauss with 1160 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1160 traces tagged "gauss"
FrameSaver: q=136490 n=15729 tag=gauss
wclsFrameSaver: saving 1574 traces tagged "wiener"
FrameSaver: q=153615 n=13553 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1157 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1157 traces tagged "gauss"
FrameSaver: q=138687 n=16060 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=158076 n=13962 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1080 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1080 traces tagged "gauss"
FrameSaver: q=137449 n=15510 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=155221 n=13350 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1613 traces, 1613 summary
Retagger: tagging trace set: gauss with 1144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1144 traces tagged "gauss"
FrameSaver: q=138090 n=16226 tag=gauss
wclsFrameSaver: saving 1613 traces tagged "wiener"
FrameSaver: q=157352 n=13982 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1678 traces, 1678 summary
Retagger: tagging trace set: gauss with 1282 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1282 traces tagged "gauss"
FrameSaver: q=150633 n=18078 tag=gauss
wclsFrameSaver: saving 1678 traces tagged "wiener"
FrameSaver: q=167440 n=15852 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1546 traces, 1546 summary
Retagger: tagging trace set: gauss with 1155 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1155 traces tagged "gauss"
FrameSaver: q=139349 n=16012 tag=gauss
wclsFrameSaver: saving 1546 traces tagged "wiener"
FrameSaver: q=155852 n=13921 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1614 traces, 1614 summary
Retagger: tagging trace set: gauss with 1182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1182 traces tagged "gauss"
FrameSaver: q=139931 n=16233 tag=gauss
wclsFrameSaver: saving 1614 traces tagged "wiener"
FrameSaver: q=158849 n=14203 tag=wiener
27-Mar-2025 12:35:04 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166657_146_1743077940_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:35:04 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166657_146_1743077940_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       38.7738       43.7462       48.1889       43.6512       3.19835        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000291892   0.000448615   0.000809698   0.000400621   0.000156419      10     
reco:rns:RandomNumberSaver                     2.7824e-05    7.20686e-05   0.000349192   4.4315e-05    9.26651e-05      10     
reco:wclsdatavd:WireCellToolkit                  38.4538       43.2126       47.7371       42.5055       3.18235        10     
reco:gaushit:GausHitFinder                      0.0134328     0.022441      0.029401      0.0243864    0.00522194       10     
reco:reco3d:SpacePointSolver                    0.0450805     0.0759265     0.105913      0.0761778     0.0164605       10     
reco:hitpdune:DisambigFromSpacePoints          0.00144852    0.00253522    0.00471734    0.00235993    0.000852267      10     
reco:pandora:StandardPandora                    0.120013      0.381857       1.90502      0.239356      0.509958        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00108623    0.00267797    0.00398009    0.00282285    0.00071328       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000163535   0.000471384   0.00229788    0.000249154   0.000617068      10     
reco:pandoraStdcalo:Calorimetry                 0.0012236    0.00187549    0.00342486    0.00179727    0.000574969      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000712291    0.0013299    0.00212085    0.00132785    0.000341596      10     
[art]:TriggerResults:TriggerResultInserter     1.4197e-05    2.13481e-05   4.3954e-05    1.9733e-05    7.83767e-06      10     
end_path:out1:RootOutput                        3.737e-06    5.5907e-06    1.3636e-05     4.864e-06    2.73495e-06      10     
end_path:out1:RootOutput(write)                 0.025317      0.0394529     0.0458094     0.0423149    0.00692062       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3453.8 MB
  Peak resident set size usage (VmHWM): 1619.23 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_146_1743077940_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078418.272584, 'end_time': 1743078905.4227998}
justIN time: 2025-04-03 08:30:26 UTC       justIN version: 01.03.00