justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.147@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.147@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:35:43
From worker nodeHostnamet2wn003.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:14:53
Input filesmonte-carlo-005746-000246
JobscriptExit code0
Real time19m (1185s)
CPU time19m (1140s = 96%)
Max RSS bytes1690079232 (1611 MiB)
Outputting started2025-03-27 12:34:50
Output files
Finished2025-03-27 12:35:43
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

tion...
Begin: 7.35138e+06
0 7.34698e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:26:12 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_147_1743077707_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:26:13 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_147_1743077707_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1529 traces, 1529 summary
Retagger: tagging trace set: gauss with 1086 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1086 traces tagged "gauss"
FrameSaver: q=136771 n=15418 tag=gauss
wclsFrameSaver: saving 1529 traces tagged "wiener"
FrameSaver: q=155517 n=13184 tag=wiener
27-Mar-2025 12:27:03 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_147_1743077707_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1800 traces, 1800 summary
Retagger: tagging trace set: gauss with 1318 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1318 traces tagged "gauss"
FrameSaver: q=148387 n=17980 tag=gauss
wclsFrameSaver: saving 1800 traces tagged "wiener"
FrameSaver: q=167530 n=15586 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1656 traces, 1656 summary
Retagger: tagging trace set: gauss with 1177 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1177 traces tagged "gauss"
FrameSaver: q=141085 n=16638 tag=gauss
wclsFrameSaver: saving 1656 traces tagged "wiener"
FrameSaver: q=159553 n=14281 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1646 traces, 1646 summary
Retagger: tagging trace set: gauss with 1167 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1167 traces tagged "gauss"
FrameSaver: q=142560 n=16457 tag=gauss
wclsFrameSaver: saving 1646 traces tagged "wiener"
FrameSaver: q=162025 n=14185 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1545 traces, 1545 summary
Retagger: tagging trace set: gauss with 1118 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1118 traces tagged "gauss"
FrameSaver: q=136887 n=15691 tag=gauss
wclsFrameSaver: saving 1545 traces tagged "wiener"
FrameSaver: q=155539 n=13588 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1730 traces, 1730 summary
Retagger: tagging trace set: gauss with 1301 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1301 traces tagged "gauss"
FrameSaver: q=155593 n=17540 tag=gauss
wclsFrameSaver: saving 1730 traces tagged "wiener"
FrameSaver: q=174290 n=15381 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1661 traces, 1661 summary
Retagger: tagging trace set: gauss with 1216 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1216 traces tagged "gauss"
FrameSaver: q=152056 n=16799 tag=gauss
wclsFrameSaver: saving 1661 traces tagged "wiener"
FrameSaver: q=170413 n=14677 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1597 traces, 1597 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=139341 n=16263 tag=gauss
wclsFrameSaver: saving 1597 traces tagged "wiener"
FrameSaver: q=156726 n=14208 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1571 traces, 1571 summary
Retagger: tagging trace set: gauss with 1135 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1135 traces tagged "gauss"
FrameSaver: q=137355 n=15843 tag=gauss
wclsFrameSaver: saving 1571 traces tagged "wiener"
FrameSaver: q=155625 n=13633 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1112 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1112 traces tagged "gauss"
FrameSaver: q=137209 n=15789 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=155698 n=13503 tag=wiener
27-Mar-2025 12:34:37 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_147_1743077707_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:34:37 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_147_1743077707_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       47.8878       49.959        51.9327       49.8589       1.16073        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000389751   0.000608681   0.000843685   0.000615786   0.00014374       10     
reco:rns:RandomNumberSaver                      4.696e-05    9.17236e-05   0.000380924   5.6133e-05    9.70275e-05      10     
reco:wclsdatavd:WireCellToolkit                  46.8425       49.277        51.4972       49.1604       1.4227         10     
reco:gaushit:GausHitFinder                      0.0242445     0.0267911     0.0298422     0.0260774    0.00188016       10     
reco:reco3d:SpacePointSolver                    0.080416      0.097035      0.122215      0.0930499     0.0127134       10     
reco:hitpdune:DisambigFromSpacePoints          0.00239936    0.00317926    0.00723993    0.00275145     0.0013716       10     
reco:pandora:StandardPandora                    0.151534      0.493711       2.72901       0.26054      0.746216        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00252238    0.00368272    0.00604364    0.00334473    0.00105313       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000256614   0.000601974   0.00238093    0.000305884   0.000625864      10     
reco:pandoraStdcalo:Calorimetry                0.00197691    0.00247484    0.00507287    0.00222268    0.000875971      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00135378    0.00171479    0.00363886    0.00154119    0.000648267      10     
[art]:TriggerResults:TriggerResultInserter     1.9046e-05    2.82304e-05   6.8692e-05    2.4583e-05    1.37844e-05      10     
end_path:out1:RootOutput                        4.259e-06    6.8151e-06    2.3224e-05     4.594e-06    5.5221e-06       10     
end_path:out1:RootOutput(write)                 0.0435578     0.0457319     0.0486243     0.0453197    0.00163355       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3467.93 MB
  Peak resident set size usage (VmHWM): 1666.49 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_147_1743077707_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078330.113617, 'end_time': 1743078878.0694933}
justIN time: 2025-04-03 08:29:35 UTC       justIN version: 01.03.00