justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166656.59@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166656.59@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:45
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-03-27 12:36:50
From worker nodeHostnamer21-n18.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU X5650 @ 2.67GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:55:06
Input filesmonte-carlo-005747-000139
JobscriptExit code0
Real time28m (1728s)
CPU time28m (1713s = 99%)
Max RSS bytes1627561984 (1552 MiB)
Outputting started2025-03-27 12:23:55
Output files
Finished2025-03-27 12:36:50
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ow with regularization...
Begin: 2.46818e+07
0 2.46803e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:10:53 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_166656_59_1743076513_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:10:54 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_166656_59_1743076513_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4158 traces, 4158 summary
Retagger: tagging trace set: gauss with 2679 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2679 traces tagged "gauss"
FrameSaver: q=489141 n=41152 tag=gauss
wclsFrameSaver: saving 4158 traces tagged "wiener"
FrameSaver: q=531251 n=34178 tag=wiener
27-Mar-2025 12:12:15 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_166656_59_1743076513_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3957 traces, 3957 summary
Retagger: tagging trace set: gauss with 2427 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2427 traces tagged "gauss"
FrameSaver: q=424335 n=37630 tag=gauss
wclsFrameSaver: saving 3957 traces tagged "wiener"
FrameSaver: q=466430 n=30830 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3994 traces, 3994 summary
Retagger: tagging trace set: gauss with 2495 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2495 traces tagged "gauss"
FrameSaver: q=434018 n=38506 tag=gauss
wclsFrameSaver: saving 3994 traces tagged "wiener"
FrameSaver: q=474268 n=31737 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4297 traces, 4297 summary
Retagger: tagging trace set: gauss with 2732 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2732 traces tagged "gauss"
FrameSaver: q=518605 n=43728 tag=gauss
wclsFrameSaver: saving 4297 traces tagged "wiener"
FrameSaver: q=561619 n=36375 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3914 traces, 3914 summary
Retagger: tagging trace set: gauss with 2469 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2469 traces tagged "gauss"
FrameSaver: q=430847 n=37812 tag=gauss
wclsFrameSaver: saving 3914 traces tagged "wiener"
FrameSaver: q=470978 n=31076 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4500 traces, 4500 summary
Retagger: tagging trace set: gauss with 2843 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2843 traces tagged "gauss"
FrameSaver: q=558971 n=45924 tag=gauss
wclsFrameSaver: saving 4500 traces tagged "wiener"
FrameSaver: q=604898 n=38021 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4214 traces, 4214 summary
Retagger: tagging trace set: gauss with 2592 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2592 traces tagged "gauss"
FrameSaver: q=507875 n=42022 tag=gauss
wclsFrameSaver: saving 4214 traces tagged "wiener"
FrameSaver: q=551595 n=34768 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4072 traces, 4072 summary
Retagger: tagging trace set: gauss with 2550 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2550 traces tagged "gauss"
FrameSaver: q=441879 n=38808 tag=gauss
wclsFrameSaver: saving 4072 traces tagged "wiener"
FrameSaver: q=484359 n=31955 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4198 traces, 4198 summary
Retagger: tagging trace set: gauss with 2557 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2557 traces tagged "gauss"
FrameSaver: q=461959 n=40618 tag=gauss
wclsFrameSaver: saving 4198 traces tagged "wiener"
FrameSaver: q=504209 n=33377 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3835 traces, 3835 summary
Retagger: tagging trace set: gauss with 2389 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2389 traces tagged "gauss"
FrameSaver: q=428416 n=36982 tag=gauss
wclsFrameSaver: saving 3835 traces tagged "wiener"
FrameSaver: q=467437 n=30471 tag=wiener
27-Mar-2025 12:23:53 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_166656_59_1743076513_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:23:53 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_166656_59_1743076513_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       75.8859       77.3597       80.8067       77.0087       1.50731        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000509267   0.000838923   0.00100939    0.000871694   0.000138207      10     
reco:rns:RandomNumberSaver                     3.8477e-05    6.3211e-05    0.000240896   4.3888e-05    5.9302e-05       10     
reco:wclsdatavd:WireCellToolkit                  75.1307       76.4742       78.5024       76.2947       1.1392         10     
reco:gaushit:GausHitFinder                      0.0426377     0.0489553     0.0615595     0.0467192    0.00604704       10     
reco:reco3d:SpacePointSolver                    0.258448      0.282664      0.319264      0.275199      0.0185565       10     
reco:hitpdune:DisambigFromSpacePoints          0.00486697     0.0056695    0.00764427    0.00543577    0.000816332      10     
reco:pandora:StandardPandora                    0.230721      0.453191       2.08288      0.257865      0.544073        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00452023    0.00713595     0.0103007    0.00655209    0.00177488       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00174304     0.0041398    0.00618132    0.00346624    0.00152089       10     
reco:pandoraStdcalo:Calorimetry                0.00863398    0.00985098     0.0122228    0.00952368     0.0010856       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00307608      0.00335     0.00493101    0.00315631    0.000532992      10     
[art]:TriggerResults:TriggerResultInserter      2.219e-05    2.56743e-05   5.0912e-05     2.298e-05    8.4253e-06       10     
end_path:out1:RootOutput                        7.033e-06    8.4312e-06    1.7893e-05    7.4455e-06    3.16033e-06      10     
end_path:out1:RootOutput(write)                 0.0586205     0.0647896     0.0752749     0.063409     0.00509676       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3456.02 MB
  Peak resident set size usage (VmHWM): 1445.59 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_166656_59_1743076513_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077415.7745147, 'end_time': 1743078234.644793}
justIN time: 2025-04-03 13:30:43 UTC       justIN version: 01.03.00