justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.118@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.118@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:42
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-03-27 12:21:21
From worker nodeHostnamer22-n09.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5620 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:39:19
Input filesmonte-carlo-005746-000007
JobscriptExit code0
Real time29m (1745s)
CPU time28m (1734s = 99%)
Max RSS bytes1600045056 (1525 MiB)
Outputting started2025-03-27 12:08:25
Output files
Finished2025-03-27 12:21:21
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Begin: 8.63446e+06
0 8.46484e+06
1 8.45358e+06
2 8.45313e+06
Now with regularization...
Begin: 7.82561e+06
0 7.82558e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:53:24 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_118_1743075567_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:53:25 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166655_118_1743075567_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1667 traces, 1667 summary
Retagger: tagging trace set: gauss with 1146 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1146 traces tagged "gauss"
FrameSaver: q=142435 n=16789 tag=gauss
wclsFrameSaver: saving 1667 traces tagged "wiener"
FrameSaver: q=162050 n=14441 tag=wiener
27-Mar-2025 11:54:58 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_118_1743075567_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1558 traces, 1558 summary
Retagger: tagging trace set: gauss with 1159 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1159 traces tagged "gauss"
FrameSaver: q=142343 n=16556 tag=gauss
wclsFrameSaver: saving 1558 traces tagged "wiener"
FrameSaver: q=158504 n=14398 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1661 traces, 1661 summary
Retagger: tagging trace set: gauss with 1208 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1208 traces tagged "gauss"
FrameSaver: q=141311 n=16922 tag=gauss
wclsFrameSaver: saving 1661 traces tagged "wiener"
FrameSaver: q=159102 n=14751 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1556 traces, 1556 summary
Retagger: tagging trace set: gauss with 1176 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1176 traces tagged "gauss"
FrameSaver: q=142585 n=16536 tag=gauss
wclsFrameSaver: saving 1556 traces tagged "wiener"
FrameSaver: q=157546 n=14311 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1550 traces, 1550 summary
Retagger: tagging trace set: gauss with 1242 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1242 traces tagged "gauss"
FrameSaver: q=139387 n=16517 tag=gauss
wclsFrameSaver: saving 1550 traces tagged "wiener"
FrameSaver: q=154262 n=14733 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1653 traces, 1653 summary
Retagger: tagging trace set: gauss with 1160 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1160 traces tagged "gauss"
FrameSaver: q=156297 n=16573 tag=gauss
wclsFrameSaver: saving 1653 traces tagged "wiener"
FrameSaver: q=175910 n=14276 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1657 traces, 1657 summary
Retagger: tagging trace set: gauss with 1214 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1214 traces tagged "gauss"
FrameSaver: q=150045 n=16789 tag=gauss
wclsFrameSaver: saving 1657 traces tagged "wiener"
FrameSaver: q=167852 n=14603 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1677 traces, 1677 summary
Retagger: tagging trace set: gauss with 1208 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1208 traces tagged "gauss"
FrameSaver: q=146722 n=16792 tag=gauss
wclsFrameSaver: saving 1677 traces tagged "wiener"
FrameSaver: q=166246 n=14682 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1707 traces, 1707 summary
Retagger: tagging trace set: gauss with 1238 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1238 traces tagged "gauss"
FrameSaver: q=154050 n=17417 tag=gauss
wclsFrameSaver: saving 1707 traces tagged "wiener"
FrameSaver: q=174307 n=15272 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1562 traces, 1562 summary
Retagger: tagging trace set: gauss with 1273 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1273 traces tagged "gauss"
FrameSaver: q=142072 n=16790 tag=gauss
wclsFrameSaver: saving 1562 traces tagged "wiener"
FrameSaver: q=156593 n=15225 tag=wiener
27-Mar-2025 12:08:24 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166655_118_1743075567_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:08:24 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166655_118_1743075567_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       88.4543       89.3445       92.7694       88.9685       1.19351        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000454096   0.000575093   0.000924041   0.000535374   0.000124528      10     
reco:rns:RandomNumberSaver                     4.5068e-05    7.97851e-05   0.000361295    4.709e-05    9.39033e-05      10     
reco:wclsdatavd:WireCellToolkit                  88.0061        88.68        90.2813       88.5195      0.621308        10     
reco:gaushit:GausHitFinder                      0.0228814     0.0253386     0.0285502     0.0251235    0.00164917       10     
reco:reco3d:SpacePointSolver                    0.0557262     0.0795663     0.123255      0.0737966     0.0210371       10     
reco:hitpdune:DisambigFromSpacePoints          0.00205282    0.00299518    0.00586114    0.00261109    0.00105918       10     
reco:pandora:StandardPandora                     0.28053      0.503741       2.26704      0.303037      0.588182        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00184586    0.00315382    0.00557944    0.00297573    0.000899869      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000283212   0.000507626   0.00182919    0.000299366   0.00047008       10     
reco:pandoraStdcalo:Calorimetry                0.00177716    0.00226909    0.00485705    0.00197502    0.000873439      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00115584    0.00141653    0.00308852     0.0012331    0.000559268      10     
[art]:TriggerResults:TriggerResultInserter     2.6211e-05    3.1437e-05    5.9064e-05    2.73285e-05   9.63021e-06      10     
end_path:out1:RootOutput                        8.499e-06    1.11902e-05   2.1608e-05    9.0065e-06    4.81289e-06      10     
end_path:out1:RootOutput(write)                 0.037857      0.0402291     0.0456123     0.0396641    0.00217713       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3413.5 MB
  Peak resident set size usage (VmHWM): 1422.48 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_118_1743075567_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076361.8223033, 'end_time': 1743077305.379543}
justIN time: 2025-04-03 13:41:33 UTC       justIN version: 01.03.00