justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.25@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.25@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:33:03
From worker nodeHostnamet2wn065.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:10:40
Input filesmonte-carlo-005746-000157
JobscriptExit code0
Real time20m (1231s)
CPU time16m (975s = 79%)
Max RSS bytes1645215744 (1569 MiB)
Outputting started2025-03-27 12:31:12
Output files
Finished2025-03-27 12:33:03
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

6.23576e+06
Now with regularization...
Begin: 5.22218e+06
0 5.21021e+06
1 5.20219e+06
2 5.19558e+06
3 5.1902e+06
4 5.18584e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:24:05 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_25_1743077482_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:24:07 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166657_25_1743077482_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1587 traces, 1587 summary
Retagger: tagging trace set: gauss with 1140 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1140 traces tagged "gauss"
FrameSaver: q=137362 n=16101 tag=gauss
wclsFrameSaver: saving 1587 traces tagged "wiener"
FrameSaver: q=155376 n=13952 tag=wiener
27-Mar-2025 12:24:50 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_25_1743077482_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1699 traces, 1699 summary
Retagger: tagging trace set: gauss with 1242 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1242 traces tagged "gauss"
FrameSaver: q=148967 n=17032 tag=gauss
wclsFrameSaver: saving 1699 traces tagged "wiener"
FrameSaver: q=168478 n=14797 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1645 traces, 1645 summary
Retagger: tagging trace set: gauss with 1178 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1178 traces tagged "gauss"
FrameSaver: q=139328 n=16245 tag=gauss
wclsFrameSaver: saving 1645 traces tagged "wiener"
FrameSaver: q=158752 n=13991 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1599 traces, 1599 summary
Retagger: tagging trace set: gauss with 1169 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1169 traces tagged "gauss"
FrameSaver: q=137353 n=15878 tag=gauss
wclsFrameSaver: saving 1599 traces tagged "wiener"
FrameSaver: q=155501 n=13867 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1823 traces, 1823 summary
Retagger: tagging trace set: gauss with 1298 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1298 traces tagged "gauss"
FrameSaver: q=147232 n=18054 tag=gauss
wclsFrameSaver: saving 1823 traces tagged "wiener"
FrameSaver: q=166966 n=15591 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1649 traces, 1649 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=139994 n=16509 tag=gauss
wclsFrameSaver: saving 1649 traces tagged "wiener"
FrameSaver: q=157558 n=14423 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1613 traces, 1613 summary
Retagger: tagging trace set: gauss with 1208 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1208 traces tagged "gauss"
FrameSaver: q=141324 n=16889 tag=gauss
wclsFrameSaver: saving 1613 traces tagged "wiener"
FrameSaver: q=157252 n=14757 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1579 traces, 1579 summary
Retagger: tagging trace set: gauss with 1120 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1120 traces tagged "gauss"
FrameSaver: q=137851 n=15665 tag=gauss
wclsFrameSaver: saving 1579 traces tagged "wiener"
FrameSaver: q=156713 n=13445 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1692 traces, 1692 summary
Retagger: tagging trace set: gauss with 1250 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1250 traces tagged "gauss"
FrameSaver: q=152357 n=17295 tag=gauss
wclsFrameSaver: saving 1692 traces tagged "wiener"
FrameSaver: q=171045 n=15082 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1643 traces, 1643 summary
Retagger: tagging trace set: gauss with 1144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1144 traces tagged "gauss"
FrameSaver: q=148072 n=16179 tag=gauss
wclsFrameSaver: saving 1643 traces tagged "wiener"
FrameSaver: q=167614 n=14005 tag=wiener
27-Mar-2025 12:31:11 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166657_25_1743077482_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:31:11 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166657_25_1743077482_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.3943       41.5505       43.023        41.4817      0.935873        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000412701   0.000575641   0.00117439    0.000523027   0.00021122       10     
reco:rns:RandomNumberSaver                     4.1738e-05    8.74454e-05   0.000415777   4.86155e-05   0.000109739      10     
reco:wclsdatavd:WireCellToolkit                  38.3609       40.9712       42.6473       41.1286       1.25663        10     
reco:gaushit:GausHitFinder                      0.0206374     0.0227776     0.0262487     0.0225831    0.00159346       10     
reco:reco3d:SpacePointSolver                    0.0546412     0.0735913     0.0964193     0.0723473     0.0118668       10     
reco:hitpdune:DisambigFromSpacePoints          0.00167041    0.00259637    0.00511917    0.00237013    0.000883789      10     
reco:pandora:StandardPandora                    0.175743      0.425248       2.25302      0.220247      0.609599        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00191162    0.00305483    0.00801945    0.00264448    0.00168816       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000207568   0.000532953   0.00217288    0.000250077   0.000615331      10     
reco:pandoraStdcalo:Calorimetry                0.00151736    0.00220019    0.00617514     0.0018016    0.00133453       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000978128   0.00145748    0.00419983    0.00112882    0.000925708      10     
[art]:TriggerResults:TriggerResultInserter     1.7092e-05    2.68012e-05   8.9457e-05    1.89705e-05   2.10565e-05      10     
end_path:out1:RootOutput                        4.008e-06     9.613e-06    3.3072e-05     7.514e-06    8.03585e-06      10     
end_path:out1:RootOutput(write)                 0.0385963     0.0425506     0.0453779     0.0431577    0.00245977       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3402.9 MB
  Peak resident set size usage (VmHWM): 1518.5 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_25_1743077482_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078155.4320223, 'end_time': 1743078672.5077477}
justIN time: 2025-04-03 08:21:40 UTC       justIN version: 01.03.00