justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.124@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.124@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:42
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-03-27 12:13:09
From worker nodeHostnamewn-d20-004.beowulf.cluster
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:39:55
Input filesmonte-carlo-005746-000102
JobscriptExit code0
Real time19m (1184s)
CPU time16m (969s = 81%)
Max RSS bytes1685757952 (1607 MiB)
Outputting started2025-03-27 11:59:40
Output files
Finished2025-03-27 12:13:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

06
1 6.96076e+06
2 6.93916e+06
3 6.93056e+06
4 6.92655e+06
Now with regularization...
Begin: 5.91147e+06
0 5.90821e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:52:43 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_124_1743075628_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:52:43 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166655_124_1743075628_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1563 traces, 1563 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=136185 n=15626 tag=gauss
wclsFrameSaver: saving 1563 traces tagged "wiener"
FrameSaver: q=154043 n=13542 tag=wiener
27-Mar-2025 11:53:25 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_124_1743075628_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1559 traces, 1559 summary
Retagger: tagging trace set: gauss with 1136 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1136 traces tagged "gauss"
FrameSaver: q=136365 n=15436 tag=gauss
wclsFrameSaver: saving 1559 traces tagged "wiener"
FrameSaver: q=154071 n=13544 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1538 traces, 1538 summary
Retagger: tagging trace set: gauss with 1121 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1121 traces tagged "gauss"
FrameSaver: q=137718 n=15527 tag=gauss
wclsFrameSaver: saving 1538 traces tagged "wiener"
FrameSaver: q=156167 n=13431 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1699 traces, 1699 summary
Retagger: tagging trace set: gauss with 1304 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1304 traces tagged "gauss"
FrameSaver: q=158181 n=18184 tag=gauss
wclsFrameSaver: saving 1699 traces tagged "wiener"
FrameSaver: q=174773 n=15942 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1685 traces, 1685 summary
Retagger: tagging trace set: gauss with 1213 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1213 traces tagged "gauss"
FrameSaver: q=139322 n=16751 tag=gauss
wclsFrameSaver: saving 1685 traces tagged "wiener"
FrameSaver: q=158541 n=14560 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1624 traces, 1624 summary
Retagger: tagging trace set: gauss with 1213 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1213 traces tagged "gauss"
FrameSaver: q=138900 n=16614 tag=gauss
wclsFrameSaver: saving 1624 traces tagged "wiener"
FrameSaver: q=157633 n=14589 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1199 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1199 traces tagged "gauss"
FrameSaver: q=138077 n=16205 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=156686 n=14281 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1616 traces, 1616 summary
Retagger: tagging trace set: gauss with 1152 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1152 traces tagged "gauss"
FrameSaver: q=141448 n=16509 tag=gauss
wclsFrameSaver: saving 1616 traces tagged "wiener"
FrameSaver: q=159938 n=14217 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1672 traces, 1672 summary
Retagger: tagging trace set: gauss with 1169 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1169 traces tagged "gauss"
FrameSaver: q=146001 n=17014 tag=gauss
wclsFrameSaver: saving 1672 traces tagged "wiener"
FrameSaver: q=164953 n=14682 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1681 traces, 1681 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=152518 n=17267 tag=gauss
wclsFrameSaver: saving 1681 traces tagged "wiener"
FrameSaver: q=171390 n=15048 tag=wiener
27-Mar-2025 11:59:39 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166655_124_1743075628_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 11:59:39 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166655_124_1743075628_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       39.7733       40.7099       41.3261       40.7933      0.436902        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000547983    0.0028656     0.0141357    0.00070286     0.0043854       10     
reco:rns:RandomNumberSaver                     3.9955e-05    0.000101931   0.000493039   6.03185e-05   0.00013058       10     
reco:wclsdatavd:WireCellToolkit                  38.256        40.0428       40.8839       40.2784      0.729693        10     
reco:gaushit:GausHitFinder                      0.0203829     0.022137      0.0244239     0.0218336    0.00136513       10     
reco:reco3d:SpacePointSolver                    0.0664346     0.0760731     0.0935628     0.0735601    0.00720736       10     
reco:hitpdune:DisambigFromSpacePoints          0.00219144    0.00273509    0.00585977     0.0024261    0.00104758       10     
reco:pandora:StandardPandora                    0.241506      0.512974       2.70965       0.25584      0.732795        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00143644    0.00277346    0.00891915    0.00219999    0.00208418       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000197823   0.000564757   0.00317477    0.000280739   0.000872658      10     
reco:pandoraStdcalo:Calorimetry                0.00135022    0.00223064    0.00692533    0.00177381    0.00157733       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000906999   0.00156343    0.00516909    0.00118252    0.00120886       10     
[art]:TriggerResults:TriggerResultInserter     1.6952e-05    2.75237e-05   0.000103635   1.83795e-05   2.54717e-05      10     
end_path:out1:RootOutput                        3.477e-06     8.041e-06    3.3343e-05     4.528e-06    8.59221e-06      10     
end_path:out1:RootOutput(write)                 0.0343517     0.0393042     0.0447667     0.039057     0.00346252       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3295.95 MB
  Peak resident set size usage (VmHWM): 1493.05 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_124_1743075628_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076316.5204687, 'end_time': 1743076779.9331765}
justIN time: 2025-04-03 08:27:23 UTC       justIN version: 01.03.00