justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.81@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.81@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-03-27 12:21:25
From worker nodeHostnamewn-d20-009.beowulf.cluster
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:48:14
Input filesmonte-carlo-005746-000209
JobscriptExit code0
Real time19m (1191s)
CPU time16m (1012s = 84%)
Max RSS bytes1705582592 (1626 MiB)
Outputting started2025-03-27 12:08:06
Output files
Finished2025-03-27 12:21:25
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

regularization...
Begin: 2.06987e+07
0 1.97976e+07
1 1.97873e+07
Now with regularization...
Begin: 1.9089e+07
0 1.90889e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:00:38 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_81_1743076128_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:00:39 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_81_1743076128_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1632 traces, 1632 summary
Retagger: tagging trace set: gauss with 1182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1182 traces tagged "gauss"
FrameSaver: q=143304 n=16405 tag=gauss
wclsFrameSaver: saving 1632 traces tagged "wiener"
FrameSaver: q=162889 n=14274 tag=wiener
27-Mar-2025 12:01:20 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_81_1743076128_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1540 traces, 1540 summary
Retagger: tagging trace set: gauss with 1106 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1106 traces tagged "gauss"
FrameSaver: q=135065 n=15193 tag=gauss
wclsFrameSaver: saving 1540 traces tagged "wiener"
FrameSaver: q=153476 n=13174 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1630 traces, 1630 summary
Retagger: tagging trace set: gauss with 1149 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1149 traces tagged "gauss"
FrameSaver: q=139767 n=15966 tag=gauss
wclsFrameSaver: saving 1630 traces tagged "wiener"
FrameSaver: q=159033 n=13875 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1566 traces, 1566 summary
Retagger: tagging trace set: gauss with 1138 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1138 traces tagged "gauss"
FrameSaver: q=137363 n=15831 tag=gauss
wclsFrameSaver: saving 1566 traces tagged "wiener"
FrameSaver: q=155547 n=13777 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1643 traces, 1643 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=137489 n=16132 tag=gauss
wclsFrameSaver: saving 1643 traces tagged "wiener"
FrameSaver: q=157042 n=13938 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1703 traces, 1703 summary
Retagger: tagging trace set: gauss with 1224 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1224 traces tagged "gauss"
FrameSaver: q=141504 n=17074 tag=gauss
wclsFrameSaver: saving 1703 traces tagged "wiener"
FrameSaver: q=161083 n=14757 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1663 traces, 1663 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=140347 n=16539 tag=gauss
wclsFrameSaver: saving 1663 traces tagged "wiener"
FrameSaver: q=159271 n=14286 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1568 traces, 1568 summary
Retagger: tagging trace set: gauss with 1139 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1139 traces tagged "gauss"
FrameSaver: q=139145 n=15874 tag=gauss
wclsFrameSaver: saving 1568 traces tagged "wiener"
FrameSaver: q=155419 n=13804 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1461 traces, 1461 summary
Retagger: tagging trace set: gauss with 1204 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1204 traces tagged "gauss"
FrameSaver: q=138675 n=15804 tag=gauss
wclsFrameSaver: saving 1461 traces tagged "wiener"
FrameSaver: q=153624 n=14252 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1720 traces, 1720 summary
Retagger: tagging trace set: gauss with 1295 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1295 traces tagged "gauss"
FrameSaver: q=156911 n=18101 tag=gauss
wclsFrameSaver: saving 1720 traces tagged "wiener"
FrameSaver: q=174732 n=15811 tag=wiener
27-Mar-2025 12:08:05 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_81_1743076128_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:08:05 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_81_1743076128_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.6397       43.9725       45.3272       44.8373       1.49392        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000563954   0.000743169   0.00138601    0.000671747   0.000220821      10     
reco:rns:RandomNumberSaver                     5.3581e-05    9.58258e-05   0.000421334   5.92165e-05   0.000108579      10     
reco:wclsdatavd:WireCellToolkit                  37.5758       43.3128       44.9246       44.3925       2.15136        10     
reco:gaushit:GausHitFinder                      0.0203482     0.0225061     0.0277413      0.02209     0.00198767       10     
reco:reco3d:SpacePointSolver                    0.0547234     0.0752857      0.09994      0.0739134     0.0131359       10     
reco:hitpdune:DisambigFromSpacePoints           0.0019515    0.00270587    0.00606383    0.00228034     0.0011579       10     
reco:pandora:StandardPandora                    0.154398      0.502288       2.84463      0.258759      0.781848        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00168697    0.00400418     0.0177887     0.0023919    0.00462586       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000208453   0.000449985   0.00162513    0.000232729   0.000444012      10     
reco:pandoraStdcalo:Calorimetry                0.00159092    0.00231331    0.00710717    0.00175272     0.0016082       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00107447    0.00166756     0.0052314    0.00128255    0.00119364       10     
[art]:TriggerResults:TriggerResultInserter     1.7843e-05    2.94254e-05   9.8857e-05    2.04285e-05   2.34623e-05      10     
end_path:out1:RootOutput                        6.542e-06    1.00809e-05   3.2711e-05     7.584e-06    7.57422e-06      10     
end_path:out1:RootOutput(write)                 0.0402428     0.0432571     0.0518105     0.0424872    0.00301694       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3388.72 MB
  Peak resident set size usage (VmHWM): 1593.51 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_81_1743076128_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076790.3848855, 'end_time': 1743077286.3124747}
justIN time: 2025-04-03 08:20:39 UTC       justIN version: 01.03.00