justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.168@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.168@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:30:59
From worker nodeHostnamee4057
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:15:50
Input filesmonte-carlo-005746-000090
JobscriptExit code0
Real time14m (892s)
CPU time14m (858s = 96%)
Max RSS bytes1710735360 (1631 MiB)
Outputting started2025-03-27 12:30:44
Output files
Finished2025-03-27 12:30:59
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

058e+06
Now with regularization...
Begin: 5.26987e+06
0 5.26143e+06
1 5.254e+06
2 5.24663e+06
3 5.24015e+06
4 5.23505e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:23:59 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_168_1743077761_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:24:00 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_168_1743077761_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1206 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1206 traces tagged "gauss"
FrameSaver: q=136965 n=16291 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=155829 n=14199 tag=wiener
27-Mar-2025 07:24:40 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_168_1743077761_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1546 traces, 1546 summary
Retagger: tagging trace set: gauss with 1104 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1104 traces tagged "gauss"
FrameSaver: q=146694 n=15737 tag=gauss
wclsFrameSaver: saving 1546 traces tagged "wiener"
FrameSaver: q=164884 n=13513 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1131 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1131 traces tagged "gauss"
FrameSaver: q=139836 n=16176 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=158169 n=13954 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1729 traces, 1729 summary
Retagger: tagging trace set: gauss with 1248 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1248 traces tagged "gauss"
FrameSaver: q=153520 n=17471 tag=gauss
wclsFrameSaver: saving 1729 traces tagged "wiener"
FrameSaver: q=172578 n=15324 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1561 traces, 1561 summary
Retagger: tagging trace set: gauss with 1076 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1076 traces tagged "gauss"
FrameSaver: q=143473 n=15600 tag=gauss
wclsFrameSaver: saving 1561 traces tagged "wiener"
FrameSaver: q=162447 n=13462 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1516 traces, 1516 summary
Retagger: tagging trace set: gauss with 1229 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1229 traces tagged "gauss"
FrameSaver: q=138349 n=16334 tag=gauss
wclsFrameSaver: saving 1516 traces tagged "wiener"
FrameSaver: q=152559 n=14514 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1724 traces, 1724 summary
Retagger: tagging trace set: gauss with 1246 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1246 traces tagged "gauss"
FrameSaver: q=147696 n=17494 tag=gauss
wclsFrameSaver: saving 1724 traces tagged "wiener"
FrameSaver: q=166937 n=15221 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1611 traces, 1611 summary
Retagger: tagging trace set: gauss with 1209 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1209 traces tagged "gauss"
FrameSaver: q=140024 n=16565 tag=gauss
wclsFrameSaver: saving 1611 traces tagged "wiener"
FrameSaver: q=157086 n=14559 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1115 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1115 traces tagged "gauss"
FrameSaver: q=137186 n=15872 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=156894 n=13675 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1613 traces, 1613 summary
Retagger: tagging trace set: gauss with 1126 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1126 traces tagged "gauss"
FrameSaver: q=144029 n=16050 tag=gauss
wclsFrameSaver: saving 1613 traces tagged "wiener"
FrameSaver: q=163745 n=13840 tag=wiener
27-Mar-2025 07:30:42 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_168_1743077761_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:30:42 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_168_1743077761_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       34.6403       39.1996       43.5649       39.0502       2.28291        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000478165   0.000803139   0.00153039    0.000726114   0.000277499      10     
reco:rns:RandomNumberSaver                     3.9965e-05    8.49112e-05   0.000358411   5.4838e-05    9.18725e-05      10     
reco:wclsdatavd:WireCellToolkit                  34.3532       38.6831       43.1972       38.2519       2.24585        10     
reco:gaushit:GausHitFinder                      0.0139691     0.0191292     0.0232927     0.020342     0.00333614       10     
reco:reco3d:SpacePointSolver                    0.0542144     0.0690097     0.0916548     0.068003      0.010044        10     
reco:hitpdune:DisambigFromSpacePoints          0.00161376    0.00224887    0.00454263    0.00196066    0.000822477      10     
reco:pandora:StandardPandora                     0.15507      0.373295       2.06763      0.184413      0.565265        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0013309    0.00287742    0.00557997    0.00268064    0.00103754       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000174577   0.000478904   0.00217356    0.000225177   0.000594259      10     
reco:pandoraStdcalo:Calorimetry                0.00128255    0.00197412    0.00500462    0.00167696     0.0010269       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000872443   0.00125817    0.00323181    0.00106955    0.000664706      10     
[art]:TriggerResults:TriggerResultInserter     1.5008e-05    2.2899e-05    5.8159e-05     1.85e-05     1.21061e-05      10     
end_path:out1:RootOutput                        3.396e-06    7.6893e-06    2.4576e-05     5.886e-06    5.92412e-06      10     
end_path:out1:RootOutput(write)                 0.0320033     0.0401415     0.0586303     0.0389261    0.00743823       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3427 MB
  Peak resident set size usage (VmHWM): 1589.72 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_168_1743077761_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078206.8939364, 'end_time': 1743078643.2724152}
justIN time: 2025-04-03 08:28:32 UTC       justIN version: 01.03.00