justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.2@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.2@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:21:55
From worker nodeHostnamee4037
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:06:51
Input filesmonte-carlo-005746-000025
JobscriptExit code0
Real time14m (885s)
CPU time14m (841s = 95%)
Max RSS bytes1730072576 (1649 MiB)
Outputting started2025-03-27 12:21:37
Output files
Finished2025-03-27 12:21:55
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

in: 2.76112e+07
0 1.99668e+07
1 1.9776e+07
2 1.97546e+07
3 1.97457e+07
Now with regularization...
Begin: 1.89588e+07
0 1.89539e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:15:22 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_2_1743077221_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:15:22 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_2_1743077221_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1692 traces, 1692 summary
Retagger: tagging trace set: gauss with 1241 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1241 traces tagged "gauss"
FrameSaver: q=149277 n=16982 tag=gauss
wclsFrameSaver: saving 1692 traces tagged "wiener"
FrameSaver: q=169097 n=14823 tag=wiener
27-Mar-2025 07:16:00 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_2_1743077221_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1644 traces, 1644 summary
Retagger: tagging trace set: gauss with 1206 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1206 traces tagged "gauss"
FrameSaver: q=142389 n=16697 tag=gauss
wclsFrameSaver: saving 1644 traces tagged "wiener"
FrameSaver: q=159810 n=14539 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1578 traces, 1578 summary
Retagger: tagging trace set: gauss with 1148 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1148 traces tagged "gauss"
FrameSaver: q=138651 n=16141 tag=gauss
wclsFrameSaver: saving 1578 traces tagged "wiener"
FrameSaver: q=158666 n=13945 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1615 traces, 1615 summary
Retagger: tagging trace set: gauss with 1142 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1142 traces tagged "gauss"
FrameSaver: q=136981 n=16081 tag=gauss
wclsFrameSaver: saving 1615 traces tagged "wiener"
FrameSaver: q=156630 n=13950 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1729 traces, 1729 summary
Retagger: tagging trace set: gauss with 1186 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1186 traces tagged "gauss"
FrameSaver: q=152253 n=17170 tag=gauss
wclsFrameSaver: saving 1729 traces tagged "wiener"
FrameSaver: q=172546 n=14756 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1764 traces, 1764 summary
Retagger: tagging trace set: gauss with 1250 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1250 traces tagged "gauss"
FrameSaver: q=152761 n=17294 tag=gauss
wclsFrameSaver: saving 1764 traces tagged "wiener"
FrameSaver: q=173627 n=15100 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1650 traces, 1650 summary
Retagger: tagging trace set: gauss with 1193 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1193 traces tagged "gauss"
FrameSaver: q=153452 n=17015 tag=gauss
wclsFrameSaver: saving 1650 traces tagged "wiener"
FrameSaver: q=172424 n=14740 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1685 traces, 1685 summary
Retagger: tagging trace set: gauss with 1184 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1184 traces tagged "gauss"
FrameSaver: q=143383 n=16728 tag=gauss
wclsFrameSaver: saving 1685 traces tagged "wiener"
FrameSaver: q=163174 n=14402 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1725 traces, 1725 summary
Retagger: tagging trace set: gauss with 1311 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1311 traces tagged "gauss"
FrameSaver: q=142387 n=17394 tag=gauss
wclsFrameSaver: saving 1725 traces tagged "wiener"
FrameSaver: q=160230 n=15242 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=138016 n=16052 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=156924 n=14076 tag=wiener
27-Mar-2025 07:21:36 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_2_1743077221_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:21:36 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_2_1743077221_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       36.2851       36.7799       38.1427       36.7189      0.500609        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000353982   0.000638634   0.00105711    0.000551437   0.000224881      10     
reco:rns:RandomNumberSaver                     3.4905e-05    6.33624e-05   0.000275325   3.9013e-05    7.08153e-05      10     
reco:wclsdatavd:WireCellToolkit                  35.8795       36.2872       36.6765       36.3228      0.252422        10     
reco:gaushit:GausHitFinder                      0.0159076     0.0174304     0.0188731     0.0174834    0.000848432      10     
reco:reco3d:SpacePointSolver                    0.0454126     0.0591419     0.0933181     0.0529427     0.0163144       10     
reco:hitpdune:DisambigFromSpacePoints          0.00152839    0.00203782    0.00372178    0.00186523    0.000607137      10     
reco:pandora:StandardPandora                    0.148236      0.368368       2.11281      0.173656      0.581768        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00105649    0.00219375    0.00493385     0.0021031    0.00101936       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000129913   0.000387853   0.00173832    0.000199353   0.000477126      10     
reco:pandoraStdcalo:Calorimetry                0.00100154    0.00159495    0.00395593     0.0013604    0.00080183       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000794796   0.00105715    0.00250751    0.000867682   0.000493083      10     
[art]:TriggerResults:TriggerResultInserter     1.2363e-05    1.74206e-05   4.3461e-05    1.41665e-05   8.80462e-06      10     
end_path:out1:RootOutput                        3.427e-06    7.4442e-06    1.9867e-05     6.638e-06    4.29491e-06      10     
end_path:out1:RootOutput(write)                 0.0269107     0.0365362     0.0568407     0.0336313    0.00802084       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3441.41 MB
  Peak resident set size usage (VmHWM): 1628.06 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_2_1743077221_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077687.3417826, 'end_time': 1743078097.1013894}
justIN time: 2025-04-03 08:15:26 UTC       justIN version: 01.03.00