justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.20@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.20@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:00:09
From worker nodeHostnamee4066
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:43:44
Input filesmonte-carlo-005746-000144
JobscriptExit code0
Real time16m (966s)
CPU time15m (920s = 95%)
Max RSS bytes1732018176 (1651 MiB)
Outputting started2025-03-27 11:59:51
Output files
Finished2025-03-27 12:00:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

23758e+07
0 2.47435e+07
1 2.41547e+07
2 2.41222e+07
3 2.41048e+07
Now with regularization...
Begin: 2.31748e+07
0 2.31656e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 06:52:33 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_20_1743075836_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 06:52:34 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_20_1743075836_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1714 traces, 1714 summary
Retagger: tagging trace set: gauss with 1227 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1227 traces tagged "gauss"
FrameSaver: q=141718 n=17137 tag=gauss
wclsFrameSaver: saving 1714 traces tagged "wiener"
FrameSaver: q=161663 n=14979 tag=wiener
27-Mar-2025 06:53:18 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_20_1743075836_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1730 traces, 1730 summary
Retagger: tagging trace set: gauss with 1257 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1257 traces tagged "gauss"
FrameSaver: q=148026 n=17494 tag=gauss
wclsFrameSaver: saving 1730 traces tagged "wiener"
FrameSaver: q=167104 n=15138 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1660 traces, 1660 summary
Retagger: tagging trace set: gauss with 1220 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1220 traces tagged "gauss"
FrameSaver: q=146076 n=17066 tag=gauss
wclsFrameSaver: saving 1660 traces tagged "wiener"
FrameSaver: q=163631 n=14855 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1690 traces, 1690 summary
Retagger: tagging trace set: gauss with 1217 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1217 traces tagged "gauss"
FrameSaver: q=140348 n=16910 tag=gauss
wclsFrameSaver: saving 1690 traces tagged "wiener"
FrameSaver: q=159784 n=14646 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1596 traces, 1596 summary
Retagger: tagging trace set: gauss with 1199 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1199 traces tagged "gauss"
FrameSaver: q=137168 n=16114 tag=gauss
wclsFrameSaver: saving 1596 traces tagged "wiener"
FrameSaver: q=155368 n=14284 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1628 traces, 1628 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=140678 n=16299 tag=gauss
wclsFrameSaver: saving 1628 traces tagged "wiener"
FrameSaver: q=159837 n=13973 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1677 traces, 1677 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=140684 n=16776 tag=gauss
wclsFrameSaver: saving 1677 traces tagged "wiener"
FrameSaver: q=160215 n=14431 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=139108 n=15824 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=155708 n=13913 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1617 traces, 1617 summary
Retagger: tagging trace set: gauss with 1188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1188 traces tagged "gauss"
FrameSaver: q=137772 n=16464 tag=gauss
wclsFrameSaver: saving 1617 traces tagged "wiener"
FrameSaver: q=156768 n=14366 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1544 traces, 1544 summary
Retagger: tagging trace set: gauss with 1125 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1125 traces tagged "gauss"
FrameSaver: q=137383 n=15669 tag=gauss
wclsFrameSaver: saving 1544 traces tagged "wiener"
FrameSaver: q=155674 n=13523 tag=wiener
27-Mar-2025 06:59:50 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_20_1743075836_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 06:59:50 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_20_1743075836_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       38.936        42.6956       44.8241       43.1005       1.60957        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000520615   0.000792198   0.00126645    0.000784549    0.0001932       10     
reco:rns:RandomNumberSaver                     4.9363e-05    8.92282e-05   0.000347992   5.79085e-05   8.71057e-05      10     
reco:wclsdatavd:WireCellToolkit                  38.7029       42.1721       44.4305       42.4266       1.50609        10     
reco:gaushit:GausHitFinder                      0.0163469     0.0213599      0.02698      0.0210318    0.00271993       10     
reco:reco3d:SpacePointSolver                    0.0539128     0.0751558     0.104643      0.0692291     0.0167767       10     
reco:hitpdune:DisambigFromSpacePoints          0.00147359    0.00250633    0.00518168    0.00227177    0.000972169      10     
reco:pandora:StandardPandora                    0.105474      0.370723       2.05832      0.201512      0.563902        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00142952    0.00291666    0.00559877    0.00260064    0.00119684       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000215885   0.000327177   0.000747039   0.000248641   0.000166347      10     
reco:pandoraStdcalo:Calorimetry                0.00144152     0.0019532    0.00465348    0.00166206    0.000909148      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000737491    0.0012735    0.00302432    0.00112201    0.000596814      10     
[art]:TriggerResults:TriggerResultInserter     1.2734e-05    2.7179e-05     5.876e-05    2.21965e-05   1.44656e-05      10     
end_path:out1:RootOutput                        3.727e-06     9.681e-06    2.8323e-05    8.2405e-06    6.52219e-06      10     
end_path:out1:RootOutput(write)                 0.031569      0.0417445     0.0534603     0.0421777    0.00749979       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3409.44 MB
  Peak resident set size usage (VmHWM): 1610.76 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_20_1743075836_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076318.3683515, 'end_time': 1743076791.357089}
justIN time: 2025-04-03 08:22:25 UTC       justIN version: 01.03.00