justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.190@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.190@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:33:40
From worker nodeHostnamee4066
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:17:16
Input filesmonte-carlo-005746-000302
JobscriptExit code0
Real time16m (966s)
CPU time15m (920s = 95%)
Max RSS bytes1733652480 (1653 MiB)
Outputting started2025-03-27 12:33:23
Output files
Finished2025-03-27 12:33:40
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

4399e+07
0 1.87108e+07
1 1.8611e+07
2 1.85796e+07
3 1.8568e+07
Now with regularization...
Begin: 1.77587e+07
0 1.77491e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:26:15 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_190_1743077847_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:26:16 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_190_1743077847_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1560 traces, 1560 summary
Retagger: tagging trace set: gauss with 1093 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1093 traces tagged "gauss"
FrameSaver: q=136066 n=15410 tag=gauss
wclsFrameSaver: saving 1560 traces tagged "wiener"
FrameSaver: q=154905 n=13306 tag=wiener
27-Mar-2025 07:27:00 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_190_1743077847_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1683 traces, 1683 summary
Retagger: tagging trace set: gauss with 1232 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1232 traces tagged "gauss"
FrameSaver: q=155616 n=17163 tag=gauss
wclsFrameSaver: saving 1683 traces tagged "wiener"
FrameSaver: q=173647 n=14952 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1523 traces, 1523 summary
Retagger: tagging trace set: gauss with 1140 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1140 traces tagged "gauss"
FrameSaver: q=139305 n=15854 tag=gauss
wclsFrameSaver: saving 1523 traces tagged "wiener"
FrameSaver: q=154874 n=13760 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1550 traces, 1550 summary
Retagger: tagging trace set: gauss with 1116 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1116 traces tagged "gauss"
FrameSaver: q=142580 n=15723 tag=gauss
wclsFrameSaver: saving 1550 traces tagged "wiener"
FrameSaver: q=162918 n=13655 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=137065 n=16354 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=156616 n=14206 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1763 traces, 1763 summary
Retagger: tagging trace set: gauss with 1250 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1250 traces tagged "gauss"
FrameSaver: q=152826 n=17867 tag=gauss
wclsFrameSaver: saving 1763 traces tagged "wiener"
FrameSaver: q=172644 n=15628 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1582 traces, 1582 summary
Retagger: tagging trace set: gauss with 1206 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1206 traces tagged "gauss"
FrameSaver: q=140589 n=16653 tag=gauss
wclsFrameSaver: saving 1582 traces tagged "wiener"
FrameSaver: q=157697 n=14714 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1571 traces, 1571 summary
Retagger: tagging trace set: gauss with 1182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1182 traces tagged "gauss"
FrameSaver: q=137276 n=15800 tag=gauss
wclsFrameSaver: saving 1571 traces tagged "wiener"
FrameSaver: q=154854 n=13721 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1766 traces, 1766 summary
Retagger: tagging trace set: gauss with 1274 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1274 traces tagged "gauss"
FrameSaver: q=160701 n=18428 tag=gauss
wclsFrameSaver: saving 1766 traces tagged "wiener"
FrameSaver: q=182380 n=16077 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1620 traces, 1620 summary
Retagger: tagging trace set: gauss with 1148 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1148 traces tagged "gauss"
FrameSaver: q=138106 n=16083 tag=gauss
wclsFrameSaver: saving 1620 traces tagged "wiener"
FrameSaver: q=156900 n=13875 tag=wiener
27-Mar-2025 07:33:22 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_190_1743077847_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:33:22 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_190_1743077847_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       38.9656       41.7371       44.4241       41.7361       2.17049        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000434443   0.000899779   0.00144608    0.000715941   0.000356583      10     
reco:rns:RandomNumberSaver                     5.5243e-05    0.00010483    0.000455864   6.6665e-05    0.000117281      10     
reco:wclsdatavd:WireCellToolkit                  38.6786       41.2427       43.7145       41.3952       2.00101        10     
reco:gaushit:GausHitFinder                      0.0176322     0.0209447     0.0287237     0.0203657    0.00313971       10     
reco:reco3d:SpacePointSolver                    0.0480937     0.0653602     0.0922008     0.0626567     0.0128996       10     
reco:hitpdune:DisambigFromSpacePoints          0.00157385     0.0023141    0.00518189    0.00209923    0.00101448       10     
reco:pandora:StandardPandora                     0.16658      0.347999       1.68403      0.196411       0.44627        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00159873    0.00270685    0.00567593    0.00250302    0.00106812       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000159589   0.00052094    0.00256033    0.000251957   0.000704166      10     
reco:pandoraStdcalo:Calorimetry                0.00131113    0.00206297     0.0048788    0.00169076    0.00104148       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000872874   0.00136084    0.00344163    0.00122107    0.000709183      10     
[art]:TriggerResults:TriggerResultInserter     1.5769e-05    2.44576e-05   5.7738e-05    2.30735e-05   1.15195e-05      10     
end_path:out1:RootOutput                        8.285e-06    1.78205e-05   9.3255e-05     9.387e-06    2.51666e-05      10     
end_path:out1:RootOutput(write)                 0.0280728     0.0457436     0.0833797     0.0440631     0.0143718       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3412.13 MB
  Peak resident set size usage (VmHWM): 1613.68 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_190_1743077847_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078335.904435, 'end_time': 1743078802.9262}
justIN time: 2025-04-03 08:29:26 UTC       justIN version: 01.03.00