justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.64@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.64@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:04:12
From worker nodeHostnamee2555
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:47:06
Input filesmonte-carlo-005746-000190
JobscriptExit code0
Real time16m (1008s)
CPU time16m (962s = 95%)
Max RSS bytes1729146880 (1649 MiB)
Outputting started2025-03-27 12:03:55
Output files
Finished2025-03-27 12:04:12
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

.63189e+07
0 3.0683e+07
1 3.03876e+07
2 3.03427e+07
3 3.03406e+07
Now with regularization...
Begin: 2.93704e+07
0 2.93695e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 06:56:12 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_64_1743076037_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 06:56:13 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_64_1743076037_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1588 traces, 1588 summary
Retagger: tagging trace set: gauss with 1159 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1159 traces tagged "gauss"
FrameSaver: q=153537 n=16498 tag=gauss
wclsFrameSaver: saving 1588 traces tagged "wiener"
FrameSaver: q=171371 n=14336 tag=wiener
27-Mar-2025 06:57:00 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_64_1743076037_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1619 traces, 1619 summary
Retagger: tagging trace set: gauss with 1161 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1161 traces tagged "gauss"
FrameSaver: q=139000 n=16494 tag=gauss
wclsFrameSaver: saving 1619 traces tagged "wiener"
FrameSaver: q=158771 n=14242 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1600 traces, 1600 summary
Retagger: tagging trace set: gauss with 1199 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1199 traces tagged "gauss"
FrameSaver: q=155841 n=17197 tag=gauss
wclsFrameSaver: saving 1600 traces tagged "wiener"
FrameSaver: q=173559 n=15057 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1739 traces, 1739 summary
Retagger: tagging trace set: gauss with 1233 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1233 traces tagged "gauss"
FrameSaver: q=142519 n=17227 tag=gauss
wclsFrameSaver: saving 1739 traces tagged "wiener"
FrameSaver: q=162487 n=14902 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1591 traces, 1591 summary
Retagger: tagging trace set: gauss with 1161 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1161 traces tagged "gauss"
FrameSaver: q=139877 n=16248 tag=gauss
wclsFrameSaver: saving 1591 traces tagged "wiener"
FrameSaver: q=158050 n=14033 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1579 traces, 1579 summary
Retagger: tagging trace set: gauss with 1188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1188 traces tagged "gauss"
FrameSaver: q=139221 n=16330 tag=gauss
wclsFrameSaver: saving 1579 traces tagged "wiener"
FrameSaver: q=156365 n=14302 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1587 traces, 1587 summary
Retagger: tagging trace set: gauss with 1151 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1151 traces tagged "gauss"
FrameSaver: q=138728 n=16086 tag=gauss
wclsFrameSaver: saving 1587 traces tagged "wiener"
FrameSaver: q=157957 n=13907 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1599 traces, 1599 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=141492 n=16092 tag=gauss
wclsFrameSaver: saving 1599 traces tagged "wiener"
FrameSaver: q=160376 n=13855 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1670 traces, 1670 summary
Retagger: tagging trace set: gauss with 1222 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1222 traces tagged "gauss"
FrameSaver: q=139935 n=16742 tag=gauss
wclsFrameSaver: saving 1670 traces tagged "wiener"
FrameSaver: q=157600 n=14592 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1554 traces, 1554 summary
Retagger: tagging trace set: gauss with 1137 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1137 traces tagged "gauss"
FrameSaver: q=145028 n=15800 tag=gauss
wclsFrameSaver: saving 1554 traces tagged "wiener"
FrameSaver: q=162057 n=13763 tag=wiener
27-Mar-2025 07:03:54 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_64_1743076037_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:03:54 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_64_1743076037_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.0171       45.1041       47.0331       45.665        1.52167        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000428193   0.000911716   0.00178774    0.000783199   0.000371133      10     
reco:rns:RandomNumberSaver                     4.8922e-05    0.000116402   0.000445025    5.823e-05    0.000123155      10     
reco:wclsdatavd:WireCellToolkit                  41.6741       44.5629       46.667        45.0166       1.4195         10     
reco:gaushit:GausHitFinder                      0.0201246     0.0236343     0.0272004     0.0233891    0.00186769       10     
reco:reco3d:SpacePointSolver                    0.063683      0.0764649     0.0870798     0.0772479    0.00786613       10     
reco:hitpdune:DisambigFromSpacePoints          0.00171427    0.00264048    0.00595648    0.00230402    0.00115845       10     
reco:pandora:StandardPandora                    0.165713      0.378502       1.8374       0.221685      0.486858        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00226117    0.00337933    0.00724344    0.00306433    0.00131321       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000186811   0.000336875   0.000925927   0.000281353   0.000199534      10     
reco:pandoraStdcalo:Calorimetry                0.00157397    0.00231834     0.005153     0.00198575    0.000978331      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00106888    0.00145799    0.00356769    0.00122387    0.000710388      10     
[art]:TriggerResults:TriggerResultInserter     1.5289e-05    2.55369e-05   6.7156e-05    1.9898e-05    1.47136e-05      10     
end_path:out1:RootOutput                        7.844e-06    1.07452e-05   2.7442e-05    9.2425e-06    5.60694e-06      10     
end_path:out1:RootOutput(write)                 0.0316931     0.0456916     0.0686889     0.0436377     0.0114027       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3423.17 MB
  Peak resident set size usage (VmHWM): 1627.93 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_64_1743076037_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076532.038131, 'end_time': 1743077034.8529353}
justIN time: 2025-04-03 08:24:41 UTC       justIN version: 01.03.00