justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.169@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.169@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:39:51
From worker nodeHostnamee4037
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:24:13
Input filesmonte-carlo-005746-000011
JobscriptExit code0
Real time15m (921s)
CPU time14m (880s = 95%)
Max RSS bytes1728421888 (1648 MiB)
Outputting started2025-03-27 12:39:36
Output files
Finished2025-03-27 12:39:51
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+06
1 7.49071e+06
2 7.46382e+06
3 7.45413e+06
4 7.44971e+06
Now with regularization...
Begin: 6.40904e+06
0 6.40576e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:32:50 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_169_1743078263_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:32:50 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358071_169_1743078263_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1563 traces, 1563 summary
Retagger: tagging trace set: gauss with 1241 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1241 traces tagged "gauss"
FrameSaver: q=142062 n=17024 tag=gauss
wclsFrameSaver: saving 1563 traces tagged "wiener"
FrameSaver: q=157166 n=15129 tag=wiener
27-Mar-2025 07:33:41 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_169_1743078263_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1726 traces, 1726 summary
Retagger: tagging trace set: gauss with 1235 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1235 traces tagged "gauss"
FrameSaver: q=142724 n=17215 tag=gauss
wclsFrameSaver: saving 1726 traces tagged "wiener"
FrameSaver: q=162021 n=14747 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1568 traces, 1568 summary
Retagger: tagging trace set: gauss with 1067 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1067 traces tagged "gauss"
FrameSaver: q=137734 n=15626 tag=gauss
wclsFrameSaver: saving 1568 traces tagged "wiener"
FrameSaver: q=156733 n=13327 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1580 traces, 1580 summary
Retagger: tagging trace set: gauss with 1134 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1134 traces tagged "gauss"
FrameSaver: q=141632 n=16404 tag=gauss
wclsFrameSaver: saving 1580 traces tagged "wiener"
FrameSaver: q=159830 n=14193 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1168 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1168 traces tagged "gauss"
FrameSaver: q=136853 n=15813 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=154557 n=13845 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1184 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1184 traces tagged "gauss"
FrameSaver: q=138153 n=16088 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=157636 n=14183 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1488 traces, 1488 summary
Retagger: tagging trace set: gauss with 1080 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1080 traces tagged "gauss"
FrameSaver: q=136784 n=15431 tag=gauss
wclsFrameSaver: saving 1488 traces tagged "wiener"
FrameSaver: q=158270 n=13469 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1157 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1157 traces tagged "gauss"
FrameSaver: q=139315 n=16018 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=158509 n=13886 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1183 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1183 traces tagged "gauss"
FrameSaver: q=149179 n=16881 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=168398 n=14711 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1683 traces, 1683 summary
Retagger: tagging trace set: gauss with 1210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1210 traces tagged "gauss"
FrameSaver: q=143029 n=17125 tag=gauss
wclsFrameSaver: saving 1683 traces tagged "wiener"
FrameSaver: q=161911 n=14779 tag=wiener
27-Mar-2025 07:39:35 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358071_169_1743078263_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:39:35 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358071_169_1743078263_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       35.7153       39.7193       50.5175       38.2258       4.55404        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000395549   0.000711241   0.00112739    0.000656738   0.000255831      10     
reco:rns:RandomNumberSaver                     2.7832e-05    6.6818e-05    0.000274273   4.24595e-05   7.0223e-05       10     
reco:wclsdatavd:WireCellToolkit                  35.4632       39.2129       48.0825       37.9442       4.04225        10     
reco:gaushit:GausHitFinder                      0.0165033     0.0187307     0.0248123     0.0172394     0.0028481       10     
reco:reco3d:SpacePointSolver                    0.0433826     0.0598634     0.089529      0.0571754     0.0156908       10     
reco:hitpdune:DisambigFromSpacePoints          0.00136615    0.00201551    0.00383549    0.00177902    0.000710571      10     
reco:pandora:StandardPandora                     0.14137      0.380392       2.27251      0.175747      0.631009        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00176958    0.00254434    0.00635732    0.00217036    0.00128522       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000162343   0.000293935   0.000999499   0.000210939   0.000237929      10     
reco:pandoraStdcalo:Calorimetry                0.00114798     0.0018664    0.00550259    0.00137987    0.00124034       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000704618   0.00125407     0.0042241    0.00090098    0.000999791      10     
[art]:TriggerResults:TriggerResultInserter     1.3355e-05    2.18168e-05   7.1544e-05    1.58395e-05   1.67507e-05      10     
end_path:out1:RootOutput                        3.657e-06    8.9439e-06    2.7362e-05     7.294e-06    6.42863e-06      10     
end_path:out1:RootOutput(write)                 0.0259905     0.0348083     0.0579036     0.0307309    0.00997261       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3416.26 MB
  Peak resident set size usage (VmHWM): 1612.74 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_169_1743078263_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078737.196551, 'end_time': 1743079175.7534213}
justIN time: 2025-04-03 08:28:34 UTC       justIN version: 01.03.00