justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 167599.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID167599.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5780
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-31 02:38:17
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat2025-03-31 03:06:05
From worker nodeHostnameuct2-c537.mwt2.org
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-31 02:49:12
Input filesmonte-carlo-005780-000011
JobscriptExit code0
Real time16m (990s)
CPU time15m (943s = 95%)
Max RSS bytes1726996480 (1646 MiB)
Outputting started2025-03-31 03:05:43
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim_reco.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim_reco.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim_reco.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05780/1/015/prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1.err
Finished2025-03-31 03:06:05
Saved logsjustin-logs:167599.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rization...
Begin: 1.85687e+07
0 1.82987e+07
1 1.82987e+07
Now with regularization...
Begin: 1.74622e+07
0 1.74622e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/497fe0b9af7ebfe5ac11fff59041e9416f4d09cb/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
30-Mar-2025 21:58:36 CDT  Initiating request to open input file "prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim.root"
30-Mar-2025 21:58:36 CDT  Opened input file "prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1794 traces, 1794 summary
Retagger: tagging trace set: gauss with 1389 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1389 traces tagged "gauss"
FrameSaver: q=159739 n=18502 tag=gauss
wclsFrameSaver: saving 1794 traces tagged "wiener"
FrameSaver: q=178374 n=16266 tag=wiener
30-Mar-2025 21:59:21 CDT  Opened output file with pattern "prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1649 traces, 1649 summary
Retagger: tagging trace set: gauss with 1308 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1308 traces tagged "gauss"
FrameSaver: q=157134 n=17115 tag=gauss
wclsFrameSaver: saving 1649 traces tagged "wiener"
FrameSaver: q=174538 n=15234 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1841 traces, 1841 summary
Retagger: tagging trace set: gauss with 1383 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1383 traces tagged "gauss"
FrameSaver: q=176318 n=19450 tag=gauss
wclsFrameSaver: saving 1841 traces tagged "wiener"
FrameSaver: q=194746 n=16877 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1754 traces, 1754 summary
Retagger: tagging trace set: gauss with 1353 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1353 traces tagged "gauss"
FrameSaver: q=166175 n=18835 tag=gauss
wclsFrameSaver: saving 1754 traces tagged "wiener"
FrameSaver: q=182849 n=16425 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1831 traces, 1831 summary
Retagger: tagging trace set: gauss with 1362 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1362 traces tagged "gauss"
FrameSaver: q=163218 n=18984 tag=gauss
wclsFrameSaver: saving 1831 traces tagged "wiener"
FrameSaver: q=181647 n=16533 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1708 traces, 1708 summary
Retagger: tagging trace set: gauss with 1379 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1379 traces tagged "gauss"
FrameSaver: q=165253 n=18983 tag=gauss
wclsFrameSaver: saving 1708 traces tagged "wiener"
FrameSaver: q=180425 n=16704 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1728 traces, 1728 summary
Retagger: tagging trace set: gauss with 1324 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1324 traces tagged "gauss"
FrameSaver: q=158578 n=17982 tag=gauss
wclsFrameSaver: saving 1728 traces tagged "wiener"
FrameSaver: q=176935 n=15893 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1793 traces, 1793 summary
Retagger: tagging trace set: gauss with 1352 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1352 traces tagged "gauss"
FrameSaver: q=174769 n=19017 tag=gauss
wclsFrameSaver: saving 1793 traces tagged "wiener"
FrameSaver: q=193951 n=16544 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1844 traces, 1844 summary
Retagger: tagging trace set: gauss with 1377 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1377 traces tagged "gauss"
FrameSaver: q=178482 n=19261 tag=gauss
wclsFrameSaver: saving 1844 traces tagged "wiener"
FrameSaver: q=197010 n=16676 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1709 traces, 1709 summary
Retagger: tagging trace set: gauss with 1301 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1301 traces tagged "gauss"
FrameSaver: q=161852 n=17782 tag=gauss
wclsFrameSaver: saving 1709 traces tagged "wiener"
FrameSaver: q=178891 n=15507 tag=wiener
30-Mar-2025 22:05:42 CDT  Closed output file "prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim_reco.root"
30-Mar-2025 22:05:42 CDT  Closed input file "prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       39.2316       41.828        44.4656       41.4258       1.56583        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000355027   0.000600834   0.000847873   0.000622023   0.000167226      10     
reco:rns:RandomNumberSaver                     3.4645e-05    6.73658e-05   0.00027117     4.762e-05    6.82339e-05      10     
reco:wclsdatavd:WireCellToolkit                  38.9529       41.3338       44.2205       41.0966       1.3814         10     
reco:gaushit:GausHitFinder                      0.0123936     0.0167101     0.0213842     0.0182731    0.00309417       10     
reco:reco3d:SpacePointSolver                    0.0400865     0.0641186      0.08083       0.06223      0.0119455       10     
reco:hitpdune:DisambigFromSpacePoints          0.00134125    0.00194966    0.00413484    0.00172402    0.000791139      10     
reco:pandora:StandardPandora                    0.153253      0.366628       1.86327      0.203035      0.499463        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00161842    0.00243938    0.00495843    0.00226423    0.00089792       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000157977   0.000402709   0.00177336    0.000220594   0.000476517      10     
reco:pandoraStdcalo:Calorimetry                0.00115002    0.00178148    0.00408982    0.00158944    0.000796335      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000748827   0.00121579     0.0027926     0.0011297    0.000561989      10     
[art]:TriggerResults:TriggerResultInserter     1.1141e-05    1.85959e-05   4.9533e-05    1.5424e-05    1.05609e-05      10     
end_path:out1:RootOutput                        5.46e-06     8.4671e-06     2.126e-05    7.5195e-06    4.36105e-06      10     
end_path:out1:RootOutput(write)                 0.0296462     0.0351677     0.0460788     0.0331035     0.0049483       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3347.87 MB
  Peak resident set size usage (VmHWM): 1578.77 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_m_450MeV_protodunevd_1_167599_0_1743389371_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743389885.0648596, 'end_time': 1743390342.7230458}
justIN time: 2025-04-03 08:17:58 UTC       justIN version: 01.03.00