justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 167096.127@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID167096.127@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5781
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-28 15:45:32
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-03-28 16:29:51
From worker nodeHostnamecdr1221.int.cedar.computecanada.ca
cpuinfoIntel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-28 16:07:23
Input filesmonte-carlo-005781-000153
JobscriptExit code0
Real time20m (1243s)
CPU time18m (1080s = 86%)
Max RSS bytes1726791680 (1646 MiB)
Outputting started2025-03-28 16:28:07
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim_reco.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim_reco.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim_reco.err
Finished2025-03-28 16:29:51
Saved logsjustin-logs:167096.127-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Begin: 2.29869e+07
0 2.25215e+07
1 2.25175e+07
Now with regularization...
Begin: 2.19154e+07
0 2.19153e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/497fe0b9af7ebfe5ac11fff59041e9416f4d09cb/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
28-Mar-2025 09:18:53 PDT  Initiating request to open input file "prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim.root"
28-Mar-2025 09:18:53 PDT  Opened input file "prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1813 traces, 1813 summary
Retagger: tagging trace set: gauss with 1478 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1478 traces tagged "gauss"
FrameSaver: q=178276 n=20107 tag=gauss
wclsFrameSaver: saving 1813 traces tagged "wiener"
FrameSaver: q=193785 n=17863 tag=wiener
28-Mar-2025 09:20:23 PDT  Opened output file with pattern "prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1867 traces, 1867 summary
Retagger: tagging trace set: gauss with 1431 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1431 traces tagged "gauss"
FrameSaver: q=175095 n=19267 tag=gauss
wclsFrameSaver: saving 1867 traces tagged "wiener"
FrameSaver: q=193794 n=16828 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1670 traces, 1670 summary
Retagger: tagging trace set: gauss with 1376 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1376 traces tagged "gauss"
FrameSaver: q=163013 n=18829 tag=gauss
wclsFrameSaver: saving 1670 traces tagged "wiener"
FrameSaver: q=177878 n=16926 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1880 traces, 1880 summary
Retagger: tagging trace set: gauss with 1441 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1441 traces tagged "gauss"
FrameSaver: q=177453 n=19696 tag=gauss
wclsFrameSaver: saving 1880 traces tagged "wiener"
FrameSaver: q=195251 n=17240 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1799 traces, 1799 summary
Retagger: tagging trace set: gauss with 1374 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1374 traces tagged "gauss"
FrameSaver: q=170045 n=19246 tag=gauss
wclsFrameSaver: saving 1799 traces tagged "wiener"
FrameSaver: q=186977 n=16875 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1758 traces, 1758 summary
Retagger: tagging trace set: gauss with 1324 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1324 traces tagged "gauss"
FrameSaver: q=175861 n=18757 tag=gauss
wclsFrameSaver: saving 1758 traces tagged "wiener"
FrameSaver: q=192895 n=16331 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1846 traces, 1846 summary
Retagger: tagging trace set: gauss with 1425 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1425 traces tagged "gauss"
FrameSaver: q=175935 n=19454 tag=gauss
wclsFrameSaver: saving 1846 traces tagged "wiener"
FrameSaver: q=194799 n=17318 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1795 traces, 1795 summary
Retagger: tagging trace set: gauss with 1438 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1438 traces tagged "gauss"
FrameSaver: q=171286 n=19335 tag=gauss
wclsFrameSaver: saving 1795 traces tagged "wiener"
FrameSaver: q=187776 n=17078 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1805 traces, 1805 summary
Retagger: tagging trace set: gauss with 1428 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1428 traces tagged "gauss"
FrameSaver: q=175838 n=19475 tag=gauss
wclsFrameSaver: saving 1805 traces tagged "wiener"
FrameSaver: q=192885 n=17157 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1741 traces, 1741 summary
Retagger: tagging trace set: gauss with 1458 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1458 traces tagged "gauss"
FrameSaver: q=179652 n=20046 tag=gauss
wclsFrameSaver: saving 1741 traces tagged "wiener"
FrameSaver: q=194425 n=18067 tag=wiener
28-Mar-2025 09:28:06 PDT  Closed output file "prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim_reco.root"
28-Mar-2025 09:28:06 PDT  Closed input file "prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       48.5077       54.696        88.6878       51.1901       11.4752        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000398129   0.000531108   0.000620593   0.000533466   6.95247e-05      10     
reco:rns:RandomNumberSaver                      3.44e-05     5.72224e-05   0.000192742   4.44435e-05   4.53631e-05      10     
reco:wclsdatavd:WireCellToolkit                  47.6072       50.1253       53.4999       49.6994       2.0075         10     
reco:gaushit:GausHitFinder                      0.0222647     0.0242426     0.0300629     0.023551     0.00208501       10     
reco:reco3d:SpacePointSolver                    0.0759527     0.0943835     0.107083      0.0980107    0.00891978       10     
reco:hitpdune:DisambigFromSpacePoints          0.00213581    0.00265402    0.00433418     0.002464     0.000589422      10     
reco:pandora:StandardPandora                     0.31293       4.39722       40.9063      0.343449       12.1697        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00207418    0.00315178    0.00502609     0.0032576    0.000856243      10     
reco:pandoraShower:LArPandoraShowerCreation    0.00024935    0.000413464   0.00142341    0.000276566   0.000341735      10     
reco:pandoraStdcalo:Calorimetry                0.00164131    0.00225548     0.0037451    0.00228073    0.000563911      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00118085    0.00140861    0.00248825    0.00130664    0.000364631      10     
[art]:TriggerResults:TriggerResultInserter     1.4237e-05    1.91142e-05   4.5414e-05    1.5871e-05    8.92704e-06      10     
end_path:out1:RootOutput                        6.525e-06    8.0056e-06    1.5457e-05     7.364e-06    2.51129e-06      10     
end_path:out1:RootOutput(write)                 0.0388033     0.0401538     0.0410789     0.0403904    0.000669147      10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3325.55 MB
  Peak resident set size usage (VmHWM): 1561.81 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_p_450MeV_protodunevd_1_167096_127_1743178089_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743178687.2147882, 'end_time': 1743179287.1028216}
justIN time: 2025-04-03 07:59:02 UTC       justIN version: 01.03.00