justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 167096.188@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID167096.188@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5781
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-28 15:45:32
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-03-28 16:29:44
From worker nodeHostnamecdr1221.int.cedar.computecanada.ca
cpuinfoIntel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-28 16:09:00
Input filesmonte-carlo-005781-000170
JobscriptExit code0
Real time19m (1179s)
CPU time18m (1120s = 94%)
Max RSS bytes1733976064 (1653 MiB)
Outputting started2025-03-28 16:28:41
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim_reco.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim_reco.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/003/prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim_reco.err
Finished2025-03-28 16:29:44
Saved logsjustin-logs:167096.188-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

8e+07
0 1.19265e+07
1 1.18323e+07
2 1.18209e+07
Now with regularization...
Begin: 1.06426e+07
0 1.06389e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/497fe0b9af7ebfe5ac11fff59041e9416f4d09cb/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
28-Mar-2025 09:18:53 PDT  Initiating request to open input file "prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim.root"
28-Mar-2025 09:18:53 PDT  Opened input file "prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1823 traces, 1823 summary
Retagger: tagging trace set: gauss with 1398 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1398 traces tagged "gauss"
FrameSaver: q=173971 n=19171 tag=gauss
wclsFrameSaver: saving 1823 traces tagged "wiener"
FrameSaver: q=192297 n=16838 tag=wiener
28-Mar-2025 09:20:23 PDT  Opened output file with pattern "prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1873 traces, 1873 summary
Retagger: tagging trace set: gauss with 1463 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1463 traces tagged "gauss"
FrameSaver: q=176254 n=19967 tag=gauss
wclsFrameSaver: saving 1873 traces tagged "wiener"
FrameSaver: q=196139 n=17444 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1722 traces, 1722 summary
Retagger: tagging trace set: gauss with 1450 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1450 traces tagged "gauss"
FrameSaver: q=171726 n=19735 tag=gauss
wclsFrameSaver: saving 1722 traces tagged "wiener"
FrameSaver: q=186759 n=17925 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1783 traces, 1783 summary
Retagger: tagging trace set: gauss with 1342 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1342 traces tagged "gauss"
FrameSaver: q=167231 n=18340 tag=gauss
wclsFrameSaver: saving 1783 traces tagged "wiener"
FrameSaver: q=185313 n=16177 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1831 traces, 1831 summary
Retagger: tagging trace set: gauss with 1382 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1382 traces tagged "gauss"
FrameSaver: q=175520 n=19149 tag=gauss
wclsFrameSaver: saving 1831 traces tagged "wiener"
FrameSaver: q=193559 n=16790 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1838 traces, 1838 summary
Retagger: tagging trace set: gauss with 1438 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1438 traces tagged "gauss"
FrameSaver: q=178014 n=20228 tag=gauss
wclsFrameSaver: saving 1838 traces tagged "wiener"
FrameSaver: q=193940 n=17848 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1845 traces, 1845 summary
Retagger: tagging trace set: gauss with 1432 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1432 traces tagged "gauss"
FrameSaver: q=175869 n=19568 tag=gauss
wclsFrameSaver: saving 1845 traces tagged "wiener"
FrameSaver: q=194228 n=17359 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1857 traces, 1857 summary
Retagger: tagging trace set: gauss with 1424 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1424 traces tagged "gauss"
FrameSaver: q=175722 n=19697 tag=gauss
wclsFrameSaver: saving 1857 traces tagged "wiener"
FrameSaver: q=194192 n=17334 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1822 traces, 1822 summary
Retagger: tagging trace set: gauss with 1466 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1466 traces tagged "gauss"
FrameSaver: q=175145 n=19483 tag=gauss
wclsFrameSaver: saving 1822 traces tagged "wiener"
FrameSaver: q=192323 n=17467 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1767 traces, 1767 summary
Retagger: tagging trace set: gauss with 1351 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1351 traces tagged "gauss"
FrameSaver: q=173252 n=18744 tag=gauss
wclsFrameSaver: saving 1767 traces tagged "wiener"
FrameSaver: q=190652 n=16318 tag=wiener
28-Mar-2025 09:28:40 PDT  Closed output file "prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim_reco.root"
28-Mar-2025 09:28:40 PDT  Closed input file "prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       53.2925       58.0489       88.6795       54.0357       10.3072        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000438426   0.000563766   0.000788234   0.000544981   9.9814e-05       10     
reco:rns:RandomNumberSaver                     3.1919e-05    5.56026e-05   0.000184366   4.2589e-05    4.33558e-05      10     
reco:wclsdatavd:WireCellToolkit                  47.9095       53.5039       57.1942       53.3939       2.3399         10     
reco:gaushit:GausHitFinder                      0.0225654     0.0238874     0.0256363     0.0240169    0.000960965      10     
reco:reco3d:SpacePointSolver                    0.0769099     0.0926677     0.104948      0.0917271    0.00853754       10     
reco:hitpdune:DisambigFromSpacePoints          0.00223328    0.00269815    0.00404557    0.00260873    0.000472475      10     
reco:pandora:StandardPandora                    0.314001       4.37428       40.5959      0.350987       12.0739        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00174729    0.00303086    0.00425592    0.00322174    0.000777843      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000247926   0.000449413   0.00153674    0.000278083   0.000380752      10     
reco:pandoraStdcalo:Calorimetry                0.00181083    0.00226762    0.00402473    0.00213569    0.000605665      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00122139    0.00141991    0.00258977    0.00129258    0.000392408      10     
[art]:TriggerResults:TriggerResultInserter     1.6316e-05    2.08687e-05   4.5643e-05    1.86195e-05   8.30916e-06      10     
end_path:out1:RootOutput                        6.639e-06    7.9618e-06    1.5249e-05    7.2725e-06    2.45365e-06      10     
end_path:out1:RootOutput(write)                 0.0378611     0.0395929     0.0410438     0.0397818    0.00096372       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3348.49 MB
  Peak resident set size usage (VmHWM): 1583.57 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_p_450MeV_protodunevd_1_167096_188_1743178148_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743178695.529876, 'end_time': 1743179320.5436244}
justIN time: 2025-04-03 07:59:53 UTC       justIN version: 01.03.00