justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 167098.13@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID167098.13@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5781
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-28 15:46:42
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-03-28 16:46:29
From worker nodeHostnamecdr1215.int.cedar.computecanada.ca
cpuinfoIntel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-28 16:21:39
Input filesmonte-carlo-005781-000247
JobscriptExit code0
Real time20m (1226s)
CPU time18m (1084s = 88%)
Max RSS bytes1733283840 (1652 MiB)
Outputting started2025-03-28 16:42:06
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim_reco.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim_reco.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/005/prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim_reco.err
Finished2025-03-28 16:46:29
Saved logsjustin-logs:167098.13-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

1.23679e+07
0 1.11389e+07
1 1.10577e+07
2 1.10568e+07
Now with regularization...
Begin: 1.0294e+07
0 1.02938e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/497fe0b9af7ebfe5ac11fff59041e9416f4d09cb/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
28-Mar-2025 09:33:35 PDT  Initiating request to open input file "prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim.root"
28-Mar-2025 09:33:35 PDT  Opened input file "prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1951 traces, 1951 summary
Retagger: tagging trace set: gauss with 1421 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1421 traces tagged "gauss"
FrameSaver: q=178118 n=20173 tag=gauss
wclsFrameSaver: saving 1951 traces tagged "wiener"
FrameSaver: q=198380 n=17609 tag=wiener
28-Mar-2025 09:34:31 PDT  Opened output file with pattern "prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1783 traces, 1783 summary
Retagger: tagging trace set: gauss with 1435 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1435 traces tagged "gauss"
FrameSaver: q=175557 n=20221 tag=gauss
wclsFrameSaver: saving 1783 traces tagged "wiener"
FrameSaver: q=191866 n=18033 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1739 traces, 1739 summary
Retagger: tagging trace set: gauss with 1352 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1352 traces tagged "gauss"
FrameSaver: q=167237 n=18807 tag=gauss
wclsFrameSaver: saving 1739 traces tagged "wiener"
FrameSaver: q=183344 n=16656 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1767 traces, 1767 summary
Retagger: tagging trace set: gauss with 1388 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1388 traces tagged "gauss"
FrameSaver: q=170336 n=19015 tag=gauss
wclsFrameSaver: saving 1767 traces tagged "wiener"
FrameSaver: q=187791 n=16844 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1851 traces, 1851 summary
Retagger: tagging trace set: gauss with 1424 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1424 traces tagged "gauss"
FrameSaver: q=176461 n=19791 tag=gauss
wclsFrameSaver: saving 1851 traces tagged "wiener"
FrameSaver: q=194390 n=17345 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1684 traces, 1684 summary
Retagger: tagging trace set: gauss with 1332 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1332 traces tagged "gauss"
FrameSaver: q=169043 n=18743 tag=gauss
wclsFrameSaver: saving 1684 traces tagged "wiener"
FrameSaver: q=185092 n=16495 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1866 traces, 1866 summary
Retagger: tagging trace set: gauss with 1409 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1409 traces tagged "gauss"
FrameSaver: q=173910 n=19339 tag=gauss
wclsFrameSaver: saving 1866 traces tagged "wiener"
FrameSaver: q=194109 n=17034 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1858 traces, 1858 summary
Retagger: tagging trace set: gauss with 1464 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1464 traces tagged "gauss"
FrameSaver: q=178625 n=20299 tag=gauss
wclsFrameSaver: saving 1858 traces tagged "wiener"
FrameSaver: q=196106 n=17925 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1779 traces, 1779 summary
Retagger: tagging trace set: gauss with 1475 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1475 traces tagged "gauss"
FrameSaver: q=172767 n=19400 tag=gauss
wclsFrameSaver: saving 1779 traces tagged "wiener"
FrameSaver: q=189887 n=17256 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1778 traces, 1778 summary
Retagger: tagging trace set: gauss with 1477 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1477 traces tagged "gauss"
FrameSaver: q=177772 n=19897 tag=gauss
wclsFrameSaver: saving 1778 traces tagged "wiener"
FrameSaver: q=193235 n=17971 tag=wiener
28-Mar-2025 09:42:05 PDT  Closed output file "prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim_reco.root"
28-Mar-2025 09:42:05 PDT  Closed input file "prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       49.6893       50.5792       54.9664       50.2244       1.48554        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000476653   0.00064433    0.000815975   0.000632856   9.96674e-05      10     
reco:rns:RandomNumberSaver                     4.8128e-05    7.38656e-05   0.000278009   4.95605e-05   6.81216e-05      10     
reco:wclsdatavd:WireCellToolkit                  49.2585       49.6459       50.2096       49.6922       0.2992         10     
reco:gaushit:GausHitFinder                      0.0226472     0.0246952     0.0278472     0.0242932    0.00166001       10     
reco:reco3d:SpacePointSolver                    0.0796924     0.0958945      0.10997      0.0948313    0.00932802       10     
reco:hitpdune:DisambigFromSpacePoints          0.00238142     0.0028653    0.00521811    0.00262306    0.000799006      10     
reco:pandora:StandardPandora                    0.235129      0.756334       4.5526       0.339839       1.26595        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00183172    0.00297877    0.00565455    0.00298753    0.00105937       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000275909   0.000552611   0.00198236    0.000347012   0.00050769       10     
reco:pandoraStdcalo:Calorimetry                0.00190427    0.00242707    0.00540633    0.00215928    0.000998808      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00118311    0.00144302    0.00309458    0.00127394    0.000551465      10     
[art]:TriggerResults:TriggerResultInserter     1.7391e-05    2.3204e-05    5.7226e-05    1.93925e-05   1.14278e-05      10     
end_path:out1:RootOutput                        3.691e-06    8.2958e-06    2.0074e-05    7.3315e-06    4.10084e-06      10     
end_path:out1:RootOutput(write)                 0.0397504     0.0413753     0.0437048     0.0413094    0.00107608       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3281.13 MB
  Peak resident set size usage (VmHWM): 1525.72 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_p_450MeV_protodunevd_1_167098_13_1743178907_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743179572.881023, 'end_time': 1743180125.6390471}
justIN time: 2025-04-03 07:45:51 UTC       justIN version: 01.03.00