justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 167101.20@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID167101.20@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5781
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-28 15:49:00
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-03-28 16:58:05
From worker nodeHostnamecompute-1-3.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-28 16:31:51
Input filesmonte-carlo-005781-000490
JobscriptExit code0
Real time25m (1508s)
CPU time24m (1489s = 98%)
Max RSS bytes1599664128 (1525 MiB)
Outputting started2025-03-28 16:57:00
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim_reco.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim_reco.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen.log
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim_reco.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1.err
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ykermaid/pdvd_muon/05781/1/011/prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1.log
Finished2025-03-28 16:58:05
Saved logsjustin-logs:167101.20-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ation...
Begin: 1.5648e+07
0 1.55456e+07
1 1.55449e+07
Now with regularization...
Begin: 1.4767e+07
0 1.4767e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/497fe0b9af7ebfe5ac11fff59041e9416f4d09cb/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
28-Mar-2025 12:44:02 AST  Initiating request to open input file "prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim.root"
28-Mar-2025 12:44:02 AST  Opened input file "prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1845 traces, 1845 summary
Retagger: tagging trace set: gauss with 1416 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1416 traces tagged "gauss"
FrameSaver: q=176827 n=19314 tag=gauss
wclsFrameSaver: saving 1845 traces tagged "wiener"
FrameSaver: q=195037 n=16787 tag=wiener
28-Mar-2025 12:45:22 AST  Opened output file with pattern "prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1857 traces, 1857 summary
Retagger: tagging trace set: gauss with 1456 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1456 traces tagged "gauss"
FrameSaver: q=175709 n=19514 tag=gauss
wclsFrameSaver: saving 1857 traces tagged "wiener"
FrameSaver: q=193915 n=17141 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1780 traces, 1780 summary
Retagger: tagging trace set: gauss with 1349 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1349 traces tagged "gauss"
FrameSaver: q=169605 n=18533 tag=gauss
wclsFrameSaver: saving 1780 traces tagged "wiener"
FrameSaver: q=187407 n=16112 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1785 traces, 1785 summary
Retagger: tagging trace set: gauss with 1421 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1421 traces tagged "gauss"
FrameSaver: q=175650 n=19785 tag=gauss
wclsFrameSaver: saving 1785 traces tagged "wiener"
FrameSaver: q=191638 n=17511 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1752 traces, 1752 summary
Retagger: tagging trace set: gauss with 1313 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1313 traces tagged "gauss"
FrameSaver: q=163300 n=18087 tag=gauss
wclsFrameSaver: saving 1752 traces tagged "wiener"
FrameSaver: q=181291 n=15940 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1735 traces, 1735 summary
Retagger: tagging trace set: gauss with 1486 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1486 traces tagged "gauss"
FrameSaver: q=177002 n=20100 tag=gauss
wclsFrameSaver: saving 1735 traces tagged "wiener"
FrameSaver: q=191857 n=18300 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1737 traces, 1737 summary
Retagger: tagging trace set: gauss with 1342 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1342 traces tagged "gauss"
FrameSaver: q=170039 n=18448 tag=gauss
wclsFrameSaver: saving 1737 traces tagged "wiener"
FrameSaver: q=187092 n=16356 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1729 traces, 1729 summary
Retagger: tagging trace set: gauss with 1396 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1396 traces tagged "gauss"
FrameSaver: q=172167 n=19094 tag=gauss
wclsFrameSaver: saving 1729 traces tagged "wiener"
FrameSaver: q=187765 n=16938 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1797 traces, 1797 summary
Retagger: tagging trace set: gauss with 1349 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1349 traces tagged "gauss"
FrameSaver: q=177117 n=19125 tag=gauss
wclsFrameSaver: saving 1797 traces tagged "wiener"
FrameSaver: q=193498 n=16612 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1715 traces, 1715 summary
Retagger: tagging trace set: gauss with 1372 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1372 traces tagged "gauss"
FrameSaver: q=161700 n=18377 tag=gauss
wclsFrameSaver: saving 1715 traces tagged "wiener"
FrameSaver: q=177993 n=16206 tag=wiener
28-Mar-2025 12:56:58 AST  Closed output file "prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim_reco.root"
28-Mar-2025 12:56:58 AST  Closed input file "prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       72.7352       76.532        86.7041       74.7356       4.38601        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000333642   0.000519509   0.000971962   0.000466343   0.000198617      10     
reco:rns:RandomNumberSaver                     3.4487e-05    6.13298e-05   0.000223901   4.32595e-05   5.46569e-05      10     
reco:wclsdatavd:WireCellToolkit                  72.3498       75.9232       86.077        74.2043       4.2679         10     
reco:gaushit:GausHitFinder                      0.0204828     0.0257439     0.0450737     0.0231574     0.007255        10     
reco:reco3d:SpacePointSolver                    0.0609264     0.0805214     0.116851      0.0712504     0.0185964       10     
reco:hitpdune:DisambigFromSpacePoints          0.00191149    0.00280203    0.00492863    0.00206795    0.00114981       10     
reco:pandora:StandardPandora                    0.249032      0.447335       1.8369       0.256669      0.467761        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00176981    0.00320394    0.00594928    0.00284637    0.00125856       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000219723   0.000459479   0.00138009    0.000272467   0.000372113      10     
reco:pandoraStdcalo:Calorimetry                0.00160512    0.00253547    0.00556176    0.00195107    0.00119736       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000989263   0.00159201    0.00361691     0.0011078    0.000806785      10     
[art]:TriggerResults:TriggerResultInserter     1.7992e-05    2.55417e-05   7.1682e-05    1.9427e-05    1.56419e-05      10     
end_path:out1:RootOutput                        3.939e-06    5.9296e-06    1.9886e-05      4.2e-06     4.67141e-06      10     
end_path:out1:RootOutput(write)                 0.0314282     0.0396182     0.0632524     0.0323722     0.011701        10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3314.09 MB
  Peak resident set size usage (VmHWM): 1338.59 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_p_450MeV_protodunevd_1_167101_20_1743179519_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743180202.7868512, 'end_time': 1743181019.7651563}
justIN time: 2025-04-03 08:22:27 UTC       justIN version: 01.03.00