justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.33@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.33@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-03-27 12:36:29
From worker nodeHostnamecompute-1-3.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:11:18
Input filesmonte-carlo-005746-000166
JobscriptExit code0
Real time24m (1498s)
CPU time24m (1478s = 98%)
Max RSS bytes1600991232 (1526 MiB)
Outputting started2025-03-27 12:36:17
Output files
Finished2025-03-27 12:36:29
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

regularization...
Begin: 4.52139e+07
0 4.44118e+07
1 4.43916e+07
Now with regularization...
Begin: 4.37623e+07
0 4.37618e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 08:23:42 AST  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_33_1743077488_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 08:23:42 AST  Opened input file "prod_muon_400MeV_protodunevd_1_166657_33_1743077488_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1645 traces, 1645 summary
Retagger: tagging trace set: gauss with 1299 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1299 traces tagged "gauss"
FrameSaver: q=158210 n=17857 tag=gauss
wclsFrameSaver: saving 1645 traces tagged "wiener"
FrameSaver: q=173640 n=15955 tag=wiener
27-Mar-2025 08:24:59 AST  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_33_1743077488_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1723 traces, 1723 summary
Retagger: tagging trace set: gauss with 1237 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1237 traces tagged "gauss"
FrameSaver: q=144472 n=17413 tag=gauss
wclsFrameSaver: saving 1723 traces tagged "wiener"
FrameSaver: q=163851 n=15137 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1673 traces, 1673 summary
Retagger: tagging trace set: gauss with 1159 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1159 traces tagged "gauss"
FrameSaver: q=149320 n=16793 tag=gauss
wclsFrameSaver: saving 1673 traces tagged "wiener"
FrameSaver: q=168483 n=14597 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1652 traces, 1652 summary
Retagger: tagging trace set: gauss with 1190 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1190 traces tagged "gauss"
FrameSaver: q=152542 n=16935 tag=gauss
wclsFrameSaver: saving 1652 traces tagged "wiener"
FrameSaver: q=171951 n=14844 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1722 traces, 1722 summary
Retagger: tagging trace set: gauss with 1218 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1218 traces tagged "gauss"
FrameSaver: q=155909 n=17509 tag=gauss
wclsFrameSaver: saving 1722 traces tagged "wiener"
FrameSaver: q=175172 n=15254 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1585 traces, 1585 summary
Retagger: tagging trace set: gauss with 1220 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1220 traces tagged "gauss"
FrameSaver: q=137190 n=16174 tag=gauss
wclsFrameSaver: saving 1585 traces tagged "wiener"
FrameSaver: q=155068 n=14295 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1636 traces, 1636 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=137465 n=15891 tag=gauss
wclsFrameSaver: saving 1636 traces tagged "wiener"
FrameSaver: q=157014 n=13791 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1574 traces, 1574 summary
Retagger: tagging trace set: gauss with 1135 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1135 traces tagged "gauss"
FrameSaver: q=137067 n=15818 tag=gauss
wclsFrameSaver: saving 1574 traces tagged "wiener"
FrameSaver: q=155727 n=13696 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1143 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1143 traces tagged "gauss"
FrameSaver: q=139120 n=15613 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=155900 n=13662 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1225 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1225 traces tagged "gauss"
FrameSaver: q=142106 n=17022 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=159907 n=14763 tag=wiener
27-Mar-2025 08:36:15 AST  Closed output file "prod_muon_400MeV_protodunevd_1_166657_33_1743077488_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 08:36:15 AST  Closed input file "prod_muon_400MeV_protodunevd_1_166657_33_1743077488_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       72.3626       74.247        78.8834       73.0082       2.23628        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000307723   0.000521321   0.000701335   0.000549186   0.00013079       10     
reco:rns:RandomNumberSaver                     3.4966e-05    5.96467e-05   0.000253586   3.73635e-05   6.46977e-05      10     
reco:wclsdatavd:WireCellToolkit                  71.9804       73.6806       78.4935       72.5405       2.27603        10     
reco:gaushit:GausHitFinder                      0.0184476     0.0201544     0.0231827     0.0200209    0.00133191       10     
reco:reco3d:SpacePointSolver                    0.052794      0.0638555     0.0736512     0.0633635    0.00639307       10     
reco:hitpdune:DisambigFromSpacePoints           0.0017793     0.0023762    0.00394503    0.00217416    0.000570509      10     
reco:pandora:StandardPandora                    0.160755      0.436213       2.22773      0.245789      0.597938        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00155022    0.00307642    0.00557712    0.00299577    0.00105118       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000207548   0.000441703   0.00135373    0.000285415   0.000340349      10     
reco:pandoraStdcalo:Calorimetry                0.00142633    0.00182151    0.00379971    0.00162327    0.000671415      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000900494   0.00117433    0.00261774    0.00100827    0.000489465      10     
[art]:TriggerResults:TriggerResultInserter     1.7487e-05    2.62861e-05   5.1462e-05    1.9737e-05    1.18835e-05      10     
end_path:out1:RootOutput                        3.845e-06    5.4684e-06    1.5812e-05    4.0885e-06    3.52505e-06      10     
end_path:out1:RootOutput(write)                 0.0293405     0.0316724     0.0340785     0.031798     0.00124728       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3345.61 MB
  Peak resident set size usage (VmHWM): 1375.96 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_33_1743077488_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078185.8223994, 'end_time': 1743078977.0889795}
justIN time: 2025-04-03 08:21:26 UTC       justIN version: 01.03.00