justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.169@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.169@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:42
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-03-27 12:10:09
From worker nodeHostnamecompute-1-3.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:43:05
Input filesmonte-carlo-005746-000146
JobscriptExit code0
Real time26m (1606s)
CPU time24m (1483s = 92%)
Max RSS bytes1599131648 (1525 MiB)
Outputting started2025-03-27 12:09:52
Output files
Finished2025-03-27 12:10:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

1e+07
0 9.94789e+06
1 9.80657e+06
2 9.78725e+06
3 9.78409e+06
Now with regularization...
Begin: 8.80457e+06
0 8.80032e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:57:19 AST  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_169_1743075852_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:57:20 AST  Opened input file "prod_muon_400MeV_protodunevd_1_166655_169_1743075852_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1639 traces, 1639 summary
Retagger: tagging trace set: gauss with 1164 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1164 traces tagged "gauss"
FrameSaver: q=137793 n=16407 tag=gauss
wclsFrameSaver: saving 1639 traces tagged "wiener"
FrameSaver: q=156636 n=14108 tag=wiener
27-Mar-2025 07:58:39 AST  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_169_1743075852_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1537 traces, 1537 summary
Retagger: tagging trace set: gauss with 1126 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1126 traces tagged "gauss"
FrameSaver: q=136698 n=15600 tag=gauss
wclsFrameSaver: saving 1537 traces tagged "wiener"
FrameSaver: q=155960 n=13576 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1591 traces, 1591 summary
Retagger: tagging trace set: gauss with 1151 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1151 traces tagged "gauss"
FrameSaver: q=137027 n=15962 tag=gauss
wclsFrameSaver: saving 1591 traces tagged "wiener"
FrameSaver: q=155065 n=14003 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1618 traces, 1618 summary
Retagger: tagging trace set: gauss with 1157 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1157 traces tagged "gauss"
FrameSaver: q=140089 n=16307 tag=gauss
wclsFrameSaver: saving 1618 traces tagged "wiener"
FrameSaver: q=158458 n=13869 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1739 traces, 1739 summary
Retagger: tagging trace set: gauss with 1244 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1244 traces tagged "gauss"
FrameSaver: q=142617 n=17473 tag=gauss
wclsFrameSaver: saving 1739 traces tagged "wiener"
FrameSaver: q=162502 n=15166 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1650 traces, 1650 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=140180 n=16313 tag=gauss
wclsFrameSaver: saving 1650 traces tagged "wiener"
FrameSaver: q=159295 n=14162 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1634 traces, 1634 summary
Retagger: tagging trace set: gauss with 1202 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1202 traces tagged "gauss"
FrameSaver: q=137627 n=16280 tag=gauss
wclsFrameSaver: saving 1634 traces tagged "wiener"
FrameSaver: q=156535 n=14207 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1681 traces, 1681 summary
Retagger: tagging trace set: gauss with 1221 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1221 traces tagged "gauss"
FrameSaver: q=146909 n=16982 tag=gauss
wclsFrameSaver: saving 1681 traces tagged "wiener"
FrameSaver: q=166374 n=14883 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1154 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1154 traces tagged "gauss"
FrameSaver: q=145869 n=16165 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=164922 n=14138 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1668 traces, 1668 summary
Retagger: tagging trace set: gauss with 1180 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1180 traces tagged "gauss"
FrameSaver: q=149295 n=16770 tag=gauss
wclsFrameSaver: saving 1668 traces tagged "wiener"
FrameSaver: q=168604 n=14479 tag=wiener
27-Mar-2025 08:09:50 AST  Closed output file "prod_muon_400MeV_protodunevd_1_166655_169_1743075852_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 08:09:50 AST  Closed input file "prod_muon_400MeV_protodunevd_1_166655_169_1743075852_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       70.6976       73.3023       83.0568       71.8117       3.64307        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000264816   0.000421672   0.000734766   0.000380358   0.000128616      10     
reco:rns:RandomNumberSaver                     2.9819e-05    6.24532e-05   0.000285888   3.76515e-05   7.46004e-05      10     
reco:wclsdatavd:WireCellToolkit                  70.3164       72.6089       82.6514       71.4152       3.45227        10     
reco:gaushit:GausHitFinder                      0.0187428     0.0203695     0.0233395     0.0198975    0.00123116       10     
reco:reco3d:SpacePointSolver                    0.0590199     0.0681297     0.0973357     0.063161      0.0108928       10     
reco:hitpdune:DisambigFromSpacePoints          0.00200353     0.0025302    0.00475764    0.00232187    0.000762857      10     
reco:pandora:StandardPandora                    0.232601      0.547648       3.12614      0.264268      0.859681        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00138553    0.00253598    0.00469093    0.00240588    0.000852948      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000216359   0.000276281   0.000700311   0.000229706   0.000141927      10     
reco:pandoraStdcalo:Calorimetry                0.00143272     0.0018331    0.00395642    0.00164881    0.000720133      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000898131    0.0011412    0.00263816    0.000959939   0.000505593      10     
[art]:TriggerResults:TriggerResultInserter     1.7519e-05    2.24874e-05   4.8196e-05    1.8766e-05    9.10457e-06      10     
end_path:out1:RootOutput                        3.903e-06    5.3314e-06    1.5074e-05     4.144e-06    3.26283e-06      10     
end_path:out1:RootOutput(write)                 0.029927      0.0443263      0.16436      0.0309638     0.0400184       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3354.38 MB
  Peak resident set size usage (VmHWM): 1378.9 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_169_1743075852_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076593.66187, 'end_time': 1743077391.7540727}
justIN time: 2025-04-03 08:30:28 UTC       justIN version: 01.03.00