justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.37@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.37@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-03-27 12:36:23
From worker nodeHostnamecompute-1-3.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:11:12
Input filesmonte-carlo-005746-000160
JobscriptExit code0
Real time24m (1497s)
CPU time24m (1480s = 98%)
Max RSS bytes1599741952 (1525 MiB)
Outputting started2025-03-27 12:36:10
Output files
Finished2025-03-27 12:36:23
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

larization...
Begin: 6.84546e+06
0 6.8436e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 08:23:30 AST  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_37_1743077483_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 08:23:31 AST  Opened input file "prod_muon_400MeV_protodunevd_1_166657_37_1743077483_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1621 traces, 1621 summary
Retagger: tagging trace set: gauss with 1247 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1247 traces tagged "gauss"
FrameSaver: q=140469 n=17074 tag=gauss
wclsFrameSaver: saving 1621 traces tagged "wiener"
FrameSaver: q=156882 n=15176 tag=wiener
27-Mar-2025 08:24:54 AST  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_37_1743077483_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1519 traces, 1519 summary
Retagger: tagging trace set: gauss with 1067 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1067 traces tagged "gauss"
FrameSaver: q=135082 n=15076 tag=gauss
wclsFrameSaver: saving 1519 traces tagged "wiener"
FrameSaver: q=153214 n=13001 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1626 traces, 1626 summary
Retagger: tagging trace set: gauss with 1153 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1153 traces tagged "gauss"
FrameSaver: q=138138 n=16032 tag=gauss
wclsFrameSaver: saving 1626 traces tagged "wiener"
FrameSaver: q=156367 n=13949 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1542 traces, 1542 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=136587 n=15728 tag=gauss
wclsFrameSaver: saving 1542 traces tagged "wiener"
FrameSaver: q=154021 n=13765 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1612 traces, 1612 summary
Retagger: tagging trace set: gauss with 1149 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1149 traces tagged "gauss"
FrameSaver: q=137006 n=15697 tag=gauss
wclsFrameSaver: saving 1612 traces tagged "wiener"
FrameSaver: q=155667 n=13645 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1688 traces, 1688 summary
Retagger: tagging trace set: gauss with 1223 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1223 traces tagged "gauss"
FrameSaver: q=137158 n=16649 tag=gauss
wclsFrameSaver: saving 1688 traces tagged "wiener"
FrameSaver: q=156629 n=14564 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1693 traces, 1693 summary
Retagger: tagging trace set: gauss with 1168 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1168 traces tagged "gauss"
FrameSaver: q=145011 n=16759 tag=gauss
wclsFrameSaver: saving 1693 traces tagged "wiener"
FrameSaver: q=164645 n=14509 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1779 traces, 1779 summary
Retagger: tagging trace set: gauss with 1259 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1259 traces tagged "gauss"
FrameSaver: q=154186 n=17752 tag=gauss
wclsFrameSaver: saving 1779 traces tagged "wiener"
FrameSaver: q=173929 n=15457 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1647 traces, 1647 summary
Retagger: tagging trace set: gauss with 1215 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1215 traces tagged "gauss"
FrameSaver: q=139686 n=16599 tag=gauss
wclsFrameSaver: saving 1647 traces tagged "wiener"
FrameSaver: q=158050 n=14517 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1594 traces, 1594 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=138196 n=15871 tag=gauss
wclsFrameSaver: saving 1594 traces tagged "wiener"
FrameSaver: q=156492 n=13690 tag=wiener
27-Mar-2025 08:36:08 AST  Closed output file "prod_muon_400MeV_protodunevd_1_166657_37_1743077483_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 08:36:08 AST  Closed input file "prod_muon_400MeV_protodunevd_1_166657_37_1743077483_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       72.5532       75.0276       82.7901       72.8672       3.59627        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000311881   0.000563127   0.000983774   0.000528173   0.000202962      10     
reco:rns:RandomNumberSaver                     3.3133e-05    5.96242e-05   0.000251018   3.83085e-05   6.38558e-05      10     
reco:wclsdatavd:WireCellToolkit                  72.2123       74.5003       80.808        72.4994       3.25408        10     
reco:gaushit:GausHitFinder                      0.0182722     0.0196006     0.0213511     0.0195551    0.000856266      10     
reco:reco3d:SpacePointSolver                    0.0543916     0.064713      0.0889821     0.0608795    0.00966563       10     
reco:hitpdune:DisambigFromSpacePoints          0.00200776    0.00237524    0.00375365    0.00219065    0.000494413      10     
reco:pandora:StandardPandora                    0.162185      0.397303       1.84616      0.246289      0.483768        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00150058    0.00279859    0.00477122    0.00248488    0.000870002      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000208854   0.000407558   0.00151937    0.000228629   0.000399102      10     
reco:pandoraStdcalo:Calorimetry                0.00142339    0.00181675    0.00358217    0.00158116    0.000609798      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00088189    0.00113412    0.00241952    0.000964923   0.000438079      10     
[art]:TriggerResults:TriggerResultInserter     1.7452e-05    2.3312e-05    4.5323e-05    1.8714e-05    9.09022e-06      10     
end_path:out1:RootOutput                        4.141e-06    5.4367e-06    1.4542e-05     4.344e-06    3.04488e-06      10     
end_path:out1:RootOutput(write)                 0.0301872     0.0314739     0.0359037     0.0310979    0.00158089       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3372.74 MB
  Peak resident set size usage (VmHWM): 1396.57 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_37_1743077483_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078174.0830247, 'end_time': 1743078969.8961189}
justIN time: 2025-04-03 08:24:43 UTC       justIN version: 01.03.00