justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.77@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.77@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-03-27 12:26:11
From worker nodeHostnametd825.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:59:44
Input filesmonte-carlo-005747-000216
JobscriptExit code0
Real time26m (1562s)
CPU time25m (1500s = 96%)
Max RSS bytes1737273344 (1656 MiB)
Outputting started2025-03-27 12:25:47
Output files
Finished2025-03-27 12:26:11
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Now with regularization...
Begin: 2.68836e+07
0 2.688e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:16:11 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_77_1743076797_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:16:12 CET  Opened input file "prod_muon_5GeV_protodunevd_1_358069_77_1743076797_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4312 traces, 4312 summary
Retagger: tagging trace set: gauss with 2696 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2696 traces tagged "gauss"
FrameSaver: q=500097 n=43391 tag=gauss
wclsFrameSaver: saving 4312 traces tagged "wiener"
FrameSaver: q=541035 n=35814 tag=wiener
27-Mar-2025 13:17:12 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_77_1743076797_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4204 traces, 4204 summary
Retagger: tagging trace set: gauss with 2644 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2644 traces tagged "gauss"
FrameSaver: q=480820 n=40919 tag=gauss
wclsFrameSaver: saving 4204 traces tagged "wiener"
FrameSaver: q=523489 n=33778 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4008 traces, 4008 summary
Retagger: tagging trace set: gauss with 2480 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2480 traces tagged "gauss"
FrameSaver: q=444910 n=38844 tag=gauss
wclsFrameSaver: saving 4008 traces tagged "wiener"
FrameSaver: q=487619 n=31965 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4080 traces, 4080 summary
Retagger: tagging trace set: gauss with 2533 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2533 traces tagged "gauss"
FrameSaver: q=450875 n=39400 tag=gauss
wclsFrameSaver: saving 4080 traces tagged "wiener"
FrameSaver: q=492157 n=32454 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3964 traces, 3964 summary
Retagger: tagging trace set: gauss with 2425 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2425 traces tagged "gauss"
FrameSaver: q=459024 n=38599 tag=gauss
wclsFrameSaver: saving 3964 traces tagged "wiener"
FrameSaver: q=499087 n=31794 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3929 traces, 3929 summary
Retagger: tagging trace set: gauss with 2434 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2434 traces tagged "gauss"
FrameSaver: q=424817 n=37298 tag=gauss
wclsFrameSaver: saving 3929 traces tagged "wiener"
FrameSaver: q=465783 n=30554 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4048 traces, 4048 summary
Retagger: tagging trace set: gauss with 2501 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2501 traces tagged "gauss"
FrameSaver: q=438438 n=38906 tag=gauss
wclsFrameSaver: saving 4048 traces tagged "wiener"
FrameSaver: q=479981 n=32002 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3971 traces, 3971 summary
Retagger: tagging trace set: gauss with 2544 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2544 traces tagged "gauss"
FrameSaver: q=429248 n=38741 tag=gauss
wclsFrameSaver: saving 3971 traces tagged "wiener"
FrameSaver: q=469895 n=32234 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4635 traces, 4635 summary
Retagger: tagging trace set: gauss with 2872 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2872 traces tagged "gauss"
FrameSaver: q=671731 n=48557 tag=gauss
wclsFrameSaver: saving 4635 traces tagged "wiener"
FrameSaver: q=717752 n=40343 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4070 traces, 4070 summary
Retagger: tagging trace set: gauss with 2506 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2506 traces tagged "gauss"
FrameSaver: q=463131 n=39664 tag=gauss
wclsFrameSaver: saving 4070 traces tagged "wiener"
FrameSaver: q=505593 n=32669 tag=wiener
27-Mar-2025 13:25:44 CET  Closed output file "prod_muon_5GeV_protodunevd_1_358069_77_1743076797_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:25:44 CET  Closed input file "prod_muon_5GeV_protodunevd_1_358069_77_1743076797_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                        49.03        56.4638       62.2674       55.8778       4.05864        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000562736   0.00106704    0.00226313    0.000916434   0.000460753      10     
reco:rns:RandomNumberSaver                     4.8901e-05    0.000142896   0.000828229   7.10455e-05   0.000228605      10     
reco:wclsdatavd:WireCellToolkit                  48.1634       55.2837       61.3177       55.0029       3.92753        10     
reco:gaushit:GausHitFinder                      0.0391549     0.0593875     0.0807672     0.0577855     0.0106123       10     
reco:reco3d:SpacePointSolver                    0.385081      0.421464      0.461198      0.423898      0.0264254       10     
reco:hitpdune:DisambigFromSpacePoints          0.00594421    0.00743429     0.014596     0.00648148     0.002484        10     
reco:pandora:StandardPandora                     0.28179      0.559957       2.71306      0.312063      0.718757        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00604543    0.00984694     0.0239179    0.00815244    0.00496886       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00216304    0.00414749     0.0110856    0.00333822    0.00247087       10     
reco:pandoraStdcalo:Calorimetry                0.00943251     0.0114569     0.0216548     0.0101282    0.00350446       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00452375    0.00541292     0.0102442    0.00491411    0.00162382       10     
[art]:TriggerResults:TriggerResultInserter      2.134e-05    4.31757e-05   0.000142732   3.06955e-05   3.49661e-05      10     
end_path:out1:RootOutput                        4.74e-06     1.25003e-05    4.12e-05     9.9405e-06    9.72361e-06      10     
end_path:out1:RootOutput(write)                 0.0609156     0.0921636     0.129494      0.0926982     0.0171679       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3440.45 MB
  Peak resident set size usage (VmHWM): 1627.32 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_77_1743076797_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077725.5504007, 'end_time': 1743078347.121154}
justIN time: 2025-04-03 08:24:25 UTC       justIN version: 01.03.00