justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358067.6@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358067.6@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:50
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 11:42:45
From worker nodeHostnametds420.pic.es
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:19:53
Input filesmonte-carlo-005747-000006
JobscriptExit code0
Real time22m (1352s)
CPU time19m (1162s = 85%)
Max RSS bytes1737416704 (1656 MiB)
Outputting started2025-03-27 11:42:26
Output files
Finished2025-03-27 11:42:45
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+07
Now with regularization...
Begin: 2.29531e+07
0 2.29494e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:35:38 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358067_6_1743074460_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:35:39 CET  Opened input file "prod_muon_5GeV_protodunevd_1_358067_6_1743074460_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3884 traces, 3884 summary
Retagger: tagging trace set: gauss with 2460 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2460 traces tagged "gauss"
FrameSaver: q=403789 n=37368 tag=gauss
wclsFrameSaver: saving 3884 traces tagged "wiener"
FrameSaver: q=442806 n=30939 tag=wiener
27-Mar-2025 12:36:28 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358067_6_1743074460_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3763 traces, 3763 summary
Retagger: tagging trace set: gauss with 2343 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2343 traces tagged "gauss"
FrameSaver: q=386514 n=35644 tag=gauss
wclsFrameSaver: saving 3763 traces tagged "wiener"
FrameSaver: q=424569 n=29330 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4443 traces, 4443 summary
Retagger: tagging trace set: gauss with 2841 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2841 traces tagged "gauss"
FrameSaver: q=496956 n=44209 tag=gauss
wclsFrameSaver: saving 4443 traces tagged "wiener"
FrameSaver: q=540705 n=36892 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3952 traces, 3952 summary
Retagger: tagging trace set: gauss with 2479 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2479 traces tagged "gauss"
FrameSaver: q=427478 n=37920 tag=gauss
wclsFrameSaver: saving 3952 traces tagged "wiener"
FrameSaver: q=468736 n=31160 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3967 traces, 3967 summary
Retagger: tagging trace set: gauss with 2480 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2480 traces tagged "gauss"
FrameSaver: q=446319 n=38378 tag=gauss
wclsFrameSaver: saving 3967 traces tagged "wiener"
FrameSaver: q=487521 n=31605 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4072 traces, 4072 summary
Retagger: tagging trace set: gauss with 2516 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2516 traces tagged "gauss"
FrameSaver: q=449520 n=39332 tag=gauss
wclsFrameSaver: saving 4072 traces tagged "wiener"
FrameSaver: q=490129 n=32412 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4009 traces, 4009 summary
Retagger: tagging trace set: gauss with 2524 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2524 traces tagged "gauss"
FrameSaver: q=421368 n=37709 tag=gauss
wclsFrameSaver: saving 4009 traces tagged "wiener"
FrameSaver: q=462777 n=31022 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4158 traces, 4158 summary
Retagger: tagging trace set: gauss with 2583 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2583 traces tagged "gauss"
FrameSaver: q=496687 n=41512 tag=gauss
wclsFrameSaver: saving 4158 traces tagged "wiener"
FrameSaver: q=538195 n=34393 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3994 traces, 3994 summary
Retagger: tagging trace set: gauss with 2490 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2490 traces tagged "gauss"
FrameSaver: q=436453 n=38495 tag=gauss
wclsFrameSaver: saving 3994 traces tagged "wiener"
FrameSaver: q=478043 n=31827 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4030 traces, 4030 summary
Retagger: tagging trace set: gauss with 2504 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2504 traces tagged "gauss"
FrameSaver: q=450601 n=38883 tag=gauss
wclsFrameSaver: saving 4030 traces tagged "wiener"
FrameSaver: q=492855 n=31927 tag=wiener
27-Mar-2025 12:42:25 CET  Closed output file "prod_muon_5GeV_protodunevd_1_358067_6_1743074460_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:42:25 CET  Closed input file "prod_muon_5GeV_protodunevd_1_358067_6_1743074460_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       35.0455       39.8346       47.3765       39.0604       3.86687        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000443289   0.000665684   0.00140711    0.000600988   0.000259126      10     
reco:rns:RandomNumberSaver                     4.1091e-05    7.67625e-05   0.000343938   4.4531e-05    8.93174e-05      10     
reco:wclsdatavd:WireCellToolkit                  34.5073       38.4725       42.6936       38.1854       2.89611        10     
reco:gaushit:GausHitFinder                      0.0303673     0.0388856     0.0518437     0.0360152    0.00783571       10     
reco:reco3d:SpacePointSolver                    0.251853      0.291022      0.366137      0.267671      0.0419206       10     
reco:hitpdune:DisambigFromSpacePoints          0.00345568     0.0045744    0.00787766    0.00370993    0.00143784       10     
reco:pandora:StandardPandora                    0.175491      0.939386       7.42817      0.208241       2.1633         10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0040223    0.00568402    0.00967955    0.00476624    0.00193161       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000808687    0.0024609    0.00569772    0.00171251    0.00150741       10     
reco:pandoraStdcalo:Calorimetry                0.00530786    0.00712307     0.0104162    0.00634955    0.00188654       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00262923    0.00361734     0.006184     0.00323878    0.00109101       10     
[art]:TriggerResults:TriggerResultInserter      1.221e-05    2.17332e-05   6.2781e-05    1.74505e-05   1.40872e-05      10     
end_path:out1:RootOutput                        3.96e-06     6.8711e-06    2.3461e-05     4.605e-06    5.69431e-06      10     
end_path:out1:RootOutput(write)                 0.0545893     0.0638715     0.0822257     0.0583961    0.00990215       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3422.04 MB
  Peak resident set size usage (VmHWM): 1611.48 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358067_6_1743074460_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743075279.3387487, 'end_time': 1743075745.9645574}
justIN time: 2025-04-03 08:13:16 UTC       justIN version: 01.03.00