justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166658.16@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166658.16@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:17:00
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:40:25
From worker nodeHostnametd811.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:24:19
Input filesmonte-carlo-005747-000007
JobscriptExit code0
Real time15m (951s)
CPU time15m (925s = 97%)
Max RSS bytes1747034112 (1666 MiB)
Outputting started2025-03-27 12:40:10
Output files
Finished2025-03-27 12:40:25
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ow with regularization...
Begin: 3.21344e+07
0 3.21265e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:34:31 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_166658_16_1743078266_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:34:32 CET  Opened input file "prod_muon_5GeV_protodunevd_1_166658_16_1743078266_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3990 traces, 3990 summary
Retagger: tagging trace set: gauss with 2415 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2415 traces tagged "gauss"
FrameSaver: q=446144 n=38036 tag=gauss
wclsFrameSaver: saving 3990 traces tagged "wiener"
FrameSaver: q=488722 n=31232 tag=wiener
27-Mar-2025 13:35:02 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_166658_16_1743078266_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3911 traces, 3911 summary
Retagger: tagging trace set: gauss with 2468 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2468 traces tagged "gauss"
FrameSaver: q=420770 n=37991 tag=gauss
wclsFrameSaver: saving 3911 traces tagged "wiener"
FrameSaver: q=460389 n=31310 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3931 traces, 3931 summary
Retagger: tagging trace set: gauss with 2441 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2441 traces tagged "gauss"
FrameSaver: q=430540 n=38168 tag=gauss
wclsFrameSaver: saving 3931 traces tagged "wiener"
FrameSaver: q=472597 n=31378 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4153 traces, 4153 summary
Retagger: tagging trace set: gauss with 2629 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2629 traces tagged "gauss"
FrameSaver: q=482342 n=41781 tag=gauss
wclsFrameSaver: saving 4153 traces tagged "wiener"
FrameSaver: q=523673 n=34597 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4007 traces, 4007 summary
Retagger: tagging trace set: gauss with 2451 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2451 traces tagged "gauss"
FrameSaver: q=439226 n=38550 tag=gauss
wclsFrameSaver: saving 4007 traces tagged "wiener"
FrameSaver: q=480409 n=31588 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4055 traces, 4055 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=471386 n=40556 tag=gauss
wclsFrameSaver: saving 4055 traces tagged "wiener"
FrameSaver: q=512722 n=33678 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4195 traces, 4195 summary
Retagger: tagging trace set: gauss with 2597 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2597 traces tagged "gauss"
FrameSaver: q=468411 n=40904 tag=gauss
wclsFrameSaver: saving 4195 traces tagged "wiener"
FrameSaver: q=511268 n=33628 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4254 traces, 4254 summary
Retagger: tagging trace set: gauss with 2684 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2684 traces tagged "gauss"
FrameSaver: q=495832 n=42837 tag=gauss
wclsFrameSaver: saving 4254 traces tagged "wiener"
FrameSaver: q=538030 n=35476 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4474 traces, 4474 summary
Retagger: tagging trace set: gauss with 2774 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2774 traces tagged "gauss"
FrameSaver: q=535834 n=45239 tag=gauss
wclsFrameSaver: saving 4474 traces tagged "wiener"
FrameSaver: q=580150 n=37273 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3895 traces, 3895 summary
Retagger: tagging trace set: gauss with 2450 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2450 traces tagged "gauss"
FrameSaver: q=444241 n=37508 tag=gauss
wclsFrameSaver: saving 3895 traces tagged "wiener"
FrameSaver: q=485206 n=30795 tag=wiener
27-Mar-2025 13:40:10 CET  Closed output file "prod_muon_5GeV_protodunevd_1_166658_16_1743078266_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:40:10 CET  Closed input file "prod_muon_5GeV_protodunevd_1_166658_16_1743078266_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       30.1848       33.4881       35.0428       33.8525       1.43321        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000294433   0.000431308   0.000565795   0.000446129   8.33641e-05      10     
reco:rns:RandomNumberSaver                     3.2311e-05    5.4968e-05    0.000215552   3.8566e-05    5.3584e-05       10     
reco:wclsdatavd:WireCellToolkit                  28.4158       32.8369       34.5011       33.3692       1.73343        10     
reco:gaushit:GausHitFinder                      0.0289508     0.0321007     0.0361832     0.0321679    0.00226318       10     
reco:reco3d:SpacePointSolver                    0.209564      0.229496      0.252279      0.231426      0.0130716       10     
reco:hitpdune:DisambigFromSpacePoints          0.00319047    0.00369175    0.00474958    0.00360189    0.00042074       10     
reco:pandora:StandardPandora                    0.158814      0.314914       1.43746      0.194679      0.374591        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00333786    0.00474152    0.00636186    0.00478371    0.000802296      10     
reco:pandoraShower:LArPandoraShowerCreation    0.00128827    0.00208006     0.0032485    0.00213966    0.000623505      10     
reco:pandoraStdcalo:Calorimetry                0.00493932    0.00560502    0.00661996    0.00562444    0.000458527      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00240931    0.00259041    0.00352037    0.00249588    0.000314147      10     
[art]:TriggerResults:TriggerResultInserter      1.008e-05    1.4305e-05     3.452e-05     1.257e-05    6.8431e-06       10     
end_path:out1:RootOutput                        3.06e-06      6.128e-06     1.609e-05     5.32e-06     3.73708e-06      10     
end_path:out1:RootOutput(write)                 0.0496046     0.0526677     0.055477      0.052784     0.00219423       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3510.66 MB
  Peak resident set size usage (VmHWM): 1690.03 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_166658_16_1743078266_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078846.2268565, 'end_time': 1743079210.538773}
justIN time: 2025-04-03 08:21:34 UTC       justIN version: 01.03.00