justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.107@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.107@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:22:51
From worker nodeHostnametds420.pic.es
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:02:06
Input filesmonte-carlo-005747-000270
JobscriptExit code0
Real time20m (1225s)
CPU time19m (1186s = 96%)
Max RSS bytes1752428544 (1671 MiB)
Outputting started2025-03-27 12:22:32
Output files
Finished2025-03-27 12:22:51
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

regularization...
Begin: 2.00847e+07
0 2.00827e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:14:59 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_107_1743076937_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:15:00 CET  Opened input file "prod_muon_5GeV_protodunevd_1_358069_107_1743076937_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3885 traces, 3885 summary
Retagger: tagging trace set: gauss with 2370 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2370 traces tagged "gauss"
FrameSaver: q=423972 n=37381 tag=gauss
wclsFrameSaver: saving 3885 traces tagged "wiener"
FrameSaver: q=463469 n=30634 tag=wiener
27-Mar-2025 13:15:40 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_107_1743076937_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4258 traces, 4258 summary
Retagger: tagging trace set: gauss with 2651 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2651 traces tagged "gauss"
FrameSaver: q=451724 n=41062 tag=gauss
wclsFrameSaver: saving 4258 traces tagged "wiener"
FrameSaver: q=495031 n=33786 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4197 traces, 4197 summary
Retagger: tagging trace set: gauss with 2640 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2640 traces tagged "gauss"
FrameSaver: q=462163 n=40687 tag=gauss
wclsFrameSaver: saving 4197 traces tagged "wiener"
FrameSaver: q=505414 n=33731 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4074 traces, 4074 summary
Retagger: tagging trace set: gauss with 2563 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2563 traces tagged "gauss"
FrameSaver: q=446651 n=39627 tag=gauss
wclsFrameSaver: saving 4074 traces tagged "wiener"
FrameSaver: q=488167 n=32595 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4159 traces, 4159 summary
Retagger: tagging trace set: gauss with 2561 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2561 traces tagged "gauss"
FrameSaver: q=461228 n=40471 tag=gauss
wclsFrameSaver: saving 4159 traces tagged "wiener"
FrameSaver: q=504292 n=33237 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3889 traces, 3889 summary
Retagger: tagging trace set: gauss with 2401 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2401 traces tagged "gauss"
FrameSaver: q=438900 n=37527 tag=gauss
wclsFrameSaver: saving 3889 traces tagged "wiener"
FrameSaver: q=479940 n=30744 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4137 traces, 4137 summary
Retagger: tagging trace set: gauss with 2600 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2600 traces tagged "gauss"
FrameSaver: q=469585 n=40771 tag=gauss
wclsFrameSaver: saving 4137 traces tagged "wiener"
FrameSaver: q=513019 n=33547 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4000 traces, 4000 summary
Retagger: tagging trace set: gauss with 2506 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2506 traces tagged "gauss"
FrameSaver: q=444954 n=38974 tag=gauss
wclsFrameSaver: saving 4000 traces tagged "wiener"
FrameSaver: q=486716 n=32082 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3989 traces, 3989 summary
Retagger: tagging trace set: gauss with 2505 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2505 traces tagged "gauss"
FrameSaver: q=432630 n=38719 tag=gauss
wclsFrameSaver: saving 3989 traces tagged "wiener"
FrameSaver: q=473569 n=31757 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4048 traces, 4048 summary
Retagger: tagging trace set: gauss with 2551 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2551 traces tagged "gauss"
FrameSaver: q=436648 n=39136 tag=gauss
wclsFrameSaver: saving 4048 traces tagged "wiener"
FrameSaver: q=478167 n=32133 tag=wiener
27-Mar-2025 13:22:31 CET  Closed output file "prod_muon_5GeV_protodunevd_1_358069_107_1743076937_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:22:31 CET  Closed input file "prod_muon_5GeV_protodunevd_1_358069_107_1743076937_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.2083       44.5626       47.9557       44.9598       2.55365        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000532882   0.000871702   0.00119519    0.000855968   0.000215892      10     
reco:rns:RandomNumberSaver                     4.4291e-05    8.69568e-05   0.000427449   4.9721e-05    0.000113543      10     
reco:wclsdatavd:WireCellToolkit                  38.1522       43.6728       47.1866       44.1933       2.7615         10     
reco:gaushit:GausHitFinder                      0.0351116     0.0462638     0.0514035     0.0468401    0.00444971       10     
reco:reco3d:SpacePointSolver                    0.269623      0.339278       0.38193      0.349634      0.0322479       10     
reco:hitpdune:DisambigFromSpacePoints          0.00404871    0.00549852     0.005963     0.00564234    0.000572762      10     
reco:pandora:StandardPandora                    0.222193      0.395931       1.58218      0.272217      0.396291        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00416763    0.00607751    0.00865855    0.00611449    0.00116295       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00130839    0.00274912    0.00386331    0.00265984    0.000822351      10     
reco:pandoraStdcalo:Calorimetry                0.00609785    0.00865461     0.0103503     0.0088311    0.00114682       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00274523     0.004076      0.0049224    0.00416248    0.000570064      10     
[art]:TriggerResults:TriggerResultInserter     1.5761e-05    2.69007e-05   6.3792e-05    2.30405e-05   1.27877e-05      10     
end_path:out1:RootOutput                        4.19e-06      8.903e-06     2.298e-05     8.035e-06    4.9934e-06       10     
end_path:out1:RootOutput(write)                 0.0583737     0.0750558     0.0842743     0.0753793     0.0068926       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3410.52 MB
  Peak resident set size usage (VmHWM): 1621.19 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_107_1743076937_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077656.1332617, 'end_time': 1743078152.3737175}
justIN time: 2025-04-03 08:29:39 UTC       justIN version: 01.03.00