justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358067.8@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358067.8@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:50
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 11:45:57
From worker nodeHostnametds420.pic.es
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:22:54
Input filesmonte-carlo-005747-000007
JobscriptExit code0
Real time22m (1363s)
CPU time22m (1320s = 96%)
Max RSS bytes1742118912 (1661 MiB)
Outputting started2025-03-27 11:45:37
Output files
Finished2025-03-27 11:45:57
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+07
Now with regularization...
Begin: 3.58397e+07
0 3.58356e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:36:22 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358067_8_1743074585_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:36:23 CET  Opened input file "prod_muon_5GeV_protodunevd_1_358067_8_1743074585_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3930 traces, 3930 summary
Retagger: tagging trace set: gauss with 2413 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2413 traces tagged "gauss"
FrameSaver: q=435017 n=37850 tag=gauss
wclsFrameSaver: saving 3930 traces tagged "wiener"
FrameSaver: q=476324 n=31070 tag=wiener
27-Mar-2025 12:37:22 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358067_8_1743074585_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4344 traces, 4344 summary
Retagger: tagging trace set: gauss with 2735 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2735 traces tagged "gauss"
FrameSaver: q=490046 n=42870 tag=gauss
wclsFrameSaver: saving 4344 traces tagged "wiener"
FrameSaver: q=534299 n=35431 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3973 traces, 3973 summary
Retagger: tagging trace set: gauss with 2469 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2469 traces tagged "gauss"
FrameSaver: q=423499 n=38135 tag=gauss
wclsFrameSaver: saving 3973 traces tagged "wiener"
FrameSaver: q=464246 n=31376 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4025 traces, 4025 summary
Retagger: tagging trace set: gauss with 2536 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2536 traces tagged "gauss"
FrameSaver: q=440278 n=38783 tag=gauss
wclsFrameSaver: saving 4025 traces tagged "wiener"
FrameSaver: q=482048 n=31922 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3950 traces, 3950 summary
Retagger: tagging trace set: gauss with 2449 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2449 traces tagged "gauss"
FrameSaver: q=441488 n=38272 tag=gauss
wclsFrameSaver: saving 3950 traces tagged "wiener"
FrameSaver: q=481654 n=31263 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4377 traces, 4377 summary
Retagger: tagging trace set: gauss with 2722 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2722 traces tagged "gauss"
FrameSaver: q=512089 n=43245 tag=gauss
wclsFrameSaver: saving 4377 traces tagged "wiener"
FrameSaver: q=555746 n=36121 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4106 traces, 4106 summary
Retagger: tagging trace set: gauss with 2536 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2536 traces tagged "gauss"
FrameSaver: q=483345 n=40838 tag=gauss
wclsFrameSaver: saving 4106 traces tagged "wiener"
FrameSaver: q=524816 n=33698 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4082 traces, 4082 summary
Retagger: tagging trace set: gauss with 2516 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2516 traces tagged "gauss"
FrameSaver: q=450865 n=39820 tag=gauss
wclsFrameSaver: saving 4082 traces tagged "wiener"
FrameSaver: q=492465 n=32773 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3746 traces, 3746 summary
Retagger: tagging trace set: gauss with 2308 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2308 traces tagged "gauss"
FrameSaver: q=397746 n=35529 tag=gauss
wclsFrameSaver: saving 3746 traces tagged "wiener"
FrameSaver: q=436221 n=29128 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4029 traces, 4029 summary
Retagger: tagging trace set: gauss with 2548 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2548 traces tagged "gauss"
FrameSaver: q=462539 n=39681 tag=gauss
wclsFrameSaver: saving 4029 traces tagged "wiener"
FrameSaver: q=503384 n=32652 tag=wiener
27-Mar-2025 12:45:36 CET  Closed output file "prod_muon_5GeV_protodunevd_1_358067_8_1743074585_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:45:36 CET  Closed input file "prod_muon_5GeV_protodunevd_1_358067_8_1743074585_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       51.2053        54.67        58.6976       54.809        2.20952        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000797527   0.000973897   0.00155419    0.000856399   0.000225922      10     
reco:rns:RandomNumberSaver                     4.2611e-05    0.000116872   0.000621763   5.7796e-05    0.00016891       10     
reco:wclsdatavd:WireCellToolkit                  50.3924       53.5763       55.6166       53.9295       1.88441        10     
reco:gaushit:GausHitFinder                      0.0450008     0.0536769     0.0636006     0.0522351    0.00598661       10     
reco:reco3d:SpacePointSolver                     0.34182      0.387911      0.440515      0.392344      0.0305842       10     
reco:hitpdune:DisambigFromSpacePoints          0.00504089    0.00645481    0.00897721    0.00658548    0.00110334       10     
reco:pandora:StandardPandora                    0.239933      0.525827       2.56024      0.297407       0.67986        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00451869     0.0088603     0.0144645    0.00882856    0.00292212       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00125344    0.00321968    0.00613818    0.00264439    0.00141838       10     
reco:pandoraStdcalo:Calorimetry                0.00632813    0.00995245     0.0159789    0.00972269    0.00235893       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00292873    0.00474132    0.00840084    0.00455076    0.00135905       10     
[art]:TriggerResults:TriggerResultInserter      1.947e-05    4.08049e-05   0.000105522   2.82505e-05   2.6125e-05       10     
end_path:out1:RootOutput                        4.28e-06     7.5692e-06     2.642e-05     4.965e-06    6.39744e-06      10     
end_path:out1:RootOutput(write)                 0.065569      0.0826148     0.0934074     0.0828939     0.0081375       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3451.81 MB
  Peak resident set size usage (VmHWM): 1628.49 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358067_8_1743074585_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743075339.1607976, 'end_time': 1743075937.2140758}
justIN time: 2025-04-03 08:14:23 UTC       justIN version: 01.03.00