justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.157@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.157@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:10:14
From worker nodeHostnametd809.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:52:51
Input filesmonte-carlo-005746-000042
JobscriptExit code0
Real time17m (1022s)
CPU time16m (992s = 97%)
Max RSS bytes1708371968 (1629 MiB)
Outputting started2025-03-27 12:09:55
Output files
Finished2025-03-27 12:10:14
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

arization...
Begin: 1.17281e+07
0 1.14166e+07
1 1.14075e+07
Now with regularization...
Begin: 1.0754e+07
0 1.07536e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:02:08 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_157_1743076381_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:02:09 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358068_157_1743076381_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1658 traces, 1658 summary
Retagger: tagging trace set: gauss with 1186 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1186 traces tagged "gauss"
FrameSaver: q=141725 n=16671 tag=gauss
wclsFrameSaver: saving 1658 traces tagged "wiener"
FrameSaver: q=161161 n=14513 tag=wiener
27-Mar-2025 13:02:58 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_157_1743076381_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1189 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1189 traces tagged "gauss"
FrameSaver: q=136530 n=15799 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=152965 n=13882 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=136222 n=15547 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=155247 n=13552 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1578 traces, 1578 summary
Retagger: tagging trace set: gauss with 1131 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1131 traces tagged "gauss"
FrameSaver: q=140586 n=15962 tag=gauss
wclsFrameSaver: saving 1578 traces tagged "wiener"
FrameSaver: q=158427 n=13786 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1681 traces, 1681 summary
Retagger: tagging trace set: gauss with 1225 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1225 traces tagged "gauss"
FrameSaver: q=144295 n=17098 tag=gauss
wclsFrameSaver: saving 1681 traces tagged "wiener"
FrameSaver: q=162070 n=14876 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1578 traces, 1578 summary
Retagger: tagging trace set: gauss with 1111 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1111 traces tagged "gauss"
FrameSaver: q=137109 n=15710 tag=gauss
wclsFrameSaver: saving 1578 traces tagged "wiener"
FrameSaver: q=155988 n=13587 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1679 traces, 1679 summary
Retagger: tagging trace set: gauss with 1201 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1201 traces tagged "gauss"
FrameSaver: q=140205 n=16689 tag=gauss
wclsFrameSaver: saving 1679 traces tagged "wiener"
FrameSaver: q=159302 n=14468 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1590 traces, 1590 summary
Retagger: tagging trace set: gauss with 1148 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1148 traces tagged "gauss"
FrameSaver: q=147772 n=16217 tag=gauss
wclsFrameSaver: saving 1590 traces tagged "wiener"
FrameSaver: q=165647 n=13981 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1600 traces, 1600 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=136555 n=15841 tag=gauss
wclsFrameSaver: saving 1600 traces tagged "wiener"
FrameSaver: q=155557 n=13670 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1565 traces, 1565 summary
Retagger: tagging trace set: gauss with 1180 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1180 traces tagged "gauss"
FrameSaver: q=138601 n=16085 tag=gauss
wclsFrameSaver: saving 1565 traces tagged "wiener"
FrameSaver: q=155519 n=14033 tag=wiener
27-Mar-2025 13:09:54 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358068_157_1743076381_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:09:54 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358068_157_1743076381_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.303        45.9829       52.1759       46.1787       2.99067        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000337311   0.000549663    0.0012064    0.000466363   0.000242473      10     
reco:rns:RandomNumberSaver                      3.279e-05    8.21636e-05   0.000399803   4.74255e-05   0.000106255      10     
reco:wclsdatavd:WireCellToolkit                  42.0009       45.485        51.8519       45.8701       2.87606        10     
reco:gaushit:GausHitFinder                      0.0154313     0.0170959     0.023176      0.0160574    0.00222573       10     
reco:reco3d:SpacePointSolver                    0.0442442     0.0597276     0.0816357     0.059599     0.00934825       10     
reco:hitpdune:DisambigFromSpacePoints           0.001533     0.00202844    0.00363023    0.00186464    0.00057792       10     
reco:pandora:StandardPandora                     0.18553      0.374433       1.83006      0.206934        0.486         10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00113792    0.00220088    0.00438474    0.00193038    0.000831629      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000155981   0.000351215    0.0014247    0.000203171   0.000377029      10     
reco:pandoraStdcalo:Calorimetry                0.00118049    0.00158238    0.00339803    0.00132609    0.000624525      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000791265   0.00110317    0.00232906    0.000910896   0.000437608      10     
[art]:TriggerResults:TriggerResultInserter      1.165e-05    1.93482e-05   7.0411e-05    1.2855e-05    1.71147e-05      10     
end_path:out1:RootOutput                        3.23e-06     9.1961e-06     3.159e-05     5.135e-06    9.16245e-06      10     
end_path:out1:RootOutput(write)                 0.0287474     0.0353031     0.0456141     0.0334793    0.00474542       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3386.97 MB
  Peak resident set size usage (VmHWM): 1568.99 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_157_1743076381_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076890.9214177, 'end_time': 1743077394.8420932}
justIN time: 2025-04-03 08:28:26 UTC       justIN version: 01.03.00