justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358067.84@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358067.84@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:50
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-03-27 11:59:37
From worker nodeHostnametdm002.pic.es
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:30:20
Input filesmonte-carlo-005747-000077
JobscriptExit code0
Real time28m (1736s)
CPU time27m (1625s = 93%)
Max RSS bytes1743015936 (1662 MiB)
Outputting started2025-03-27 11:59:16
Output files
Finished2025-03-27 11:59:37
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Now with regularization...
Begin: 2.22394e+07
0 2.223e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:47:40 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358067_84_1743075033_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:47:41 CET  Opened input file "prod_muon_5GeV_protodunevd_1_358067_84_1743075033_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3978 traces, 3978 summary
Retagger: tagging trace set: gauss with 2492 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2492 traces tagged "gauss"
FrameSaver: q=437227 n=37896 tag=gauss
wclsFrameSaver: saving 3978 traces tagged "wiener"
FrameSaver: q=478457 n=31268 tag=wiener
27-Mar-2025 12:48:58 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358067_84_1743075033_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4101 traces, 4101 summary
Retagger: tagging trace set: gauss with 2555 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2555 traces tagged "gauss"
FrameSaver: q=443169 n=40058 tag=gauss
wclsFrameSaver: saving 4101 traces tagged "wiener"
FrameSaver: q=485341 n=32908 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4007 traces, 4007 summary
Retagger: tagging trace set: gauss with 2460 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2460 traces tagged "gauss"
FrameSaver: q=442882 n=38783 tag=gauss
wclsFrameSaver: saving 4007 traces tagged "wiener"
FrameSaver: q=484915 n=31906 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4083 traces, 4083 summary
Retagger: tagging trace set: gauss with 2537 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2537 traces tagged "gauss"
FrameSaver: q=443833 n=39717 tag=gauss
wclsFrameSaver: saving 4083 traces tagged "wiener"
FrameSaver: q=484849 n=32869 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4032 traces, 4032 summary
Retagger: tagging trace set: gauss with 2529 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2529 traces tagged "gauss"
FrameSaver: q=444734 n=39383 tag=gauss
wclsFrameSaver: saving 4032 traces tagged "wiener"
FrameSaver: q=486012 n=32502 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3881 traces, 3881 summary
Retagger: tagging trace set: gauss with 2432 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2432 traces tagged "gauss"
FrameSaver: q=411948 n=36756 tag=gauss
wclsFrameSaver: saving 3881 traces tagged "wiener"
FrameSaver: q=452576 n=30337 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4642 traces, 4642 summary
Retagger: tagging trace set: gauss with 2884 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2884 traces tagged "gauss"
FrameSaver: q=596767 n=48213 tag=gauss
wclsFrameSaver: saving 4642 traces tagged "wiener"
FrameSaver: q=644813 n=40104 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4347 traces, 4347 summary
Retagger: tagging trace set: gauss with 2716 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2716 traces tagged "gauss"
FrameSaver: q=535471 n=44495 tag=gauss
wclsFrameSaver: saving 4347 traces tagged "wiener"
FrameSaver: q=580063 n=36807 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4503 traces, 4503 summary
Retagger: tagging trace set: gauss with 2856 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2856 traces tagged "gauss"
FrameSaver: q=573740 n=45900 tag=gauss
wclsFrameSaver: saving 4503 traces tagged "wiener"
FrameSaver: q=618602 n=38187 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3901 traces, 3901 summary
Retagger: tagging trace set: gauss with 2386 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2386 traces tagged "gauss"
FrameSaver: q=423673 n=37318 tag=gauss
wclsFrameSaver: saving 3901 traces tagged "wiener"
FrameSaver: q=465528 n=30530 tag=wiener
27-Mar-2025 12:59:15 CET  Closed output file "prod_muon_5GeV_protodunevd_1_358067_84_1743075033_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:59:15 CET  Closed input file "prod_muon_5GeV_protodunevd_1_358067_84_1743075033_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       58.8219       66.6991       70.4312       67.1405       3.50951        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00076983    0.000935345    0.0012683    0.000868815   0.00015742       10     
reco:rns:RandomNumberSaver                     4.4717e-05    8.49213e-05   0.000359769   5.39135e-05   9.18738e-05      10     
reco:wclsdatavd:WireCellToolkit                  53.8206       65.2405       69.3419       66.037        4.44923        10     
reco:gaushit:GausHitFinder                      0.0525614     0.0666941     0.0900677     0.0618372     0.0121948       10     
reco:reco3d:SpacePointSolver                    0.414004      0.459086      0.512212      0.457341      0.0305852       10     
reco:hitpdune:DisambigFromSpacePoints          0.00694554    0.00897372     0.0106824    0.00857353    0.00130015       10     
reco:pandora:StandardPandora                    0.331932      0.796238       4.35785      0.395942       1.18809        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00616493    0.00904173     0.0133662    0.00899809    0.00195703       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00193045    0.00386459    0.00701403     0.0035352    0.00140099       10     
reco:pandoraStdcalo:Calorimetry                0.00986763     0.0126822     0.0199608     0.011763     0.00260312       10     
reco:pandoraGnocalo:GnocchiCalorimetry           0.00315     0.00552276    0.00901635    0.00539464    0.00136912       10     
[art]:TriggerResults:TriggerResultInserter     1.7376e-05    3.29442e-05   9.8335e-05    2.73425e-05   2.21848e-05      10     
end_path:out1:RootOutput                        6.166e-06    9.5913e-06     2.628e-05    7.9535e-06    5.60129e-06      10     
end_path:out1:RootOutput(write)                 0.0627009     0.0903812     0.104576      0.0903539     0.011128        10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3429.33 MB
  Peak resident set size usage (VmHWM): 1652.61 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358067_84_1743075033_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076019.5996132, 'end_time': 1743076756.6074524}
justIN time: 2025-04-03 08:21:25 UTC       justIN version: 01.03.00