justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.79@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.79@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-03-27 12:20:14
From worker nodeHostnameheplnc163.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:00:14
Input filesmonte-carlo-005747-000227
JobscriptExit code0
Real time19m (1168s)
CPU time18m (1122s = 96%)
Max RSS bytes1721208832 (1641 MiB)
Outputting started2025-03-27 12:19:43
Output files
Finished2025-03-27 12:20:04
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Now with regularization...
Begin: 2.33565e+07
0 2.33549e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:12:50 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_79_1743076825_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:12:50 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_79_1743076825_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4053 traces, 4053 summary
Retagger: tagging trace set: gauss with 2530 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2530 traces tagged "gauss"
FrameSaver: q=473497 n=40571 tag=gauss
wclsFrameSaver: saving 4053 traces tagged "wiener"
FrameSaver: q=515218 n=33614 tag=wiener
27-Mar-2025 12:13:32 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_79_1743076825_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3910 traces, 3910 summary
Retagger: tagging trace set: gauss with 2433 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2433 traces tagged "gauss"
FrameSaver: q=410074 n=37469 tag=gauss
wclsFrameSaver: saving 3910 traces tagged "wiener"
FrameSaver: q=450749 n=30681 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4173 traces, 4173 summary
Retagger: tagging trace set: gauss with 2640 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2640 traces tagged "gauss"
FrameSaver: q=486622 n=41410 tag=gauss
wclsFrameSaver: saving 4173 traces tagged "wiener"
FrameSaver: q=529165 n=34296 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4599 traces, 4599 summary
Retagger: tagging trace set: gauss with 2904 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2904 traces tagged "gauss"
FrameSaver: q=540310 n=46875 tag=gauss
wclsFrameSaver: saving 4599 traces tagged "wiener"
FrameSaver: q=584479 n=39076 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4142 traces, 4142 summary
Retagger: tagging trace set: gauss with 2605 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2605 traces tagged "gauss"
FrameSaver: q=452727 n=40683 tag=gauss
wclsFrameSaver: saving 4142 traces tagged "wiener"
FrameSaver: q=494037 n=33761 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4158 traces, 4158 summary
Retagger: tagging trace set: gauss with 2642 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2642 traces tagged "gauss"
FrameSaver: q=461488 n=41212 tag=gauss
wclsFrameSaver: saving 4158 traces tagged "wiener"
FrameSaver: q=504299 n=34199 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4568 traces, 4568 summary
Retagger: tagging trace set: gauss with 2830 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2830 traces tagged "gauss"
FrameSaver: q=542450 n=45779 tag=gauss
wclsFrameSaver: saving 4568 traces tagged "wiener"
FrameSaver: q=589587 n=37839 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4074 traces, 4074 summary
Retagger: tagging trace set: gauss with 2552 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2552 traces tagged "gauss"
FrameSaver: q=441491 n=39362 tag=gauss
wclsFrameSaver: saving 4074 traces tagged "wiener"
FrameSaver: q=482695 n=32532 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3987 traces, 3987 summary
Retagger: tagging trace set: gauss with 2486 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2486 traces tagged "gauss"
FrameSaver: q=440486 n=38721 tag=gauss
wclsFrameSaver: saving 3987 traces tagged "wiener"
FrameSaver: q=481541 n=31982 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4028 traces, 4028 summary
Retagger: tagging trace set: gauss with 2506 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2506 traces tagged "gauss"
FrameSaver: q=451317 n=39209 tag=gauss
wclsFrameSaver: saving 4028 traces tagged "wiener"
FrameSaver: q=491920 n=32421 tag=wiener
27-Mar-2025 12:19:42 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_79_1743076825_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:19:42 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_79_1743076825_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       37.0991       40.3798       43.875        40.2397       1.70068        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000419148   0.000622421   0.000931722   0.000593446   0.000153414      10     
reco:rns:RandomNumberSaver                     4.8431e-05    9.3698e-05    0.000418497   5.6541e-05    0.00010874       10     
reco:wclsdatavd:WireCellToolkit                  36.4731       39.5004       43.0694       39.5033       1.71345        10     
reco:gaushit:GausHitFinder                      0.0313871     0.0484293     0.0600981     0.0499843    0.00908345       10     
reco:reco3d:SpacePointSolver                    0.230976      0.297453      0.362788      0.306414      0.0488939       10     
reco:hitpdune:DisambigFromSpacePoints          0.00342781    0.00503707    0.00816502     0.0049511     0.0012895       10     
reco:pandora:StandardPandora                    0.178856      0.422521       2.1784       0.232656      0.585812        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00544701    0.00797868     0.0129515    0.00694335    0.00251112       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00119662     0.0035651    0.00653909    0.00357107    0.00154124       10     
reco:pandoraStdcalo:Calorimetry                0.00578784    0.00874216     0.0125435    0.00868761    0.00167817       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00219389    0.00403535    0.00686939    0.00387929    0.00110093       10     
[art]:TriggerResults:TriggerResultInserter     1.2885e-05    2.48007e-05   7.6153e-05    2.01275e-05   1.72905e-05      10     
end_path:out1:RootOutput                        4.619e-06     8.475e-06    2.9726e-05    5.5905e-06    7.24375e-06      10     
end_path:out1:RootOutput(write)                 0.0455978     0.0753198     0.101349      0.0733384     0.0148088       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3406.94 MB
  Peak resident set size usage (VmHWM): 1588.19 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_79_1743076825_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077532.78537, 'end_time': 1743077983.0599968}
justIN time: 2025-04-03 08:26:19 UTC       justIN version: 01.03.00