justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358067.141@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358067.141@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:50
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:06:38
From worker nodeHostnametdm003.pic.es
cpuinfoAMD EPYC 7662 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:35:12
Input filesmonte-carlo-005747-000009
JobscriptExit code0
Real time30m (1858s)
CPU time26m (1613s = 86%)
Max RSS bytes1700474880 (1621 MiB)
Outputting started2025-03-27 12:06:10
Output files
Finished2025-03-27 12:06:38
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

th regularization...
Begin: 2.25276e+07
0 2.25244e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:54:43 CET  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358067_141_1743075327_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:54:44 CET  Opened input file "prod_muon_5GeV_protodunevd_1_358067_141_1743075327_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3953 traces, 3953 summary
Retagger: tagging trace set: gauss with 2465 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2465 traces tagged "gauss"
FrameSaver: q=450471 n=38478 tag=gauss
wclsFrameSaver: saving 3953 traces tagged "wiener"
FrameSaver: q=491931 n=31699 tag=wiener
27-Mar-2025 12:56:20 CET  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358067_141_1743075327_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3880 traces, 3880 summary
Retagger: tagging trace set: gauss with 2410 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2410 traces tagged "gauss"
FrameSaver: q=449760 n=37888 tag=gauss
wclsFrameSaver: saving 3880 traces tagged "wiener"
FrameSaver: q=490855 n=31109 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4176 traces, 4176 summary
Retagger: tagging trace set: gauss with 2625 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2625 traces tagged "gauss"
FrameSaver: q=464514 n=40842 tag=gauss
wclsFrameSaver: saving 4176 traces tagged "wiener"
FrameSaver: q=507644 n=33834 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4509 traces, 4509 summary
Retagger: tagging trace set: gauss with 2727 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2727 traces tagged "gauss"
FrameSaver: q=536329 n=45059 tag=gauss
wclsFrameSaver: saving 4509 traces tagged "wiener"
FrameSaver: q=582579 n=37279 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4024 traces, 4024 summary
Retagger: tagging trace set: gauss with 2505 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2505 traces tagged "gauss"
FrameSaver: q=480595 n=39811 tag=gauss
wclsFrameSaver: saving 4024 traces tagged "wiener"
FrameSaver: q=522828 n=32968 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3978 traces, 3978 summary
Retagger: tagging trace set: gauss with 2426 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2426 traces tagged "gauss"
FrameSaver: q=431732 n=38261 tag=gauss
wclsFrameSaver: saving 3978 traces tagged "wiener"
FrameSaver: q=474121 n=31561 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3940 traces, 3940 summary
Retagger: tagging trace set: gauss with 2519 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2519 traces tagged "gauss"
FrameSaver: q=417955 n=37977 tag=gauss
wclsFrameSaver: saving 3940 traces tagged "wiener"
FrameSaver: q=458109 n=31315 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4020 traces, 4020 summary
Retagger: tagging trace set: gauss with 2455 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2455 traces tagged "gauss"
FrameSaver: q=427499 n=38105 tag=gauss
wclsFrameSaver: saving 4020 traces tagged "wiener"
FrameSaver: q=469122 n=31339 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4533 traces, 4533 summary
Retagger: tagging trace set: gauss with 2798 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2798 traces tagged "gauss"
FrameSaver: q=526432 n=45537 tag=gauss
wclsFrameSaver: saving 4533 traces tagged "wiener"
FrameSaver: q=571932 n=37610 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3963 traces, 3963 summary
Retagger: tagging trace set: gauss with 2533 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2533 traces tagged "gauss"
FrameSaver: q=427789 n=38185 tag=gauss
wclsFrameSaver: saving 3963 traces tagged "wiener"
FrameSaver: q=468008 n=31532 tag=wiener
27-Mar-2025 13:06:09 CET  Closed output file "prod_muon_5GeV_protodunevd_1_358067_141_1743075327_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:06:09 CET  Closed input file "prod_muon_5GeV_protodunevd_1_358067_141_1743075327_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       62.4851       66.3249       81.3563       65.0058       5.33142        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00100421    0.00115863     0.0015499     0.0011163    0.000155801      10     
reco:rns:RandomNumberSaver                     5.4771e-05    0.000113691   0.00058329    6.2191e-05    0.000156582      10     
reco:wclsdatavd:WireCellToolkit                  52.8138       62.4732       66.6113       62.5026       3.67674        10     
reco:gaushit:GausHitFinder                      0.0644481     0.0725879     0.0859865     0.0701558    0.00716072       10     
reco:reco3d:SpacePointSolver                    0.417106      0.487564       0.55822       0.48872      0.0442271       10     
reco:hitpdune:DisambigFromSpacePoints          0.00514252    0.00784864    0.00958397    0.00774423    0.00118479       10     
reco:pandora:StandardPandora                    0.334001       3.13063       27.8416      0.368502       8.23714        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00788131     0.0107938     0.0158239     0.0103798    0.00251438       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00193353    0.00435382    0.00777373    0.00387808    0.00178631       10     
reco:pandoraStdcalo:Calorimetry                 0.0107752     0.0129966     0.0183163     0.0126769    0.00197301       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00511178    0.00642964      0.01071      0.0060382    0.00146808       10     
[art]:TriggerResults:TriggerResultInserter      2.284e-05    4.52176e-05   0.000124612   2.9655e-05    3.35751e-05      10     
end_path:out1:RootOutput                        5.03e-06     1.00542e-05    4.123e-05    5.5255e-06     1.059e-05       10     
end_path:out1:RootOutput(write)                 0.0991521     0.109878      0.125577       0.10813     0.00865351       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3451.63 MB
  Peak resident set size usage (VmHWM): 1558.62 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358067_141_1743075327_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076425.04723, 'end_time': 1743077170.5152812}
justIN time: 2025-04-03 08:30:21 UTC       justIN version: 01.03.00