justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358069.87@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358069.87@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5747
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:55
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2025-03-27 12:20:46
From worker nodeHostnamen154.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:00:16
Input filesmonte-carlo-005747-000081
JobscriptExit code0
Real time20m (1215s)
CPU time18m (1086s = 89%)
Max RSS bytes1759309824 (1677 MiB)
Outputting started2025-03-27 12:20:31
Output files
Finished2025-03-27 12:20:46
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

w with regularization...
Begin: 2.31767e+07
0 2.31751e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:12:43 GMT  Initiating request to open input file "prod_muon_5GeV_protodunevd_1_358069_87_1743076863_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:12:43 GMT  Opened input file "prod_muon_5GeV_protodunevd_1_358069_87_1743076863_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4309 traces, 4309 summary
Retagger: tagging trace set: gauss with 2722 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2722 traces tagged "gauss"
FrameSaver: q=566693 n=45096 tag=gauss
wclsFrameSaver: saving 4309 traces tagged "wiener"
FrameSaver: q=609957 n=37445 tag=wiener
27-Mar-2025 12:13:26 GMT  Opened output file with pattern "prod_muon_5GeV_protodunevd_1_358069_87_1743076863_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3831 traces, 3831 summary
Retagger: tagging trace set: gauss with 2352 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2352 traces tagged "gauss"
FrameSaver: q=424982 n=37225 tag=gauss
wclsFrameSaver: saving 3831 traces tagged "wiener"
FrameSaver: q=465706 n=30553 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4192 traces, 4192 summary
Retagger: tagging trace set: gauss with 2703 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2703 traces tagged "gauss"
FrameSaver: q=492055 n=41737 tag=gauss
wclsFrameSaver: saving 4192 traces tagged "wiener"
FrameSaver: q=533862 n=34749 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3965 traces, 3965 summary
Retagger: tagging trace set: gauss with 2478 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2478 traces tagged "gauss"
FrameSaver: q=405879 n=37483 tag=gauss
wclsFrameSaver: saving 3965 traces tagged "wiener"
FrameSaver: q=447008 n=30882 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3921 traces, 3921 summary
Retagger: tagging trace set: gauss with 2434 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2434 traces tagged "gauss"
FrameSaver: q=430610 n=37647 tag=gauss
wclsFrameSaver: saving 3921 traces tagged "wiener"
FrameSaver: q=471654 n=30924 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4123 traces, 4123 summary
Retagger: tagging trace set: gauss with 2584 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2584 traces tagged "gauss"
FrameSaver: q=462372 n=40465 tag=gauss
wclsFrameSaver: saving 4123 traces tagged "wiener"
FrameSaver: q=503553 n=33234 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4019 traces, 4019 summary
Retagger: tagging trace set: gauss with 2502 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2502 traces tagged "gauss"
FrameSaver: q=447706 n=38786 tag=gauss
wclsFrameSaver: saving 4019 traces tagged "wiener"
FrameSaver: q=488718 n=32073 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 3938 traces, 3938 summary
Retagger: tagging trace set: gauss with 2455 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2455 traces tagged "gauss"
FrameSaver: q=431421 n=38020 tag=gauss
wclsFrameSaver: saving 3938 traces tagged "wiener"
FrameSaver: q=471253 n=31429 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4014 traces, 4014 summary
Retagger: tagging trace set: gauss with 2527 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2527 traces tagged "gauss"
FrameSaver: q=449567 n=38869 tag=gauss
wclsFrameSaver: saving 4014 traces tagged "wiener"
FrameSaver: q=491869 n=32118 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 4088 traces, 4088 summary
Retagger: tagging trace set: gauss with 2538 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 2538 traces tagged "gauss"
FrameSaver: q=473738 n=40054 tag=gauss
wclsFrameSaver: saving 4088 traces tagged "wiener"
FrameSaver: q=516282 n=32828 tag=wiener
27-Mar-2025 12:20:31 GMT  Closed output file "prod_muon_5GeV_protodunevd_1_358069_87_1743076863_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:20:31 GMT  Closed input file "prod_muon_5GeV_protodunevd_1_358069_87_1743076863_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       42.7054       46.4353       47.8225       46.7044       1.36859        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000438543   0.00058317    0.000715397   0.000576672   8.03396e-05      10     
reco:rns:RandomNumberSaver                     3.3506e-05    5.60952e-05   0.00022317    3.78485e-05   5.5756e-05       10     
reco:wclsdatavd:WireCellToolkit                  39.2627       45.5041       47.1117       46.0524       2.15541        10     
reco:gaushit:GausHitFinder                      0.0324953     0.0371437     0.0504201     0.0359878    0.00484468       10     
reco:reco3d:SpacePointSolver                    0.265608      0.292645      0.319544      0.288438      0.0158939       10     
reco:hitpdune:DisambigFromSpacePoints          0.00430137    0.00487396    0.00724453    0.00450528    0.000862366      10     
reco:pandora:StandardPandora                    0.228435      0.513318       2.90813      0.245763      0.798433        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00371987     0.0061772     0.0183887    0.00462267    0.00416231       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000969269   0.00312483     0.0102158    0.00271899    0.00246139       10     
reco:pandoraStdcalo:Calorimetry                0.00629641    0.00807603     0.017588     0.00705103    0.00319456       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00268437    0.00332078    0.00823424    0.00278228    0.00163871       10     
[art]:TriggerResults:TriggerResultInserter     1.4741e-05    2.25945e-05   7.8049e-05    1.6432e-05    1.85081e-05      10     
end_path:out1:RootOutput                        2.948e-06    7.8963e-06    2.5078e-05     5.779e-06    6.01056e-06      10     
end_path:out1:RootOutput(write)                 0.0509939     0.057772      0.0981376     0.0527968     0.0135637       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3408.54 MB
  Peak resident set size usage (VmHWM): 1644.14 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_5GeV_protodunevd_1_358069_87_1743076863_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077534.5765884, 'end_time': 1743078031.8129416}
justIN time: 2025-04-03 08:22:27 UTC       justIN version: 01.03.00