justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.127@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.127@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:42
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-03-27 12:20:16
From worker nodeHostnamer22-n09.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5620 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:39:00
Input filesmonte-carlo-005746-000079
JobscriptExit code0
Real time28m (1697s)
CPU time27m (1669s = 98%)
Max RSS bytes1600311296 (1526 MiB)
Outputting started2025-03-27 12:07:17
Output files
Finished2025-03-27 12:20:16
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

e+06
0 6.07787e+06
1 6.0264e+06
2 6.01865e+06
3 6.01671e+06
Now with regularization...
Begin: 4.61425e+06
0 4.61305e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:52:54 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_127_1743075552_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:52:54 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166655_127_1743075552_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1605 traces, 1605 summary
Retagger: tagging trace set: gauss with 1143 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1143 traces tagged "gauss"
FrameSaver: q=135450 n=15748 tag=gauss
wclsFrameSaver: saving 1605 traces tagged "wiener"
FrameSaver: q=154603 n=13721 tag=wiener
27-Mar-2025 11:54:24 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_127_1743075552_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1660 traces, 1660 summary
Retagger: tagging trace set: gauss with 1193 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1193 traces tagged "gauss"
FrameSaver: q=143215 n=16596 tag=gauss
wclsFrameSaver: saving 1660 traces tagged "wiener"
FrameSaver: q=162433 n=14594 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1592 traces, 1592 summary
Retagger: tagging trace set: gauss with 1221 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1221 traces tagged "gauss"
FrameSaver: q=137657 n=16204 tag=gauss
wclsFrameSaver: saving 1592 traces tagged "wiener"
FrameSaver: q=155394 n=14226 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1596 traces, 1596 summary
Retagger: tagging trace set: gauss with 1173 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1173 traces tagged "gauss"
FrameSaver: q=137361 n=15996 tag=gauss
wclsFrameSaver: saving 1596 traces tagged "wiener"
FrameSaver: q=155684 n=13904 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1671 traces, 1671 summary
Retagger: tagging trace set: gauss with 1270 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1270 traces tagged "gauss"
FrameSaver: q=137796 n=16737 tag=gauss
wclsFrameSaver: saving 1671 traces tagged "wiener"
FrameSaver: q=157074 n=14824 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1584 traces, 1584 summary
Retagger: tagging trace set: gauss with 1208 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1208 traces tagged "gauss"
FrameSaver: q=149089 n=16711 tag=gauss
wclsFrameSaver: saving 1584 traces tagged "wiener"
FrameSaver: q=165531 n=14650 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1542 traces, 1542 summary
Retagger: tagging trace set: gauss with 1116 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1116 traces tagged "gauss"
FrameSaver: q=138240 n=15578 tag=gauss
wclsFrameSaver: saving 1542 traces tagged "wiener"
FrameSaver: q=156716 n=13532 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1583 traces, 1583 summary
Retagger: tagging trace set: gauss with 1162 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1162 traces tagged "gauss"
FrameSaver: q=138934 n=16179 tag=gauss
wclsFrameSaver: saving 1583 traces tagged "wiener"
FrameSaver: q=156047 n=14183 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1156 traces tagged "gauss"
FrameSaver: q=138031 n=15937 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=155565 n=13783 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1771 traces, 1771 summary
Retagger: tagging trace set: gauss with 1300 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1300 traces tagged "gauss"
FrameSaver: q=154594 n=17842 tag=gauss
wclsFrameSaver: saving 1771 traces tagged "wiener"
FrameSaver: q=174164 n=15398 tag=wiener
27-Mar-2025 12:07:16 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166655_127_1743075552_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:07:16 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166655_127_1743075552_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       84.5166       85.5117       89.3686       85.0652       1.36931        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000466137   0.000695036   0.000834804   0.000727504   0.000102002      10     
reco:rns:RandomNumberSaver                     3.6261e-05    5.92185e-05   0.000245641    3.887e-05    6.21562e-05      10     
reco:wclsdatavd:WireCellToolkit                  84.0954       84.8381       86.515        84.6146      0.734731        10     
reco:gaushit:GausHitFinder                      0.0220765     0.0235216     0.0247981     0.0236243    0.000804406      10     
reco:reco3d:SpacePointSolver                    0.0577283     0.0656115     0.0763042     0.0639863    0.00521686       10     
reco:hitpdune:DisambigFromSpacePoints          0.00221902    0.00266681     0.0043322     0.0024844    0.000573204      10     
reco:pandora:StandardPandora                    0.269259      0.530559       2.6947        0.29516      0.721539        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00263216    0.00307343    0.00508979    0.00286594    0.00069371       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000264543   0.000478877   0.00169257    0.000274559   0.000439234      10     
reco:pandoraStdcalo:Calorimetry                0.00161228    0.00209121    0.00412195    0.00185335    0.000693989      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00113668    0.00133968    0.00272404    0.00118141    0.000462859      10     
[art]:TriggerResults:TriggerResultInserter     2.2988e-05    2.6994e-05    5.5534e-05    2.3688e-05    9.53905e-06      10     
end_path:out1:RootOutput                        6.565e-06    8.7914e-06    1.8686e-05     7.791e-06    3.32838e-06      10     
end_path:out1:RootOutput(write)                 0.0362531     0.0389358     0.046134      0.0378491     0.003069        10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3338.38 MB
  Peak resident set size usage (VmHWM): 1346.47 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_127_1743075552_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076328.7064056, 'end_time': 1743077237.164326}
justIN time: 2025-04-03 13:43:49 UTC       justIN version: 01.03.00