justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.146@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.146@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:35:42
From worker nodeHostnamet2wn003.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:15:14
Input filesmonte-carlo-005746-000126
JobscriptExit code0
Real time19m (1175s)
CPU time18m (1131s = 96%)
Max RSS bytes1706348544 (1627 MiB)
Outputting started2025-03-27 12:34:50
Output files
Finished2025-03-27 12:35:42
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

06
0 5.98378e+06
1 5.83411e+06
2 5.82334e+06
3 5.81989e+06
Now with regularization...
Begin: 4.74813e+06
0 4.74542e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:26:11 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_146_1743077728_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:26:12 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_146_1743077728_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1488 traces, 1488 summary
Retagger: tagging trace set: gauss with 1087 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1087 traces tagged "gauss"
FrameSaver: q=139109 n=15647 tag=gauss
wclsFrameSaver: saving 1488 traces tagged "wiener"
FrameSaver: q=155617 n=13483 tag=wiener
27-Mar-2025 12:27:03 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_146_1743077728_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1542 traces, 1542 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=138979 n=15873 tag=gauss
wclsFrameSaver: saving 1542 traces tagged "wiener"
FrameSaver: q=156141 n=13745 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1647 traces, 1647 summary
Retagger: tagging trace set: gauss with 1226 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1226 traces tagged "gauss"
FrameSaver: q=139679 n=16424 tag=gauss
wclsFrameSaver: saving 1647 traces tagged "wiener"
FrameSaver: q=158086 n=14309 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1625 traces, 1625 summary
Retagger: tagging trace set: gauss with 1159 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1159 traces tagged "gauss"
FrameSaver: q=137967 n=16253 tag=gauss
wclsFrameSaver: saving 1625 traces tagged "wiener"
FrameSaver: q=156146 n=14041 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1657 traces, 1657 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=143215 n=16620 tag=gauss
wclsFrameSaver: saving 1657 traces tagged "wiener"
FrameSaver: q=163173 n=14448 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1633 traces, 1633 summary
Retagger: tagging trace set: gauss with 1174 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1174 traces tagged "gauss"
FrameSaver: q=137143 n=16201 tag=gauss
wclsFrameSaver: saving 1633 traces tagged "wiener"
FrameSaver: q=155746 n=14132 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1692 traces, 1692 summary
Retagger: tagging trace set: gauss with 1220 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1220 traces tagged "gauss"
FrameSaver: q=151856 n=17238 tag=gauss
wclsFrameSaver: saving 1692 traces tagged "wiener"
FrameSaver: q=170396 n=14971 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1658 traces, 1658 summary
Retagger: tagging trace set: gauss with 1269 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1269 traces tagged "gauss"
FrameSaver: q=153087 n=17257 tag=gauss
wclsFrameSaver: saving 1658 traces tagged "wiener"
FrameSaver: q=170804 n=15241 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1552 traces, 1552 summary
Retagger: tagging trace set: gauss with 1116 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1116 traces tagged "gauss"
FrameSaver: q=137641 n=15600 tag=gauss
wclsFrameSaver: saving 1552 traces tagged "wiener"
FrameSaver: q=155301 n=13503 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1638 traces, 1638 summary
Retagger: tagging trace set: gauss with 1161 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1161 traces tagged "gauss"
FrameSaver: q=137675 n=16211 tag=gauss
wclsFrameSaver: saving 1638 traces tagged "wiener"
FrameSaver: q=156594 n=14157 tag=wiener
27-Mar-2025 12:34:49 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_146_1743077728_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:34:49 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_146_1743077728_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       49.1756       51.2284       53.0374       50.9625       1.07994        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000375704   0.000591282   0.00113609    0.000422648   0.000262421      10     
reco:rns:RandomNumberSaver                     4.8904e-05    8.70456e-05   0.000350856   5.6188e-05    8.84793e-05      10     
reco:wclsdatavd:WireCellToolkit                  48.218        50.561        52.6441       50.4069       1.33415        10     
reco:gaushit:GausHitFinder                      0.023007      0.0260353     0.0283088     0.026003     0.00144019       10     
reco:reco3d:SpacePointSolver                    0.0785363     0.0891905     0.0966693     0.088265     0.00624794       10     
reco:hitpdune:DisambigFromSpacePoints          0.00245697    0.00295436    0.00577063    0.00268994    0.000949029      10     
reco:pandora:StandardPandora                    0.161977      0.490114       2.59767      0.271361      0.703517        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00181996    0.00317045     0.0056021    0.00292354    0.000979886      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000250011   0.000542922   0.00229016    0.000298961   0.000605735      10     
reco:pandoraStdcalo:Calorimetry                0.00176682    0.00228822    0.00463422    0.00206792    0.000792322      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00136582    0.00165505    0.00340113    0.00147373    0.000584285      10     
[art]:TriggerResults:TriggerResultInserter     1.7444e-05    2.98456e-05   7.0546e-05    2.21225e-05   1.54261e-05      10     
end_path:out1:RootOutput                        4.388e-06    6.6077e-06     2.068e-05     4.904e-06    4.72474e-06      10     
end_path:out1:RootOutput(write)                 0.0431986     0.0453147     0.0485171     0.0445986    0.00168511       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3421.96 MB
  Peak resident set size usage (VmHWM): 1604.95 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_146_1743077728_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078328.3754187, 'end_time': 1743078889.7922542}
justIN time: 2025-04-03 08:28:27 UTC       justIN version: 01.03.00