justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.86@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.86@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:10:33
From worker nodeHostnamet2wn155.physics.ox.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:48:34
Input filesmonte-carlo-005746-000221
JobscriptExit code0
Real time21m (1293s)
CPU time20m (1203s = 93%)
Max RSS bytes1731563520 (1651 MiB)
Outputting started2025-03-27 12:10:08
Output files
Finished2025-03-27 12:10:33
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

n...
Begin: 2.38311e+07
0 2.26877e+07
1 2.26642e+07
2 2.26553e+07
Now with regularization...
Begin: 2.19098e+07
0 2.19042e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:01:22 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_86_1743076153_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:01:22 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_86_1743076153_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1619 traces, 1619 summary
Retagger: tagging trace set: gauss with 1195 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1195 traces tagged "gauss"
FrameSaver: q=138718 n=16469 tag=gauss
wclsFrameSaver: saving 1619 traces tagged "wiener"
FrameSaver: q=157005 n=14351 tag=wiener
27-Mar-2025 12:02:22 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_86_1743076153_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1585 traces, 1585 summary
Retagger: tagging trace set: gauss with 1154 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1154 traces tagged "gauss"
FrameSaver: q=136681 n=15859 tag=gauss
wclsFrameSaver: saving 1585 traces tagged "wiener"
FrameSaver: q=155469 n=13805 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1688 traces, 1688 summary
Retagger: tagging trace set: gauss with 1245 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1245 traces tagged "gauss"
FrameSaver: q=141708 n=17206 tag=gauss
wclsFrameSaver: saving 1688 traces tagged "wiener"
FrameSaver: q=159863 n=14934 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1698 traces, 1698 summary
Retagger: tagging trace set: gauss with 1198 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1198 traces tagged "gauss"
FrameSaver: q=137902 n=16503 tag=gauss
wclsFrameSaver: saving 1698 traces tagged "wiener"
FrameSaver: q=157830 n=14298 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1137 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1137 traces tagged "gauss"
FrameSaver: q=136660 n=15841 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=156477 n=13837 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1712 traces, 1712 summary
Retagger: tagging trace set: gauss with 1222 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1222 traces tagged "gauss"
FrameSaver: q=153787 n=17663 tag=gauss
wclsFrameSaver: saving 1712 traces tagged "wiener"
FrameSaver: q=175362 n=15154 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1651 traces, 1651 summary
Retagger: tagging trace set: gauss with 1187 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1187 traces tagged "gauss"
FrameSaver: q=140187 n=16480 tag=gauss
wclsFrameSaver: saving 1651 traces tagged "wiener"
FrameSaver: q=158914 n=14299 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1550 traces, 1550 summary
Retagger: tagging trace set: gauss with 1128 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1128 traces tagged "gauss"
FrameSaver: q=140113 n=16097 tag=gauss
wclsFrameSaver: saving 1550 traces tagged "wiener"
FrameSaver: q=156883 n=14109 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1657 traces, 1657 summary
Retagger: tagging trace set: gauss with 1253 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1253 traces tagged "gauss"
FrameSaver: q=142054 n=16921 tag=gauss
wclsFrameSaver: saving 1657 traces tagged "wiener"
FrameSaver: q=159072 n=14834 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1624 traces, 1624 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=141691 n=16668 tag=gauss
wclsFrameSaver: saving 1624 traces tagged "wiener"
FrameSaver: q=159351 n=14531 tag=wiener
27-Mar-2025 12:10:07 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_86_1743076153_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:10:07 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_86_1743076153_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       44.7661       51.8536       62.3409       50.2562       5.40667        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00045478    0.000622018   0.00115591    0.000519921   0.00020788       10     
reco:rns:RandomNumberSaver                     3.9701e-05    8.46227e-05   0.000397585   4.63085e-05   0.000104639      10     
reco:wclsdatavd:WireCellToolkit                  44.4121       51.1439       61.6873       49.7818       5.04054        10     
reco:gaushit:GausHitFinder                      0.0163694     0.0202615     0.027938      0.0183166    0.00436157       10     
reco:reco3d:SpacePointSolver                    0.058829      0.0911617     0.163329      0.0789426     0.0318101       10     
reco:hitpdune:DisambigFromSpacePoints          0.00175542    0.00299467    0.00644008    0.00222914    0.00144614       10     
reco:pandora:StandardPandora                    0.236088      0.542568       2.86746      0.262771      0.776561        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00143184    0.00334411    0.00890078    0.00266386    0.00203621       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000192125   0.000343527   0.00109223    0.000226726   0.000259447      10     
reco:pandoraStdcalo:Calorimetry                0.00123694    0.00226635    0.00603425    0.00154608    0.00138947       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000876011   0.00161694    0.00428604    0.000952427   0.00110393       10     
[art]:TriggerResults:TriggerResultInserter     1.4831e-05    2.45417e-05   7.6071e-05    1.7874e-05    1.75276e-05      10     
end_path:out1:RootOutput                        3.259e-06    5.8775e-06    2.4463e-05    3.8275e-06    6.20418e-06      10     
end_path:out1:RootOutput(write)                 0.0288712     0.0394991     0.0580418     0.0312317     0.0115737       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3430.67 MB
  Peak resident set size usage (VmHWM): 1656.45 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_86_1743076153_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076838.826084, 'end_time': 1743077408.6268976}
justIN time: 2025-04-03 08:23:45 UTC       justIN version: 01.03.00