justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.94@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.94@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:35:57
From worker nodeHostnamet2wn155.physics.ox.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:11:49
Input filesmonte-carlo-005746-000185
JobscriptExit code0
Real time23m (1421s)
CPU time23m (1384s = 97%)
Max RSS bytes1729716224 (1649 MiB)
Outputting started2025-03-27 12:35:31
Output files
Finished2025-03-27 12:35:57
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

36914e+06
2 6.34241e+06
3 6.33284e+06
4 6.32784e+06
Now with regularization...
Begin: 5.2804e+06
0 5.27308e+06
1 5.26803e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:24:38 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_94_1743077520_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:24:39 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_94_1743077520_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1161 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1161 traces tagged "gauss"
FrameSaver: q=137606 n=15913 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=156713 n=13856 tag=wiener
27-Mar-2025 12:25:26 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_94_1743077520_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1607 traces, 1607 summary
Retagger: tagging trace set: gauss with 1158 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1158 traces tagged "gauss"
FrameSaver: q=138352 n=16108 tag=gauss
wclsFrameSaver: saving 1607 traces tagged "wiener"
FrameSaver: q=156966 n=14067 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1541 traces, 1541 summary
Retagger: tagging trace set: gauss with 1078 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1078 traces tagged "gauss"
FrameSaver: q=137128 n=15410 tag=gauss
wclsFrameSaver: saving 1541 traces tagged "wiener"
FrameSaver: q=155579 n=13322 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1701 traces, 1701 summary
Retagger: tagging trace set: gauss with 1234 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1234 traces tagged "gauss"
FrameSaver: q=159809 n=18028 tag=gauss
wclsFrameSaver: saving 1701 traces tagged "wiener"
FrameSaver: q=177292 n=15651 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1635 traces, 1635 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=140580 n=16493 tag=gauss
wclsFrameSaver: saving 1635 traces tagged "wiener"
FrameSaver: q=158238 n=14290 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1617 traces, 1617 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=146642 n=16443 tag=gauss
wclsFrameSaver: saving 1617 traces tagged "wiener"
FrameSaver: q=164667 n=14284 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1740 traces, 1740 summary
Retagger: tagging trace set: gauss with 1232 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1232 traces tagged "gauss"
FrameSaver: q=154844 n=17588 tag=gauss
wclsFrameSaver: saving 1740 traces tagged "wiener"
FrameSaver: q=174670 n=15172 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1696 traces, 1696 summary
Retagger: tagging trace set: gauss with 1260 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1260 traces tagged "gauss"
FrameSaver: q=149488 n=17636 tag=gauss
wclsFrameSaver: saving 1696 traces tagged "wiener"
FrameSaver: q=168433 n=15275 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1661 traces, 1661 summary
Retagger: tagging trace set: gauss with 1174 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1174 traces tagged "gauss"
FrameSaver: q=152265 n=16945 tag=gauss
wclsFrameSaver: saving 1661 traces tagged "wiener"
FrameSaver: q=170782 n=14597 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1739 traces, 1739 summary
Retagger: tagging trace set: gauss with 1266 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1266 traces tagged "gauss"
FrameSaver: q=141127 n=17366 tag=gauss
wclsFrameSaver: saving 1739 traces tagged "wiener"
FrameSaver: q=161171 n=15131 tag=wiener
27-Mar-2025 12:35:30 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_94_1743077520_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:35:30 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_94_1743077520_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       46.3511       64.5521       77.7233       66.7927       12.3909        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000540339   0.000708285   0.00101169    0.000708349   0.000132692      10     
reco:rns:RandomNumberSaver                     4.3259e-05    9.22826e-05   0.000325976   6.4901e-05    8.02383e-05      10     
reco:wclsdatavd:WireCellToolkit                  44.2381       63.7748       76.9866       66.044        12.519         10     
reco:gaushit:GausHitFinder                      0.0169689     0.0312512     0.0384556     0.0367903    0.00918881       10     
reco:reco3d:SpacePointSolver                    0.0650428     0.107181       0.14433      0.120358      0.028717        10     
reco:hitpdune:DisambigFromSpacePoints          0.00197617    0.00408445    0.00759802    0.00439751     0.0016171       10     
reco:pandora:StandardPandora                    0.257303      0.566229       1.92887       0.46303      0.465408        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00243932    0.00363686    0.00604606    0.00351453    0.00107787       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000214447   0.000630848   0.00244429    0.000423144   0.000645469      10     
reco:pandoraStdcalo:Calorimetry                0.00147833    0.00268254    0.00449433    0.00290309    0.000903553      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000847781   0.00207962    0.00363599    0.00219139    0.000898393      10     
[art]:TriggerResults:TriggerResultInserter      1.246e-05    3.09658e-05   9.8212e-05    2.65465e-05   2.36217e-05      10     
end_path:out1:RootOutput                        2.998e-06    9.1145e-06    2.6211e-05     7.603e-06    6.02639e-06      10     
end_path:out1:RootOutput(write)                 0.0291339     0.0537252     0.0722149     0.0596128     0.0164197       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3402.78 MB
  Peak resident set size usage (VmHWM): 1621.84 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_94_1743077520_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078237.9653625, 'end_time': 1743078931.8565123}
justIN time: 2025-04-03 08:24:32 UTC       justIN version: 01.03.00