justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.77@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.77@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2025-03-27 12:37:21
From worker nodeHostnamen154.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:21:33
Input filesmonte-carlo-005746-000420
JobscriptExit code0
Real time15m (932s)
CPU time15m (921s = 98%)
Max RSS bytes1731153920 (1650 MiB)
Outputting started2025-03-27 12:37:06
Output files
Finished2025-03-27 12:37:21
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

1.10708e+07
4 1.10445e+07
5 1.10275e+07
6 1.10162e+07
7 1.10082e+07
Now with regularization...
Begin: 1.00254e+07
0 1.00176e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:29:24 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_77_1743078099_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:29:24 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358071_77_1743078099_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1646 traces, 1646 summary
Retagger: tagging trace set: gauss with 1201 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1201 traces tagged "gauss"
FrameSaver: q=151464 n=17266 tag=gauss
wclsFrameSaver: saving 1646 traces tagged "wiener"
FrameSaver: q=172874 n=15291 tag=wiener
27-Mar-2025 12:30:06 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_77_1743078099_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1239 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1239 traces tagged "gauss"
FrameSaver: q=138673 n=16477 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=156611 n=14641 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1707 traces, 1707 summary
Retagger: tagging trace set: gauss with 1214 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1214 traces tagged "gauss"
FrameSaver: q=142906 n=16733 tag=gauss
wclsFrameSaver: saving 1707 traces tagged "wiener"
FrameSaver: q=162341 n=14564 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1664 traces, 1664 summary
Retagger: tagging trace set: gauss with 1149 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1149 traces tagged "gauss"
FrameSaver: q=139808 n=16440 tag=gauss
wclsFrameSaver: saving 1664 traces tagged "wiener"
FrameSaver: q=159240 n=14156 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1683 traces, 1683 summary
Retagger: tagging trace set: gauss with 1233 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1233 traces tagged "gauss"
FrameSaver: q=155830 n=17446 tag=gauss
wclsFrameSaver: saving 1683 traces tagged "wiener"
FrameSaver: q=175066 n=15319 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1716 traces, 1716 summary
Retagger: tagging trace set: gauss with 1193 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1193 traces tagged "gauss"
FrameSaver: q=144049 n=17067 tag=gauss
wclsFrameSaver: saving 1716 traces tagged "wiener"
FrameSaver: q=164079 n=14553 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1713 traces, 1713 summary
Retagger: tagging trace set: gauss with 1283 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1283 traces tagged "gauss"
FrameSaver: q=158408 n=18278 tag=gauss
wclsFrameSaver: saving 1713 traces tagged "wiener"
FrameSaver: q=175980 n=15999 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1711 traces, 1711 summary
Retagger: tagging trace set: gauss with 1254 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1254 traces tagged "gauss"
FrameSaver: q=155175 n=17627 tag=gauss
wclsFrameSaver: saving 1711 traces tagged "wiener"
FrameSaver: q=173864 n=15324 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1642 traces, 1642 summary
Retagger: tagging trace set: gauss with 1211 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1211 traces tagged "gauss"
FrameSaver: q=139266 n=16504 tag=gauss
wclsFrameSaver: saving 1642 traces tagged "wiener"
FrameSaver: q=157681 n=14430 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1119 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1119 traces tagged "gauss"
FrameSaver: q=138544 n=15743 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=157405 n=13463 tag=wiener
27-Mar-2025 12:37:05 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358071_77_1743078099_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:37:05 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358071_77_1743078099_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.9336       45.8563       47.7428       46.5635       1.92308        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000349789   0.000449879   0.000549693   0.000452159   6.39576e-05      10     
reco:rns:RandomNumberSaver                     3.4867e-05    5.52934e-05   0.000215904   3.70535e-05   5.36051e-05      10     
reco:wclsdatavd:WireCellToolkit                  39.148        45.3284       47.3173       46.164         2.285         10     
reco:gaushit:GausHitFinder                      0.0157465     0.0177242     0.021969      0.0172036    0.00182779       10     
reco:reco3d:SpacePointSolver                    0.0495817     0.0670362     0.081635      0.0671191    0.00780973       10     
reco:hitpdune:DisambigFromSpacePoints          0.00191261    0.00229552    0.00381271    0.00218824    0.000531612      10     
reco:pandora:StandardPandora                    0.240407      0.402133       1.64901      0.267506      0.415857        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00137879    0.00239892    0.00421718     0.0023886    0.000729268      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000184378   0.000250883   0.000627272   0.000210365   0.000125871      10     
reco:pandoraStdcalo:Calorimetry                0.00126936    0.00163428    0.00350198     0.0014388    0.000630483      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000849221   0.00107525    0.00243274    0.000930211   0.000454746      10     
[art]:TriggerResults:TriggerResultInserter     1.3851e-05    1.76421e-05   4.6611e-05    1.4272e-05    9.67093e-06      10     
end_path:out1:RootOutput                        5.188e-06    6.5381e-06    1.5489e-05    5.6165e-06    2.99107e-06      10     
end_path:out1:RootOutput(write)                 0.0281552     0.0292318     0.0317371     0.0290866     0.0010477       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3373.55 MB
  Peak resident set size usage (VmHWM): 1606.52 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_77_1743078099_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078537.09873, 'end_time': 1743079026.398138}
justIN time: 2025-04-03 08:24:37 UTC       justIN version: 01.03.00