justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.41@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.41@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-03-27 12:03:30
From worker nodeHostnameheplnc148.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:44:37
Input filesmonte-carlo-005746-000161
JobscriptExit code0
Real time18m (1110s)
CPU time17m (1022s = 92%)
Max RSS bytes1717755904 (1638 MiB)
Outputting started2025-03-27 12:03:08
Output files
Finished2025-03-27 12:03:30
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

.08189e+07
2 1.07813e+07
3 1.07643e+07
4 1.07534e+07
5 1.07462e+07
Now with regularization...
Begin: 9.71082e+06
0 9.70383e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:55:39 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_41_1743075888_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:55:40 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_41_1743075888_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1673 traces, 1673 summary
Retagger: tagging trace set: gauss with 1301 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1301 traces tagged "gauss"
FrameSaver: q=140119 n=16948 tag=gauss
wclsFrameSaver: saving 1673 traces tagged "wiener"
FrameSaver: q=157673 n=15033 tag=wiener
27-Mar-2025 11:56:22 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_41_1743075888_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1587 traces, 1587 summary
Retagger: tagging trace set: gauss with 1128 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1128 traces tagged "gauss"
FrameSaver: q=139818 n=16257 tag=gauss
wclsFrameSaver: saving 1587 traces tagged "wiener"
FrameSaver: q=157661 n=13968 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1645 traces, 1645 summary
Retagger: tagging trace set: gauss with 1144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1144 traces tagged "gauss"
FrameSaver: q=141188 n=16408 tag=gauss
wclsFrameSaver: saving 1645 traces tagged "wiener"
FrameSaver: q=161245 n=14260 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1633 traces, 1633 summary
Retagger: tagging trace set: gauss with 1201 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1201 traces tagged "gauss"
FrameSaver: q=155737 n=17172 tag=gauss
wclsFrameSaver: saving 1633 traces tagged "wiener"
FrameSaver: q=172138 n=14796 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1774 traces, 1774 summary
Retagger: tagging trace set: gauss with 1291 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1291 traces tagged "gauss"
FrameSaver: q=160455 n=18315 tag=gauss
wclsFrameSaver: saving 1774 traces tagged "wiener"
FrameSaver: q=178806 n=16059 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1612 traces, 1612 summary
Retagger: tagging trace set: gauss with 1171 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1171 traces tagged "gauss"
FrameSaver: q=148520 n=16577 tag=gauss
wclsFrameSaver: saving 1612 traces tagged "wiener"
FrameSaver: q=167778 n=14429 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1615 traces, 1615 summary
Retagger: tagging trace set: gauss with 1192 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1192 traces tagged "gauss"
FrameSaver: q=142381 n=16863 tag=gauss
wclsFrameSaver: saving 1615 traces tagged "wiener"
FrameSaver: q=160907 n=14597 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1621 traces, 1621 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=138671 n=16290 tag=gauss
wclsFrameSaver: saving 1621 traces tagged "wiener"
FrameSaver: q=156955 n=14111 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1722 traces, 1722 summary
Retagger: tagging trace set: gauss with 1265 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1265 traces tagged "gauss"
FrameSaver: q=139941 n=17317 tag=gauss
wclsFrameSaver: saving 1722 traces tagged "wiener"
FrameSaver: q=159332 n=15105 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1660 traces, 1660 summary
Retagger: tagging trace set: gauss with 1224 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1224 traces tagged "gauss"
FrameSaver: q=141488 n=16915 tag=gauss
wclsFrameSaver: saving 1660 traces tagged "wiener"
FrameSaver: q=160577 n=14634 tag=wiener
27-Mar-2025 12:03:07 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_41_1743075888_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:03:07 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_41_1743075888_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       40.5725       44.0288       45.9823       44.4614       1.71958        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000479484   0.000655065   0.00143194    0.000551651   0.000282995      10     
reco:rns:RandomNumberSaver                     5.3481e-05    0.000130434   0.000537834   6.4236e-05    0.000149235      10     
reco:wclsdatavd:WireCellToolkit                  37.8483       43.2957       45.5342       44.0447       2.32589        10     
reco:gaushit:GausHitFinder                      0.020436      0.0233833     0.0259322     0.0233662     0.0014431       10     
reco:reco3d:SpacePointSolver                    0.0737412     0.0818235     0.0885737     0.0827445    0.00490146       10     
reco:hitpdune:DisambigFromSpacePoints          0.00212491    0.00289601    0.00622919    0.00254802    0.00114396       10     
reco:pandora:StandardPandora                    0.203209      0.566114       3.22799       0.27775      0.887647        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00184651    0.00319748    0.00741548    0.00269042    0.00153017       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000213773   0.00060713    0.00231383    0.000283405   0.000624564      10     
reco:pandoraStdcalo:Calorimetry                0.00171246    0.00228318    0.00577181    0.00187913    0.00117365       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00107614    0.00151015    0.00360885    0.00130597    0.000711543      10     
[art]:TriggerResults:TriggerResultInserter     1.6772e-05    2.40894e-05   5.8089e-05    2.02585e-05   1.16235e-05      10     
end_path:out1:RootOutput                        4.028e-06    1.39761e-05   5.6466e-05     7.549e-06    1.52072e-05      10     
end_path:out1:RootOutput(write)                 0.0383714     0.0458101     0.0490549     0.0465218     0.002882        10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3384.34 MB
  Peak resident set size usage (VmHWM): 1582.09 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_41_1743075888_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076499.180561, 'end_time': 1743076988.496028}
justIN time: 2025-04-03 08:24:23 UTC       justIN version: 01.03.00