justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.134@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.134@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:13:33
From worker nodeHostnamet2wn003.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:51:15
Input filesmonte-carlo-005746-000248
JobscriptExit code0
Real time21m (1289s)
CPU time19m (1190s = 92%)
Max RSS bytes1684414464 (1606 MiB)
Outputting started2025-03-27 12:12:45
Output files
Finished2025-03-27 12:13:33
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Begin: 5.77844e+07
0 5.56887e+07
1 5.55474e+07
2 5.55167e+07
Now with regularization...
Begin: 5.47174e+07
0 5.46985e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:04:03 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_134_1743076290_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:04:03 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_134_1743076290_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1587 traces, 1587 summary
Retagger: tagging trace set: gauss with 1124 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1124 traces tagged "gauss"
FrameSaver: q=142192 n=15998 tag=gauss
wclsFrameSaver: saving 1587 traces tagged "wiener"
FrameSaver: q=161241 n=13746 tag=wiener
27-Mar-2025 12:04:57 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_134_1743076290_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1551 traces, 1551 summary
Retagger: tagging trace set: gauss with 1122 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1122 traces tagged "gauss"
FrameSaver: q=136671 n=15515 tag=gauss
wclsFrameSaver: saving 1551 traces tagged "wiener"
FrameSaver: q=155186 n=13272 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1659 traces, 1659 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=155399 n=17533 tag=gauss
wclsFrameSaver: saving 1659 traces tagged "wiener"
FrameSaver: q=173726 n=15192 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1647 traces, 1647 summary
Retagger: tagging trace set: gauss with 1251 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1251 traces tagged "gauss"
FrameSaver: q=143731 n=17030 tag=gauss
wclsFrameSaver: saving 1647 traces tagged "wiener"
FrameSaver: q=161061 n=14971 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1660 traces, 1660 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=147109 n=16978 tag=gauss
wclsFrameSaver: saving 1660 traces tagged "wiener"
FrameSaver: q=165684 n=14732 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1528 traces, 1528 summary
Retagger: tagging trace set: gauss with 1085 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1085 traces tagged "gauss"
FrameSaver: q=137199 n=15455 tag=gauss
wclsFrameSaver: saving 1528 traces tagged "wiener"
FrameSaver: q=154894 n=13349 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1577 traces, 1577 summary
Retagger: tagging trace set: gauss with 1144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1144 traces tagged "gauss"
FrameSaver: q=137356 n=15742 tag=gauss
wclsFrameSaver: saving 1577 traces tagged "wiener"
FrameSaver: q=155841 n=13716 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1205 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1205 traces tagged "gauss"
FrameSaver: q=152073 n=17275 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=168118 n=15328 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1521 traces, 1521 summary
Retagger: tagging trace set: gauss with 1117 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1117 traces tagged "gauss"
FrameSaver: q=143847 n=15643 tag=gauss
wclsFrameSaver: saving 1521 traces tagged "wiener"
FrameSaver: q=161089 n=13706 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1548 traces, 1548 summary
Retagger: tagging trace set: gauss with 1140 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1140 traces tagged "gauss"
FrameSaver: q=137890 n=15887 tag=gauss
wclsFrameSaver: saving 1548 traces tagged "wiener"
FrameSaver: q=156020 n=13853 tag=wiener
27-Mar-2025 12:12:44 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_134_1743076290_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:12:44 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_134_1743076290_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       46.8645       50.4452       53.9778       50.3732       2.5035         10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.00037911    0.000565192   0.00125754     0.0004644    0.000255723      10     
reco:rns:RandomNumberSaver                     4.9014e-05    9.89358e-05   0.000494563   5.2326e-05    0.000132112      10     
reco:wclsdatavd:WireCellToolkit                  46.4469       49.694        53.546        49.724        2.30464        10     
reco:gaushit:GausHitFinder                      0.0218411     0.0262208     0.0307513     0.0262435    0.00285573       10     
reco:reco3d:SpacePointSolver                    0.0756981     0.0895458     0.128128      0.0851773     0.0142945       10     
reco:hitpdune:DisambigFromSpacePoints          0.00215072    0.00297419    0.00540667    0.00268152    0.000876764      10     
reco:pandora:StandardPandora                    0.148553      0.574877        3.33        0.276504      0.919675        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00240097    0.00350559    0.00858603    0.00293838    0.00173534       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000236846   0.000664082   0.00272865    0.000307212   0.000739015      10     
reco:pandoraStdcalo:Calorimetry                0.00158398    0.00237683    0.00598956    0.00194296    0.00124468       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000850959   0.00163348    0.00423195    0.00132915    0.000899827      10     
[art]:TriggerResults:TriggerResultInserter     1.8415e-05    2.85038e-05   8.1938e-05    2.3079e-05    1.79517e-05      10     
end_path:out1:RootOutput                        4.229e-06    1.02108e-05   3.5929e-05     5.005e-06    1.06158e-05      10     
end_path:out1:RootOutput(write)                 0.0384438     0.0430494     0.050072      0.0420781    0.00378228       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3430.96 MB
  Peak resident set size usage (VmHWM): 1603.3 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_134_1743076290_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076980.9069226, 'end_time': 1743077565.076009}
justIN time: 2025-04-03 08:29:29 UTC       justIN version: 01.03.00