justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.83@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.83@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-03-27 12:31:16
From worker nodeHostnamer21-n02.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5620 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:48:43
Input filesmonte-carlo-005746-000022
JobscriptExit code0
Real time29m (1776s)
CPU time29m (1763s = 99%)
Max RSS bytes1599053824 (1524 MiB)
Outputting started2025-03-27 12:18:20
Output files
Finished2025-03-27 12:31:16
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

9.40677e+06
1 9.2626e+06
2 9.23083e+06
3 9.22096e+06
4 9.21697e+06
Now with regularization...
Begin: 8.24126e+06
0 8.23727e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:02:51 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_83_1743076131_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:02:52 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_83_1743076131_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1656 traces, 1656 summary
Retagger: tagging trace set: gauss with 1239 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1239 traces tagged "gauss"
FrameSaver: q=139978 n=16789 tag=gauss
wclsFrameSaver: saving 1656 traces tagged "wiener"
FrameSaver: q=158182 n=14663 tag=wiener
27-Mar-2025 12:04:25 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_83_1743076131_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1589 traces, 1589 summary
Retagger: tagging trace set: gauss with 1110 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1110 traces tagged "gauss"
FrameSaver: q=137169 n=15970 tag=gauss
wclsFrameSaver: saving 1589 traces tagged "wiener"
FrameSaver: q=155469 n=13765 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1548 traces, 1548 summary
Retagger: tagging trace set: gauss with 1102 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1102 traces tagged "gauss"
FrameSaver: q=136691 n=15392 tag=gauss
wclsFrameSaver: saving 1548 traces tagged "wiener"
FrameSaver: q=154891 n=13269 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1586 traces, 1586 summary
Retagger: tagging trace set: gauss with 1105 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1105 traces tagged "gauss"
FrameSaver: q=135604 n=15490 tag=gauss
wclsFrameSaver: saving 1586 traces tagged "wiener"
FrameSaver: q=155083 n=13302 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1638 traces, 1638 summary
Retagger: tagging trace set: gauss with 1139 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1139 traces tagged "gauss"
FrameSaver: q=140054 n=16174 tag=gauss
wclsFrameSaver: saving 1638 traces tagged "wiener"
FrameSaver: q=159634 n=13964 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1533 traces, 1533 summary
Retagger: tagging trace set: gauss with 1085 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1085 traces tagged "gauss"
FrameSaver: q=137665 n=15304 tag=gauss
wclsFrameSaver: saving 1533 traces tagged "wiener"
FrameSaver: q=155660 n=13209 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1655 traces, 1655 summary
Retagger: tagging trace set: gauss with 1209 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1209 traces tagged "gauss"
FrameSaver: q=137972 n=16447 tag=gauss
wclsFrameSaver: saving 1655 traces tagged "wiener"
FrameSaver: q=156569 n=14385 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1661 traces, 1661 summary
Retagger: tagging trace set: gauss with 1165 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1165 traces tagged "gauss"
FrameSaver: q=138391 n=16612 tag=gauss
wclsFrameSaver: saving 1661 traces tagged "wiener"
FrameSaver: q=157806 n=14426 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1618 traces, 1618 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=145929 n=16351 tag=gauss
wclsFrameSaver: saving 1618 traces tagged "wiener"
FrameSaver: q=164129 n=14162 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1611 traces, 1611 summary
Retagger: tagging trace set: gauss with 1137 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1137 traces tagged "gauss"
FrameSaver: q=137424 n=15945 tag=gauss
wclsFrameSaver: saving 1611 traces tagged "wiener"
FrameSaver: q=156435 n=13694 tag=wiener
27-Mar-2025 12:18:18 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_83_1743076131_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:18:18 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_83_1743076131_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       91.3879       91.966        92.677        91.9793      0.397316        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000479865   0.000713457   0.000953979   0.000735913   0.000159764      10     
reco:rns:RandomNumberSaver                     4.1352e-05    7.14507e-05   0.000285313   4.50835e-05   7.16535e-05      10     
reco:wclsdatavd:WireCellToolkit                  90.2161       91.297        91.9415       91.2963      0.478337        10     
reco:gaushit:GausHitFinder                      0.0227697     0.0237528     0.0246276     0.0238574    0.00067988       10     
reco:reco3d:SpacePointSolver                    0.0670369     0.0755213     0.0918296     0.0737931    0.00679639       10     
reco:hitpdune:DisambigFromSpacePoints          0.00248422    0.00304289    0.00433228    0.00305061    0.000502223      10     
reco:pandora:StandardPandora                    0.283338      0.513758       2.29262      0.317739      0.593523        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0018345    0.00292218    0.00553812    0.00268281    0.000959295      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000270234   0.000338055   0.000824497   0.000286504   0.000162516      10     
reco:pandoraStdcalo:Calorimetry                0.00177859    0.00224017    0.00462963    0.00200125    0.000814012      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00107374    0.00136732    0.00304824    0.00119708    0.000563946      10     
[art]:TriggerResults:TriggerResultInserter     2.4626e-05    2.88551e-05   5.6635e-05    2.56425e-05   9.28403e-06      10     
end_path:out1:RootOutput                        7.884e-06    1.01635e-05   2.7748e-05    8.0755e-06    5.87471e-06      10     
end_path:out1:RootOutput(write)                 0.0382391     0.0409448      0.04556      0.0402577    0.00242685       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3300.73 MB
  Peak resident set size usage (VmHWM): 1319 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_83_1743076131_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076929.6701095, 'end_time': 1743077899.6306791}
justIN time: 2025-04-03 13:35:49 UTC       justIN version: 01.03.00