justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.5@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.5@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-03-27 12:24:58
From worker nodeHostnamer22-n11.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5620 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:43:11
Input filesmonte-carlo-005746-000009
JobscriptExit code0
Real time28m (1729s)
CPU time28m (1694s = 97%)
Max RSS bytes1599746048 (1525 MiB)
Outputting started2025-03-27 12:12:00
Output files
Finished2025-03-27 12:24:58
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+06
1 5.672e+06
2 5.6517e+06
3 5.64537e+06
4 5.64221e+06
Now with regularization...
Begin: 4.62233e+06
0 4.61713e+06
1 4.61422e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:57:24 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_5_1743075810_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:57:25 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_5_1743075810_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1697 traces, 1697 summary
Retagger: tagging trace set: gauss with 1224 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1224 traces tagged "gauss"
FrameSaver: q=138427 n=16895 tag=gauss
wclsFrameSaver: saving 1697 traces tagged "wiener"
FrameSaver: q=157476 n=14733 tag=wiener
27-Mar-2025 11:58:59 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_5_1743075810_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1670 traces, 1670 summary
Retagger: tagging trace set: gauss with 1192 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1192 traces tagged "gauss"
FrameSaver: q=142399 n=16805 tag=gauss
wclsFrameSaver: saving 1670 traces tagged "wiener"
FrameSaver: q=161354 n=14503 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1717 traces, 1717 summary
Retagger: tagging trace set: gauss with 1234 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1234 traces tagged "gauss"
FrameSaver: q=157504 n=17632 tag=gauss
wclsFrameSaver: saving 1717 traces tagged "wiener"
FrameSaver: q=176729 n=15271 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1620 traces, 1620 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=138338 n=16032 tag=gauss
wclsFrameSaver: saving 1620 traces tagged "wiener"
FrameSaver: q=157527 n=13837 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1701 traces, 1701 summary
Retagger: tagging trace set: gauss with 1281 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1281 traces tagged "gauss"
FrameSaver: q=153838 n=17166 tag=gauss
wclsFrameSaver: saving 1701 traces tagged "wiener"
FrameSaver: q=172612 n=15106 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1677 traces, 1677 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=151211 n=16911 tag=gauss
wclsFrameSaver: saving 1677 traces tagged "wiener"
FrameSaver: q=170181 n=14846 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1689 traces, 1689 summary
Retagger: tagging trace set: gauss with 1209 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1209 traces tagged "gauss"
FrameSaver: q=141785 n=16959 tag=gauss
wclsFrameSaver: saving 1689 traces tagged "wiener"
FrameSaver: q=160593 n=14655 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1571 traces, 1571 summary
Retagger: tagging trace set: gauss with 1154 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1154 traces tagged "gauss"
FrameSaver: q=135495 n=15507 tag=gauss
wclsFrameSaver: saving 1571 traces tagged "wiener"
FrameSaver: q=153830 n=13596 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1589 traces, 1589 summary
Retagger: tagging trace set: gauss with 1137 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1137 traces tagged "gauss"
FrameSaver: q=137825 n=15982 tag=gauss
wclsFrameSaver: saving 1589 traces tagged "wiener"
FrameSaver: q=157597 n=13904 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1611 traces, 1611 summary
Retagger: tagging trace set: gauss with 1152 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1152 traces tagged "gauss"
FrameSaver: q=138265 n=15982 tag=gauss
wclsFrameSaver: saving 1611 traces tagged "wiener"
FrameSaver: q=157996 n=13908 tag=wiener
27-Mar-2025 12:11:59 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_5_1743075810_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:11:59 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_5_1743075810_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       84.4836       86.832        94.1047       85.3016       3.49347        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000500429   0.000626943   0.000940952   0.000567621   0.000133617      10     
reco:rns:RandomNumberSaver                     3.9723e-05    6.92486e-05   0.000265995   4.29875e-05   6.60453e-05      10     
reco:wclsdatavd:WireCellToolkit                  84.0954       86.1154       92.9742       84.8198       3.00853        10     
reco:gaushit:GausHitFinder                      0.0212021     0.0240154      0.02694      0.0237159    0.00161058       10     
reco:reco3d:SpacePointSolver                    0.0581559     0.0716683     0.086895      0.0732942    0.00795538       10     
reco:hitpdune:DisambigFromSpacePoints           0.0025018    0.00301532    0.00474275     0.0028551    0.000606919      10     
reco:pandora:StandardPandora                    0.247782      0.565741       2.85648      0.317346      0.764059        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00256352    0.00321488     0.005555      0.0029146    0.000843875      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000265075   0.000353485   0.00101352    0.000275823   0.000220415      10     
reco:pandoraStdcalo:Calorimetry                0.00176941    0.00221273    0.00459869    0.00187526    0.000829288      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.00109419    0.00140708    0.00326574    0.00116669    0.000625699      10     
[art]:TriggerResults:TriggerResultInserter     2.4727e-05    3.0533e-05    6.6226e-05    2.59325e-05   1.20511e-05      10     
end_path:out1:RootOutput                        7.421e-06    1.07014e-05   3.1875e-05     8.481e-06    7.06686e-06      10     
end_path:out1:RootOutput(write)                 0.0361963     0.0396953     0.0448405     0.0382694    0.00275058       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3429.19 MB
  Peak resident set size usage (VmHWM): 1434.94 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_5_1743075810_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076600.4487352, 'end_time': 1743077520.4537587}
justIN time: 2025-04-03 12:56:01 UTC       justIN version: 01.03.00