justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.3@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.3@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:41
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-03-27 11:54:14
From worker nodeHostnamer21-n18.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU X5650 @ 2.67GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:15:40
Input filesmonte-carlo-005746-000006
JobscriptExit code0
Real time25m (1536s)
CPU time25m (1513s = 98%)
Max RSS bytes1598722048 (1524 MiB)
Outputting started2025-03-27 11:41:17
Output files
Finished2025-03-27 11:54:14
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rization...
Begin: 2.39922e+07
0 2.27759e+07
1 2.26984e+07
2 2.26909e+07
Now with regularization...
Begin: 2.1946e+07
0 2.19445e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 11:28:11 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_3_1743074150_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 11:28:11 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_166655_3_1743074150_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1722 traces, 1722 summary
Retagger: tagging trace set: gauss with 1229 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1229 traces tagged "gauss"
FrameSaver: q=152772 n=17084 tag=gauss
wclsFrameSaver: saving 1722 traces tagged "wiener"
FrameSaver: q=173295 n=14888 tag=wiener
27-Mar-2025 11:29:32 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_3_1743074150_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1588 traces, 1588 summary
Retagger: tagging trace set: gauss with 1252 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1252 traces tagged "gauss"
FrameSaver: q=142116 n=16958 tag=gauss
wclsFrameSaver: saving 1588 traces tagged "wiener"
FrameSaver: q=157390 n=15080 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1648 traces, 1648 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=141540 n=16694 tag=gauss
wclsFrameSaver: saving 1648 traces tagged "wiener"
FrameSaver: q=160652 n=14529 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1598 traces, 1598 summary
Retagger: tagging trace set: gauss with 1236 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1236 traces tagged "gauss"
FrameSaver: q=137370 n=16195 tag=gauss
wclsFrameSaver: saving 1598 traces tagged "wiener"
FrameSaver: q=154681 n=14257 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1622 traces, 1622 summary
Retagger: tagging trace set: gauss with 1115 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1115 traces tagged "gauss"
FrameSaver: q=148662 n=16536 tag=gauss
wclsFrameSaver: saving 1622 traces tagged "wiener"
FrameSaver: q=168182 n=14291 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1654 traces, 1654 summary
Retagger: tagging trace set: gauss with 1243 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1243 traces tagged "gauss"
FrameSaver: q=152184 n=17093 tag=gauss
wclsFrameSaver: saving 1654 traces tagged "wiener"
FrameSaver: q=169866 n=14911 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1591 traces, 1591 summary
Retagger: tagging trace set: gauss with 1246 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1246 traces tagged "gauss"
FrameSaver: q=139927 n=16788 tag=gauss
wclsFrameSaver: saving 1591 traces tagged "wiener"
FrameSaver: q=155617 n=14888 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1556 traces, 1556 summary
Retagger: tagging trace set: gauss with 1126 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1126 traces tagged "gauss"
FrameSaver: q=138064 n=15757 tag=gauss
wclsFrameSaver: saving 1556 traces tagged "wiener"
FrameSaver: q=155736 n=13811 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1583 traces, 1583 summary
Retagger: tagging trace set: gauss with 1162 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1162 traces tagged "gauss"
FrameSaver: q=141512 n=16227 tag=gauss
wclsFrameSaver: saving 1583 traces tagged "wiener"
FrameSaver: q=159265 n=14032 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1553 traces, 1553 summary
Retagger: tagging trace set: gauss with 1169 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1169 traces tagged "gauss"
FrameSaver: q=138660 n=15824 tag=gauss
wclsFrameSaver: saving 1553 traces tagged "wiener"
FrameSaver: q=155319 n=13910 tag=wiener
27-Mar-2025 11:41:16 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_166655_3_1743074150_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 11:41:16 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_166655_3_1743074150_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       76.5145       77.8864       80.8665       77.5993       1.30605        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000630762   0.000734577   0.000846863    0.0007349    6.63527e-05      10     
reco:rns:RandomNumberSaver                     3.0908e-05    5.55663e-05   0.000230363   3.51145e-05   5.8537e-05       10     
reco:wclsdatavd:WireCellToolkit                  76.1623       77.2395       78.4385       77.1936      0.873214        10     
reco:gaushit:GausHitFinder                      0.0195604     0.0222301     0.0266843     0.021357     0.00226605       10     
reco:reco3d:SpacePointSolver                    0.051822      0.062368      0.0793104     0.0599896    0.00887541       10     
reco:hitpdune:DisambigFromSpacePoints          0.00185258    0.00237308     0.0041901    0.00214363    0.000649063      10     
reco:pandora:StandardPandora                    0.229918      0.514427       2.78079      0.256238      0.755866        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00140919    0.00267136    0.00556338    0.00246997    0.00101994       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000223775   0.000301766    0.0008328    0.000245354   0.000177217      10     
reco:pandoraStdcalo:Calorimetry                0.00147777    0.00192517    0.00423693    0.00161552    0.000788011      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000934129   0.00118646    0.00268752    0.00102038    0.000502773      10     
[art]:TriggerResults:TriggerResultInserter     1.9996e-05    2.46986e-05   4.8536e-05    2.1845e-05    8.13712e-06      10     
end_path:out1:RootOutput                        8.255e-06    9.6638e-06    1.9121e-05     8.654e-06    3.16316e-06      10     
end_path:out1:RootOutput(write)                 0.0314793     0.0345674     0.0405745     0.0337998    0.00249153       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3431.73 MB
  Peak resident set size usage (VmHWM): 1422.98 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_3_1743074150_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743074850.5456774, 'end_time': 1743075677.267082}
justIN time: 2025-04-03 12:56:01 UTC       justIN version: 01.03.00