justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.80@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.80@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-03-27 12:10:09
From worker nodeHostnamet2wn155.physics.ox.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:49:07
Input filesmonte-carlo-005746-000224
JobscriptExit code0
Real time20m (1214s)
CPU time18m (1124s = 92%)
Max RSS bytes1731239936 (1651 MiB)
Outputting started2025-03-27 12:09:21
Output files
Finished2025-03-27 12:10:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

n...
Begin: 6.51092e+06
0 6.17179e+06
1 6.12653e+06
2 6.12101e+06
Now with regularization...
Begin: 5.0261e+06
0 5.02516e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:00:16 GMT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_80_1743076158_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:00:16 GMT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_80_1743076158_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1576 traces, 1576 summary
Retagger: tagging trace set: gauss with 1208 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1208 traces tagged "gauss"
FrameSaver: q=139541 n=16219 tag=gauss
wclsFrameSaver: saving 1576 traces tagged "wiener"
FrameSaver: q=155622 n=14221 tag=wiener
27-Mar-2025 12:01:24 GMT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_80_1743076158_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1624 traces, 1624 summary
Retagger: tagging trace set: gauss with 1084 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1084 traces tagged "gauss"
FrameSaver: q=150228 n=16078 tag=gauss
wclsFrameSaver: saving 1624 traces tagged "wiener"
FrameSaver: q=168993 n=13913 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1607 traces, 1607 summary
Retagger: tagging trace set: gauss with 1169 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1169 traces tagged "gauss"
FrameSaver: q=138495 n=15854 tag=gauss
wclsFrameSaver: saving 1607 traces tagged "wiener"
FrameSaver: q=156746 n=13800 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1630 traces, 1630 summary
Retagger: tagging trace set: gauss with 1150 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1150 traces tagged "gauss"
FrameSaver: q=148548 n=16213 tag=gauss
wclsFrameSaver: saving 1630 traces tagged "wiener"
FrameSaver: q=167245 n=13985 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1139 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1139 traces tagged "gauss"
FrameSaver: q=136458 n=15689 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=155208 n=13606 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1640 traces, 1640 summary
Retagger: tagging trace set: gauss with 1247 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1247 traces tagged "gauss"
FrameSaver: q=138190 n=16464 tag=gauss
wclsFrameSaver: saving 1640 traces tagged "wiener"
FrameSaver: q=156033 n=14451 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1640 traces, 1640 summary
Retagger: tagging trace set: gauss with 1135 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1135 traces tagged "gauss"
FrameSaver: q=138614 n=16060 tag=gauss
wclsFrameSaver: saving 1640 traces tagged "wiener"
FrameSaver: q=157709 n=13806 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1602 traces, 1602 summary
Retagger: tagging trace set: gauss with 1116 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1116 traces tagged "gauss"
FrameSaver: q=151964 n=16280 tag=gauss
wclsFrameSaver: saving 1602 traces tagged "wiener"
FrameSaver: q=171249 n=13855 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1542 traces, 1542 summary
Retagger: tagging trace set: gauss with 1109 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1109 traces tagged "gauss"
FrameSaver: q=135436 n=15417 tag=gauss
wclsFrameSaver: saving 1542 traces tagged "wiener"
FrameSaver: q=153908 n=13385 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1531 traces, 1531 summary
Retagger: tagging trace set: gauss with 1140 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1140 traces tagged "gauss"
FrameSaver: q=135780 n=15396 tag=gauss
wclsFrameSaver: saving 1531 traces tagged "wiener"
FrameSaver: q=153251 n=13428 tag=wiener
27-Mar-2025 12:09:20 GMT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_80_1743076158_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 12:09:20 GMT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_80_1743076158_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       43.3784       51.0813       59.4156       52.987        5.47388        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000407131   0.000587958   0.000986469   0.000520357   0.000170397      10     
reco:rns:RandomNumberSaver                     3.5553e-05    7.30812e-05   0.000337046   4.75635e-05   8.81651e-05      10     
reco:wclsdatavd:WireCellToolkit                  43.0732       50.3559       58.8937       52.5958       5.15936        10     
reco:gaushit:GausHitFinder                      0.0146726     0.0201486     0.0269251     0.0186138    0.00438233       10     
reco:reco3d:SpacePointSolver                    0.0509783     0.079541      0.0965564     0.0836556     0.014708        10     
reco:hitpdune:DisambigFromSpacePoints          0.00169415    0.00290149    0.00531015    0.00278965    0.000939001      10     
reco:pandora:StandardPandora                    0.200012      0.571349       3.14744      0.296982      0.860223        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00175828    0.00314086    0.00657967    0.00260808    0.00138545       10     
reco:pandoraShower:LArPandoraShowerCreation    0.00019258    0.000320154   0.000911374   0.000255732   0.000204386      10     
reco:pandoraStdcalo:Calorimetry                 0.0012699    0.00207351    0.00482807     0.0018983    0.00100131       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000771965   0.00140772    0.00348062    0.00122986    0.000760829      10     
[art]:TriggerResults:TriggerResultInserter     1.2633e-05    2.60711e-05   6.2359e-05    2.18605e-05   1.42644e-05      10     
end_path:out1:RootOutput                        3.015e-06    5.0437e-06    1.7466e-05     3.67e-06     4.1645e-06       10     
end_path:out1:RootOutput(write)                 0.0276722     0.0393881     0.0496176     0.0411482     0.0094604       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3365.88 MB
  Peak resident set size usage (VmHWM): 1589.22 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_80_1743076158_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076768.0380144, 'end_time': 1743077361.8370354}
justIN time: 2025-04-03 08:24:44 UTC       justIN version: 01.03.00