justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.158@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.158@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:10:25
From worker nodeHostnametd809.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:52:44
Input filesmonte-carlo-005746-000008
JobscriptExit code0
Real time17m (1041s)
CPU time16m (1008s = 96%)
Max RSS bytes1712967680 (1633 MiB)
Outputting started2025-03-27 12:10:06
Output files
Finished2025-03-27 12:10:25
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ularization...
Begin: 9.89609e+07
0 9.86884e+07
1 9.8681e+07
Now with regularization...
Begin: 9.82436e+07
0 9.82432e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:02:24 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_158_1743076376_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:02:24 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358068_158_1743076376_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1659 traces, 1659 summary
Retagger: tagging trace set: gauss with 1193 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1193 traces tagged "gauss"
FrameSaver: q=150160 n=16952 tag=gauss
wclsFrameSaver: saving 1659 traces tagged "wiener"
FrameSaver: q=169418 n=14661 tag=wiener
27-Mar-2025 13:03:10 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_158_1743076376_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1631 traces, 1631 summary
Retagger: tagging trace set: gauss with 1184 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1184 traces tagged "gauss"
FrameSaver: q=138272 n=16209 tag=gauss
wclsFrameSaver: saving 1631 traces tagged "wiener"
FrameSaver: q=156619 n=14172 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1645 traces, 1645 summary
Retagger: tagging trace set: gauss with 1136 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1136 traces tagged "gauss"
FrameSaver: q=137730 n=16288 tag=gauss
wclsFrameSaver: saving 1645 traces tagged "wiener"
FrameSaver: q=158200 n=14074 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1711 traces, 1711 summary
Retagger: tagging trace set: gauss with 1252 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1252 traces tagged "gauss"
FrameSaver: q=146507 n=17466 tag=gauss
wclsFrameSaver: saving 1711 traces tagged "wiener"
FrameSaver: q=164310 n=15206 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1632 traces, 1632 summary
Retagger: tagging trace set: gauss with 1183 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1183 traces tagged "gauss"
FrameSaver: q=137182 n=16159 tag=gauss
wclsFrameSaver: saving 1632 traces tagged "wiener"
FrameSaver: q=155974 n=14143 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1124 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1124 traces tagged "gauss"
FrameSaver: q=136946 n=15868 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=156203 n=13671 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1206 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1206 traces tagged "gauss"
FrameSaver: q=153916 n=17502 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=173366 n=15232 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1607 traces, 1607 summary
Retagger: tagging trace set: gauss with 1191 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1191 traces tagged "gauss"
FrameSaver: q=137922 n=16069 tag=gauss
wclsFrameSaver: saving 1607 traces tagged "wiener"
FrameSaver: q=156611 n=14025 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1654 traces, 1654 summary
Retagger: tagging trace set: gauss with 1292 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1292 traces tagged "gauss"
FrameSaver: q=139466 n=17072 tag=gauss
wclsFrameSaver: saving 1654 traces tagged "wiener"
FrameSaver: q=156788 n=15127 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1609 traces, 1609 summary
Retagger: tagging trace set: gauss with 1228 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1228 traces tagged "gauss"
FrameSaver: q=150342 n=17341 tag=gauss
wclsFrameSaver: saving 1609 traces tagged "wiener"
FrameSaver: q=166693 n=15396 tag=wiener
27-Mar-2025 13:10:05 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358068_158_1743076376_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:10:05 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358068_158_1743076376_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       38.1413       45.5541       49.972        46.8496       3.84703        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000368642   0.000549921   0.00101344    0.000509768   0.000171551      10     
reco:rns:RandomNumberSaver                      4.34e-05     8.66005e-05   0.000403063   5.39655e-05   0.00010565       10     
reco:wclsdatavd:WireCellToolkit                  37.8208       44.9172       49.4911       46.5047       3.90002        10     
reco:gaushit:GausHitFinder                      0.0159854     0.0219577     0.0279382     0.0227918     0.0043642       10     
reco:reco3d:SpacePointSolver                    0.0520229     0.0735763     0.0938845     0.0751635     0.0125331       10     
reco:hitpdune:DisambigFromSpacePoints          0.00151117    0.00259491     0.0050064    0.00244793    0.00096835       10     
reco:pandora:StandardPandora                     0.14074      0.488639       2.64036      0.267314      0.719181        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0014377    0.00262396    0.00688757    0.00213526    0.00148681       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000168561   0.000334984   0.000941146   0.000272237   0.000218516      10     
reco:pandoraStdcalo:Calorimetry                0.00126557    0.00213366    0.00561414    0.00188486    0.00121654       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000841855   0.00151097    0.00453086    0.00127841    0.00103988       10     
[art]:TriggerResults:TriggerResultInserter      1.246e-05    2.37692e-05   7.2601e-05    1.8025e-05    1.68704e-05      10     
end_path:out1:RootOutput                        4.27e-06      9.084e-06     2.908e-05     7.28e-06     6.82972e-06      10     
end_path:out1:RootOutput(write)                 0.0313679     0.0393559     0.0474273     0.0420046    0.00621794       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3418.66 MB
  Peak resident set size usage (VmHWM): 1613.9 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_158_1743076376_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076903.5505981, 'end_time': 1743077405.838145}
justIN time: 2025-04-03 08:31:37 UTC       justIN version: 01.03.00