justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.128@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.128@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-03-27 12:35:17
From worker nodeHostnametd814.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:18:39
Input filesmonte-carlo-005746-000035
JobscriptExit code0
Real time16m (978s)
CPU time15m (955s = 97%)
Max RSS bytes1707446272 (1628 MiB)
Outputting started2025-03-27 12:34:58
Output files
Finished2025-03-27 12:35:17
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+07
4 1.50584e+07
5 1.50486e+07
Now with regularization...
Begin: 1.43359e+07
0 1.43132e+07
1 1.42973e+07
2 1.42862e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:27:13 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_128_1743077929_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:27:14 CET  Opened input file "prod_muon_400MeV_protodunevd_1_166657_128_1743077929_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1185 traces tagged "gauss"
FrameSaver: q=138875 n=16298 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=156036 n=14324 tag=wiener
27-Mar-2025 13:28:03 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_128_1743077929_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1658 traces, 1658 summary
Retagger: tagging trace set: gauss with 1210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1210 traces tagged "gauss"
FrameSaver: q=140327 n=16689 tag=gauss
wclsFrameSaver: saving 1658 traces tagged "wiener"
FrameSaver: q=158178 n=14594 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1680 traces, 1680 summary
Retagger: tagging trace set: gauss with 1216 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1216 traces tagged "gauss"
FrameSaver: q=142407 n=16745 tag=gauss
wclsFrameSaver: saving 1680 traces tagged "wiener"
FrameSaver: q=161330 n=14229 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1738 traces, 1738 summary
Retagger: tagging trace set: gauss with 1300 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1300 traces tagged "gauss"
FrameSaver: q=154486 n=17890 tag=gauss
wclsFrameSaver: saving 1738 traces tagged "wiener"
FrameSaver: q=173002 n=15699 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1564 traces, 1564 summary
Retagger: tagging trace set: gauss with 1129 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1129 traces tagged "gauss"
FrameSaver: q=139997 n=16140 tag=gauss
wclsFrameSaver: saving 1564 traces tagged "wiener"
FrameSaver: q=157219 n=13945 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1577 traces, 1577 summary
Retagger: tagging trace set: gauss with 1123 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1123 traces tagged "gauss"
FrameSaver: q=136589 n=15714 tag=gauss
wclsFrameSaver: saving 1577 traces tagged "wiener"
FrameSaver: q=155631 n=13633 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1502 traces, 1502 summary
Retagger: tagging trace set: gauss with 1059 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1059 traces tagged "gauss"
FrameSaver: q=135294 n=15158 tag=gauss
wclsFrameSaver: saving 1502 traces tagged "wiener"
FrameSaver: q=153660 n=13022 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1636 traces, 1636 summary
Retagger: tagging trace set: gauss with 1183 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1183 traces tagged "gauss"
FrameSaver: q=139431 n=16215 tag=gauss
wclsFrameSaver: saving 1636 traces tagged "wiener"
FrameSaver: q=158977 n=14009 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1685 traces, 1685 summary
Retagger: tagging trace set: gauss with 1288 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1288 traces tagged "gauss"
FrameSaver: q=148083 n=17932 tag=gauss
wclsFrameSaver: saving 1685 traces tagged "wiener"
FrameSaver: q=165139 n=15788 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1729 traces, 1729 summary
Retagger: tagging trace set: gauss with 1240 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1240 traces tagged "gauss"
FrameSaver: q=145409 n=17692 tag=gauss
wclsFrameSaver: saving 1729 traces tagged "wiener"
FrameSaver: q=165446 n=15069 tag=wiener
27-Mar-2025 13:34:57 CET  Closed output file "prod_muon_400MeV_protodunevd_1_166657_128_1743077929_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:34:57 CET  Closed input file "prod_muon_400MeV_protodunevd_1_166657_128_1743077929_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       43.9819       45.8596       50.7477       45.4644       2.02099        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000304683   0.000525685   0.00126527    0.000403859   0.000285671      10     
reco:rns:RandomNumberSaver                      3.188e-05    7.23985e-05   0.000380153   3.86105e-05   0.00010273       10     
reco:wclsdatavd:WireCellToolkit                  43.6734       45.3473       50.1558       45.1267       1.81841        10     
reco:gaushit:GausHitFinder                      0.0154989     0.0189363     0.0282952     0.0175003    0.00378251       10     
reco:reco3d:SpacePointSolver                    0.0532166     0.0734188     0.165969      0.0614479     0.0322786       10     
reco:hitpdune:DisambigFromSpacePoints          0.00161247    0.00221974    0.00409689    0.00188549    0.000804568      10     
reco:pandora:StandardPandora                    0.188062      0.374643       1.70067      0.216259      0.444087        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00125502    0.00239524    0.00517241    0.00207072    0.00108811       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000163312   0.000368397   0.00145463    0.000181641   0.000392664      10     
reco:pandoraStdcalo:Calorimetry                0.00115103    0.00156618    0.00372493    0.00123711    0.000757397      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000813557   0.00112117    0.00267293    0.000885563   0.000545681      10     
[art]:TriggerResults:TriggerResultInserter      1.152e-05    2.02181e-05    5.152e-05     1.414e-05    1.30327e-05      10     
end_path:out1:RootOutput                        3.02e-06     5.9432e-06     1.934e-05     4.636e-06    4.60721e-06      10     
end_path:out1:RootOutput(write)                 0.0292265     0.0332781     0.045932      0.0321119     0.0047834       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3455.68 MB
  Peak resident set size usage (VmHWM): 1602.3 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_128_1743077929_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078397.2220857, 'end_time': 1743078898.3602808}
justIN time: 2025-04-03 08:31:35 UTC       justIN version: 01.03.00