Jobsub ID 108728.13@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108728.13@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:01:57
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:18:12
From worker node	Hostname	n257.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:04:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_750_20231203T132606Z_gen_g4_detsim_hitreco__20240508T052429Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4429s)
	CPU time	1h (3676s = 82%)
Outputting started	2024-11-23 05:17:57
Output files
Finished	2024-11-23 05:18:12
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
Cannot attach back to inner node of other track.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:36 GMT run: 50574832 subRun: 1 event: 75100
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 706, 802, 748
minwire 0: 274
minwire 1: 2108
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.13078, 0.652122, 0.217098, 
Output 1: 0.573818, 0.375815, 0.0429611, 0.00740681, 
Output 2: 0.0787667, 0.921233, 

556
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1706!
23-Nov-2024 05:17:56 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_750_20231203T132606Z_gen_g4_detsim_hitreco__20240508T052429Z_reco2_reco_data_2024-11-23T_040411Z.root"
23-Nov-2024 05:17:56 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/26/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_750_20231203T132606Z_gen_g4_detsim_hitreco__20240508T052429Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.29325       37.2072       296.642       11.1038       65.281         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0350549     0.0685965     0.111337      0.0775739     0.0190957       100    
reco:linecluster:LineCluster                            0.21415      0.594234       6.42347      0.363293      0.838005        100    
reco:trajcluster:TrajCluster                           0.100923       1.12424       14.5181      0.119005       3.03488        100    
reco:pandora:StandardPandora                           0.114119       2.72599       66.8308       1.27872       6.87078        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000151277   0.00838943     0.0843312    0.00188071     0.0174143       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000214828    0.0216165     0.278583     0.00334096     0.0479116       100    
reco:pandoracalo:Calorimetry                          0.000118556    0.0246297      0.26199     0.00139125     0.0589664       100    
reco:pandorapid:Chi2ParticleID                        4.3332e-05    0.000198438   0.00206017    0.000148943   0.000235508      100    
reco:trkshowersplit:TrackShowerHits                   0.000132652    0.0393104     0.499671     0.00119048      0.10947        100    
reco:pmtrack:PMAlgTrackMaker                          0.00130982      3.68751       40.2597      0.246145       8.60436        100    
reco:pmtrackcalo:Calorimetry                          7.8409e-05     0.0140247     0.165179     0.000998943    0.0348247       100    
reco:pmtrackpid:Chi2ParticleID                        3.0849e-05    0.000372271   0.00242817    0.000161272   0.000542424      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000219266     3.06412       31.5801       0.36712       6.65814        100    
reco:pmtrajfitcalo:Calorimetry                        7.8679e-05     0.0252286     0.266354     0.00193774     0.0597785       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6008e-05    0.000219809   0.000881284   0.000170028   0.000185407      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0011287      4.47789       51.6247      0.203756       10.8901        100    
reco:pmtracktccalo:Calorimetry                        7.4312e-05     0.011114      0.185617     0.00114759     0.0287963       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9746e-05    0.000329378   0.00217569    0.000171155   0.000410934      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000200741     2.19535       21.8186       0.36055       4.38731        100    
reco:pmtrajfittccalo:Calorimetry                      7.4973e-05     0.020818      0.349629     0.00147875     0.0539694       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.1861e-05    0.000459525   0.00314526    0.000199955   0.00065904       100    
reco:blurredcluster:BlurredClustering                 8.3468e-05      1.2104        8.78035      0.595128       1.59011        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000133504     13.9568       122.751       3.70403       24.8606        100    
reco:emshower:EMShower                                0.000193678     0.03479      0.246582     0.00730757     0.0548476       100    
reco:cvnmap:CVNMapper                                 2.2694e-05    0.00897149     0.0482524    0.00795178    0.00820145       100    
reco:cvneva:CVNEvaluator                              2.0438e-05     0.644043       1.19101      0.811126       0.37827        100    
reco:energyrecnumu:EnergyReco                         0.000355295    0.0444371      1.34308     0.00455611     0.151437        100    
reco:energyrecnue:EnergyReco                          0.000159463   0.00264792     0.0267089    0.000586044   0.00525068       100    
reco:energyrecnc:EnergyReco                            0.0026506    0.00539567     0.0298888    0.00326325    0.00555403       100    
reco:energyrecnumurange:EnergyReco                    0.000170374    0.0421764      1.33152     0.00289233     0.150235        100    
reco:energyrecnumumcs:EnergyReco                      0.00167943     0.0439685      1.33234     0.00415608     0.150569        100    
reco:opdec:Deconvolution                               0.124237      0.424531      0.665985      0.417212      0.108126        100    
reco:ophitspe:OpHitFinderDeco                           0.4921        2.3983        4.31131       2.44561       0.85861        100    
reco:opflash:OpFlashFinder                            0.000318295   0.00153865    0.00696322    0.00149436    0.000821159      100    
reco:rns:RandomNumberSaver                            1.8976e-05    4.09259e-05   0.000540927   3.4807e-05    5.13408e-05      100    
reco:anglereconue:NuAngularReco                       0.000186514   0.00204201     0.0240349    0.000455385    0.0044134       100    
reco:anglereconumu:NuAngularReco                      8.2167e-05    0.00183296     0.0223213    0.000287651   0.00430936       100    
reco:anglereconuepfps:NuAngularReco                   0.000138233   0.00197993     0.0228767    0.000414512   0.00431225       100    
reco:anglereconumupfps:NuAngularReco                  0.00012626    0.00189972     0.0222045    0.000374837    0.0042962       100    
[art]:TriggerResults:TriggerResultInserter             9.027e-06    1.32337e-05   6.2178e-05    1.2003e-05    5.74371e-06      100    
end_path:out1:RootOutput                               2.835e-06    6.40002e-06   3.8563e-05     6.608e-06    4.17236e-06      100    
end_path:out1:RootOutput(write)                        0.0190018     0.274693      0.963408      0.253664      0.191182        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38386.7 MB
  Peak resident set size usage (VmHWM): 2325.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_750_20231203T132606Z_gen_g4_detsim_hitreco__20240508T052429Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_750_20231203T132606Z_gen_g4_detsim_hitreco__20240508T052429Z_reco2_reco_2024-11-23T_040411Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_750_20231203T132606Z_gen_g4_detsim_hitreco__20240508T052429Z_reco2_reco_data_2024-11-23T_040411Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work