Jobsub ID 108736.85@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108736.85@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:19:19
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 05:30:20
From worker node	Hostname	n246.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:29:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_665_20231207T062654Z_gen_g4_detsim_hitreco__20240510T044847Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3661s)
	CPU time	48m (2908s = 79%)
Outputting started	2024-11-23 05:30:07
Output files
Finished	2024-11-23 05:30:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

eginModule  23-Nov-2024 05:30:00 GMT run: 6467222 subRun: 1 event: 66600
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:00 GMT run: 6467222 subRun: 1 event: 66600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:00 GMT run: 6467222 subRun: 1 event: 66600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:01 GMT run: 6467222 subRun: 1 event: 66600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:01 GMT run: 6467222 subRun: 1 event: 66600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:30:03 GMT run: 6467222 subRun: 1 event: 66600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:30:03 GMT run: 6467222 subRun: 1 event: 66600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 165, 87, 97
minwire 0: 1287
minwire 1: 1109
minwire 2: 1351
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0192815, 0.000567499, 0.980151, 
Output 1: 0.931139, 0.0683431, 0.000495817, 2.23059e-05, 
Output 2: 0.979976, 0.0200238, 

488
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1573!
23-Nov-2024 05:30:06 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_665_20231207T062654Z_gen_g4_detsim_hitreco__20240510T044847Z_reco2_reco_data_2024-11-23T_042910Z.root"
23-Nov-2024 05:30:06 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/60/f1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_665_20231207T062654Z_gen_g4_detsim_hitreco__20240510T044847Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.68232       29.4615       888.795       9.83128       90.4418        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0346654     0.0700456      0.18536      0.0684963     0.0227463       100    
reco:linecluster:LineCluster                           0.341416       0.87963       41.0685      0.373812       4.06322        100    
reco:trajcluster:TrajCluster                           0.101211      0.935438       58.3535      0.111453       5.83579        100    
reco:pandora:StandardPandora                           0.877891       2.94951       145.517       1.37108       14.3534        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000118043    0.0067807     0.215205     0.00151777     0.022254        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000152679    0.0172789     0.482487     0.00289328     0.0541356       100    
reco:pandoracalo:Calorimetry                          0.000110329    0.0248323      1.68735     0.00105256     0.168028        100    
reco:pandorapid:Chi2ParticleID                        3.6809e-05    0.000224376   0.00542196    0.000121455   0.000551999      100    
reco:trkshowersplit:TrackShowerHits                   0.000109527    0.027246       1.66181     0.000615951    0.167121        100    
reco:pmtrack:PMAlgTrackMaker                          0.000892228     2.7553        125.444      0.179417       12.7224        100    
reco:pmtrackcalo:Calorimetry                          7.6205e-05     0.0131013     0.758192     0.000910838    0.0760832       100    
reco:pmtrackpid:Chi2ParticleID                        2.5758e-05    0.000296674    0.0050571    0.000143757   0.000549769      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000158199     2.15814       77.622       0.334161       8.06401        100    
reco:pmtrajfitcalo:Calorimetry                        5.8781e-05     0.0254895      1.69962      0.0013622     0.169226        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3515e-05    0.000198764   0.00127377    0.000143657   0.000173405      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00112461      2.96951       129.106      0.270576       13.1616        100    
reco:pmtracktccalo:Calorimetry                        7.9831e-05    0.00836807      0.38294     0.000834969    0.0389203       100    
reco:pmtracktcpid:Chi2ParticleID                      3.8433e-05    0.000281027   0.00394201    0.000146943   0.000445563      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000196381     1.69526       64.6971      0.338231       6.56261        100    
reco:pmtrajfittccalo:Calorimetry                      7.5403e-05     0.0161589      0.94044     0.000935955    0.0940976       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5496e-05    0.000360076    0.0061092    0.000156091   0.00066304       100    
reco:blurredcluster:BlurredClustering                 0.00807694      1.01962       10.3004      0.461066       1.49608        100    
reco:emtrkmichelid:EmTrackMichelId                     0.224244       10.5107       216.113       2.36803       24.1983        100    
reco:emshower:EMShower                                0.000243721    0.0259479     0.457611     0.00533589     0.0549171       100    
reco:cvnmap:CVNMapper                                 2.2943e-05     0.011844      0.0547942     0.0129532     0.0100908       100    
reco:cvneva:CVNEvaluator                               1.558e-05     0.572328       1.13949      0.780084      0.382738        100    
reco:energyrecnumu:EnergyReco                         0.000286903    0.0511234      1.43879     0.00437996     0.182315        100    
reco:energyrecnue:EnergyReco                          0.000119065   0.00193481     0.0615089    0.000473265    0.0063211       100    
reco:energyrecnc:EnergyReco                           0.00139421    0.00414356     0.0660952    0.00290606    0.00662719       100    
reco:energyrecnumurange:EnergyReco                    0.000134865    0.0490755      1.44644     0.00278676     0.182458        100    
reco:energyrecnumumcs:EnergyReco                      0.00140906     0.0503006      1.42412     0.00342196     0.180631        100    
reco:opdec:Deconvolution                               0.207191      0.417263       0.83399      0.409928      0.128691        100    
reco:ophitspe:OpHitFinderDeco                          0.666128       1.95135       4.70804       1.69202       0.94742        100    
reco:opflash:OpFlashFinder                            0.000442336   0.00139878    0.00445572    0.00117033    0.000804282      100    
reco:rns:RandomNumberSaver                            1.6812e-05    4.00591e-05   0.000406238   3.71755e-05   3.87924e-05      100    
reco:anglereconue:NuAngularReco                       0.000146356   0.00162019     0.0540568    0.000425179   0.00552099       100    
reco:anglereconumu:NuAngularReco                      5.9823e-05    0.00134982     0.0535351    0.000202874   0.00546291       100    
reco:anglereconuepfps:NuAngularReco                   9.8836e-05    0.00151448     0.0539517    0.000377814   0.00549729       100    
reco:anglereconumupfps:NuAngularReco                  7.8077e-05    0.00141777     0.0534818    0.000280551   0.00546174       100    
[art]:TriggerResults:TriggerResultInserter             7.915e-06    1.30893e-05    6.365e-05    1.1767e-05    6.44233e-06      100    
end_path:out1:RootOutput                               2.465e-06    6.40526e-06   2.4396e-05     6.057e-06    3.47891e-06      100    
end_path:out1:RootOutput(write)                        0.0304223     0.233147       1.47636      0.238366      0.187834        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38594 MB
  Peak resident set size usage (VmHWM): 2519.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_665_20231207T062654Z_gen_g4_detsim_hitreco__20240510T044847Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_665_20231207T062654Z_gen_g4_detsim_hitreco__20240510T044847Z_reco2_reco_2024-11-23T_042910Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_665_20231207T062654Z_gen_g4_detsim_hitreco__20240510T044847Z_reco2_reco_data_2024-11-23T_042910Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work