Jobsub ID 108735.95@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108735.95@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:18:15
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 05:31:09
From worker node	Hostname	n258.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:30:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_820_20231201T210815Z_gen_g4_detsim_hitreco__20240507T223623Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3634s)
	CPU time	54m (3277s = 90%)
Outputting started	2024-11-23 05:30:54
Output files
Finished	2024-11-23 05:31:09
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ed hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:15 GMT run: 6390196 subRun: 1 event: 82100
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 1527, 980, 1371
minwire 0: 1366
minwire 1: 239
minwire 2: 1129
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0278228, 0.0116403, 0.960537, 
Output 1: 0.609674, 0.315472, 0.063754, 0.0110994, 
Output 2: 0.811823, 0.188177, 

766
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2950!
23-Nov-2024 05:30:53 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_820_20231201T210815Z_gen_g4_detsim_hitreco__20240507T223623Z_reco2_reco_data_2024-11-23T_043023Z.root"
23-Nov-2024 05:30:53 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/7b/06/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_820_20231201T210815Z_gen_g4_detsim_hitreco__20240507T223623Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.41817       32.5317       1055.51       9.62362       110.909        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0178314     0.029076      0.0582683     0.0349255     0.0105088       100    
reco:linecluster:LineCluster                           0.160652      0.959423       66.047       0.184098       6.55681        100    
reco:trajcluster:TrajCluster                           0.0509389      1.28872       82.7296      0.0607463      8.3247         100    
reco:pandora:StandardPandora                            1.07442       4.84446       312.608       1.19076       31.0258        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000140064   0.00599635     0.117222     0.00126057     0.0169567       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000186392    0.0113511     0.200415     0.00268123     0.0272087       100    
reco:pandoracalo:Calorimetry                          0.000108836    0.0259635      1.23639     0.00102699     0.131272        100    
reco:pandorapid:Chi2ParticleID                        4.4203e-05    0.000165261   0.00113867    0.000130165   0.000150792      100    
reco:trkshowersplit:TrackShowerHits                   0.000125056    0.0380398      1.85641     0.000718976    0.195196        100    
reco:pmtrack:PMAlgTrackMaker                          0.00122016      3.0129        131.429      0.180893       13.7766        100    
reco:pmtrackcalo:Calorimetry                          7.6334e-05     0.0116328     0.411393     0.000974698    0.0467825       100    
reco:pmtrackpid:Chi2ParticleID                        3.1199e-05    0.00030282    0.00369681    0.000160193   0.000529512      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000179408     2.47143       78.2202      0.262059       8.87848        100    
reco:pmtrajfitcalo:Calorimetry                        6.8699e-05     0.0263346      1.21122     0.00128205     0.129119        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8885e-05    0.000199226   0.000922819   0.000154702   0.000170186      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00104047      3.42737       131.705      0.218782       14.1946        100    
reco:pmtracktccalo:Calorimetry                        7.5102e-05     0.0072419     0.163348     0.00082735     0.022968        100    
reco:pmtracktcpid:Chi2ParticleID                       3.149e-05    0.000285211   0.00292301    0.000141787   0.000468409      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000173807     1.72644       45.1228      0.268715       5.26978        100    
reco:pmtrajfittccalo:Calorimetry                      6.8269e-05     0.0172136     0.621928     0.00103522     0.0707743       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8183e-05    0.00039108    0.00476631    0.000158795   0.000715266      100    
reco:blurredcluster:BlurredClustering                  0.0081695     0.784825       7.91727      0.303548       1.23127        100    
reco:emtrkmichelid:EmTrackMichelId                     0.136569       10.6552       183.845       2.82865       24.9436        100    
reco:emshower:EMShower                                0.000310114    0.0299882     0.319399     0.00551868     0.0553387       100    
reco:cvnmap:CVNMapper                                 2.6249e-05     0.0102423     0.0439675     0.0073962     0.0100747       100    
reco:cvneva:CVNEvaluator                              1.7844e-05     0.567174       1.16094      0.778961       0.37555        100    
reco:energyrecnumu:EnergyReco                         0.000270149    0.0388197      1.07712     0.00423716     0.149535        100    
reco:energyrecnue:EnergyReco                          0.000145885   0.00198792     0.0355427    0.000473888   0.00528042       100    
reco:energyrecnc:EnergyReco                           0.00251982    0.00449214     0.0392764    0.00300493    0.00553896       100    
reco:energyrecnumurange:EnergyReco                    0.000147928    0.0366803      1.07834     0.00270039     0.147769        100    
reco:energyrecnumumcs:EnergyReco                       0.0019332     0.0384307      1.04567     0.00348755      0.14714        100    
reco:opdec:Deconvolution                               0.084874      0.277919      0.541172      0.251892      0.097071        100    
reco:ophitspe:OpHitFinderDeco                          0.322488       1.93756       3.86567       1.75295      0.833797        100    
reco:opflash:OpFlashFinder                            0.000371591   0.00138575     0.0037966    0.00118547    0.000680583      100    
reco:rns:RandomNumberSaver                            1.9597e-05    3.80313e-05   0.000294655   3.58425e-05   2.79126e-05      100    
reco:anglereconue:NuAngularReco                       0.000168948   0.00148015     0.0269522    0.000382286   0.00417667       100    
reco:anglereconumu:NuAngularReco                      8.6603e-05    0.00133029     0.0290915    0.000238384   0.00430113       100    
reco:anglereconuepfps:NuAngularReco                   0.000141687   0.00143549     0.0254997    0.000347565   0.00406538       100    
reco:anglereconumupfps:NuAngularReco                  0.000118083   0.00137564     0.0270355    0.000289922   0.00418342       100    
[art]:TriggerResults:TriggerResultInserter             9.448e-06    1.35514e-05   5.0204e-05    1.2258e-05    5.60596e-06      100    
end_path:out1:RootOutput                               2.645e-06    5.60396e-06   1.7523e-05     5.811e-06    2.43798e-06      100    
end_path:out1:RootOutput(write)                        0.0228628     0.234344       1.19714      0.234407      0.182207        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38519.5 MB
  Peak resident set size usage (VmHWM): 2447.74 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_820_20231201T210815Z_gen_g4_detsim_hitreco__20240507T223623Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_820_20231201T210815Z_gen_g4_detsim_hitreco__20240507T223623Z_reco2_reco_2024-11-23T_043023Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_820_20231201T210815Z_gen_g4_detsim_hitreco__20240507T223623Z_reco2_reco_data_2024-11-23T_043023Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work