Jobsub ID 108517.123@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108517.123@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 22:34:24
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2024-11-22 23:25:20
From worker node	Hostname	wn1904250.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 22:40:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_535_20231208T052338Z_gen_g4_detsim_hitreco__20240510T065446Z_reco2.root
Outputting started	2024-11-22 23:21:20
Output files
Finished	2024-11-22 23:25:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

bled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:59 GMT run: 50649927 subRun: 1 event: 53600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:59 GMT run: 50649927 subRun: 1 event: 53600
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:12 GMT run: 50649927 subRun: 1 event: 53600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:12 GMT run: 50649927 subRun: 1 event: 53600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:12 GMT run: 50649927 subRun: 1 event: 53600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:12 GMT run: 50649927 subRun: 1 event: 53600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:12 GMT run: 50649927 subRun: 1 event: 53600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 609, 632, 395
minwire 0: 1859
minwire 1: 1603
minwire 2: 1488
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.107421, 0.840956, 0.0516234, 
Output 1: 0.178152, 0.530643, 0.258806, 0.0323991, 
Output 2: 0.792829, 0.207171, 

717
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3096!
22-Nov-2024 23:21:18 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_535_20231208T052338Z_gen_g4_detsim_hitreco__20240510T065446Z_reco2_reco_data_2024-11-22T_224039Z.root"
22-Nov-2024 23:21:18 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/e0/a9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_535_20231208T052338Z_gen_g4_detsim_hitreco__20240510T065446Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.370421       22.6069       262.25        11.4095       38.0422        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00989484     0.0303489     0.0586872     0.0315244     0.0113473       100    
reco:linecluster:LineCluster                           0.0538215     0.165708       2.2161       0.107166      0.293799        100    
reco:trajcluster:TrajCluster                           0.0268333     0.343243       11.1016      0.0432579      1.32147        100    
reco:pandora:StandardPandora                           0.0392308      1.83208       15.3943       1.48974       1.65725        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000133951   0.00525767     0.0770575    0.00194238     0.0108117       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000175269    0.0166471     0.283474     0.00416662     0.0381285       100    
reco:pandoracalo:Calorimetry                          0.000105649   0.00997611      0.21791     0.00150267     0.0288909       100    
reco:pandorapid:Chi2ParticleID                        3.9594e-05    0.000197122   0.00118753    0.000150363   0.000157818      100    
reco:trkshowersplit:TrackShowerHits                   8.9488e-05     0.0137562     0.321998     0.00105605     0.0461251       100    
reco:pmtrack:PMAlgTrackMaker                          0.00110096      1.71151       34.605       0.254422       4.90251        100    
reco:pmtrackcalo:Calorimetry                           7.456e-05    0.00686035     0.213764     0.00118676     0.0246089       100    
reco:pmtrackpid:Chi2ParticleID                        2.9075e-05    0.000284882   0.00306632    0.000165856   0.000403148      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000179467     1.26716       16.7609      0.439066       2.64544        100    
reco:pmtrajfitcalo:Calorimetry                        6.4651e-05     0.0115266     0.248892     0.00196802     0.0328467       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6059e-05    0.00022865    0.00103675    0.000184031    0.0001918       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000912184     2.02178       44.112       0.224062       6.3006         100    
reco:pmtracktccalo:Calorimetry                        6.5553e-05    0.00641201     0.147616     0.00122875     0.0190501       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5588e-05    0.000303147   0.00261982    0.00018656    0.000394532      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000191861     1.25439       18.5137      0.326847       2.82056        100    
reco:pmtrajfittccalo:Calorimetry                      8.7454e-05    0.00955675     0.221883     0.00156593     0.0283296       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4464e-05    0.000359676   0.00302264    0.000191846   0.000499482      100    
reco:blurredcluster:BlurredClustering                 0.000110929     1.08904       11.0929      0.564628       1.61793        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000125376     9.09968       107.509       3.51969       16.7219        100    
reco:emshower:EMShower                                0.000188044    0.0286297     0.336743     0.00597046     0.0529194       100    
reco:cvnmap:CVNMapper                                 1.9637e-05     0.0159306     0.0595159     0.0136796     0.0142628       100    
reco:cvneva:CVNEvaluator                              1.5239e-05     0.813282       1.62499       1.10288      0.557105        100    
reco:energyrecnumu:EnergyReco                         0.000496794    0.0248158     0.607686     0.00551806     0.0718736       100    
reco:energyrecnue:EnergyReco                          0.000149621   0.00210118     0.0261758    0.000801918   0.00377271       100    
reco:energyrecnc:EnergyReco                            0.0018199    0.00494396     0.0267616    0.00387648    0.00373175       100    
reco:energyrecnumurange:EnergyReco                    0.00027682     0.0225378      0.58326     0.00371247     0.0696074       100    
reco:energyrecnumumcs:EnergyReco                      0.00073438     0.0241109     0.582719     0.00404019     0.0694684       100    
reco:opdec:Deconvolution                                0.03781      0.239891      0.395464      0.249241      0.0837318       100    
reco:ophitspe:OpHitFinderDeco                          0.0915363      2.22014       4.31185       2.17149      0.969766        100    
reco:opflash:OpFlashFinder                            0.000237427   0.00173093    0.00384479    0.00179212    0.000722064      100    
reco:rns:RandomNumberSaver                            1.8475e-05    4.67373e-05   0.000372881   4.2635e-05    3.41761e-05      100    
reco:anglereconue:NuAngularReco                       0.00015438     0.0014528     0.0242345    0.000535247   0.00312439       100    
reco:anglereconumu:NuAngularReco                      7.0663e-05    0.00116279     0.0233416    0.000299343   0.00299503       100    
reco:anglereconuepfps:NuAngularReco                   0.000132108   0.00135713      0.02363     0.000456744   0.00301682       100    
reco:anglereconumupfps:NuAngularReco                  0.000102753   0.00125577     0.0237224    0.000386336   0.00303861       100    
[art]:TriggerResults:TriggerResultInserter             9.689e-06    1.77926e-05   7.2256e-05    1.6025e-05    7.7008e-06       100    
end_path:out1:RootOutput                               3.056e-06    9.3283e-06    2.9776e-05    9.2375e-06    3.41737e-06      100    
end_path:out1:RootOutput(write)                        0.0152714     0.304645       1.0052       0.318697      0.201707        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21685 MB
  Peak resident set size usage (VmHWM): 2232.68 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_535_20231208T052338Z_gen_g4_detsim_hitreco__20240510T065446Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_535_20231208T052338Z_gen_g4_detsim_hitreco__20240510T065446Z_reco2_reco_2024-11-22T_224039Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_535_20231208T052338Z_gen_g4_detsim_hitreco__20240510T065446Z_reco2_reco_data_2024-11-22T_224039Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work