Jobsub ID 301703.9@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301703.9@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:29:03
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:33:53
From worker node	Hostname	n253.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:29:53
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_666_20231208T033453Z_gen_g4_detsim_hitreco__20240510T064906Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3826s)
	CPU time	56m (3413s = 89%)
Outputting started	2024-11-23 05:33:40
Output files
Finished	2024-11-23 05:33:53
List job events Wrapper job log
Jobscript log (last 10,000 characters)

er: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:28 GMT run: 50642276 subRun: 1 event: 66700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:28 GMT run: 50642276 subRun: 1 event: 66700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:29 GMT run: 50642276 subRun: 1 event: 66700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:29 GMT run: 50642276 subRun: 1 event: 66700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:29 GMT run: 50642276 subRun: 1 event: 66700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:29 GMT run: 50642276 subRun: 1 event: 66700
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:33:33 GMT run: 50642276 subRun: 1 event: 66700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 192, 160, 168
minwire 0: 936
minwire 1: 1487
minwire 2: 694
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0645996, 0.930765, 0.00463498, 
Output 1: 0.961881, 0.0380311, 8.61729e-05, 1.70778e-06, 
Output 2: 0.999278, 0.000721631, 

606
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2015!
23-Nov-2024 05:33:38 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_666_20231208T033453Z_gen_g4_detsim_hitreco__20240510T064906Z_reco2_reco_data_2024-11-23T_042958Z.root"
23-Nov-2024 05:33:38 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/91/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_666_20231208T033453Z_gen_g4_detsim_hitreco__20240510T064906Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.596816       34.0144       442.343       9.73268       67.764         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179338     0.0374971     0.0877471     0.0367042     0.0152708       100    
reco:linecluster:LineCluster                           0.101157      0.525807       21.9405      0.186274       2.19603        100    
reco:trajcluster:TrajCluster                           0.0507318     0.845787       21.7828      0.0612412      3.13692        100    
reco:pandora:StandardPandora                           0.0599413      2.66877       48.3485       1.26292       6.10713        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00015307    0.00817625     0.181637     0.00160826     0.021712        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000203926    0.0167786     0.216652     0.00303835     0.0373688       100    
reco:pandoracalo:Calorimetry                          0.000109467    0.0272961     0.879026     0.00124017     0.102607        100    
reco:pandorapid:Chi2ParticleID                        4.6568e-05    0.000191356   0.00185565    0.000132517   0.000217899      100    
reco:trkshowersplit:TrackShowerHits                   9.1684e-05     0.033429      0.824647     0.000794635    0.120755        100    
reco:pmtrack:PMAlgTrackMaker                          0.00141719      3.37015       85.3894      0.221172       11.1826        100    
reco:pmtrackcalo:Calorimetry                          8.1294e-05     0.0166376     0.459349     0.00103265     0.0552302       100    
reco:pmtrackpid:Chi2ParticleID                        3.1951e-05    0.00037457    0.00319096    0.000151813   0.000572711      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00021081      2.43146       44.1569      0.357336       5.68642        100    
reco:pmtrajfitcalo:Calorimetry                        6.9541e-05     0.0252273     0.533499     0.00177064     0.078496        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9155e-05    0.000228171   0.00147449    0.00017161    0.000204788      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0012151      3.73592       79.9982      0.213456       10.2852        100    
reco:pmtracktccalo:Calorimetry                        8.3128e-05     0.0117051     0.309273     0.00102641     0.0372322       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5267e-05    0.000336627   0.00312757    0.00016371    0.000456903      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000196262     2.16268       40.4639      0.356099       5.30383        100    
reco:pmtrajfittccalo:Calorimetry                      8.1524e-05     0.0219653     0.606343     0.00133572     0.0734413       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4065e-05    0.000468341   0.00491772    0.000191267   0.000738394      100    
reco:blurredcluster:BlurredClustering                  8.427e-05      1.12245       6.09471      0.650901       1.2984         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000128443     13.2587       162.224       2.8714        24.8783        100    
reco:emshower:EMShower                                0.000190301    0.0371844     0.469117     0.00787276     0.0681006       100    
reco:cvnmap:CVNMapper                                 2.1641e-05     0.0100667     0.0473137    0.00786739    0.00872067       100    
reco:cvneva:CVNEvaluator                              1.9437e-05     0.635316       1.14939      0.807265      0.353844        100    
reco:energyrecnumu:EnergyReco                         0.00032173     0.0456837      2.14048     0.00498414     0.215639        100    
reco:energyrecnue:EnergyReco                          0.000180462   0.00271219     0.0597213    0.000476628   0.00709337       100    
reco:energyrecnc:EnergyReco                           0.00180293    0.00543451     0.0626873    0.00328229     0.0073088       100    
reco:energyrecnumurange:EnergyReco                    0.000180472    0.0441255      2.18109      0.0030159     0.219687        100    
reco:energyrecnumumcs:EnergyReco                      0.00237352     0.0463659      2.21404      0.0040967     0.222713        100    
reco:opdec:Deconvolution                               0.0569883     0.298241      0.549792      0.292464      0.104993        100    
reco:ophitspe:OpHitFinderDeco                          0.214912       2.27306       4.43565       2.23531      0.980074        100    
reco:opflash:OpFlashFinder                            0.000264401    0.0015218    0.00554119     0.0014147    0.000763322      100    
reco:rns:RandomNumberSaver                            2.0709e-05    3.59802e-05   0.000381673   2.95165e-05   3.57788e-05      100    
reco:anglereconue:NuAngularReco                       0.000173879   0.00217071     0.0531136    0.000400293   0.00624295       100    
reco:anglereconumu:NuAngularReco                      8.6865e-05    0.00195874     0.0520444    0.000275908   0.00612341       100    
reco:anglereconuepfps:NuAngularReco                   0.000141859   0.00209519     0.0516007    0.000372622   0.00609668       100    
reco:anglereconumupfps:NuAngularReco                  0.000139494   0.00203259     0.0521051    0.000323042   0.00613049       100    
[art]:TriggerResults:TriggerResultInserter             9.307e-06    1.50224e-05   0.00016878    1.2859e-05    1.56261e-05      100    
end_path:out1:RootOutput                               2.996e-06    6.45392e-06   6.0695e-05     5.155e-06    6.25908e-06      100    
end_path:out1:RootOutput(write)                        0.0130486     0.282332       1.2784       0.263035      0.206503        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38366.8 MB
  Peak resident set size usage (VmHWM): 2306.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_666_20231208T033453Z_gen_g4_detsim_hitreco__20240510T064906Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_666_20231208T033453Z_gen_g4_detsim_hitreco__20240510T064906Z_reco2_reco_2024-11-23T_042958Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_666_20231208T033453Z_gen_g4_detsim_hitreco__20240510T064906Z_reco2_reco_data_2024-11-23T_042958Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work