Jobsub ID 108746.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108746.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:39:54
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:33:32
From worker node	Hostname	b9p09p9768.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:40:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1045_20231203T072503Z_gen_g4_detsim_hitreco__20240508T072817Z_reco2.root
Jobscript	Exit code	0
	Real time	52m (3161s)
	CPU time	54m (3242s = 102%)
Outputting started	2024-11-23 05:33:15
Output files
Finished	2024-11-23 05:33:32
List job events Wrapper job log
Jobscript log (last 10,000 characters)

50572032 subRun: 1 event: 104600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:04 CET run: 50572032 subRun: 1 event: 104600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:04 CET run: 50572032 subRun: 1 event: 104600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:04 CET run: 50572032 subRun: 1 event: 104600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:04 CET run: 50572032 subRun: 1 event: 104600
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:33:10 CET run: 50572032 subRun: 1 event: 104600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:33:10 CET run: 50572032 subRun: 1 event: 104600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:33:10 CET run: 50572032 subRun: 1 event: 104600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 324, 366, 380
minwire 0: 2419
minwire 1: 465
minwire 2: 2587
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0235043, 0.974616, 0.00188011, 
Output 1: 0.0287891, 0.968631, 0.00255004, 2.96505e-05, 
Output 2: 0.918806, 0.0811938, 

417
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1343!
23-Nov-2024 06:33:14 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1045_20231203T072503Z_gen_g4_detsim_hitreco__20240508T072817Z_reco2_reco_data_2024-11-23T_044038Z.root"
23-Nov-2024 06:33:14 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cf/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1045_20231203T072503Z_gen_g4_detsim_hitreco__20240508T072817Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.532413       29.6305       791.609       9.71651       83.1829        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0166671     0.0422082     0.122033      0.040275      0.0144376       100    
reco:linecluster:LineCluster                           0.0944863     0.747343       49.4491      0.179891       4.90648        100    
reco:trajcluster:TrajCluster                           0.0475469      1.60206       115.587      0.0610805      11.5237        100    
reco:pandora:StandardPandora                           0.056597       2.18375       62.8529       1.30565       6.20517        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000126671   0.00547344     0.0709257    0.00163519     0.0101296       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000175192   0.00911023     0.0852274    0.00310798     0.0145887       100    
reco:pandoracalo:Calorimetry                           9.71e-05      0.0215968     0.941628     0.00102062     0.098215        100    
reco:pandorapid:Chi2ParticleID                          3.4e-05     0.000166336    0.0012895    0.000125226   0.000146914      100    
reco:trkshowersplit:TrackShowerHits                    7.462e-05     0.0315613      1.6167      0.000718925    0.164575        100    
reco:pmtrack:PMAlgTrackMaker                          0.00102472      3.19723       151.722      0.267652       15.3424        100    
reco:pmtrackcalo:Calorimetry                           6.237e-05    0.00938236     0.215152     0.000852252    0.0264428       100    
reco:pmtrackpid:Chi2ParticleID                        2.5191e-05    0.000299913   0.00198024    0.000144091   0.000397012      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000223632     2.71607       88.6203      0.289461       9.73502        100    
reco:pmtrajfitcalo:Calorimetry                        8.3391e-05     0.0270283      1.50335     0.00158448     0.151191        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.565e-05    0.000195728   0.00121796    0.000146606   0.00016725       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00124694      3.47051       135.717      0.266125       14.1125        100    
reco:pmtracktccalo:Calorimetry                         6.226e-05    0.00558901     0.0656401    0.000974137    0.0112729       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3801e-05    0.000271252   0.00160615    0.000161891   0.000312176      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000204712     1.51049       32.7719      0.280449       3.77694        100    
reco:pmtrajfittccalo:Calorimetry                       7.633e-05     0.0126457     0.385977     0.00123153     0.0421163       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.204e-05    0.000346433   0.00312957    0.000165726   0.000470458      100    
reco:blurredcluster:BlurredClustering                 0.000110011     1.02516       6.25149      0.460817       1.34188        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000143741     9.84018       137.059       2.59787       18.8802        100    
reco:emshower:EMShower                                0.000215292    0.025547      0.227439     0.00641975     0.041058        100    
reco:cvnmap:CVNMapper                                  2.185e-05    0.00929219     0.0319842     0.010958     0.00709593       100    
reco:cvneva:CVNEvaluator                               1.523e-05     0.667764        1.173       0.784794      0.414602        100    
reco:energyrecnumu:EnergyReco                         0.000315033    0.029944       0.73057     0.00486333     0.0966585       100    
reco:energyrecnue:EnergyReco                          0.000127851   0.00199768     0.0223852    0.000496319   0.00367314       100    
reco:energyrecnc:EnergyReco                           0.00166566    0.00437267     0.0339191    0.00323622    0.00436097       100    
reco:energyrecnumurange:EnergyReco                    0.000146411    0.0287125      0.77995     0.00301602      0.10403        100    
reco:energyrecnumumcs:EnergyReco                      0.00100621     0.0284264      0.78846     0.00358517     0.095079        100    
reco:opdec:Deconvolution                               0.0517278      0.27607      0.468461      0.280552      0.0955884       100    
reco:ophitspe:OpHitFinderDeco                          0.150509       1.85561       4.2444        1.7397       0.868553        100    
reco:opflash:OpFlashFinder                            0.000156981   0.00147285     0.0039116    0.00140914    0.000737528      100    
reco:rns:RandomNumberSaver                             1.682e-05    3.6594e-05    0.000318282     3.3e-05     3.04169e-05      100    
reco:anglereconue:NuAngularReco                       0.000155072   0.00143931     0.0235512    0.000448838   0.00297674       100    
reco:anglereconumu:NuAngularReco                       7.258e-05    0.00121287     0.0232114    0.000222187   0.00293188       100    
reco:anglereconuepfps:NuAngularReco                   0.000127421   0.00137917     0.0238392    0.000383753   0.00298455       100    
reco:anglereconumupfps:NuAngularReco                  0.000104471    0.001282      0.0231765    0.000319257   0.00293986       100    
[art]:TriggerResults:TriggerResultInserter             8.68e-06     1.39503e-05    3.973e-05     1.306e-05    4.64531e-06      100    
end_path:out1:RootOutput                               3.13e-06     7.57139e-06    2.179e-05     7.546e-06    3.43322e-06      100    
end_path:out1:RootOutput(write)                        0.0122006     0.235272       1.20756      0.207279      0.178865        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21659.9 MB
  Peak resident set size usage (VmHWM): 2250.54 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1045_20231203T072503Z_gen_g4_detsim_hitreco__20240508T072817Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1045_20231203T072503Z_gen_g4_detsim_hitreco__20240508T072817Z_reco2_reco_2024-11-23T_044038Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1045_20231203T072503Z_gen_g4_detsim_hitreco__20240508T072817Z_reco2_reco_data_2024-11-23T_044038Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work