Jobsub ID 108737.72@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108737.72@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:21:28
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:01:23
From worker node	Hostname	b9p08p3365.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:27:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_893_20231203T120748Z_gen_g4_detsim_hitreco__20240509T192551Z_reco2.root
Jobscript	Exit code	0
	Real time	33m (1994s)
	CPU time	35m (2154s = 108%)
Outputting started	2024-11-23 05:01:09
Output files
Finished	2024-11-23 05:01:23
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Fitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:48 CET run: 74510902 subRun: 1 event: 89400
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:48 CET run: 74510902 subRun: 1 event: 89400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:01:05 CET run: 74510902 subRun: 1 event: 89400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:01:05 CET run: 74510902 subRun: 1 event: 89400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:01:05 CET run: 74510902 subRun: 1 event: 89400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:01:05 CET run: 74510902 subRun: 1 event: 89400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:01:05 CET run: 74510902 subRun: 1 event: 89400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1408, 1155, 796
minwire 0: 2414
minwire 1: 280
minwire 2: 2425
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.11345, 0.875025, 0.0115252, 
Output 1: 0.664825, 0.328355, 0.00620078, 0.000619143, 
Output 2: 0.767077, 0.232923, 

618
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2188!
23-Nov-2024 06:01:08 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_893_20231203T120748Z_gen_g4_detsim_hitreco__20240509T192551Z_reco2_reco_data_2024-11-23T_042759Z.root"
23-Nov-2024 06:01:09 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/1d/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_893_20231203T120748Z_gen_g4_detsim_hitreco__20240509T192551Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.499454       18.0887       341.873       4.73075       44.9241        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0164114     0.0463327      0.81026      0.0382731     0.0775422       100    
reco:linecluster:LineCluster                           0.0934595     0.276462       6.34401      0.170744      0.635123        100    
reco:trajcluster:TrajCluster                           0.0469054      0.60672       25.4047      0.0510038      2.81832        100    
reco:pandora:StandardPandora                           0.052507       1.51592       24.576        0.91997       2.85592        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112641   0.00434783     0.073166     0.000769121    0.0118749       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000153891   0.00653474     0.097271     0.00144697     0.0161358       100    
reco:pandoracalo:Calorimetry                          8.8411e-05     0.0156448     0.513926     0.000375284    0.0658545       100    
reco:pandorapid:Chi2ParticleID                         3.092e-05    0.000123009   0.00217369    8.61605e-05   0.000218533      100    
reco:trkshowersplit:TrackShowerHits                    4.789e-05     0.0168706     0.581104     0.000317118    0.0698374       100    
reco:pmtrack:PMAlgTrackMaker                          0.000881488     1.48797       44.4503      0.0570947      5.31393        100    
reco:pmtrackcalo:Calorimetry                          5.6131e-05    0.00708307      0.19582     0.000375153    0.0266167       100    
reco:pmtrackpid:Chi2ParticleID                         2.217e-05     0.0001896    0.00200257    7.85405e-05   0.000315137      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000155071     1.8665        66.0586      0.0773591      7.56865        100    
reco:pmtrajfitcalo:Calorimetry                        5.4781e-05     0.0174142     0.588855     0.000480864    0.074754        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.235e-05    0.000122626   0.000577825   8.7911e-05    0.000112526      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000721986     1.99031       50.8971      0.0670003      6.88054        100    
reco:pmtracktccalo:Calorimetry                         5.503e-05    0.00375713     0.0791066    0.000421843    0.0119414       100    
reco:pmtracktcpid:Chi2ParticleID                      2.1691e-05    0.000162661   0.00132289    8.9406e-05    0.000222613      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000130381    0.995757       18.5106      0.101882       2.70815        100    
reco:pmtrajfittccalo:Calorimetry                       5.252e-05     0.0105028     0.285094     0.00050314     0.0396276       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1081e-05    0.000229164   0.00239587    9.5231e-05    0.000382259      100    
reco:blurredcluster:BlurredClustering                  8.57e-05      0.610597       5.15144      0.193671      0.964868        100    
reco:emtrkmichelid:EmTrackMichelId                     8.309e-05      6.61136       93.3835       1.02485       15.3941        100    
reco:emshower:EMShower                                0.000128941    0.0181494     0.160464     0.00475428     0.0288266       100    
reco:cvnmap:CVNMapper                                  1.52e-05     0.00618645     0.0316673    0.00439219    0.00706832       100    
reco:cvneva:CVNEvaluator                               1.204e-05      0.42665       1.02777      0.672684      0.358702        100    
reco:energyrecnumu:EnergyReco                         0.000260433    0.0263833     0.772901     0.00282469     0.0902983       100    
reco:energyrecnue:EnergyReco                          0.000108291   0.00170549     0.0378431    0.000341073    0.0046988       100    
reco:energyrecnc:EnergyReco                           0.00135908    0.00316438     0.040764      0.0017695    0.00504799       100    
reco:energyrecnumurange:EnergyReco                    0.000122961    0.0245292     0.739727     0.00167086     0.0867146       100    
reco:energyrecnumumcs:EnergyReco                       0.0014467     0.0256946      0.76614     0.00209581     0.0888709       100    
reco:opdec:Deconvolution                               0.0481507     0.220913      0.464174       0.22416      0.0714854       100    
reco:ophitspe:OpHitFinderDeco                          0.140629       1.08896       2.40124       1.02554      0.471132        100    
reco:opflash:OpFlashFinder                            9.0971e-05    0.000950936   0.00447543    0.000831447   0.000568329      100    
reco:rns:RandomNumberSaver                             1.698e-05    2.79697e-05   0.000252812   2.1815e-05    2.39258e-05      100    
reco:anglereconue:NuAngularReco                       0.000117471   0.00112998     0.0201252    0.000238752   0.00301605       100    
reco:anglereconumu:NuAngularReco                      5.8881e-05    0.000978256    0.0189326    0.000130071   0.00288925       100    
reco:anglereconuepfps:NuAngularReco                   9.8171e-05    0.00107911     0.0210166    0.000216242    0.0030197       100    
reco:anglereconumupfps:NuAngularReco                  8.0671e-05    0.00102898     0.019408     0.000172641   0.00294129       100    
[art]:TriggerResults:TriggerResultInserter             7.75e-06     1.0238e-05     4.635e-05     9.04e-06     4.22548e-06      100    
end_path:out1:RootOutput                                2.5e-06     4.87595e-06    2.207e-05     3.94e-06     2.91105e-06      100    
end_path:out1:RootOutput(write)                       0.00729864     0.148936      0.686927      0.144072      0.129203        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21646.9 MB
  Peak resident set size usage (VmHWM): 2217.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_893_20231203T120748Z_gen_g4_detsim_hitreco__20240509T192551Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_893_20231203T120748Z_gen_g4_detsim_hitreco__20240509T192551Z_reco2_reco_2024-11-23T_042759Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_893_20231203T120748Z_gen_g4_detsim_hitreco__20240509T192551Z_reco2_reco_data_2024-11-23T_042759Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work