Jobsub ID 301751.102@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301751.102@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:14:46
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 07:27:52
From worker node	Hostname	b9p10p4272.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:16:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_163_20231121T153419Z_gen_g4_detsim_hitreco__20240503T053955Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7880s)
	CPU time	2h (7731s = 98%)
Outputting started	2024-11-23 07:27:33
Output files
Finished	2024-11-23 07:27:52
List job events Wrapper job log
Jobscript log (last 10,000 characters)

v-2024 08:27:03 CET run: 50258005 subRun: 1 event: 32800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:03 CET run: 50258005 subRun: 1 event: 32800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:04 CET run: 50258005 subRun: 1 event: 32800
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:04 CET run: 50258005 subRun: 1 event: 32800
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:04 CET run: 50258005 subRun: 1 event: 32800
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:04 CET run: 50258005 subRun: 1 event: 32800
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:04 CET run: 50258005 subRun: 1 event: 32800
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:27:04 CET run: 50258005 subRun: 1 event: 32800
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 796, 737, 803
minwire 0: 194
minwire 1: 1347
minwire 2: 150
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.031359, 0.035979, 0.932662, 
Output 1: 0.0427781, 0.146033, 0.234812, 0.576377, 
Output 2: 0.114033, 0.885967, 

664
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2366!
23-Nov-2024 08:27:32 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_163_20231121T153419Z_gen_g4_detsim_hitreco__20240503T053955Z_reco2_reco_data_2024-11-23T_051619Z.root"
23-Nov-2024 08:27:32 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/ed/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_163_20231121T153419Z_gen_g4_detsim_hitreco__20240503T053955Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.543311       37.589        862.915       8.65112       110.779        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0171117     0.0398457     0.120123      0.0409801     0.012503        200    
reco:linecluster:LineCluster                           0.0951185      1.11257       60.9134      0.176771       5.50129        200    
reco:trajcluster:TrajCluster                           0.0476129      1.85762       71.4277      0.0596782      8.79174        200    
reco:pandora:StandardPandora                           0.0568756      3.41904       102.759       1.26131       11.5467        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000106592   0.00895819     0.340393     0.00156496     0.0326982       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000143381    0.0138311      0.41042     0.00296786     0.0374367       200    
reco:pandoracalo:Calorimetry                          8.7261e-05     0.0411947      1.72224     0.00104775      0.19213        200    
reco:pandorapid:Chi2ParticleID                        3.3641e-05    0.000161879   0.00134128    0.000123427   0.000165441      200    
reco:trkshowersplit:TrackShowerHits                   7.9681e-05     0.0546481      1.87195     0.000730133    0.241677        200    
reco:pmtrack:PMAlgTrackMaker                          0.000939321     4.34979       155.717      0.143958       17.4086        200    
reco:pmtrackcalo:Calorimetry                          5.7531e-05     0.0159788     0.840257     0.000908201    0.0849516       200    
reco:pmtrackpid:Chi2ParticleID                        2.3961e-05    0.000307699    0.0066368    0.000129077   0.000681783      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000157162     3.91493       161.164      0.298633       16.1304        200    
reco:pmtrajfitcalo:Calorimetry                        5.3901e-05     0.0375743      1.72909     0.00145739     0.170252        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.278e-05    0.000187626   0.00127195    0.000140761   0.00018011       200    
reco:pmtracktc:PMAlgTrackMaker                        0.000740109     4.67017       130.858      0.217113       17.191         200    
reco:pmtracktccalo:Calorimetry                        5.6621e-05    0.00975532     0.366892     0.000850157    0.0392483       200    
reco:pmtracktcpid:Chi2ParticleID                      2.3021e-05    0.000297305   0.00420199    0.000136626   0.000526883      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000140711     2.33128       71.8878      0.215914       8.18201        200    
reco:pmtrajfittccalo:Calorimetry                       5.287e-05     0.0248295      1.14994     0.00106697     0.113478        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1821e-05    0.000399937   0.00660043    0.000144492   0.000847109      200    
reco:blurredcluster:BlurredClustering                 0.000116181     1.04938       13.0491      0.415158       1.70455        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000133122     11.2372       211.296       2.21796       26.9961        200    
reco:emshower:EMShower                                0.000195153    0.0290936     0.417738      0.0028837     0.0608821       200    
reco:cvnmap:CVNMapper                                  1.991e-05    0.00909998     0.0489523    0.00624352    0.00919048       200    
reco:cvneva:CVNEvaluator                               1.429e-05     0.773973       6.14627       0.93105      0.823566        200    
reco:energyrecnumu:EnergyReco                         0.000317774    0.0406165     0.962807     0.00481078     0.129614        200    
reco:energyrecnue:EnergyReco                          0.000115141   0.00253312     0.0857856    0.000562476   0.00784979       200    
reco:energyrecnc:EnergyReco                           0.00144475    0.00490119     0.107831     0.00315168    0.00915585       200    
reco:energyrecnumurange:EnergyReco                    0.000146211    0.0374487     0.971023     0.00299567     0.130701        200    
reco:energyrecnumumcs:EnergyReco                      0.00081398     0.0405036      1.09174     0.00342213     0.138437        200    
reco:opdec:Deconvolution                               0.0499832     0.279461      0.706827      0.281208      0.0964899       200    
reco:ophitspe:OpHitFinderDeco                          0.144212       1.93507       3.98658       1.86365      0.911255        200    
reco:opflash:OpFlashFinder                            0.000130132   0.00143534    0.00581221    0.00129419    0.000787524      200    
reco:rns:RandomNumberSaver                             1.701e-05    3.45985e-05   0.000320544   3.1916e-05    2.34557e-05      200    
reco:anglereconue:NuAngularReco                       0.000134562   0.00203327     0.0974541    0.000385285   0.00791406       200    
reco:anglereconumu:NuAngularReco                      5.9841e-05    0.00184703     0.0970187    0.000222087   0.00795588       200    
reco:anglereconuepfps:NuAngularReco                   0.000103161    0.002021      0.0980306    0.000343384   0.00815306       200    
reco:anglereconumupfps:NuAngularReco                  8.8901e-05    0.00194371     0.0983282    0.000281718   0.00811195       200    
[art]:TriggerResults:TriggerResultInserter             8.19e-06     1.34696e-05   7.2351e-05     1.238e-05    6.29029e-06      200    
end_path:out1:RootOutput                               2.69e-06      5.702e-06     3.262e-05     5.71e-06     2.7875e-06       200    
end_path:out1:RootOutput(write)                        0.0125084     0.234946       1.50657      0.220055      0.209775        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22178.9 MB
  Peak resident set size usage (VmHWM): 2740.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_163_20231121T153419Z_gen_g4_detsim_hitreco__20240503T053955Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_163_20231121T153419Z_gen_g4_detsim_hitreco__20240503T053955Z_reco2_reco_2024-11-23T_051619Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_163_20231121T153419Z_gen_g4_detsim_hitreco__20240503T053955Z_reco2_reco_data_2024-11-23T_051619Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work