Jobsub ID 108779.100@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.100@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce13-multicore
Last heartbeat	2024-11-23 06:28:37
From worker node	Hostname	td810.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:05:16
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_53_20231208T063508Z_gen_g4_detsim_hitreco__20240510T055347Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4980s)
	CPU time	1h (4917s = 98%)
Outputting started	2024-11-23 06:28:17
Output files
Finished	2024-11-23 06:28:37
List job events Wrapper job log
Jobscript log (last 10,000 characters)

01  y: -0.571206  z: 0.758636 length: 0.977682
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.96918, Dist. 0.994757, nMCHits 23 (8, 12, 3)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 11 (4, 4, 3), nPfoHits 11 (4, 4, 3)

------------------------------------------------------------------------------------------------

---INTERPRETED-MATCHING-OUTPUT------------------(my version)-----------------------------------------
Found: 2212 with KE=0.0309078 GeV, E=0.96918 GeV, p2=0.0589552
NCQEL_P (Nuance 1002, Nu 1, CR 0)
IsCorrectNu (NNuMatches: 1) 
Parent Neutrino: 14
Neutrino Energy: 0.977678
Neutrino Momentum:   x: 0.232501  y: -0.571206  z: 0.758636 length: 0.977682
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.96918, Dist. 0.994757, nMCHits 23 (8, 12, 3)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 11 (4, 4, 3), nPfoHits 11 (4, 4, 3)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.0309078 GeV, E=0.96918 GeV, p2=0.0589552
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:28:13 CET run: 6494524 subRun: 1 event: 5400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:28:13 CET run: 6494524 subRun: 1 event: 5400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:28:13 CET run: 6494524 subRun: 1 event: 5400
Comparing two wires in the same plane: return failure
%MSG
184
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 337!
23-Nov-2024 07:28:16 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_53_20231208T063508Z_gen_g4_detsim_hitreco__20240510T055347Z_reco2_reco_data_2024-11-23T_050522Z.root"
23-Nov-2024 07:28:16 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/6e/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_53_20231208T063508Z_gen_g4_detsim_hitreco__20240510T055347Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.3324        48.8238       837.842       11.976        121.877        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00142275    0.00280709     0.0098205    0.00247246    0.00126271       100    
reco:linecluster:LineCluster                           0.0110701     0.471629       17.6079      0.023638       2.20855        100    
reco:trajcluster:TrajCluster                          0.00180153      2.60422       132.84       0.0173678      13.9467        100    
reco:pandora:StandardPandora                            1.09673       4.21574       83.6902       1.65587       10.9106        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000175101    0.0128029     0.282486     0.00193243     0.0374533       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000244223    0.0218137     0.349013     0.00415686     0.055338        100    
reco:pandoracalo:Calorimetry                          0.000136401    0.0365918      1.03372     0.00154758     0.136238        100    
reco:pandorapid:Chi2ParticleID                        5.0821e-05    0.000212099   0.00120259    0.000145077   0.000200211      100    
reco:trkshowersplit:TrackShowerHits                   0.000137141    0.0486297      1.15269     0.000897609    0.167822        100    
reco:pmtrack:PMAlgTrackMaker                          0.00172343      5.19551       115.916      0.222421       16.5256        100    
reco:pmtrackcalo:Calorimetry                          9.3141e-05     0.0250007     0.682793     0.00136228     0.0957081       100    
reco:pmtrackpid:Chi2ParticleID                         3.599e-05    0.000453671   0.00508632    0.000167836   0.000796711      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000278073     3.27004       45.5911       0.40557       7.68543        100    
reco:pmtrajfitcalo:Calorimetry                         8.416e-05     0.0339389     0.843969     0.00178327      0.11616        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.705e-05    0.000256891   0.00207974    0.000172291   0.000273873      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00164595       6.228        149.404       0.28485       20.9779        100    
reco:pmtracktccalo:Calorimetry                         9.765e-05     0.0148312     0.366441     0.00118129     0.0511432       100    
reco:pmtracktcpid:Chi2ParticleID                      4.0711e-05    0.00040132    0.00391549    0.000176481   0.00062342       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000298573     3.29869       72.1283        0.389        10.0845        100    
reco:pmtrajfittccalo:Calorimetry                      9.5961e-05     0.0401576      1.23516     0.00156791     0.163787        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.939e-05    0.000595456   0.00760153    0.000177457   0.00115032       100    
reco:blurredcluster:BlurredClustering                 0.00971985      1.32732       11.4114      0.496481       2.12969        100    
reco:emtrkmichelid:EmTrackMichelId                     0.346732       18.3268       255.562       3.9711        40.4608        100    
reco:emshower:EMShower                                0.000329173    0.0540574     0.668378     0.00592642     0.104579        100    
reco:cvnmap:CVNMapper                                  2.882e-05     0.0108501     0.045886     0.00827405    0.00980723       100    
reco:cvneva:CVNEvaluator                               1.877e-05     0.769042       1.48526      0.980194      0.472837        100    
reco:energyrecnumu:EnergyReco                         0.000344823    0.038338      0.748284     0.00658193     0.0977375       100    
reco:energyrecnue:EnergyReco                          0.000151171   0.00453962     0.140751     0.000647431    0.0153783       100    
reco:energyrecnc:EnergyReco                           0.00211846    0.00766398     0.145952     0.00376261     0.0159336       100    
reco:energyrecnumurange:EnergyReco                    0.000211442    0.0355981     0.730703     0.00384249     0.0965704       100    
reco:energyrecnumumcs:EnergyReco                       0.0009872     0.0377222     0.734271     0.00469152     0.0974292       100    
reco:opdec:Deconvolution                               0.0757149     0.232252       0.44771      0.229945      0.0903156       100    
reco:ophitspe:OpHitFinderDeco                           0.58632       2.08842       4.32361       2.07055      0.922294        100    
reco:opflash:OpFlashFinder                            0.000472994   0.00166407    0.00437015    0.00156532    0.000775662      100    
reco:rns:RandomNumberSaver                             1.812e-05    3.4377e-05    0.000364953   2.7345e-05    3.49274e-05      100    
reco:anglereconue:NuAngularReco                       0.000170922   0.00364442     0.122916     0.000397743    0.0137085       100    
reco:anglereconumu:NuAngularReco                       8.632e-05     0.0034387     0.120273     0.000254828    0.0135006       100    
reco:anglereconuepfps:NuAngularReco                   0.000139591   0.00358443     0.120467     0.000400259    0.0134892       100    
reco:anglereconumupfps:NuAngularReco                  0.000121761   0.00353604     0.121836     0.000337324    0.0136662       100    
[art]:TriggerResults:TriggerResultInserter             9.809e-06    1.41184e-05    4.654e-05    1.34455e-05   4.06054e-06      100    
end_path:out1:RootOutput                               2.96e-06     5.29339e-06    2.126e-05     3.92e-06     2.83104e-06      100    
end_path:out1:RootOutput(write)                        0.0464813     0.350601       1.97596       0.31767      0.318024        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21623.9 MB
  Peak resident set size usage (VmHWM): 2488.26 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_53_20231208T063508Z_gen_g4_detsim_hitreco__20240510T055347Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_53_20231208T063508Z_gen_g4_detsim_hitreco__20240510T055347Z_reco2_reco_2024-11-23T_050522Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_53_20231208T063508Z_gen_g4_detsim_hitreco__20240510T055347Z_reco2_reco_data_2024-11-23T_050522Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work