Jobsub ID 301625.157@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301625.157@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:33:55
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat	2024-11-23 02:21:43
From worker node	Hostname	iut2-c417.iu.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:42:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2.root
Jobscript	Exit code	0
	Real time	39m (2359s)
	CPU time	36m (2186s = 92%)
Outputting started	2024-11-23 02:21:30
Output files
Finished	2024-11-23 02:21:43
List job events Wrapper job log
Jobscript log (last 10,000 characters)

---------------------------------------------------------

---INTERPRETED-MATCHING-OUTPUT------------------(my version)-----------------------------------------
Found: 2212 with KE=0.0130976 GeV, E=0.95137 GeV, p2=0.0247498
NCQEL_P (Nuance 1002, Nu 1, CR 0)
IsCorrectNu (NNuMatches: 1) 
Parent Neutrino: -12
Neutrino Energy: 0.633305
Neutrino Momentum:   x: -0.127574  y: 0.618571  z: -0.0465863 length: 0.633306
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.95137, Dist. 0.216125, nMCHits 17 (5, 10, 2)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 5 (3, 2, 0), nPfoHits 5 (3, 2, 0)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.0130976 GeV, E=0.95137 GeV, p2=0.0247498
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  22-Nov-2024 21:21:26 EST run: 50568178 subRun: 1 event: 87400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  22-Nov-2024 21:21:26 EST run: 50568178 subRun: 1 event: 87400
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:21:26 EST run: 50568178 subRun: 1 event: 87400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:21:26 EST run: 50568178 subRun: 1 event: 87400
Comparing two wires in the same plane: return failure
%MSG
212
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 473!
22-Nov-2024 21:21:28 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2_reco_data_2024-11-23T_014213Z.root"
22-Nov-2024 21:21:28 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/ac/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.932493       20.7462       327.476       8.82936       38.054         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0269705     0.0746779     0.130842      0.0763849     0.0267699       100    
reco:linecluster:LineCluster                           0.153409      0.319466       2.88469      0.265544      0.274872        100    
reco:trajcluster:TrajCluster                           0.0767537     0.547987       28.0676      0.0863059      2.82054        100    
reco:pandora:StandardPandora                           0.0895231      1.51323       16.4783       1.21283       1.6314         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000143149   0.00415482     0.0884754    0.00132321    0.00955549       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000179427   0.00787844     0.0863684    0.00249685     0.013845        100    
reco:pandoracalo:Calorimetry                          0.000100879   0.00936229     0.244272     0.000847187    0.0287737       100    
reco:pandorapid:Chi2ParticleID                        3.9003e-05    0.000161576   0.00149606    0.000125942   0.000163732      100    
reco:trkshowersplit:TrackShowerHits                   0.000123432    0.0119703     0.405783     0.000721085    0.044309        100    
reco:pmtrack:PMAlgTrackMaker                          0.000998465     1.55305       38.7031      0.171884       4.46698        100    
reco:pmtrackcalo:Calorimetry                          6.3079e-05    0.00652775     0.235342     0.000885874    0.0242664       100    
reco:pmtrackpid:Chi2ParticleID                         2.653e-05    0.000272857   0.00235827    0.000163357   0.00033604       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000174498     1.38412       37.1697      0.197883       4.17525        100    
reco:pmtrajfitcalo:Calorimetry                        6.3179e-05     0.0105387     0.292119     0.00122953     0.0329749       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3855e-05    0.000190703   0.00123915    0.000156184   0.000157874      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000759587     2.13253         56         0.156933       6.64496        100    
reco:pmtracktccalo:Calorimetry                        6.2577e-05    0.00482211     0.145414     0.000842979    0.0152388       100    
reco:pmtracktcpid:Chi2ParticleID                      2.6059e-05    0.000256649   0.00238605    0.000144952   0.000312861      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000170651     1.03975       19.6259      0.244346       2.33241        100    
reco:pmtrajfittccalo:Calorimetry                      5.9792e-05    0.00764586     0.193641     0.00103655     0.021996        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4596e-05    0.00029704    0.00206665    0.000172283   0.000336076      100    
reco:blurredcluster:BlurredClustering                 0.000120186    0.744044       4.78574      0.390364       0.8842         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000124213     8.2724        113.577       2.57635       15.0127        100    
reco:emshower:EMShower                                0.000182382    0.0289086     0.302738     0.00523607     0.0498549       100    
reco:cvnmap:CVNMapper                                 2.4185e-05    0.00745059     0.0441804    0.00582435    0.00720025       100    
reco:cvneva:CVNEvaluator                              1.6701e-05     0.584394       1.1521       0.722723      0.358768        100    
reco:energyrecnumu:EnergyReco                         0.000303059    0.0244029     0.613849     0.00409654     0.0704277       100    
reco:energyrecnue:EnergyReco                          0.000124774   0.00164187     0.0398909    0.000546086   0.00410439       100    
reco:energyrecnc:EnergyReco                           0.00156634    0.00361378     0.0426647    0.00279056    0.00424776       100    
reco:energyrecnumurange:EnergyReco                    0.000181431    0.0215288     0.527554     0.00261964     0.0629697       100    
reco:energyrecnumumcs:EnergyReco                      0.00158892     0.0226979     0.540099     0.00326356     0.062928        100    
reco:opdec:Deconvolution                               0.0914808     0.349854      0.690535      0.319044      0.125837        100    
reco:ophitspe:OpHitFinderDeco                          0.370885       1.83422       4.18474       1.77137       0.74576        100    
reco:opflash:OpFlashFinder                            0.000341952   0.00137207    0.00321967     0.0012741    0.000569169      100    
reco:rns:RandomNumberSaver                            1.9646e-05    3.62321e-05   0.000269536   3.0833e-05    2.55676e-05      100    
reco:anglereconue:NuAngularReco                       0.000150452   0.00120216     0.0402067    0.000365682   0.00403493       100    
reco:anglereconumu:NuAngularReco                      6.3729e-05    0.00101339     0.0401097    0.000223189    0.0040289       100    
reco:anglereconuepfps:NuAngularReco                   0.000108224   0.00115602     0.039929     0.000337925   0.00400927       100    
reco:anglereconumupfps:NuAngularReco                  0.000102362   0.00107865     0.0402204    0.000281535   0.00403573       100    
[art]:TriggerResults:TriggerResultInserter             8.226e-06    1.33662e-05   4.2901e-05    1.2268e-05    4.52684e-06      100    
end_path:out1:RootOutput                               2.915e-06    5.62874e-06   1.8976e-05     4.809e-06    2.74906e-06      100    
end_path:out1:RootOutput(write)                        0.0169029     0.214129       1.03902       0.22549      0.143595        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17217.9 MB
  Peak resident set size usage (VmHWM): 2248.02 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2_reco_2024-11-23T_014213Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2_reco_data_2024-11-23T_014213Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work