Jobsub ID 108844.8@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108844.8@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:57:59
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 08:15:09
From worker node	Hostname	heplnc168.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:59:24
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8109s)
	CPU time	2h (8316s = 102%)
Outputting started	2024-11-23 08:14:33
Output files
Finished	2024-11-23 08:15:09
List job events Wrapper job log
Jobscript log (last 10,000 characters)

--------------------------------
Found: 22 with KE=0.00700639 GeV, E=0.00700639 GeV, p2=4.90896e-05
OTHER_INTERACTION (Nuance 1002, Nu 1, CR 0)
IsLost (NNuLosses: 1) 
Parent Neutrino: -16
Neutrino Energy: 0.224145
Neutrino Momentum:   x: 0.148105  y: -0.127107  z: 0.110226 length: 0.224146
PrimaryId 1, Nu 1, CR 0, MCPDG 22, Energy 0.00700639, Dist. 95.0936, nMCHits 16 (5, 6, 5)
-No matched Pfo

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 0/1, Fraction: 0
#Lost: 1 
------------------------------------------------------------------------------------------------

Found: 22 with KE=0.00700639 GeV, E=0.00700639 GeV, p2=4.90896e-05
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 08:14:24 GMT run: 6220954 subRun: 1 event: 20400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 08:14:24 GMT run: 6220954 subRun: 1 event: 20400
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 08:14:24 GMT run: 6220954 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:14:26 GMT run: 6220954 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:14:26 GMT run: 6220954 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
156
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 234!
23-Nov-2024 08:14:29 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2_reco_data_2024-11-23T_055934Z.root"
23-Nov-2024 08:14:29 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/0e/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.23368       77.7091       1289.82       22.3938       167.865        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0241695     0.0555409     0.156113      0.0573913      0.02002        100    
reco:linecluster:LineCluster                            0.20595      0.948233       58.315       0.238499       5.7986         100    
reco:trajcluster:TrajCluster                           0.0663032      1.29629       63.7711      0.0886295      6.54953        100    
reco:pandora:StandardPandora                            1.69464       5.8146        143.104       3.37915       14.7492        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000322876    0.0145783     0.140922     0.00436781     0.0262236       100    
reco:pandoraShower:LArPandoraModularShowerCreation     0.0004528     0.0298145     0.509216     0.00654815     0.0648334       100    
reco:pandoracalo:Calorimetry                          0.000220944    0.0409802      1.25914     0.00252359     0.150014        100    
reco:pandorapid:Chi2ParticleID                        7.6644e-05    0.00049122    0.00551321    0.000379798   0.000588006      100    
reco:trkshowersplit:TrackShowerHits                   0.000220965    0.0462724      1.96007     0.00193544     0.205831        100    
reco:pmtrack:PMAlgTrackMaker                          0.00193932      5.36171       150.368      0.406858       17.9147        100    
reco:pmtrackcalo:Calorimetry                          0.000103344    0.0212982     0.380326      0.0029104     0.0580169       100    
reco:pmtrackpid:Chi2ParticleID                        3.7881e-05    0.000776002   0.00660743    0.000418782   0.00104083       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000403477     4.05174       154.385      0.278757       15.9807        100    
reco:pmtrajfitcalo:Calorimetry                        0.00011175     0.0418329      1.20562     0.00352604     0.148166        100    
reco:pmtrajfitpid:Chi2ParticleID                      5.0125e-05    0.000572777    0.005272     0.000396981   0.000640745      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00141659      5.39909       131.489      0.470738       17.7219        100    
reco:pmtracktccalo:Calorimetry                        0.000103394    0.0161366     0.278579     0.00279188     0.0409108       100    
reco:pmtracktcpid:Chi2ParticleID                      4.9332e-05    0.000742206   0.00467038    0.00045305    0.000908495      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000327324     2.78894       53.4896      0.525694       7.35455        100    
reco:pmtrajfittccalo:Calorimetry                      0.000106691    0.0319518     0.810945     0.00314923     0.105044        100    
reco:pmtrajfittcpid:Chi2ParticleID                    5.2529e-05    0.000940485   0.00831448    0.000489258   0.00139759       100    
reco:blurredcluster:BlurredClustering                 0.00880421       2.195        15.5114      0.944108       3.02757        100    
reco:emtrkmichelid:EmTrackMichelId                     0.222738       43.9597       503.808       9.76045       86.5207        100    
reco:emshower:EMShower                                0.000580169    0.0475138     0.541922     0.00690043     0.0851034       100    
reco:cvnmap:CVNMapper                                 4.9974e-05     0.0366612     0.114782      0.0391739     0.0314009       100    
reco:cvneva:CVNEvaluator                              3.8362e-05      1.8666        5.68756       2.03991       1.51399        100    
reco:energyrecnumu:EnergyReco                         0.000649559    0.101075       1.67386      0.0103879     0.276918        100    
reco:energyrecnue:EnergyReco                          0.000190607   0.00552562      0.10142     0.00114266     0.0135958       100    
reco:energyrecnc:EnergyReco                           0.00222592    0.00830467     0.0885051    0.00462444     0.0121417       100    
reco:energyrecnumurange:EnergyReco                    0.000194666    0.0926031      1.70081     0.00371261     0.274741        100    
reco:energyrecnumumcs:EnergyReco                      0.00164215     0.0922403      1.70483      0.0051756      0.27328        100    
reco:opdec:Deconvolution                               0.115989      0.381003      0.728648      0.381372      0.131948        100    
reco:ophitspe:OpHitFinderDeco                           0.49842       2.53218       4.68345       2.5863        1.05052        100    
reco:opflash:OpFlashFinder                            0.000971964   0.00394056     0.0149049    0.00321456    0.00255835       100    
reco:rns:RandomNumberSaver                            4.6778e-05    0.000130027   0.00118208    0.000107527   0.000116034      100    
reco:anglereconue:NuAngularReco                       0.000436349   0.00426598     0.0531982    0.00118852    0.00855773       100    
reco:anglereconumu:NuAngularReco                      0.000110968   0.00288133     0.0404386    0.000691252   0.00650797       100    
reco:anglereconuepfps:NuAngularReco                   0.000240983   0.00332206     0.0413121    0.000939007   0.00684718       100    
reco:anglereconumupfps:NuAngularReco                  0.000150843   0.00308605     0.0409803    0.000680943   0.00674827       100    
[art]:TriggerResults:TriggerResultInserter            1.6171e-05    3.99897e-05   0.000190898   2.9335e-05    2.82242e-05      100    
end_path:out1:RootOutput                               7.695e-06    2.15724e-05   7.1364e-05    1.66255e-05   1.34141e-05      100    
end_path:out1:RootOutput(write)                        0.0363309     0.404299       1.76818       0.3881       0.333312        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39564.8 MB
  Peak resident set size usage (VmHWM): 2357.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2_reco_2024-11-23T_055934Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2_reco_data_2024-11-23T_055934Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work