Jobsub ID 108834.132@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108834.132@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:43:00
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 07:11:21
From worker node	Hostname	heplnc149.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:45:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_704_20231203T112332Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5110s)
	CPU time	1h (4916s = 96%)
Outputting started	2024-11-23 07:10:56
Output files
Finished	2024-11-23 07:11:21
List job events Wrapper job log
Jobscript log (last 10,000 characters)

---------------------------------------------
#CorrectNu: 0/1, Fraction: 0
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.0208601 GeV, E=0.959132 GeV, p2=0.03958
Found: 22 with KE=0.00387683 GeV, E=0.00387778 GeV, p2=1.50372e-05
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:10:51 GMT run: 6418365 subRun: 1 event: 70500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:10:51 GMT run: 6418365 subRun: 1 event: 70500
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:10:51 GMT run: 6418365 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:51 GMT run: 6418365 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:51 GMT run: 6418365 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:51 GMT run: 6418365 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:10:52 GMT run: 6418365 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
237
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 558!
23-Nov-2024 07:10:54 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_704_20231203T112332Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2_reco_data_2024-11-23T_054552Z.root"
23-Nov-2024 07:10:54 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/ce/20/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_704_20231203T112332Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.766853       48.2332       504.054       21.0387       83.4805        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0302783     0.0507387     0.0822026     0.052687      0.0136162       100    
reco:linecluster:LineCluster                            0.13641      0.354331       3.6481       0.244099       0.4375         100    
reco:trajcluster:TrajCluster                           0.0650731      1.37045       39.594       0.108445       5.2146         100    
reco:pandora:StandardPandora                           0.0923871      3.76745       87.8561       2.08327       9.1033         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000189978    0.0119394     0.123939     0.00410726     0.0214047       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000258988    0.0269036     0.309371     0.00717522     0.052022        100    
reco:pandoracalo:Calorimetry                          0.000181753    0.0282285     0.506192     0.00368827     0.0808973       100    
reco:pandorapid:Chi2ParticleID                        5.4824e-05    0.00034629    0.00360605    0.000265125   0.000401339      100    
reco:trkshowersplit:TrackShowerHits                   0.000174228    0.0291137     0.470292     0.00257007     0.0838586       100    
reco:pmtrack:PMAlgTrackMaker                          0.00167286      3.65543       49.6933      0.756522       8.63475        100    
reco:pmtrackcalo:Calorimetry                          8.9869e-05     0.0182506     0.331051     0.00303703     0.0503947       100    
reco:pmtrackpid:Chi2ParticleID                        3.1229e-05    0.000544285   0.00417014    0.000335252   0.000692648      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000224614     3.76628       74.5534      0.829006       9.99917        100    
reco:pmtrajfitcalo:Calorimetry                        7.5122e-05     0.0316211      0.69247     0.00452604     0.0974617       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8915e-05    0.000374353   0.00175093    0.000319918   0.000284938      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00118264      4.69027       76.082       0.739293       12.2325        100    
reco:pmtracktccalo:Calorimetry                        8.9138e-05     0.0125624     0.307895     0.00287465     0.0361269       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3133e-05    0.000524551   0.00484146    0.000359408   0.000643123      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000211168     2.47063       33.7855      0.818821       4.90148        100    
reco:pmtrajfittccalo:Calorimetry                      6.9621e-05     0.0237007     0.422679     0.00358512     0.0673481       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8534e-05    0.000668351   0.00463597    0.000393868    0.0008314       100    
reco:blurredcluster:BlurredClustering                 0.000181993     1.51867       7.32763       1.07044       1.60195        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000209435     22.0016       202.151       9.28798       35.5436        100    
reco:emshower:EMShower                                0.000256474    0.0504894     0.371886      0.0189845     0.0711469       100    
reco:cvnmap:CVNMapper                                 2.9144e-05     0.0226638     0.0528714     0.0262503     0.013093        100    
reco:cvneva:CVNEvaluator                              2.4907e-05      1.26441       2.23193       1.45645      0.655408        100    
reco:energyrecnumu:EnergyReco                         0.000395546    0.0577392      1.56163     0.00942001     0.184006        100    
reco:energyrecnue:EnergyReco                          0.000198655   0.00309265     0.0328082    0.00136085    0.00551339       100    
reco:energyrecnc:EnergyReco                           0.00230675    0.00601586     0.0352083    0.00423508    0.00573181       100    
reco:energyrecnumurange:EnergyReco                    0.000398241    0.0542701      1.53573     0.00471847      0.18236        100    
reco:energyrecnumumcs:EnergyReco                      0.00231869     0.0556443      1.47609     0.00553774     0.177603        100    
reco:opdec:Deconvolution                               0.0861269     0.365357      0.625587       0.37139      0.100612        100    
reco:ophitspe:OpHitFinderDeco                          0.198926       2.16112       4.00694       2.22524      0.806835        100    
reco:opflash:OpFlashFinder                            0.000287683   0.00226956    0.00753793    0.00215415    0.000985762      100    
reco:rns:RandomNumberSaver                            2.8865e-05    6.43878e-05   0.000499893   5.2023e-05    5.02231e-05      100    
reco:anglereconue:NuAngularReco                       0.000238661   0.00246566     0.0391199    0.000962987   0.00500234       100    
reco:anglereconumu:NuAngularReco                      0.000103815   0.00204335     0.0404263     0.000504     0.00509882       100    
reco:anglereconuepfps:NuAngularReco                   0.000177685   0.00241405     0.0453123    0.000761461   0.00559488       100    
reco:anglereconumupfps:NuAngularReco                  0.000148229   0.00226617     0.0397353    0.000588089   0.00531465       100    
[art]:TriggerResults:TriggerResultInserter            1.1762e-05    1.99097e-05   0.000102634   1.6912e-05     1.066e-05       100    
end_path:out1:RootOutput                               4.499e-06    1.0698e-05    4.8702e-05     9.593e-06    6.22988e-06      100    
end_path:out1:RootOutput(write)                        0.0170514     0.346478       1.3652       0.350514      0.212556        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21605.8 MB
  Peak resident set size usage (VmHWM): 2175.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_704_20231203T112332Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_704_20231203T112332Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2_reco_2024-11-23T_054552Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_704_20231203T112332Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2_reco_data_2024-11-23T_054552Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work