Jobsub ID 108705.98@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108705.98@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:10:05
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 04:37:08
From worker node	Hostname	heplnc159.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:22:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_831_20231204T003658Z_gen_g4_detsim_hitreco__20240509T210925Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4466s)
	CPU time	1h (5156s = 115%)
Outputting started	2024-11-23 04:36:46
Output files
Finished	2024-11-23 04:37:08
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:49 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:36 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:36 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:36 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:36 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:36 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:36 GMT run: 50577091 subRun: 1 event: 83200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1599, 2302, 1939
minwire 0: 213
minwire 1: 1574
minwire 2: 30
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.109077, 0.165392, 0.725531, 
Output 1: 0.342465, 0.411882, 0.175285, 0.0703678, 
Output 2: 0.0722795, 0.92772, 

636
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2364!
23-Nov-2024 04:36:44 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_831_20231204T003658Z_gen_g4_detsim_hitreco__20240509T210925Z_reco2_reco_data_2024-11-23T_032225Z.root"
23-Nov-2024 04:36:44 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/60/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_831_20231204T003658Z_gen_g4_detsim_hitreco__20240509T210925Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.247779       42.0227       954.213       9.94052       113.392        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.004079      0.115158      0.925185      0.0330756     0.161644        100    
reco:linecluster:LineCluster                           0.0213599      1.21532       60.9635      0.095612       6.10018        100    
reco:trajcluster:TrajCluster                           0.0119549      1.90615       124.22       0.0221761      12.4804        100    
reco:pandora:StandardPandora                           0.0232175      3.3594        84.1292       1.43646       10.1042        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000133973   0.00741408     0.120836     0.00158933     0.0169709       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000180309    0.0155231     0.231319     0.00311744     0.0374801       100    
reco:pandoracalo:Calorimetry                          0.000112472    0.0291783     0.755208     0.00104386     0.102802        100    
reco:pandorapid:Chi2ParticleID                        4.5876e-05    0.000205534   0.00192502    0.000141427   0.000220586      100    
reco:trkshowersplit:TrackShowerHits                   0.000126407    0.036467       1.24413     0.000812849    0.143746        100    
reco:pmtrack:PMAlgTrackMaker                          0.00122859      4.63616       181.375      0.143906       19.6236        100    
reco:pmtrackcalo:Calorimetry                          7.5262e-05     0.0204405     0.510066     0.00106633     0.0761743       100    
reco:pmtrackpid:Chi2ParticleID                        2.8634e-05    0.000440774   0.00525298    0.000160461   0.000853924      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000183866     3.06127       86.8212      0.196027       9.94833        100    
reco:pmtrajfitcalo:Calorimetry                        6.7717e-05     0.0335822      1.02464     0.00138816     0.128658        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5949e-05    0.000224585   0.00112259    0.000163713   0.000209543      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102112      5.61409       178.421      0.173184       20.7943        100    
reco:pmtracktccalo:Calorimetry                        8.1063e-05     0.011027      0.216917     0.000974068    0.0326912       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2592e-05    0.000368175   0.00323962    0.000159951   0.000522639      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000208602     2.42787       47.9638      0.230323       6.95069        100    
reco:pmtrajfittccalo:Calorimetry                       7.421e-05      0.02737      0.806855      0.0010863      0.10749        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.2021e-05    0.000508388   0.00597053    0.000168301   0.000919584      100    
reco:blurredcluster:BlurredClustering                 0.000204085     1.10753       7.48092      0.442367       1.53712        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000173486     14.2232       191.922       2.50859       30.5405        100    
reco:emshower:EMShower                                0.000234131    0.0351743     0.460837      0.0063405     0.0668783       100    
reco:cvnmap:CVNMapper                                 2.6209e-05     0.013406      0.0838877     0.0142231     0.0135832       100    
reco:cvneva:CVNEvaluator                              1.9707e-05     0.926898       1.75484       1.20105      0.638804        100    
reco:energyrecnumu:EnergyReco                         0.000471768    0.0304372      1.00154     0.00524018     0.106313        100    
reco:energyrecnue:EnergyReco                          0.000154621   0.00254113     0.0405321    0.000658003   0.00554411       100    
reco:energyrecnc:EnergyReco                           0.00174963    0.00509989     0.0442595    0.00356702    0.00579452       100    
reco:energyrecnumurange:EnergyReco                    0.000200017    0.0293711      1.0965      0.00341453     0.115234        100    
reco:energyrecnumumcs:EnergyReco                      0.00165366     0.0313848      1.10148     0.00371467     0.116021        100    
reco:opdec:Deconvolution                               0.0138568     0.451964       2.90212      0.239295      0.571314        100    
reco:ophitspe:OpHitFinderDeco                          0.0313506      2.34601       5.98591       2.15232       1.23685        100    
reco:opflash:OpFlashFinder                            0.000183475   0.00185372    0.00432345    0.00165706    0.000863738      100    
reco:rns:RandomNumberSaver                            1.6872e-05    5.24326e-05   0.000443144   4.73845e-05   4.21364e-05      100    
reco:anglereconue:NuAngularReco                       0.000163658   0.00220044     0.0512655    0.000473922   0.00613715       100    
reco:anglereconumu:NuAngularReco                       8.477e-05    0.00185513     0.0481649    0.000227528   0.00582617       100    
reco:anglereconuepfps:NuAngularReco                   0.000132259   0.00209067     0.051766     0.00042491    0.00611976       100    
reco:anglereconumupfps:NuAngularReco                  0.000117902   0.00192847     0.0481154    0.000320723   0.00583028       100    
[art]:TriggerResults:TriggerResultInserter             9.137e-06    1.67496e-05   6.4191e-05     1.576e-05    6.91699e-06      100    
end_path:out1:RootOutput                               2.926e-06    8.62759e-06   3.5527e-05    7.7045e-06    5.02539e-06      100    
end_path:out1:RootOutput(write)                        0.0183542     0.288989       1.40463      0.269509      0.243991        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39414.6 MB
  Peak resident set size usage (VmHWM): 2252.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_831_20231204T003658Z_gen_g4_detsim_hitreco__20240509T210925Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_831_20231204T003658Z_gen_g4_detsim_hitreco__20240509T210925Z_reco2_reco_2024-11-23T_032225Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_831_20231204T003658Z_gen_g4_detsim_hitreco__20240509T210925Z_reco2_reco_data_2024-11-23T_032225Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work