Jobsub ID 108858.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108858.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:25:55
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 07:32:54
From worker node	Hostname	comp05-21
	cpuinfo	Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:27:31
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_550_20231204T071701Z_gen_g4_detsim_hitreco__20240509T210235Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3912s)
	CPU time	50m (3035s = 77%)
Outputting started	2024-11-23 07:32:43
Output files
Finished	2024-11-23 07:32:54
List job events Wrapper job log
Jobscript log (last 10,000 characters)

vent: 55100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:33 GMT run: 74517585 subRun: 1 event: 55100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:33 GMT run: 74517585 subRun: 1 event: 55100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:33 GMT run: 74517585 subRun: 1 event: 55100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:33 GMT run: 74517585 subRun: 1 event: 55100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:33 GMT run: 74517585 subRun: 1 event: 55100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:33 GMT run: 74517585 subRun: 1 event: 55100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 232, 390, 224
minwire 0: 1897
minwire 1: 482
minwire 2: 2241
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0273243, 0.0137969, 0.958879, 
Output 1: 0.0719557, 0.866275, 0.0600018, 0.00176768, 
Output 2: 0.243137, 0.756863, 

438
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1339!
23-Nov-2024 07:32:42 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_550_20231204T071701Z_gen_g4_detsim_hitreco__20240509T210235Z_reco2_reco_data_2024-11-23T_062733Z.root"
23-Nov-2024 07:32:42 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a5/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_550_20231204T071701Z_gen_g4_detsim_hitreco__20240509T210235Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.25236       30.9101       813.094       9.31394       89.1358        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0280308     0.0608212     0.319574      0.0640371     0.0303078       100    
reco:linecluster:LineCluster                           0.274566      0.666953       21.9828      0.325067       2.19627        100    
reco:trajcluster:TrajCluster                           0.0813212      1.39848       65.1661      0.0920771      7.18597        100    
reco:pandora:StandardPandora                            1.0057        3.1676        122.929       1.14631       13.1514        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000143871   0.00851911     0.381875      0.0015697     0.0385677       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000181777    0.0122967     0.166913      0.0019272     0.0313841       100    
reco:pandoracalo:Calorimetry                          0.00010812     0.0399792      2.67843     0.000713067    0.269078        100    
reco:pandorapid:Chi2ParticleID                        3.8434e-05    0.000165749   0.00161973     0.0001173    0.00019592       100    
reco:trkshowersplit:TrackShowerHits                   0.000124837    0.0240897      1.15119     0.000570978    0.122012        100    
reco:pmtrack:PMAlgTrackMaker                          0.00128703      2.19794       83.0413      0.136838       9.02609        100    
reco:pmtrackcalo:Calorimetry                          7.5799e-05     0.0113444     0.435576     0.000859792    0.0470488       100    
reco:pmtrackpid:Chi2ParticleID                        3.0638e-05    0.000278882   0.00349435    0.000139406   0.000460375      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000209612     2.11793       78.5539      0.127217       8.34685        100    
reco:pmtrajfitcalo:Calorimetry                        6.2067e-05     0.0411928      2.77275     0.00093684     0.278392        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7571e-05    0.000178515    0.0013401    0.000128264   0.000169634      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00106683      2.87944       96.7333      0.172262       11.2855        100    
reco:pmtracktccalo:Calorimetry                        7.0144e-05    0.00763146     0.259802     0.00078465     0.0280616       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7988e-05    0.000257178   0.00304494    0.00014216    0.00038273       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000196767     1.50693       47.8784      0.161819       5.14645        100    
reco:pmtrajfittccalo:Calorimetry                      6.1305e-05     0.0180497     0.826503     0.00103999     0.086114        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.554e-05    0.000348547    0.0047801    0.000148542   0.000622508      100    
reco:blurredcluster:BlurredClustering                 0.00350611      1.17278       10.0166      0.547816       1.62601        100    
reco:emtrkmichelid:EmTrackMichelId                     0.271698       12.788        264.421       3.07748       31.0618        100    
reco:emshower:EMShower                                0.00025311     0.0322164     0.649394     0.00619325     0.0766826       100    
reco:cvnmap:CVNMapper                                 2.2874e-05     0.0180212     0.0743792     0.0205316     0.0157141       100    
reco:cvneva:CVNEvaluator                              1.6252e-05     0.518956      0.910473      0.718369      0.341555        100    
reco:energyrecnumu:EnergyReco                         0.000322158    0.0560024      2.31276     0.00529651     0.272858        100    
reco:energyrecnue:EnergyReco                          0.000121196   0.00272792      0.13174     0.000441574    0.0132045       100    
reco:energyrecnc:EnergyReco                           0.00156354    0.00445891     0.136605     0.00216413     0.0135382       100    
reco:energyrecnumurange:EnergyReco                    0.000228866    0.0538468      2.36132      0.0019381     0.277294        100    
reco:energyrecnumumcs:EnergyReco                      0.00142513     0.0538141      2.24669      0.0022976      0.26624        100    
reco:opdec:Deconvolution                               0.122521      0.407498      0.840655      0.400034      0.145682        100    
reco:ophitspe:OpHitFinderDeco                          0.329785       1.41474       2.69063       1.37591      0.552469        100    
reco:opflash:OpFlashFinder                            0.000234485   0.00141174    0.00438975    0.00130192    0.000697943      100    
reco:rns:RandomNumberSaver                            1.7983e-05    3.93482e-05    0.0002365    3.74905e-05   2.45362e-05      100    
reco:anglereconue:NuAngularReco                       0.000162645   0.00234712     0.125376     0.000329998    0.0125389       100    
reco:anglereconumu:NuAngularReco                      6.6483e-05    0.00200174     0.114878     0.000173276    0.011501        100    
reco:anglereconuepfps:NuAngularReco                   0.000116971   0.00205803     0.108761     0.000284941    0.0109033       100    
reco:anglereconumupfps:NuAngularReco                  9.6709e-05    0.00209619      0.11925     0.00023046     0.0119262       100    
[art]:TriggerResults:TriggerResultInserter             8.439e-06    1.26085e-05   5.3631e-05    1.13145e-05   5.38467e-06      100    
end_path:out1:RootOutput                               2.751e-06    7.0066e-06    2.4352e-05    7.0645e-06    3.70262e-06      100    
end_path:out1:RootOutput(write)                        0.0131484     0.214562       1.19756      0.212665      0.170544        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 13024.9 MB
  Peak resident set size usage (VmHWM): 2640.06 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_550_20231204T071701Z_gen_g4_detsim_hitreco__20240509T210235Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_550_20231204T071701Z_gen_g4_detsim_hitreco__20240509T210235Z_reco2_reco_2024-11-23T_062733Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_550_20231204T071701Z_gen_g4_detsim_hitreco__20240509T210235Z_reco2_reco_data_2024-11-23T_062733Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work