Jobsub ID 301783.11@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301783.11@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:24:30
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2024-11-23 06:54:28
From worker node	Hostname	b9p07p1560.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:25:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_386_20231201T105701Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5289s)
	CPU time	1h (5161s = 97%)
Outputting started	2024-11-23 06:54:08
Output files
Finished	2024-11-23 06:54:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:54:00 CET run: 74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:54:00 CET run: 74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:54:00 CET run: 74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:54:00 CET run: 74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:54:03 CET run: 74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:54:03 CET run: 74481105 subRun: 1 event: 38700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 98, 88, 92
minwire 0: 2422
minwire 1: 105
minwire 2: 2785
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000374734, 0.999251, 0.000374577, 
Output 1: 0.00217658, 0.996825, 0.000994271, 3.90622e-06, 
Output 2: 0.998174, 0.00182577, 

275
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 729!
23-Nov-2024 07:54:07 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_386_20231201T105701Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2_reco_data_2024-11-23T_052605Z.root"
23-Nov-2024 07:54:07 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cd/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_386_20231201T105701Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.574888       50.5468       1535.06       11.6377       169.789        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0224672     0.0432172     0.331255      0.0415042     0.0311196       100    
reco:linecluster:LineCluster                           0.0943057     0.798058       22.526       0.179351       3.19219        100    
reco:trajcluster:TrajCluster                           0.0475153      2.14883       94.9548      0.0632574      10.7399        100    
reco:pandora:StandardPandora                           0.0559584      9.75457       652.43        1.32985       65.5579        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000140151    0.0105319      0.32487     0.00147706     0.0363734       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000263842    0.0182317     0.630325     0.00309628     0.0648984       100    
reco:pandoracalo:Calorimetry                          0.000113561    0.0487361      1.96556     0.00107636     0.217325        100    
reco:pandorapid:Chi2ParticleID                         3.531e-05    0.000188101   0.00120661    0.000144291   0.000157914      100    
reco:trkshowersplit:TrackShowerHits                   0.000106951    0.0594135      2.07814     0.00084115     0.256856        100    
reco:pmtrack:PMAlgTrackMaker                          0.00109826      5.04835       148.59       0.196341       18.2012        100    
reco:pmtrackcalo:Calorimetry                           6.668e-05     0.0198888      0.83735     0.00110758     0.0889907       100    
reco:pmtrackpid:Chi2ParticleID                        2.7411e-05    0.000446585   0.00617096    0.000174921   0.000918017      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000191491     3.75767       90.6393      0.411992       13.0543        100    
reco:pmtrajfitcalo:Calorimetry                         6.528e-05     0.0428175      1.52952     0.00168443     0.180206        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.714e-05    0.000221095   0.00154511    0.000169941   0.000212213      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000887755     6.19012       172.496      0.255643       21.4057        100    
reco:pmtracktccalo:Calorimetry                         6.644e-05     0.0111106      0.3016      0.000998986    0.037045        100    
reco:pmtracktcpid:Chi2ParticleID                       2.768e-05    0.000340608   0.00359866    0.000170431   0.000519801      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000163661     3.12078       81.2061      0.288192       10.0007        100    
reco:pmtrajfittccalo:Calorimetry                       6.167e-05     0.0320185      1.14693     0.00140494     0.133983        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.499e-05    0.000520237   0.00679121    0.000195971   0.000995128      100    
reco:blurredcluster:BlurredClustering                 0.000172451     1.24158       9.2397       0.575417       1.79355        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000173741     14.2608       248.332       3.1674        32.7755        100    
reco:emshower:EMShower                                0.000264731    0.0418587     0.605269     0.00700925     0.0798989       100    
reco:cvnmap:CVNMapper                                  2.459e-05     0.0121643     0.0468413     0.0122826     0.0106639       100    
reco:cvneva:CVNEvaluator                               1.784e-05     0.870322       5.63513       1.03418       0.86745        100    
reco:energyrecnumu:EnergyReco                         0.000350052    0.0433948     0.574314     0.00540379     0.105589        100    
reco:energyrecnue:EnergyReco                          0.000129361   0.00332589     0.105353     0.000536733    0.011498        100    
reco:energyrecnc:EnergyReco                           0.00161128    0.00612833     0.109805     0.00349158     0.0117743       100    
reco:energyrecnumurange:EnergyReco                    0.000153971    0.0389309     0.565752     0.00314646     0.0974168       100    
reco:energyrecnumumcs:EnergyReco                      0.00165477     0.0412348     0.565037     0.00408718     0.0980389       100    
reco:opdec:Deconvolution                               0.069146      0.301413       0.50515      0.300332      0.0965978       100    
reco:ophitspe:OpHitFinderDeco                          0.143609       2.27977       4.26764       2.20644      0.975341        100    
reco:opflash:OpFlashFinder                            0.00013209    0.00165772    0.00492825    0.00158892    0.000799596      100    
reco:rns:RandomNumberSaver                             2.073e-05    4.5559e-05    0.000384662   4.18755e-05   3.56457e-05      100    
reco:anglereconue:NuAngularReco                       0.000181751   0.00251927     0.0708712    0.000478268   0.00819901       100    
reco:anglereconumu:NuAngularReco                       8.691e-05    0.00212902     0.0637668    0.000226447   0.00752701       100    
reco:anglereconuepfps:NuAngularReco                   0.000140141   0.00230433     0.0621927    0.000423902   0.00741018       100    
reco:anglereconumupfps:NuAngularReco                  0.000112041   0.00221615     0.0622774    0.000311472    0.0074328       100    
[art]:TriggerResults:TriggerResultInserter             9.82e-06     1.55702e-05   7.7851e-05     1.467e-05    6.82114e-06      100    
end_path:out1:RootOutput                               3.38e-06     6.95396e-06    2.363e-05     6.92e-06     2.17985e-06      100    
end_path:out1:RootOutput(write)                        0.0135671     0.286636       1.30267      0.272944      0.230119        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21825 MB
  Peak resident set size usage (VmHWM): 2382.95 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_386_20231201T105701Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_386_20231201T105701Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2_reco_2024-11-23T_052605Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_386_20231201T105701Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2_reco_data_2024-11-23T_052605Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work