Jobsub ID 108743.26@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108743.26@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:34:30
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 05:57:52
From worker node	Hostname	b9p12p3947.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:38:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_432_20231208T080351Z_gen_g4_detsim_hitreco__20240510T062441Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4760s)
	CPU time	1h (4870s = 102%)
Outputting started	2024-11-23 05:57:34
Output files
Finished	2024-11-23 05:57:52
List job events Wrapper job log
Jobscript log (last 10,000 characters)

30 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:57:26 CET run: 74581430 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:57:26 CET run: 74581430 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:57:26 CET run: 74581430 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:57:26 CET run: 74581430 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:57:26 CET run: 74581430 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:57:26 CET run: 74581430 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 203, 68, 147
minwire 0: 1329
minwire 1: 713
minwire 2: 1481
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00558028, 0.000199719, 0.99422, 
Output 1: 0.042094, 0.949067, 0.00876653, 7.26571e-05, 
Output 2: 0.99463, 0.00537018, 

484
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1555!
23-Nov-2024 06:57:33 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_432_20231208T080351Z_gen_g4_detsim_hitreco__20240510T062441Z_reco2_reco_data_2024-11-23T_043819Z.root"
23-Nov-2024 06:57:33 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/a7/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_432_20231208T080351Z_gen_g4_detsim_hitreco__20240510T062441Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.8587        46.663        1029.03       9.14509       152.96         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00353017     0.0064125     0.0210308    0.00644994    0.00240534       100    
reco:linecluster:LineCluster                           0.035752      0.589043       20.8841      0.0447561      2.84272        100    
reco:trajcluster:TrajCluster                          0.00948096      2.63759       104.084      0.0209555      13.5306        100    
reco:pandora:StandardPandora                           0.812389       6.58739       307.321       1.22783       32.8549        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000132613    0.0121833     0.384815     0.00162355     0.0460028       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000247375    0.013896      0.185139     0.00302579     0.0311813       100    
reco:pandoracalo:Calorimetry                          9.5962e-05     0.0628744      2.22489     0.00118672     0.292483        100    
reco:pandorapid:Chi2ParticleID                         3.483e-05    0.000185747   0.00143457    0.000139414   0.000187244      100    
reco:trkshowersplit:TrackShowerHits                   0.000103662    0.0652339      2.27142     0.000800817    0.304131        100    
reco:pmtrack:PMAlgTrackMaker                           0.0010855      5.01938       130.556      0.180055       20.2129        100    
reco:pmtrackcalo:Calorimetry                          6.7431e-05     0.028501       1.28567      0.0011257     0.141371        100    
reco:pmtrackpid:Chi2ParticleID                         2.64e-05     0.000379109   0.00516207    0.000173849   0.000745239      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000253455     4.74183       135.521      0.386752       18.4934        100    
reco:pmtrajfitcalo:Calorimetry                        7.9821e-05     0.0625711      1.98353      0.0017225     0.276496        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2521e-05    0.00022565    0.00120316    0.000177999   0.000215899      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0012123      6.24041       167.107      0.275302       24.7132        100    
reco:pmtracktccalo:Calorimetry                        8.0022e-05     0.0170471     0.670314     0.00120712     0.0767907       100    
reco:pmtracktcpid:Chi2ParticleID                      3.1141e-05    0.000311639   0.00269864    0.000182309   0.000494397      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000211185     2.68628       70.9322      0.412683       9.50493        100    
reco:pmtrajfittccalo:Calorimetry                      7.9181e-05     0.0347272      1.45355     0.00142644      0.16494        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.626e-05    0.000433075    0.0062929     0.0001968    0.000887498      100    
reco:blurredcluster:BlurredClustering                 0.00871058      1.11755       11.3806      0.440674       1.89472        100    
reco:emtrkmichelid:EmTrackMichelId                     0.234603       13.2958       252.011       2.7587        36.4312        100    
reco:emshower:EMShower                                0.000299546    0.0474308     0.781796     0.00839181     0.111742        100    
reco:cvnmap:CVNMapper                                 2.9811e-05     0.0126421     0.143015     0.00875212     0.0170068       100    
reco:cvneva:CVNEvaluator                               1.622e-05     0.735486       1.42516      0.960196      0.473861        100    
reco:energyrecnumu:EnergyReco                         0.000335797    0.0481205      2.29747     0.00522888     0.241204        100    
reco:energyrecnue:EnergyReco                          0.000119913   0.00411761     0.118846     0.000544637    0.0149021       100    
reco:energyrecnc:EnergyReco                           0.00174237    0.00680356     0.125469     0.00335194     0.0154419       100    
reco:energyrecnumurange:EnergyReco                    0.000168003    0.0453853      2.26398     0.00308841      0.23771        100    
reco:energyrecnumumcs:EnergyReco                      0.00146062     0.0482153      2.30761     0.00374476     0.242299        100    
reco:opdec:Deconvolution                               0.0196639     0.184152      0.360555      0.180989      0.0723805       100    
reco:ophitspe:OpHitFinderDeco                          0.149966       2.01513       4.36356       1.93409      0.915771        100    
reco:opflash:OpFlashFinder                            0.000355028   0.00157203    0.00395541    0.00154899    0.000696549      100    
reco:rns:RandomNumberSaver                             1.815e-05    4.27731e-05   0.000318127   3.99705e-05   3.33376e-05      100    
reco:anglereconue:NuAngularReco                       0.000145193   0.00316333     0.110775     0.000434165    0.0125707       100    
reco:anglereconumu:NuAngularReco                      6.5641e-05    0.00289196     0.110249     0.00025904     0.012409        100    
reco:anglereconuepfps:NuAngularReco                   0.000114162   0.00306373      0.10995     0.000401164    0.0123887       100    
reco:anglereconumupfps:NuAngularReco                  9.2202e-05     0.002943      0.109891     0.000315252    0.0123362       100    
[art]:TriggerResults:TriggerResultInserter             9.06e-06     1.50198e-05   4.2811e-05    1.37805e-05   6.16435e-06      100    
end_path:out1:RootOutput                               3.17e-06     6.36322e-06    1.854e-05     6.665e-06    2.24083e-06      100    
end_path:out1:RootOutput(write)                        0.0217272     0.279374       2.09984      0.255032      0.278965        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21873.4 MB
  Peak resident set size usage (VmHWM): 2441.54 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_432_20231208T080351Z_gen_g4_detsim_hitreco__20240510T062441Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_432_20231208T080351Z_gen_g4_detsim_hitreco__20240510T062441Z_reco2_reco_2024-11-23T_043819Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_432_20231208T080351Z_gen_g4_detsim_hitreco__20240510T062441Z_reco2_reco_data_2024-11-23T_043819Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work