Jobsub ID 108727.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108727.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:57:40
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:17:15
From worker node	Hostname	n256.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:01:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_419_20231207T000922Z_gen_g4_detsim_hitreco__20240510T061422Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4547s)
	CPU time	1h (3754s = 82%)
Outputting started	2024-11-23 05:16:58
Output files
Finished	2024-11-23 05:17:15
List job events Wrapper job log
Jobscript log (last 10,000 characters)

WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:16:50 GMT run: 66027687 subRun: 1 event: 42000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:16:50 GMT run: 66027687 subRun: 1 event: 42000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:16:50 GMT run: 66027687 subRun: 1 event: 42000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:16:50 GMT run: 66027687 subRun: 1 event: 42000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:16:50 GMT run: 66027687 subRun: 1 event: 42000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 40, 40, 31
minwire 0: 2137
minwire 1: 505
minwire 2: 1906
Used alternate method to get min and max wires due to vertex determination failure: 187, 386
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 76, 275
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 423, 622
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.985982, 0.000992173, 0.0130261, 
Output 1: 0.925282, 0.066683, 0.00661595, 0.00141859, 
Output 2: 0.993636, 0.0063645, 

336
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 985!
23-Nov-2024 05:16:57 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_419_20231207T000922Z_gen_g4_detsim_hitreco__20240510T061422Z_reco2_reco_data_2024-11-23T_040118Z.root"
23-Nov-2024 05:16:57 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/30/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_419_20231207T000922Z_gen_g4_detsim_hitreco__20240510T061422Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.15881       37.8163       876.714       13.7992       97.2716        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0349998     0.0678142     0.178566      0.073571      0.0250368       100    
reco:linecluster:LineCluster                           0.201634      0.908818       40.4425      0.380921       4.01135        100    
reco:trajcluster:TrajCluster                           0.101148       1.84633       126.305      0.115449       12.7131        100    
reco:pandora:StandardPandora                           0.113566       2.77019       61.0387       1.51472       6.45668        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000185659   0.00786734     0.108431     0.00186521     0.0170066       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000253056    0.016232      0.244776     0.00469229     0.0314703       100    
reco:pandoracalo:Calorimetry                          0.000128643    0.0253819      1.01129      0.0014419     0.109959        100    
reco:pandorapid:Chi2ParticleID                        5.1206e-05    0.000231361   0.00208435    0.000171658   0.000231217      100    
reco:trkshowersplit:TrackShowerHits                   9.2684e-05     0.0396042      1.71271     0.00126037     0.189291        100    
reco:pmtrack:PMAlgTrackMaker                          0.00167127      3.22061       120.262      0.303885       12.9395        100    
reco:pmtrackcalo:Calorimetry                          0.000103204     0.01357      0.430455     0.00121689     0.0510335       100    
reco:pmtrackpid:Chi2ParticleID                        4.2359e-05    0.000378918   0.00493634    0.000202095   0.000623644      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000256553     2.98762       92.5487      0.375769       9.94771        100    
reco:pmtrajfitcalo:Calorimetry                        9.9167e-05     0.0247673      1.02785     0.00177448      0.10901        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.0266e-05    0.000255185   0.00108076    0.000203352   0.00019801       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00152101      4.30544       182.361       0.31093       19.459         100    
reco:pmtracktccalo:Calorimetry                        9.7614e-05    0.00906664     0.142449     0.00129416     0.024067        100    
reco:pmtracktcpid:Chi2ParticleID                      3.9344e-05    0.000356158   0.00273287    0.00019381    0.000472674      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000235834     2.1483        52.3764      0.441828       6.13306        100    
reco:pmtrajfittccalo:Calorimetry                      9.5489e-05     0.0185391     0.524624     0.00149291     0.0679967       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.8522e-05    0.000468749   0.00461309    0.000207275   0.000750776      100    
reco:blurredcluster:BlurredClustering                 9.3837e-05      1.39955       9.9003       0.771094       1.82835        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000152157     13.3095       176.94        3.94343       26.5103        100    
reco:emshower:EMShower                                0.000227248    0.039406      0.490365      0.010187      0.0727587       100    
reco:cvnmap:CVNMapper                                  2.634e-05     0.0126734     0.0497668     0.0089624     0.0106873       100    
reco:cvneva:CVNEvaluator                              2.2573e-05     0.772366       1.26032      0.949651       0.3911         100    
reco:energyrecnumu:EnergyReco                         0.00050481     0.0357962     0.556575     0.00543017     0.091216        100    
reco:energyrecnue:EnergyReco                          0.000220294   0.00245063     0.0342955    0.000622757   0.00499255       100    
reco:energyrecnc:EnergyReco                           0.00302786    0.00570107     0.0392432    0.00384418    0.00546402       100    
reco:energyrecnumurange:EnergyReco                    0.000272814    0.033356      0.551772     0.00340364     0.0907038       100    
reco:energyrecnumumcs:EnergyReco                      0.00147196     0.035611      0.561694     0.00431646     0.0910798       100    
reco:opdec:Deconvolution                               0.172047      0.477558      0.828044      0.470058      0.140118        100    
reco:ophitspe:OpHitFinderDeco                          0.314722       2.9395        5.2395        2.95059       1.09435        100    
reco:opflash:OpFlashFinder                            0.000210837   0.00179864    0.00686851    0.00180292    0.000871954      100    
reco:rns:RandomNumberSaver                            2.3614e-05    4.17869e-05   0.000397628   3.73605e-05   3.71776e-05      100    
reco:anglereconue:NuAngularReco                       0.00021265     0.0018319     0.0284531    0.000522694   0.00393793       100    
reco:anglereconumu:NuAngularReco                      0.000111721   0.00162427     0.0282435    0.000335466   0.00387447       100    
reco:anglereconuepfps:NuAngularReco                   0.000172505   0.00179097     0.0284917    0.000460627    0.0039154       100    
reco:anglereconumupfps:NuAngularReco                  0.000156044   0.00171012     0.0286783    0.000405674   0.00392646       100    
[art]:TriggerResults:TriggerResultInserter            1.1582e-05    1.61938e-05   7.2907e-05    1.5043e-05    6.38846e-06      100    
end_path:out1:RootOutput                               3.296e-06    6.71668e-06   3.0958e-05     5.08e-06     4.14256e-06      100    
end_path:out1:RootOutput(write)                        0.0127751     0.329752       1.37866      0.314874      0.213355        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38364.5 MB
  Peak resident set size usage (VmHWM): 2297.59 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_419_20231207T000922Z_gen_g4_detsim_hitreco__20240510T061422Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_419_20231207T000922Z_gen_g4_detsim_hitreco__20240510T061422Z_reco2_reco_2024-11-23T_040118Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_419_20231207T000922Z_gen_g4_detsim_hitreco__20240510T061422Z_reco2_reco_data_2024-11-23T_040118Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work