Jobsub ID 108752.18@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108752.18@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:46:20
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 05:39:29
From worker node	Hostname	b9p12p0328.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:47:27
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_795_20231204T083035Z_gen_g4_detsim_hitreco__20240509T194746Z_reco2.root
Jobscript	Exit code	0
	Real time	51m (3105s)
	CPU time	55m (3317s = 106%)
Outputting started	2024-11-23 05:39:12
Output files
Finished	2024-11-23 05:39:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

rajfittc@BeginModule  23-Nov-2024 06:38:23 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:03 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:03 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:03 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:03 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:04 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:04 CET run: 6422672 subRun: 1 event: 79600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 2049, 2838, 2225
minwire 0: 944
minwire 1: 314
minwire 2: 607
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00536092, 0.986809, 0.00783008, 
Output 1: 0.192967, 0.493635, 0.229223, 0.0841751, 
Output 2: 0.0280769, 0.971923, 

784
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3023!
23-Nov-2024 06:39:11 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_795_20231204T083035Z_gen_g4_detsim_hitreco__20240509T194746Z_reco2_reco_data_2024-11-23T_044731Z.root"
23-Nov-2024 06:39:11 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/97/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_795_20231204T083035Z_gen_g4_detsim_hitreco__20240509T194746Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.540605       29.0996       502.97        8.17556       64.7913        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0175296     0.0424712     0.110045      0.0424433     0.0151639       100    
reco:linecluster:LineCluster                           0.104152       1.00884       59.4904      0.174544       5.98307        100    
reco:trajcluster:TrajCluster                           0.047452       1.17539       44.101       0.0627726      4.93709        100    
reco:pandora:StandardPandora                           0.0524953      2.18491       46.5872       0.9689        5.3917         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000114051   0.00571845     0.0530489    0.00158807     0.0105225       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000160942    0.0102203     0.0982606    0.00285847     0.017823        100    
reco:pandoracalo:Calorimetry                          8.7191e-05     0.0262936     0.835916      0.001196      0.0993881       100    
reco:pandorapid:Chi2ParticleID                         3.271e-05    0.000135296   0.000992919   0.00010022    0.000122438      100    
reco:trkshowersplit:TrackShowerHits                   6.6671e-05     0.0357284     0.907601     0.000883233    0.126538        100    
reco:pmtrack:PMAlgTrackMaker                          0.000914758     3.23823       91.2442       0.30552       10.7859        100    
reco:pmtrackcalo:Calorimetry                          5.7011e-05    0.00848794     0.174875     0.000902228    0.0216762       100    
reco:pmtrackpid:Chi2ParticleID                         2.246e-05    0.00024284    0.00213408    0.000120901   0.000329296      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000157831     2.42262       54.2268       0.30006       7.10544        100    
reco:pmtrajfitcalo:Calorimetry                        5.9311e-05     0.0260756     0.837697      0.001369      0.0982533       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.392e-05    0.000151936   0.000659706   0.000115991   0.000121391      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000801367     3.6231        77.9036       0.22786       10.6955        100    
reco:pmtracktccalo:Calorimetry                        5.5881e-05    0.00565193      0.10827     0.000962169    0.0132855       100    
reco:pmtracktcpid:Chi2ParticleID                       2.29e-05     0.000226439   0.00132313    0.000130836   0.000255361      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000133241     1.55627       19.7459       0.25676        3.301         100    
reco:pmtrajfittccalo:Calorimetry                       5.576e-05     0.0161334     0.314865      0.0011263     0.0451499       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.293e-05    0.000329785   0.00244027    0.000137026   0.000475282      100    
reco:blurredcluster:BlurredClustering                 0.000119391    0.846025       5.55866      0.404628       1.08317        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000120141     10.3395       98.5926       2.77296       18.174         100    
reco:emshower:EMShower                                0.000139411    0.0267505     0.315401     0.00381621     0.0506123       100    
reco:cvnmap:CVNMapper                                  1.925e-05    0.00814506     0.0301957    0.00612502    0.00731944       100    
reco:cvneva:CVNEvaluator                               1.478e-05     0.556819       1.04568      0.691031      0.340002        100    
reco:energyrecnumu:EnergyReco                         0.000260822    0.0210864     0.374202     0.00415093     0.0550048       100    
reco:energyrecnue:EnergyReco                          0.000114741   0.00195375     0.0189727    0.000456249   0.00361681       100    
reco:energyrecnc:EnergyReco                           0.00134747    0.00372094     0.021748     0.00207147    0.00392535       100    
reco:energyrecnumurange:EnergyReco                    0.000122751    0.0197139     0.377011     0.00209138     0.0561662       100    
reco:energyrecnumumcs:EnergyReco                      0.000701136    0.0208984     0.375843      0.0035572     0.0555634       100    
reco:opdec:Deconvolution                               0.0495399     0.272186       0.60589      0.260798      0.0984375       100    
reco:ophitspe:OpHitFinderDeco                           0.14235       1.39589       3.75481       1.39264      0.604133        100    
reco:opflash:OpFlashFinder                            0.000119471    0.0011218    0.00354069    0.00104587    0.000540458      100    
reco:rns:RandomNumberSaver                             1.65e-05     2.79514e-05   0.000294673   2.6125e-05    2.75845e-05      100    
reco:anglereconue:NuAngularReco                       0.000125021   0.00142137     0.018498     0.000325543   0.00287595       100    
reco:anglereconumu:NuAngularReco                      5.7131e-05    0.00125656     0.0184317    0.000199327    0.0028108       100    
reco:anglereconuepfps:NuAngularReco                   9.5391e-05     0.0013458     0.0174332    0.000294268   0.00274743       100    
reco:anglereconumupfps:NuAngularReco                  8.1981e-05    0.00125369     0.0155368    0.000249588   0.00256396       100    
[art]:TriggerResults:TriggerResultInserter             8.45e-06     1.10265e-05    3.601e-05     1.002e-05    3.57028e-06      100    
end_path:out1:RootOutput                               2.54e-06     4.87338e-06   1.6671e-05     4.05e-06     2.38175e-06      100    
end_path:out1:RootOutput(write)                        0.0146389     0.191721      0.689963      0.179813      0.133247        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21652.4 MB
  Peak resident set size usage (VmHWM): 2227.2 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_795_20231204T083035Z_gen_g4_detsim_hitreco__20240509T194746Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_795_20231204T083035Z_gen_g4_detsim_hitreco__20240509T194746Z_reco2_reco_2024-11-23T_044731Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_795_20231204T083035Z_gen_g4_detsim_hitreco__20240509T194746Z_reco2_reco_data_2024-11-23T_044731Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work