Jobsub ID 108744.159@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.159@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:12:19
From worker node	Hostname	b9p05p5810.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:44:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2.root
Jobscript	Exit code	0
	Real time	27m (1672s)
	CPU time	34m (2048s = 122%)
Outputting started	2024-11-23 05:12:05
Output files
Finished	2024-11-23 05:12:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

turn failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:11:53 CET run: 66027687 subRun: 1 event: 48100
Hit sorting problem.
%MSG
Boundary wire vector sizes: 546, 362, 531
minwire 0: 861
minwire 1: 1318
minwire 2: 954
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00280248, 0.000427788, 0.99677, 
Output 1: 0.100892, 0.878001, 0.0186435, 0.0024641, 
Output 2: 0.983396, 0.0166039, 

489
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1629!
23-Nov-2024 06:12:04 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2_reco_data_2024-11-23T_044416Z.root"
23-Nov-2024 06:12:04 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/05/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.14472       15.9472       195.068       6.46162       29.7244        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00322455    0.00539712     0.0257922    0.00555058    0.00252404       100    
reco:linecluster:LineCluster                           0.0308771     0.0771159      0.87186      0.0381665     0.135461        100    
reco:trajcluster:TrajCluster                          0.00861904     0.229888       5.06601      0.0166135     0.709974        100    
reco:pandora:StandardPandora                           0.730868       1.58803       46.3017      0.900811       4.57247        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112511   0.00334613     0.0385974    0.00109891    0.00644355       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000153701   0.00565184     0.050202       0.0022      0.00864104       100    
reco:pandoracalo:Calorimetry                          8.7591e-05    0.00954024     0.210013     0.000651303    0.0298868       100    
reco:pandorapid:Chi2ParticleID                         3.225e-05    0.000123692   0.000860821   0.000100376   0.000101333      100    
reco:trkshowersplit:TrackShowerHits                   7.9671e-05    0.00996967     0.194386     0.000534071    0.0290284       100    
reco:pmtrack:PMAlgTrackMaker                          0.00088329      1.13252       20.9956      0.109803       3.07654        100    
reco:pmtrackcalo:Calorimetry                           5.588e-05     0.0056112     0.172424     0.000540411    0.0201256       100    
reco:pmtrackpid:Chi2ParticleID                        2.2041e-05    0.000200285    0.0028826    0.000105291   0.000338179      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000143651     1.17399       20.3932       0.17968       2.97139        100    
reco:pmtrajfitcalo:Calorimetry                         5.014e-05    0.00920469     0.176027     0.00110136     0.0274174       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.227e-05    0.000129196   0.000412755   0.000105221   8.79844e-05      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000749628     1.45556       23.6259      0.161467       3.92524        100    
reco:pmtracktccalo:Calorimetry                        5.5591e-05    0.00404682     0.0998321    0.000615547    0.0132492       100    
reco:pmtracktcpid:Chi2ParticleID                      2.2531e-05    0.000170672   0.00141522    9.9701e-05    0.000210208      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000128442    0.813419       12.3694      0.169207       1.91941        100    
reco:pmtrajfittccalo:Calorimetry                       5.072e-05    0.00698755     0.137912     0.000775949    0.0210184       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1731e-05    0.000234027   0.00180036    0.000120442   0.000325982      100    
reco:blurredcluster:BlurredClustering                 0.00395208     0.730414       5.79683      0.327939       1.00916        100    
reco:emtrkmichelid:EmTrackMichelId                     0.133295       6.51036       70.8664       1.98587       12.6414        100    
reco:emshower:EMShower                                0.000224113    0.0199884     0.222267     0.00388123     0.0367111       100    
reco:cvnmap:CVNMapper                                  1.99e-05      0.0093771     0.035869     0.00973559    0.00762667       100    
reco:cvneva:CVNEvaluator                               1.417e-05     0.568505       1.16421      0.708206      0.349654        100    
reco:energyrecnumu:EnergyReco                         0.000293143    0.016008      0.308136     0.00313326     0.0383168       100    
reco:energyrecnue:EnergyReco                          0.000111552   0.00119754     0.018744     0.000335194   0.00243337       100    
reco:energyrecnc:EnergyReco                           0.00136986    0.00285176     0.0194334    0.00192952    0.00244519       100    
reco:energyrecnumurange:EnergyReco                    0.000143442    0.0133106     0.220088     0.00159537     0.0315003       100    
reco:energyrecnumumcs:EnergyReco                      0.00133552     0.0146458     0.256481     0.00262463     0.0338202       100    
reco:opdec:Deconvolution                               0.0170845     0.144676      0.331919      0.142625      0.0570321       100    
reco:ophitspe:OpHitFinderDeco                          0.0770075      1.21021       3.56844       1.23265      0.554079        100    
reco:opflash:OpFlashFinder                            0.000170132   0.00103534    0.00358602    0.00100513    0.000504804      100    
reco:rns:RandomNumberSaver                             1.601e-05    3.12824e-05   0.000319264   2.8805e-05    3.03357e-05      100    
reco:anglereconue:NuAngularReco                       0.000114652   0.000776068   0.00983479    0.000288858   0.00147078       100    
reco:anglereconumu:NuAngularReco                       5.413e-05    0.000618141   0.00931914    0.000167592    0.0013997       100    
reco:anglereconuepfps:NuAngularReco                   9.8672e-05    0.000722515   0.00877146    0.000274203   0.00136894       100    
reco:anglereconumupfps:NuAngularReco                  8.1411e-05    0.000660031   0.00862442    0.000226148   0.00134389       100    
[art]:TriggerResults:TriggerResultInserter             7.38e-06     9.92373e-06   3.6751e-05     9.095e-06    3.24358e-06      100    
end_path:out1:RootOutput                               2.53e-06     5.69543e-06    1.934e-05     5.985e-06    2.50194e-06      100    
end_path:out1:RootOutput(write)                        0.0120169     0.169277       0.55094      0.174858      0.102428        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21821.2 MB
  Peak resident set size usage (VmHWM): 2379.69 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2_reco_2024-11-23T_044416Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2_reco_data_2024-11-23T_044416Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work