Jobsub ID 108748.25@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108748.25@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:40:58
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 06:17:52
From worker node	Hostname	td811.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:46:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_31_20231204T232403Z_gen_g4_detsim_hitreco__20240509T222336Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5453s)
	CPU time	1h (4799s = 88%)
Outputting started	2024-11-23 06:17:31
Output files
Finished	2024-11-23 06:17:52
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ilure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:17:19 CET run: 50601160 subRun: 1 event: 3200
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:17:19 CET run: 50601160 subRun: 1 event: 3200
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:17:19 CET run: 50601160 subRun: 1 event: 3200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:17:19 CET run: 50601160 subRun: 1 event: 3200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 67, 69, 63
minwire 0: 1736
minwire 1: 909
minwire 2: 1708
Used alternate method to get min and max wires due to vertex determination failure: 19, 218
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 229, 428
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.99158, 0.000495195, 0.00792438, 
Output 1: 0.903336, 0.087077, 0.00791357, 0.00167329, 
Output 2: 0.994763, 0.00523679, 

480
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1486!
23-Nov-2024 07:17:29 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_31_20231204T232403Z_gen_g4_detsim_hitreco__20240509T222336Z_reco2_reco_data_2024-11-23T_044642Z.root"
23-Nov-2024 07:17:29 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/2b/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_31_20231204T232403Z_gen_g4_detsim_hitreco__20240509T222336Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.67143       49.9527       701.479       14.1092       125.677        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0459107     0.0862927     0.244827      0.0915148     0.0289086       100    
reco:linecluster:LineCluster                           0.268797      0.908418       15.5859      0.442184       2.1832         100    
reco:trajcluster:TrajCluster                           0.134641       2.41628       52.5888       0.15169       9.63298        100    
reco:pandora:StandardPandora                           0.154523       3.88995       54.9404       1.93037       8.5377         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000158381   0.00908756      0.21138     0.00194896     0.0250743       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000287273    0.0160673     0.192754     0.00469998     0.0313831       100    
reco:pandoracalo:Calorimetry                          0.000124061    0.0335142     0.745009     0.00139435      0.12347        100    
reco:pandorapid:Chi2ParticleID                        4.2621e-05    0.000238301   0.00204546    0.000176442   0.000254053      100    
reco:trkshowersplit:TrackShowerHits                   0.00010687     0.0443518     0.902847     0.00115752     0.169328        100    
reco:pmtrack:PMAlgTrackMaker                          0.00148605      5.02737       103.354      0.441116       17.0301        100    
reco:pmtrackcalo:Calorimetry                          8.1691e-05     0.0197947     0.509292     0.00134832     0.0733603       100    
reco:pmtrackpid:Chi2ParticleID                        3.4411e-05    0.000458209   0.00458282    0.000196272   0.000841172      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000276453     4.6204        106.451      0.434476       15.764         100    
reco:pmtrajfitcalo:Calorimetry                        0.000104171    0.0393031     0.855046     0.00178603     0.145737        100    
reco:pmtrajfitpid:Chi2ParticleID                       4.324e-05    0.000299764   0.00219523    0.000217902   0.000331954      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00134202      6.93236       145.634      0.332261       24.7336        100    
reco:pmtracktccalo:Calorimetry                        7.9241e-05     0.0114542     0.199155     0.00156157      0.03469        100    
reco:pmtracktcpid:Chi2ParticleID                       3.352e-05    0.000414142   0.00341944    0.000224977   0.000596681      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000256973     3.14969       51.1813      0.571435       9.42655        100    
reco:pmtrajfittccalo:Calorimetry                      9.8961e-05     0.0294498     0.613995      0.0018699     0.106891        100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.114e-05    0.000590551   0.00562364    0.000233342   0.00104135       100    
reco:blurredcluster:BlurredClustering                 0.000134722     1.25597       6.46226      0.719301       1.45676        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000163062     16.3347       202.014       4.13271       37.1949        100    
reco:emshower:EMShower                                0.000295683    0.043687      0.351721      0.0128373     0.0688446       100    
reco:cvnmap:CVNMapper                                 2.7191e-05     0.0113876     0.0529554    0.00888811     0.0102169       100    
reco:cvneva:CVNEvaluator                               2.025e-05     0.973357       1.74119       1.22685      0.512369        100    
reco:energyrecnumu:EnergyReco                         0.000384153    0.0697884      3.36107     0.00640487     0.357221        100    
reco:energyrecnue:EnergyReco                          0.000151512    0.0031668     0.0664783    0.000579835   0.00875708       100    
reco:energyrecnc:EnergyReco                           0.00267443     0.0068788     0.0749208    0.00420135    0.00960104       100    
reco:energyrecnumurange:EnergyReco                    0.000273702    0.0668447      3.38665     0.00407655     0.358214        100    
reco:energyrecnumumcs:EnergyReco                      0.00115329     0.0694592      3.37688     0.00569734     0.357454        100    
reco:opdec:Deconvolution                               0.163029      0.574671       1.02016      0.582503      0.138175        100    
reco:ophitspe:OpHitFinderDeco                          0.656912       2.90786       5.54589       2.97951      0.992703        100    
reco:opflash:OpFlashFinder                            0.000346073    0.0019536    0.00786685    0.00177707    0.00095967       100    
reco:rns:RandomNumberSaver                             2.194e-05    4.56186e-05   0.000499835    3.384e-05    5.20904e-05      100    
reco:anglereconue:NuAngularReco                       0.000205432   0.00240814     0.0632248    0.000447069   0.00743153       100    
reco:anglereconumu:NuAngularReco                      0.000113981   0.00220023     0.0637264    0.000271898    0.0074545       100    
reco:anglereconuepfps:NuAngularReco                   0.000183121   0.00240985     0.062968     0.00044357    0.00742414       100    
reco:anglereconumupfps:NuAngularReco                  0.000166272   0.00229884     0.0634007    0.000373013   0.00741471       100    
[art]:TriggerResults:TriggerResultInserter             1.064e-05    1.74369e-05   9.6161e-05    1.5391e-05    9.13617e-06      100    
end_path:out1:RootOutput                                3.3e-06     6.49267e-06   3.4651e-05     4.81e-06     4.50377e-06      100    
end_path:out1:RootOutput(write)                        0.0289903     0.384917       2.09182      0.356039      0.296247        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21414.3 MB
  Peak resident set size usage (VmHWM): 2276.86 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_31_20231204T232403Z_gen_g4_detsim_hitreco__20240509T222336Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_31_20231204T232403Z_gen_g4_detsim_hitreco__20240509T222336Z_reco2_reco_2024-11-23T_044642Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_31_20231204T232403Z_gen_g4_detsim_hitreco__20240509T222336Z_reco2_reco_data_2024-11-23T_044642Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work