Jobsub ID 301868.30@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301868.30@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:20:29
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce01
Last heartbeat	2024-11-23 08:15:13
From worker node	Hostname	dune001-3732425.0-lcg2713.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:21:52
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_644_20231202T190922Z_gen_g4_detsim_hitreco__20240508T063244Z_reco2.root
Outputting started	2024-11-23 08:15:13
Output files
Finished	2024-11-24 13:17:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

vent: 64500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:15:03 UTC run: 50571149 subRun: 1 event: 64500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:15:03 UTC run: 50571149 subRun: 1 event: 64500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:15:04 UTC run: 50571149 subRun: 1 event: 64500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:15:04 UTC run: 50571149 subRun: 1 event: 64500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:15:04 UTC run: 50571149 subRun: 1 event: 64500
1st wire C:0 T:23 P:2 W:509 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:15:08 UTC run: 50571149 subRun: 1 event: 64500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 221, 219, 249
minwire 0: 2269
minwire 1: 572
minwire 2: 2281
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00340563, 0.996485, 0.000108928, 
Output 1: 0.995275, 0.00459792, 9.96668e-05, 2.70252e-05, 
Output 2: 0.999017, 0.000983395, 

443
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1370!
23-Nov-2024 08:15:12 UTC  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_644_20231202T190922Z_gen_g4_detsim_hitreco__20240508T063244Z_reco2_reco_data_2024-11-23T_072156Z.root"
23-Nov-2024 08:15:12 UTC  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/45/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_644_20231202T190922Z_gen_g4_detsim_hitreco__20240508T063244Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.22339       29.4112       486.238       8.99849       71.6525        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0259259     0.0408039     0.0584346     0.0306328     0.0131508       100    
reco:linecluster:LineCluster                           0.233643       1.07507       34.4184      0.242257       4.71762        100    
reco:trajcluster:TrajCluster                           0.0757794      1.02352       36.1777      0.0828316      4.74568        100    
reco:pandora:StandardPandora                            1.06943       2.2268        37.8313       1.30542       4.36639        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000158198   0.00559567     0.050081     0.00133826     0.0104834       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000210125    0.0164143     0.287906      0.0020752     0.0434629       100    
reco:pandoracalo:Calorimetry                          0.000112449    0.0198695     0.474399     0.000697177    0.0743001       100    
reco:pandorapid:Chi2ParticleID                        4.2464e-05    0.000186377   0.00206347    0.00013961    0.00021627       100    
reco:trkshowersplit:TrackShowerHits                   0.000110065    0.0295604     0.792573     0.000695621    0.121614        100    
reco:pmtrack:PMAlgTrackMaker                          0.00105861      3.61811       101.322      0.169625       13.8239        100    
reco:pmtrackcalo:Calorimetry                          5.7517e-05    0.00927065     0.264799     0.000734139    0.0318225       100    
reco:pmtrackpid:Chi2ParticleID                        2.1963e-05    0.000301141   0.00355524    0.000145789   0.000489564      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000211378     2.91446       83.9338       0.15867       10.491         100    
reco:pmtrajfitcalo:Calorimetry                        6.4517e-05     0.0238799     0.656846      0.0010311     0.0962623       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5629e-05    0.00020361    0.000816357   0.000164972   0.000156656      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00126128      3.40547       93.1026      0.137558       12.4297        100    
reco:pmtracktccalo:Calorimetry                        7.4231e-05    0.00576133     0.069775     0.000793978    0.0126809       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9765e-05    0.000271609   0.00200807    0.000164572   0.000312333      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000185209     1.49891       23.2697      0.197525       3.67842        100    
reco:pmtrajfittccalo:Calorimetry                      6.5148e-05     0.0144122      0.3737      0.000861594    0.0496108       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7581e-05    0.000368618   0.00330411    0.000183335   0.00054418       100    
reco:blurredcluster:BlurredClustering                  0.0084117     0.973031       5.12078      0.615613       1.13829        100    
reco:emtrkmichelid:EmTrackMichelId                      0.22987       9.48576       113.896       2.25127       18.9343        100    
reco:emshower:EMShower                                0.000322314    0.0299399      0.30839     0.00885889     0.0504246       100    
reco:cvnmap:CVNMapper                                 2.6069e-05     0.0109346     0.0478398    0.00581582     0.011163        100    
reco:cvneva:CVNEvaluator                              1.9009e-05     0.563704       1.03404      0.812989      0.414023        100    
reco:energyrecnumu:EnergyReco                         0.000424999    0.0296226     0.871856     0.00646418     0.100606        100    
reco:energyrecnue:EnergyReco                          0.000133921   0.00200819     0.0207737    0.000531929   0.00361892       100    
reco:energyrecnc:EnergyReco                            0.0015556    0.00409391     0.0254363    0.00255609    0.00407458       100    
reco:energyrecnumurange:EnergyReco                    0.000260391    0.0260805     0.880782     0.00245341     0.101641        100    
reco:energyrecnumumcs:EnergyReco                      0.00160948     0.0272215      0.87945     0.00292941     0.101366        100    
reco:opdec:Deconvolution                               0.0821537     0.310187      0.500128      0.316654      0.0864665       100    
reco:ophitspe:OpHitFinderDeco                          0.276809       1.78035       3.29867       1.81336      0.731853        100    
reco:opflash:OpFlashFinder                            0.000392359   0.00154887    0.00681705    0.00150488    0.000823224      100    
reco:rns:RandomNumberSaver                            1.9259e-05    4.17097e-05   0.000467813   3.55385e-05   4.44558e-05      100    
reco:anglereconue:NuAngularReco                       0.000182274   0.00149922     0.0158966    0.000435054    0.0029556       100    
reco:anglereconumu:NuAngularReco                      6.5098e-05     0.0011851     0.0156032    0.000187171   0.00279712       100    
reco:anglereconuepfps:NuAngularReco                   0.000121623   0.00137813     0.0163569    0.000343866   0.00293462       100    
reco:anglereconumupfps:NuAngularReco                  9.5964e-05    0.00126819     0.0156133    0.000254212   0.00288894       100    
[art]:TriggerResults:TriggerResultInserter             8.883e-06    1.62172e-05   8.2434e-05    1.3926e-05    9.45609e-06      100    
end_path:out1:RootOutput                               3.255e-06    4.96573e-06   2.9384e-05     4.011e-06    2.90643e-06      100    
end_path:out1:RootOutput(write)                        0.0184754     0.229945      0.991852      0.230743      0.178268        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 40288.1 MB
  Peak resident set size usage (VmHWM): 2525.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_644_20231202T190922Z_gen_g4_detsim_hitreco__20240508T063244Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_644_20231202T190922Z_gen_g4_detsim_hitreco__20240508T063244Z_reco2_reco_2024-11-23T_072156Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_644_20231202T190922Z_gen_g4_detsim_hitreco__20240508T063244Z_reco2_reco_data_2024-11-23T_072156Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work