Jobsub ID 301687.55@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301687.55@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:55:32
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:10:38
From worker node	Hostname	n257.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:56:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_565_20231208T032032Z_gen_g4_detsim_hitreco__20240510T064425Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4422s)
	CPU time	1h (3955s = 89%)
Outputting started	2024-11-23 05:10:20
Output files
Finished	2024-11-23 05:10:38
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Shower:emshower@BeginModule  23-Nov-2024 05:10:11 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:10:11 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:10:11 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:10:12 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:10:12 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:10:12 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:10:12 GMT run: 50642276 subRun: 1 event: 56600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 3442, 4445, 2293
minwire 0: 1476
minwire 1: 829
minwire 2: 1434
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0345286, 0.331317, 0.634154, 
Output 1: 0.184545, 0.474896, 0.226245, 0.114314, 
Output 2: 0.298153, 0.701847, 

835
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3285!
23-Nov-2024 05:10:19 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_565_20231208T032032Z_gen_g4_detsim_hitreco__20240510T064425Z_reco2_reco_data_2024-11-23T_035643Z.root"
23-Nov-2024 05:10:19 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/e9/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_565_20231208T032032Z_gen_g4_detsim_hitreco__20240510T064425Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.675575       39.515        431.026       10.3268       75.8707        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179589     0.0392741     0.0674466     0.0377924     0.0126495       100    
reco:linecluster:LineCluster                           0.161529      0.459926       7.38001      0.189473      0.951152        100    
reco:trajcluster:TrajCluster                           0.0511378      1.37781       38.2043      0.0610902      4.84045        100    
reco:pandora:StandardPandora                           0.0651123      2.50011       25.0887        1.249        4.0911         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000146658   0.00905853     0.191174     0.00211659      0.02313        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000192065    0.016332      0.285052     0.00337155     0.0378158       100    
reco:pandoracalo:Calorimetry                          0.000109227    0.0296893     0.561765     0.00132873     0.0846456       100    
reco:pandorapid:Chi2ParticleID                        4.4023e-05    0.000199373   0.00136297    0.000144245   0.000194213      100    
reco:trkshowersplit:TrackShowerHits                   0.000120449    0.0426471     0.649051     0.000953848     0.11655        100    
reco:pmtrack:PMAlgTrackMaker                          0.00137093      4.04306       59.5514      0.239493       10.2989        100    
reco:pmtrackcalo:Calorimetry                          8.3047e-05     0.0193544     0.544379     0.00114605     0.066986        100    
reco:pmtrackpid:Chi2ParticleID                        3.2212e-05    0.000418046   0.00359898    0.000166185   0.000677263      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000228535     2.94773       35.7257      0.422822       6.51012        100    
reco:pmtrajfitcalo:Calorimetry                        6.9863e-05     0.0310123     0.598756     0.00215274     0.0880343       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.0508e-05    0.000231071   0.00133531    0.000177617   0.000209251      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0012138      5.37809       95.5688      0.228735       14.1299        100    
reco:pmtracktccalo:Calorimetry                        7.9661e-05     0.0120694     0.308295     0.00111485     0.0363076       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6339e-05    0.000377268   0.00298139    0.000180067   0.000527776      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000218976     2.63784       35.6831      0.315999       6.0577         100    
reco:pmtrajfittccalo:Calorimetry                      8.4611e-05     0.0255499     0.431219     0.00140495     0.0698774       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.1851e-05    0.000551543   0.00391073    0.000202424   0.000824828      100    
reco:blurredcluster:BlurredClustering                 8.6614e-05      1.10941       7.91891      0.504336       1.49965        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0309928      15.193        164.431       3.19287       27.8241        100    
reco:emshower:EMShower                                0.000302044    0.0450726     0.637711     0.00770708     0.0964182       100    
reco:cvnmap:CVNMapper                                 3.1049e-05    0.00972724     0.0394969    0.00717157    0.00986262       100    
reco:cvneva:CVNEvaluator                              1.9948e-05     0.594255       1.20577      0.847802      0.421102        100    
reco:energyrecnumu:EnergyReco                         0.000328574    0.0490636      1.12693     0.00515287     0.146642        100    
reco:energyrecnue:EnergyReco                          0.000159573   0.00311197     0.0650841    0.000758822   0.00765434       100    
reco:energyrecnc:EnergyReco                           0.00259303     0.0057346     0.0681629    0.00330269    0.00790224       100    
reco:energyrecnumurange:EnergyReco                    0.000163471    0.046822       1.11918     0.00300892     0.145714        100    
reco:energyrecnumumcs:EnergyReco                      0.00270154     0.0485753      1.12073     0.00450864     0.145891        100    
reco:opdec:Deconvolution                               0.0606366      0.29813      0.501249      0.297474      0.101334        100    
reco:ophitspe:OpHitFinderDeco                          0.237291       2.25683       3.89282       2.2406       0.942878        100    
reco:opflash:OpFlashFinder                            0.000260575   0.00155427    0.00457399     0.0014633    0.000736217      100    
reco:rns:RandomNumberSaver                            1.8355e-05    3.73411e-05   0.000302965   3.19655e-05   2.83011e-05      100    
reco:anglereconue:NuAngularReco                       0.000183629   0.00231818     0.0554879    0.00045848     0.0064631       100    
reco:anglereconumu:NuAngularReco                      8.2667e-05    0.00209623     0.0544002    0.000294134   0.00635262       100    
reco:anglereconuepfps:NuAngularReco                   0.000139405   0.00222271     0.0550667    0.000409798    0.0063932       100    
reco:anglereconumupfps:NuAngularReco                  0.000124266    0.0021584     0.0547727    0.000375328   0.00639072       100    
[art]:TriggerResults:TriggerResultInserter            1.0711e-05    1.4218e-05    5.0415e-05    1.29245e-05   4.6579e-06       100    
end_path:out1:RootOutput                               2.735e-06    5.17551e-06   1.7092e-05    4.2035e-06    2.2557e-06       100    
end_path:out1:RootOutput(write)                        0.0133987     0.271426       1.23689       0.25737      0.216287        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38481.8 MB
  Peak resident set size usage (VmHWM): 2410.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_565_20231208T032032Z_gen_g4_detsim_hitreco__20240510T064425Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_565_20231208T032032Z_gen_g4_detsim_hitreco__20240510T064425Z_reco2_reco_2024-11-23T_035643Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_565_20231208T032032Z_gen_g4_detsim_hitreco__20240510T064425Z_reco2_reco_data_2024-11-23T_035643Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work