Jobsub ID 108506.68@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108506.68@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 21:52:28
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce04
Last heartbeat	2024-11-22 23:29:33
From worker node	Hostname	dune001-2994635.0-lcg2652.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 22:08:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_784_20231208T094159Z_gen_g4_detsim_hitreco__20240510T064516Z_reco2.root
Outputting started	2024-11-22 23:27:45
Output files
Finished	2024-11-22 23:29:33
List job events Wrapper job log
Jobscript log (last 10,000 characters)

trajfittc@BeginModule  22-Nov-2024 23:26:36 UTC run: 6472782 subRun: 1 event: 78500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:36 UTC run: 6472782 subRun: 1 event: 78500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:41 UTC run: 6472782 subRun: 1 event: 78500
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:41 UTC run: 6472782 subRun: 1 event: 78500
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:41 UTC run: 6472782 subRun: 1 event: 78500
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:41 UTC run: 6472782 subRun: 1 event: 78500
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:41 UTC run: 6472782 subRun: 1 event: 78500
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:26:41 UTC run: 6472782 subRun: 1 event: 78500
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 652, 1221, 1101
minwire 0: 2206
minwire 1: 434
minwire 2: 1801
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00329077, 0.00142586, 0.995283, 
Output 1: 0.178413, 0.67965, 0.0972161, 0.0447206, 
Output 2: 0.823762, 0.176238, 

676
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2609!
22-Nov-2024 23:27:14 UTC  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_784_20231208T094159Z_gen_g4_detsim_hitreco__20240510T064516Z_reco2_reco_data_2024-11-22T_220844Z.root"
22-Nov-2024 23:27:14 UTC  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/f1/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_784_20231208T094159Z_gen_g4_detsim_hitreco__20240510T064516Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.3457        45.4773       666.528       11.1534       110.068        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00207901     0.0860173     0.329934      0.0648653     0.0605115       100    
reco:linecluster:LineCluster                           0.0540772     0.968087       30.0415      0.194638       4.25922        100    
reco:trajcluster:TrajCluster                          0.00662832      2.0181        51.6761      0.0893019      8.48179        100    
reco:pandora:StandardPandora                            1.08257       4.08817       115.531       1.47499       13.2218        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000174988   0.00879719     0.103091     0.00190068     0.0183292       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000230953    0.0287689     0.954465     0.00422531     0.102882        100    
reco:pandoracalo:Calorimetry                          0.000129884    0.038551      0.820843     0.00144763     0.133737        100    
reco:pandorapid:Chi2ParticleID                        5.0735e-05    0.000226566   0.00260901    0.00016276    0.000278695      100    
reco:trkshowersplit:TrackShowerHits                   0.000142127    0.0504092      1.59863     0.000813876    0.205022        100    
reco:pmtrack:PMAlgTrackMaker                           0.0015833      5.02008       111.054       0.27262       17.1689        100    
reco:pmtrackcalo:Calorimetry                          0.000100208    0.0224339     0.593284     0.00122694     0.0802477       100    
reco:pmtrackpid:Chi2ParticleID                        4.0997e-05    0.000425661   0.00525551    0.000168576   0.000780494      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000247474     4.06195       66.5307      0.366783       12.2584        100    
reco:pmtrajfitcalo:Calorimetry                        8.6061e-05     0.0411266      0.90002     0.00181912     0.150702        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.4425e-05    0.000260429   0.00174345    0.000188728   0.000252077      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00132884      5.87687       118.219      0.310669       19.5359        100    
reco:pmtracktccalo:Calorimetry                        9.5549e-05     0.0147216     0.320347     0.00114393     0.0415239       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6839e-05    0.000395453   0.00439835    0.000175584   0.000648055      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000226184     2.85704       59.3577      0.362492       7.96625        100    
reco:pmtrajfittccalo:Calorimetry                      9.7773e-05     0.0287128     0.788576      0.0012714     0.103516        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.7379e-05    0.000556915    0.0108696    0.000194014   0.00125327       100    
reco:blurredcluster:BlurredClustering                  0.0134078      1.33733       9.54844      0.662139       1.6971         100    
reco:emtrkmichelid:EmTrackMichelId                     0.380656       14.5932       185.801       3.1448        29.7197        100    
reco:emshower:EMShower                                0.000351449    0.0282801     0.227879     0.00666017     0.0481946       100    
reco:cvnmap:CVNMapper                                 2.5899e-05     0.0117372     0.050931     0.00687952     0.0121253       100    
reco:cvneva:CVNEvaluator                              1.7312e-05     0.989703       2.18042       1.24862      0.590834        100    
reco:energyrecnumu:EnergyReco                         0.000426951    0.0548965      1.12168     0.00604388     0.162253        100    
reco:energyrecnue:EnergyReco                          0.000178665   0.00319815     0.0518075    0.000645726   0.00726157       100    
reco:energyrecnc:EnergyReco                           0.00200361     0.0061264     0.053855     0.00366889    0.00748329       100    
reco:energyrecnumurange:EnergyReco                    0.000184596    0.0511093     0.999609      0.0035473     0.155651        100    
reco:energyrecnumumcs:EnergyReco                      0.00209099     0.0529477     0.972743     0.00475447     0.154446        100    
reco:opdec:Deconvolution                               0.0386528      0.35123      0.971467      0.327986      0.171655        100    
reco:ophitspe:OpHitFinderDeco                          0.131261       2.4598        4.51265       2.46199      0.959042        100    
reco:opflash:OpFlashFinder                            0.000353523   0.00182819    0.00567792    0.00172239    0.000831745      100    
reco:rns:RandomNumberSaver                            2.3885e-05    4.01234e-05   0.00041049     3.454e-05    3.82267e-05      100    
reco:anglereconue:NuAngularReco                       0.000182843   0.00266226     0.0709235    0.000486833   0.00805829       100    
reco:anglereconumu:NuAngularReco                      8.7294e-05    0.00237866     0.0706502    0.000302302   0.00792186       100    
reco:anglereconuepfps:NuAngularReco                   0.000124283   0.00256155     0.0724161    0.000446206   0.00807772       100    
reco:anglereconumupfps:NuAngularReco                  0.000107652   0.00246529     0.0707334    0.000385463   0.00793066       100    
[art]:TriggerResults:TriggerResultInserter            1.0369e-05    1.68275e-05   5.5665e-05    1.52935e-05   6.37554e-06      100    
end_path:out1:RootOutput                               3.296e-06    6.48821e-06   2.3153e-05     5.285e-06    2.96163e-06      100    
end_path:out1:RootOutput(write)                        0.0201899     0.311543       1.35231      0.296077      0.236873        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39580.3 MB
  Peak resident set size usage (VmHWM): 2358 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_784_20231208T094159Z_gen_g4_detsim_hitreco__20240510T064516Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_784_20231208T094159Z_gen_g4_detsim_hitreco__20240510T064516Z_reco2_reco_2024-11-22T_220844Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_784_20231208T094159Z_gen_g4_detsim_hitreco__20240510T064516Z_reco2_reco_data_2024-11-22T_220844Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work