Jobsub ID 108744.166@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.166@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 05:49:38
From worker node	Hostname	heplnc160.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:40:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4133s)
	CPU time	1h (5334s = 129%)
Outputting started	2024-11-23 05:49:09
Output files
Finished	2024-11-23 05:49:38
List job events Wrapper job log
Jobscript log (last 10,000 characters)

T:12 P:2 W:600 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:48:49 GMT run: 50515154 subRun: 1 event: 26700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:48:49 GMT run: 50515154 subRun: 1 event: 26700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:48:49 GMT run: 50515154 subRun: 1 event: 26700
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:48:58 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:48:58 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:48:58 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:48:58 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 239, 254, 128
minwire 0: 1643
minwire 1: 1624
minwire 2: 1366
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0227036, 0.976552, 0.00074478, 
Output 1: 0.828445, 0.163169, 0.00781772, 0.000568328, 
Output 2: 0.947701, 0.0522989, 

634
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2354!
23-Nov-2024 05:49:06 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2_reco_data_2024-11-23T_044021Z.root"
23-Nov-2024 05:49:06 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/12/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.467418       39.4906       951.486       10.8862       112.329        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00502572     0.0460453     0.413732      0.0182373     0.0671038       100    
reco:linecluster:LineCluster                           0.024623      0.563833       19.6766      0.0832919      2.19055        100    
reco:trajcluster:TrajCluster                           0.0120813      1.40715       64.1081      0.0261733      7.05884        100    
reco:pandora:StandardPandora                           0.0253118      2.84904       66.9523       1.45653       7.4033         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000182391   0.00716096      0.13713     0.00161455     0.0174421       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000249878    0.0149551     0.297647     0.00340798     0.0385109       100    
reco:pandoracalo:Calorimetry                          0.000126506    0.0271322     0.940298     0.00117785     0.109853        100    
reco:pandorapid:Chi2ParticleID                        5.2027e-05    0.000231683   0.00259215    0.000169883    0.0002696       100    
reco:trkshowersplit:TrackShowerHits                   8.6122e-05     0.0331448     0.824236     0.00073732     0.127149        100    
reco:pmtrack:PMAlgTrackMaker                          0.00157054      3.25344       87.1789      0.129432       11.0828        100    
reco:pmtrackcalo:Calorimetry                          8.5169e-05     0.0175653     0.831897     0.00110393     0.0857203       100    
reco:pmtrackpid:Chi2ParticleID                        3.5216e-05    0.000398548   0.00604315    0.000204066   0.000724635      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000302396     4.34701       202.473      0.328478       21.2188        100    
reco:pmtrajfitcalo:Calorimetry                        9.5268e-05     0.0325144      1.5611      0.00180422     0.162888        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.5325e-05    0.000264068   0.00159737    0.000218333   0.000204737      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00145133      4.10195       128.271      0.190077       15.0392        100    
reco:pmtracktccalo:Calorimetry                        9.9466e-05     0.0127376      0.54934     0.00108578     0.0577151       100    
reco:pmtracktcpid:Chi2ParticleID                      4.2559e-05    0.000382178   0.00541635    0.000218809   0.000673911      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000239217     2.14858       52.9346      0.277731       6.55341        100    
reco:pmtrajfittccalo:Calorimetry                      9.7834e-05     0.020993      0.802536     0.00142241     0.0885083       100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.3662e-05    0.00049088    0.00665482    0.000223363   0.000902028      100    
reco:blurredcluster:BlurredClustering                 0.000247092     1.12034       14.1364      0.528385       1.87758        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000170599     15.3512       311.662       3.13776       41.5403        100    
reco:emshower:EMShower                                0.000253495    0.0327909     0.513991     0.00423445     0.0770345       100    
reco:cvnmap:CVNMapper                                 2.8784e-05     0.0131926     0.0965031    0.00891096     0.013688        100    
reco:cvneva:CVNEvaluator                              1.8805e-05      1.02646       2.78915       1.28597      0.644723        100    
reco:energyrecnumu:EnergyReco                         0.000499715    0.0580427      1.78863      0.0040477     0.217882        100    
reco:energyrecnue:EnergyReco                          0.000143579   0.00275232     0.0763868    0.000557297   0.00854081       100    
reco:energyrecnc:EnergyReco                           0.00181225    0.00535456     0.0730131     0.0035813    0.00792946       100    
reco:energyrecnumurange:EnergyReco                    0.000177702    0.0547252      1.76768     0.00227697     0.214943        100    
reco:energyrecnumumcs:EnergyReco                      0.00125305     0.0539848      1.66667      0.0037743     0.204558        100    
reco:opdec:Deconvolution                               0.0209906     0.286484       1.32648      0.232235      0.210015        100    
reco:ophitspe:OpHitFinderDeco                          0.0963326      2.29195       4.68581       2.35275       1.07034        100    
reco:opflash:OpFlashFinder                            0.000146254   0.00184444    0.00664669     0.0017103    0.00105517       100    
reco:rns:RandomNumberSaver                            2.4435e-05    5.81573e-05   0.000466392   5.06045e-05   4.64994e-05      100    
reco:anglereconue:NuAngularReco                       0.000236052   0.00205681     0.0586847    0.000510886   0.00642469       100    
reco:anglereconumu:NuAngularReco                      8.3726e-05    0.00175758     0.0554875    0.000268999   0.00621045       100    
reco:anglereconuepfps:NuAngularReco                   0.000170368    0.0018856     0.0544894    0.000475404   0.00599541       100    
reco:anglereconumupfps:NuAngularReco                  0.000135414    0.0017297     0.0531756    0.00035291    0.00584093       100    
[art]:TriggerResults:TriggerResultInserter             1.06e-05     1.84863e-05   6.9519e-05    1.7152e-05    8.34829e-06      100    
end_path:out1:RootOutput                               3.907e-06    8.72435e-06   3.7511e-05     8.361e-06    5.38364e-06      100    
end_path:out1:RootOutput(write)                        0.0208038     0.296058       1.72944      0.299569      0.230677        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39588.7 MB
  Peak resident set size usage (VmHWM): 2506.97 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2_reco_2024-11-23T_044021Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2_reco_data_2024-11-23T_044021Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work