Jobsub ID 301674.48@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301674.48@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:25:05
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 04:25:45
From worker node	Hostname	heplnc147.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:28:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_762_20231208T084459Z_gen_g4_detsim_hitreco__20240510T060539Z_reco2.root
Jobscript	Exit code	0
	Real time	57m (3422s)
	CPU time	54m (3258s = 95%)
Outputting started	2024-11-23 04:25:24
Output files
Finished	2024-11-23 04:25:45
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ot exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:09 GMT run: 6472706 subRun: 1 event: 76300
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 444, 215, 213
minwire 0: 1944
minwire 1: 836
minwire 2: 2039
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0233884, 0.00215469, 0.974457, 
Output 1: 0.352273, 0.52041, 0.103449, 0.0238677, 
Output 2: 0.528355, 0.471645, 

545
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1952!
23-Nov-2024 04:25:22 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_762_20231208T084459Z_gen_g4_detsim_hitreco__20240510T060539Z_reco2_reco_data_2024-11-23T_032827Z.root"
23-Nov-2024 04:25:22 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/be/0d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_762_20231208T084459Z_gen_g4_detsim_hitreco__20240510T060539Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.815312       31.9838       701.237       12.5479       75.2336        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00687347     0.127642      0.659019      0.0495603     0.152232        100    
reco:linecluster:LineCluster                           0.0430598     0.396617       14.1537      0.0828387      1.43849        100    
reco:trajcluster:TrajCluster                           0.012372      0.664054        48.25       0.0277076      4.80214        100    
reco:pandora:StandardPandora                           0.0546972      2.96215       42.4552       2.18227       4.50904        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000172575   0.00700106     0.111444     0.00222645     0.0138844       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000262355    0.0143813     0.182228     0.00357138     0.0271467       100    
reco:pandoracalo:Calorimetry                          0.000131779    0.0126934      0.45423     0.00134245     0.0477828       100    
reco:pandorapid:Chi2ParticleID                        4.5155e-05    0.000273468   0.00296415    0.000176353   0.000338646      100    
reco:trkshowersplit:TrackShowerHits                   0.00015381     0.012752      0.489388     0.000724562    0.0518579       100    
reco:pmtrack:PMAlgTrackMaker                          0.00169617      1.90948       76.7916      0.121834       7.8729         100    
reco:pmtrackcalo:Calorimetry                           8.967e-05    0.00872934     0.352241     0.00096473     0.0363493       100    
reco:pmtrackpid:Chi2ParticleID                        3.8101e-05    0.000362736   0.00445005     0.0001958    0.00054695       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000256735      2.318        94.6818       0.21717       9.78006        100    
reco:pmtrajfitcalo:Calorimetry                        8.2125e-05     0.0159936     0.757708     0.00130121     0.0763981       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.4966e-05    0.000275256   0.00132133    0.000203008   0.000232538      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00136379      2.52503       113.325      0.169076       11.5496        100    
reco:pmtracktccalo:Calorimetry                         7.934e-05     0.0064564     0.170784     0.00107406     0.0191728       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3232e-05    0.000333986    0.0028273    0.000205492   0.000415018      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000232599     1.44046       32.4081       0.22345       3.71529        100    
reco:pmtrajfittccalo:Calorimetry                      7.8238e-05     0.0135253     0.591934     0.00114675     0.0606807       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3072e-05    0.000466977   0.00539324    0.000227504   0.000730752      100    
reco:blurredcluster:BlurredClustering                 0.00021764      1.15246       8.89702      0.594329       1.54826        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000207552     14.7408       258.001       3.80822       31.3834        100    
reco:emshower:EMShower                                0.000321878    0.0249057      0.31942     0.00424875     0.0463893       100    
reco:cvnmap:CVNMapper                                  3.174e-05     0.0146923     0.0601122     0.0171211     0.012767        100    
reco:cvneva:CVNEvaluator                              2.2683e-05      1.05744       2.7686        1.43232      0.844547        100    
reco:energyrecnumu:EnergyReco                         0.00049834      0.03375      0.737676     0.00606683     0.110661        100    
reco:energyrecnue:EnergyReco                          0.000177104    0.0022997     0.0403054    0.00090883    0.00457263       100    
reco:energyrecnc:EnergyReco                           0.00209318    0.00509431     0.0428273    0.00375392    0.00478969       100    
reco:energyrecnumurange:EnergyReco                    0.000312851    0.0297502      0.71698     0.00344635     0.106474        100    
reco:energyrecnumumcs:EnergyReco                      0.00191572     0.0311037     0.692803     0.00402671     0.105694        100    
reco:opdec:Deconvolution                               0.0490005     0.368039       1.57387      0.241396      0.340865        100    
reco:ophitspe:OpHitFinderDeco                          0.292427       1.81361       4.44235       1.71817       0.84862        100    
reco:opflash:OpFlashFinder                            0.000505234   0.00207101    0.00568227    0.00188708    0.00105731       100    
reco:rns:RandomNumberSaver                            2.3935e-05    7.24494e-05   0.000658323   5.58255e-05   7.80023e-05      100    
reco:anglereconue:NuAngularReco                       0.000202311   0.00154908     0.0309041    0.000637794   0.00344209       100    
reco:anglereconumu:NuAngularReco                      8.6664e-05     0.0012676     0.0292905     0.0002459    0.00371897       100    
reco:anglereconuepfps:NuAngularReco                   0.000145375   0.00145577     0.0306599    0.000460579   0.00348453       100    
reco:anglereconumupfps:NuAngularReco                   0.0001219    0.00135911     0.0314306    0.000335087    0.0036235       100    
[art]:TriggerResults:TriggerResultInserter            1.1412e-05    2.16172e-05   0.000139413   1.7633e-05    1.49847e-05      100    
end_path:out1:RootOutput                               4.338e-06    1.13868e-05   4.4144e-05     9.813e-06    6.99308e-06      100    
end_path:out1:RootOutput(write)                        0.0255143      0.26179       1.22095      0.288089      0.190449        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21695.1 MB
  Peak resident set size usage (VmHWM): 2275.18 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_762_20231208T084459Z_gen_g4_detsim_hitreco__20240510T060539Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_762_20231208T084459Z_gen_g4_detsim_hitreco__20240510T060539Z_reco2_reco_2024-11-23T_032827Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_762_20231208T084459Z_gen_g4_detsim_hitreco__20240510T060539Z_reco2_reco_data_2024-11-23T_032827Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work