Jobsub ID 108742.53@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.53@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 06:17:51
From worker node	Hostname	b9p12p9920.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:33:03
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_106_20231121T152847Z_gen_g4_detsim_hitreco__20240503T054044Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6268s)
	CPU time	1h (6308s = 100%)
Outputting started	2024-11-23 06:17:32
Output files
Finished	2024-11-23 06:17:51
List job events Wrapper job log
Jobscript log (last 10,000 characters)

2024 07:17:18 CET run: 50258005 subRun: 1 event: 21400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:17:18 CET run: 50258005 subRun: 1 event: 21400
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:17:18 CET run: 50258005 subRun: 1 event: 21400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:17:18 CET run: 50258005 subRun: 1 event: 21400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:17:18 CET run: 50258005 subRun: 1 event: 21400
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:26 CET run: 50258005 subRun: 1 event: 21400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:26 CET run: 50258005 subRun: 1 event: 21400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:26 CET run: 50258005 subRun: 1 event: 21400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 318, 319, 221
minwire 0: 213
minwire 1: 1235
minwire 2: 92
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.549794, 0.419989, 0.0302178, 
Output 1: 0.224227, 0.594791, 0.17451, 0.0064722, 
Output 2: 0.940521, 0.0594793, 

580
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1860!
23-Nov-2024 07:17:30 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_106_20231121T152847Z_gen_g4_detsim_hitreco__20240503T054044Z_reco2_reco_data_2024-11-23T_043309Z.root"
23-Nov-2024 07:17:30 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/52/af/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_106_20231121T152847Z_gen_g4_detsim_hitreco__20240503T054044Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.603777       29.6243       787.887       9.58132        80.12         200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0166571     0.0409832     0.100441      0.0424002     0.0127316       200    
reco:linecluster:LineCluster                           0.0952754     0.667884       63.8297      0.179327       4.55214        200    
reco:trajcluster:TrajCluster                           0.0475431     0.922697       52.4195      0.0585565      4.64525        200    
reco:pandora:StandardPandora                           0.0584738      2.45647       74.7589       1.24373       7.34457        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000115771   0.00625408     0.157046     0.00140683     0.0168545       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000161901     0.01221      0.144703     0.00316666     0.0246079       200    
reco:pandoracalo:Calorimetry                          9.1301e-05     0.0325689      1.9994      0.00112915     0.180947        200    
reco:pandorapid:Chi2ParticleID                         3.332e-05    0.000158723    0.0010795    0.000128701   0.000133022      200    
reco:trkshowersplit:TrackShowerHits                   8.4371e-05     0.0314046      2.01189     0.00071812     0.162552        200    
reco:pmtrack:PMAlgTrackMaker                          0.000970937     3.14021       103.865      0.222608       11.8135        200    
reco:pmtrackcalo:Calorimetry                          5.9471e-05     0.013265      0.636549     0.000859355    0.0618466       200    
reco:pmtrackpid:Chi2ParticleID                         2.408e-05    0.000287932   0.00405152    0.000142766   0.000535742      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000156781     2.93711       167.159      0.355526       13.4142        200    
reco:pmtrajfitcalo:Calorimetry                        5.5521e-05     0.0316956      2.07159     0.00145061     0.177421        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.259e-05    0.000200819   0.00216488    0.000150201   0.000209375      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000784466     3.3682        101.771       0.24053       12.1466        200    
reco:pmtracktccalo:Calorimetry                         5.743e-05     0.007106      0.204747     0.000902531    0.0245123       200    
reco:pmtracktcpid:Chi2ParticleID                       2.444e-05    0.000259222   0.00301096    0.000135536   0.000379594      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000147651     1.70459       36.7983      0.302173       4.98711        200    
reco:pmtrajfittccalo:Calorimetry                       5.617e-05     0.0163987     0.648742     0.00111128     0.0698601       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3061e-05    0.000335785   0.00492572    0.000157046   0.000565787      200    
reco:blurredcluster:BlurredClustering                 0.000103471     1.02113       8.72243       0.49023       1.36017        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000178551     9.68796       160.96        2.36057       21.9971        200    
reco:emshower:EMShower                                0.000220621    0.030809      0.446866     0.00635062     0.0567355       200    
reco:cvnmap:CVNMapper                                  2.438e-05     0.0105652     0.0475955     0.0113435    0.00818403       200    
reco:cvneva:CVNEvaluator                               1.539e-05     0.765895       4.62512      0.896991      0.538442        200    
reco:energyrecnumu:EnergyReco                         0.000348202    0.044348       1.18631     0.00451523     0.145807        200    
reco:energyrecnue:EnergyReco                          0.000124951   0.00247606     0.0760132    0.000575479    0.0071393       200    
reco:energyrecnc:EnergyReco                           0.00163542    0.00484533     0.0813566    0.00324165    0.00750192       200    
reco:energyrecnumurange:EnergyReco                    0.000173341    0.0408316      1.13185     0.00297946     0.139742        200    
reco:energyrecnumumcs:EnergyReco                      0.000521903    0.0426722      1.28496     0.00373627      0.14362        200    
reco:opdec:Deconvolution                               0.0501747     0.280934       0.50736      0.284427      0.0876674       200    
reco:ophitspe:OpHitFinderDeco                           0.16505       2.04441       4.76818       1.92772      0.893574        200    
reco:opflash:OpFlashFinder                            0.000287272   0.00145886    0.00620477    0.00134889    0.000671642      200    
reco:rns:RandomNumberSaver                             1.816e-05    4.03642e-05   0.000334212   3.8905e-05    2.3196e-05       200    
reco:anglereconue:NuAngularReco                       0.000136251   0.00166756     0.0706404    0.000428448   0.00566904       200    
reco:anglereconumu:NuAngularReco                       5.876e-05     0.0014474     0.0691468    0.000235642   0.00555083       200    
reco:anglereconuepfps:NuAngularReco                   0.000102301   0.00159172     0.0690677    0.000390083   0.00555627       200    
reco:anglereconumupfps:NuAngularReco                   8.48e-05     0.00149796     0.0684391    0.000313887   0.00550102       200    
[art]:TriggerResults:TriggerResultInserter             8.66e-06     1.46525e-05    6.989e-05     1.384e-05    5.5489e-06       200    
end_path:out1:RootOutput                               2.84e-06     6.99329e-06   2.4821e-05     7.025e-06    2.36276e-06      200    
end_path:out1:RootOutput(write)                        0.0160649     0.247248       1.41748      0.249635      0.179809        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21975.6 MB
  Peak resident set size usage (VmHWM): 2548.05 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_106_20231121T152847Z_gen_g4_detsim_hitreco__20240503T054044Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_106_20231121T152847Z_gen_g4_detsim_hitreco__20240503T054044Z_reco2_reco_2024-11-23T_043309Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_106_20231121T152847Z_gen_g4_detsim_hitreco__20240503T054044Z_reco2_reco_data_2024-11-23T_043309Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work