Jobsub ID 108742.29@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.29@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 05:35:37
From worker node	Hostname	td810.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:31:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3843s)
	CPU time	1h (3809s = 99%)
Outputting started	2024-11-23 05:35:17
Output files
Finished	2024-11-23 05:35:37
List job events Wrapper job log
Jobscript log (last 10,000 characters)

y.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:05 CET run: 6414162 subRun: 1 event: 96900
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:06 CET run: 6414162 subRun: 1 event: 96900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:06 CET run: 6414162 subRun: 1 event: 96900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:06 CET run: 6414162 subRun: 1 event: 96900
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:06 CET run: 6414162 subRun: 1 event: 96900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:06 CET run: 6414162 subRun: 1 event: 96900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:06 CET run: 6414162 subRun: 1 event: 96900
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:35:08 CET run: 6414162 subRun: 1 event: 96900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 3444, 3472, 1922
minwire 0: 1681
minwire 1: 435
minwire 2: 1906
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.129044, 0.176866, 0.69409, 
Output 1: 0.548745, 0.348461, 0.0814559, 0.021338, 
Output 2: 0.0942523, 0.905748, 

784
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2976!
23-Nov-2024 06:35:16 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2_reco_data_2024-11-23T_043117Z.root"
23-Nov-2024 06:35:16 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/95/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.01586       37.5317       332.908       14.6915       61.3804        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00116939    0.00295639    0.00758749    0.00259754    0.00113118       100    
reco:linecluster:LineCluster                           0.0115452     0.364359       11.6267      0.0254201      1.70671        100    
reco:trajcluster:TrajCluster                          0.00206078      1.18837       25.4331      0.0206468      4.21331        100    
reco:pandora:StandardPandora                            1.10156       2.70373       33.8377       1.72847       3.8524         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136111   0.00884603     0.103061     0.00249827     0.0162796       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000185792    0.015149      0.113381     0.00459455     0.0230649       100    
reco:pandoracalo:Calorimetry                          0.000106161    0.0224115     0.386454     0.00162835     0.0588809       100    
reco:pandorapid:Chi2ParticleID                         4.148e-05    0.00023697    0.00403749    0.000170511   0.000404352      100    
reco:trkshowersplit:TrackShowerHits                   0.000119301    0.0314734     0.688956     0.00145907     0.0963209       100    
reco:pmtrack:PMAlgTrackMaker                          0.00132763      3.68026       41.9933      0.356557       8.40787        100    
reco:pmtrackcalo:Calorimetry                          7.2121e-05     0.0108679      0.12139     0.00152363     0.0229086       100    
reco:pmtrackpid:Chi2ParticleID                         2.961e-05    0.00040738    0.00229732    0.000205697   0.000491912      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000208932     3.16309       35.2802      0.421552       7.11582        100    
reco:pmtrajfitcalo:Calorimetry                         6.389e-05     0.0238237     0.445146     0.00239222     0.0635677       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8791e-05    0.00027591    0.00149535    0.000194597   0.000251582      100    
reco:pmtracktc:PMAlgTrackMaker                          0.00117       4.14973       54.0692      0.467781       9.67414        100    
reco:pmtracktccalo:Calorimetry                        7.2311e-05    0.00690812     0.0789796     0.0015743     0.0135848       100    
reco:pmtracktcpid:Chi2ParticleID                       2.94e-05     0.000354343   0.00195946    0.000205942   0.000370468      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000184992     2.12927       19.9188      0.447959       3.95304        100    
reco:pmtrajfittccalo:Calorimetry                      6.4601e-05     0.0184306     0.330239     0.00194852     0.0461661       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.709e-05    0.000525839   0.00419019    0.000231277   0.000712193      100    
reco:blurredcluster:BlurredClustering                 0.00638174      1.30614       6.87277      0.691951       1.48852        100    
reco:emtrkmichelid:EmTrackMichelId                     0.231628       15.0085       115.859       5.2404        22.8515        100    
reco:emshower:EMShower                                0.000281912    0.0375694      0.29128     0.00894773     0.0576369       100    
reco:cvnmap:CVNMapper                                 2.6141e-05     0.0119268     0.0492796    0.00970095     0.0106853       100    
reco:cvneva:CVNEvaluator                               1.931e-05     0.751976       1.43652      0.980464      0.481321        100    
reco:energyrecnumu:EnergyReco                         0.000505505    0.0646665      1.05246     0.00632912      0.18401        100    
reco:energyrecnue:EnergyReco                          0.000164162   0.00297568      0.02784     0.000888843   0.00507747       100    
reco:energyrecnc:EnergyReco                           0.00234974    0.00614322     0.032093     0.00400896     0.0055166       100    
reco:energyrecnumurange:EnergyReco                    0.000267053    0.0622685     0.997118     0.00386075     0.181547        100    
reco:energyrecnumumcs:EnergyReco                      0.00172706     0.0616028     0.963698     0.00471047     0.170125        100    
reco:opdec:Deconvolution                               0.0267161     0.229741      0.460712      0.233189      0.0991418       100    
reco:ophitspe:OpHitFinderDeco                          0.182255       2.11968       4.24601       2.13497       1.03481        100    
reco:opflash:OpFlashFinder                            0.000406144   0.00173087    0.00661571    0.00163785    0.000915157      100    
reco:rns:RandomNumberSaver                             1.793e-05    4.25056e-05   0.000459254   3.31805e-05   4.78969e-05      100    
reco:anglereconue:NuAngularReco                       0.000182232   0.00211941     0.0184966    0.000484955   0.00395891       100    
reco:anglereconumu:NuAngularReco                       7.125e-05    0.00190299     0.0182652    0.000318968   0.00389373       100    
reco:anglereconuepfps:NuAngularReco                   0.000127431   0.00207881     0.0181675    0.000439164   0.00402894       100    
reco:anglereconumupfps:NuAngularReco                  0.000103411   0.00204882      0.02177     0.000434429   0.00412751       100    
[art]:TriggerResults:TriggerResultInserter             1.05e-05     1.64753e-05   8.8911e-05     1.488e-05    8.73993e-06      100    
end_path:out1:RootOutput                               3.22e-06     6.39175e-06    3.867e-05     4.805e-06    4.49006e-06      100    
end_path:out1:RootOutput(write)                        0.0248817     0.334427       1.06751      0.331813       0.23179        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21209 MB
  Peak resident set size usage (VmHWM): 2072.26 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2_reco_2024-11-23T_043117Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2_reco_data_2024-11-23T_043117Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work