Jobsub ID 301622.13@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301622.13@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:30:34
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 02:42:40
From worker node	Hostname	b9p05p5799.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:32:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_291_20231201T212729Z_gen_g4_detsim_hitreco__20240508T044525Z_reco2.root
Outputting started	2024-11-23 02:31:45
Output files
Finished	2024-11-23 03:36:44
List job events Wrapper job log
Jobscript log (last 10,000 characters)

culation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:31:29 CET run: 74496296 subRun: 1 event: 29200
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 765, 403, 545
minwire 0: 2229
minwire 1: 1
minwire 2: 2580
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.354494, 0.360277, 0.285229, 
Output 1: 0.355965, 0.626489, 0.0166175, 0.000928797, 
Output 2: 0.230685, 0.769315, 

441
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1275!
23-Nov-2024 03:31:44 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_291_20231201T212729Z_gen_g4_detsim_hitreco__20240508T044525Z_reco2_reco_data_2024-11-23T_013243Z.root"
23-Nov-2024 03:31:44 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cb/5e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_291_20231201T212729Z_gen_g4_detsim_hitreco__20240508T044525Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.867896       33.2246       1083.85       9.56575       111.772        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0167822     0.0525831     0.506184      0.0437975     0.0525754       100    
reco:linecluster:LineCluster                           0.149319      0.457701       13.6205      0.191907       1.43558        100    
reco:trajcluster:TrajCluster                           0.0483859     0.952581       43.4341      0.0715992      4.53258        100    
reco:pandora:StandardPandora                           0.0839999      6.62133       503.671       1.17136       49.9837        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000122541   0.00630185     0.147503     0.00148125     0.0171212       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000179552    0.0170757     0.634173     0.00279386     0.0651871       100    
reco:pandoracalo:Calorimetry                           9.712e-05     0.0270159      1.47373     0.00132797     0.149065        100    
reco:pandorapid:Chi2ParticleID                        3.1971e-05    0.000160231   0.00129408    0.000119276   0.000157288      100    
reco:trkshowersplit:TrackShowerHits                    9.153e-05     0.0262143     0.988942     0.000793411    0.108362        100    
reco:pmtrack:PMAlgTrackMaker                          0.00100149      2.72339       86.7562      0.298234       9.71587        100    
reco:pmtrackcalo:Calorimetry                          6.0241e-05     0.015371       0.69465     0.000968407    0.0756403       100    
reco:pmtrackpid:Chi2ParticleID                         2.517e-05    0.000283637   0.00371663    0.000149872   0.000485398      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000169562     2.39386       63.6398      0.357907       7.21051        100    
reco:pmtrajfitcalo:Calorimetry                         6.852e-05     0.0291589      1.55043     0.00171697     0.157506        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6441e-05    0.000181464   0.000806027   0.000144321   0.000139254      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000891327     3.30207       108.841      0.247098       11.9773        100    
reco:pmtracktccalo:Calorimetry                         6.266e-05     0.0093873     0.474657     0.00113094     0.0485815       100    
reco:pmtracktcpid:Chi2ParticleID                       2.473e-05    0.00025926    0.00454743    0.000150111   0.000475413      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000150641     1.76156       49.3428      0.310943       5.40602        100    
reco:pmtrajfittccalo:Calorimetry                      5.5351e-05     0.0210046      1.12902     0.00131424     0.114168        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.373e-05    0.000345057   0.00431142    0.000162021   0.000571194      100    
reco:blurredcluster:BlurredClustering                 0.000116481     1.01553       7.92076      0.421825        1.484         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000144691     10.6451       193.443       2.88953       23.7327        100    
reco:emshower:EMShower                                0.000193332    0.030996      0.330984     0.00334123     0.0619122       100    
reco:cvnmap:CVNMapper                                  2.339e-05     0.0112826     0.0322736     0.0120821    0.00813159       100    
reco:cvneva:CVNEvaluator                               1.834e-05     0.640908       1.18817      0.776714      0.373762        100    
reco:energyrecnumu:EnergyReco                         0.000296332    0.0286741     0.387423     0.00432495     0.0614639       100    
reco:energyrecnue:EnergyReco                          0.000125351   0.00230597     0.0451695    0.00050796    0.00579462       100    
reco:energyrecnc:EnergyReco                           0.00156688    0.00444676     0.0490551    0.00294667    0.00613418       100    
reco:energyrecnumurange:EnergyReco                    0.000127401    0.0264233     0.382475     0.00207349     0.0596861       100    
reco:energyrecnumumcs:EnergyReco                      0.00148553     0.0273115     0.379314     0.00344635     0.0581656       100    
reco:opdec:Deconvolution                               0.0647054     0.339018       3.83386      0.281844      0.378463        100    
reco:ophitspe:OpHitFinderDeco                          0.235764       1.80474       5.67689       1.75657      0.774309        100    
reco:opflash:OpFlashFinder                            0.000152451   0.00136317    0.00319476    0.00131266    0.000520828      100    
reco:rns:RandomNumberSaver                            1.6421e-05    3.4233e-05    0.000248202    3.15e-05     2.27856e-05      100    
reco:anglereconue:NuAngularReco                       0.000165461   0.00161875     0.0430444    0.000371788   0.00482534       100    
reco:anglereconumu:NuAngularReco                      8.6391e-05    0.00136411     0.0373181    0.000217332   0.00431043       100    
reco:anglereconuepfps:NuAngularReco                   0.000137211   0.00150374     0.0380402    0.000368478   0.00435774       100    
reco:anglereconumupfps:NuAngularReco                  0.000112521   0.00143634     0.0381732    0.000257447    0.0043631       100    
[art]:TriggerResults:TriggerResultInserter             8.47e-06     1.20633e-05    3.476e-05    1.1295e-05    3.84653e-06      100    
end_path:out1:RootOutput                               2.69e-06     5.32406e-06   1.5451e-05     5.405e-06    1.5789e-06       100    
end_path:out1:RootOutput(write)                        0.0125402     0.220889       1.3021       0.207645      0.161805        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21914.3 MB
  Peak resident set size usage (VmHWM): 2484.74 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_291_20231201T212729Z_gen_g4_detsim_hitreco__20240508T044525Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_291_20231201T212729Z_gen_g4_detsim_hitreco__20240508T044525Z_reco2_reco_2024-11-23T_013243Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_291_20231201T212729Z_gen_g4_detsim_hitreco__20240508T044525Z_reco2_reco_data_2024-11-23T_013243Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work