Jobsub ID 301447.75@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301447.75@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 22:25:06
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 00:27:34
From worker node	Hostname	tdm002.pic.es
	cpuinfo	Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 22:50:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_838_20231204T040236Z_gen_g4_detsim_hitreco__20240509T211231Z_reco2.root
Outputting started	2024-11-23 00:27:34
Output files
Finished	2024-11-23 00:27:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:27:21 CET run: 6420853 subRun: 1 event: 83900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:27:21 CET run: 6420853 subRun: 1 event: 83900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:27:21 CET run: 6420853 subRun: 1 event: 83900
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:27:21 CET run: 6420853 subRun: 1 event: 83900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:27:21 CET run: 6420853 subRun: 1 event: 83900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 01:27:27 CET run: 6420853 subRun: 1 event: 83900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 01:27:27 CET run: 6420853 subRun: 1 event: 83900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 146, 150, 149
minwire 0: 897
minwire 1: 1277
minwire 2: 988
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00419189, 0.990945, 0.00486307, 
Output 1: 0.0505505, 0.523886, 0.408698, 0.0168658, 
Output 2: 0.515399, 0.484601, 

422
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1258!
23-Nov-2024 01:27:32 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_838_20231204T040236Z_gen_g4_detsim_hitreco__20240509T211231Z_reco2_reco_data_2024-11-22T_225056Z.root"
23-Nov-2024 01:27:32 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/01/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_838_20231204T040236Z_gen_g4_detsim_hitreco__20240509T211231Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              4.03092       56.762        1013.63       20.313        129.786        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00161102     0.0200435     0.278523      0.0192042     0.0273468       100    
reco:linecluster:LineCluster                           0.0118519      1.24669        95.58       0.0354037      9.65552        100    
reco:trajcluster:TrajCluster                           0.0022422      2.98354       146.267      0.0301196      16.4259        100    
reco:pandora:StandardPandora                            1.6317        3.76817       38.6723       2.53049       5.15723        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000213894    0.0119051     0.142909     0.00360518     0.0246025       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000322844    0.0284029     0.302709     0.00645872     0.0599003       100    
reco:pandoracalo:Calorimetry                          0.000119079    0.0440674      1.5624      0.00285307      0.18121        100    
reco:pandorapid:Chi2ParticleID                        4.5921e-05    0.000285895    0.004029     0.000221415   0.000412885      100    
reco:trkshowersplit:TrackShowerHits                   0.000143245    0.0511688      1.7288      0.00175867     0.217574        100    
reco:pmtrack:PMAlgTrackMaker                           0.0019984      5.4356        141.321      0.629241       17.7311        100    
reco:pmtrackcalo:Calorimetry                          0.000106398    0.0195018     0.445118     0.00200003     0.0577836       100    
reco:pmtrackpid:Chi2ParticleID                        4.1492e-05    0.000491622   0.00418096    0.000253935   0.000717489      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000392707     3.94651       77.3274      0.696206       10.3605        100    
reco:pmtrajfitcalo:Calorimetry                        0.000115229    0.0403475      1.12977     0.00336364      0.1512         100    
reco:pmtrajfitpid:Chi2ParticleID                      4.9777e-05    0.000305366   0.00154722    0.000240076   0.000252073      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00230873      7.29177       227.461      0.528311       28.2213        100    
reco:pmtracktccalo:Calorimetry                        0.00014854     0.0123048     0.178349     0.00221945     0.0316049       100    
reco:pmtracktcpid:Chi2ParticleID                      5.5352e-05    0.000439333   0.00363529    0.000271077   0.000562453      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000457938     2.87638       45.1681      0.862689       6.09362        100    
reco:pmtrajfittccalo:Calorimetry                      0.000149724    0.0261756     0.649708     0.00311834     0.0816736       100    
reco:pmtrajfittcpid:Chi2ParticleID                    5.7363e-05    0.00057294    0.00500162    0.000295613   0.000781135      100    
reco:blurredcluster:BlurredClustering                  0.0104229      1.63466       9.12641      0.993326       1.94227        100    
reco:emtrkmichelid:EmTrackMichelId                     0.439122       22.6155       240.886       7.73787        39.65         100    
reco:emshower:EMShower                                0.000432993    0.0511705     0.356964      0.0136986     0.0787194       100    
reco:cvnmap:CVNMapper                                 3.2895e-05     0.023546      0.0676119     0.0256813     0.017383        100    
reco:cvneva:CVNEvaluator                              2.3192e-05     0.998687       1.69425       1.32467      0.626695        100    
reco:energyrecnumu:EnergyReco                         0.000618886    0.0952559      2.61361     0.00867083     0.359266        100    
reco:energyrecnue:EnergyReco                          0.00027977    0.00355596     0.0388116    0.00123075    0.00671093       100    
reco:energyrecnc:EnergyReco                           0.00216763    0.00665082     0.041526     0.00419009    0.00659862       100    
reco:energyrecnumurange:EnergyReco                    0.000410677    0.0928304      2.81853     0.00468289     0.367516        100    
reco:energyrecnumumcs:EnergyReco                       0.001267      0.0940437      3.05205     0.00679478     0.375575        100    
reco:opdec:Deconvolution                               0.052178      0.316213      0.798325      0.311652      0.125189        100    
reco:ophitspe:OpHitFinderDeco                          0.268898       2.60323       6.05197       2.5255        1.15767        100    
reco:opflash:OpFlashFinder                            0.000468091   0.00232281    0.00867489    0.00210123    0.00115463       100    
reco:rns:RandomNumberSaver                            2.3116e-05    5.48572e-05   0.000554125   5.24935e-05   5.25152e-05      100    
reco:anglereconue:NuAngularReco                       0.000201043   0.00242758     0.0348317    0.000740993    0.0051671       100    
reco:anglereconumu:NuAngularReco                      9.7762e-05    0.00214569     0.0347874    0.000485152   0.00510419       100    
reco:anglereconuepfps:NuAngularReco                   0.000151025   0.00234505     0.0343398    0.000668939   0.00515122       100    
reco:anglereconumupfps:NuAngularReco                  0.000124276   0.00226584     0.0343034    0.000608093   0.00509929       100    
[art]:TriggerResults:TriggerResultInserter            1.1542e-05    1.95016e-05   0.000105661   1.78755e-05   9.78885e-06      100    
end_path:out1:RootOutput                               2.85e-06     8.10993e-06   3.5169e-05    8.3805e-06    4.32142e-06      100    
end_path:out1:RootOutput(write)                        0.038805      0.407498       1.74126      0.409245       0.28315        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10617.4 MB
  Peak resident set size usage (VmHWM): 2305.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_838_20231204T040236Z_gen_g4_detsim_hitreco__20240509T211231Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_838_20231204T040236Z_gen_g4_detsim_hitreco__20240509T211231Z_reco2_reco_2024-11-22T_225056Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_838_20231204T040236Z_gen_g4_detsim_hitreco__20240509T211231Z_reco2_reco_data_2024-11-22T_225056Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work