Jobsub ID 108593.78@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108593.78@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:25:00
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 00:16:09
From worker node	Hostname	tdm004.pic.es
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:36:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_288_20231203T123133Z_gen_g4_detsim_hitreco__20240508T050351Z_reco2.root
Outputting started	2024-11-23 00:15:57
Output files
Finished	2024-11-23 00:34:32
List job events Wrapper job log
Jobscript log (last 10,000 characters)

pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:15:43 CET run: 50574832 subRun: 1 event: 28900
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 01:15:51 CET run: 50574832 subRun: 1 event: 28900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 156, 380, 261
minwire 0: 2485
minwire 1: 853
minwire 2: 2109
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.1268, 0.871455, 0.00174465, 
Output 1: 0.837093, 0.141667, 0.016775, 0.00446451, 
Output 2: 0.989515, 0.010485, 

554
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2063!
23-Nov-2024 01:15:55 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_288_20231203T123133Z_gen_g4_detsim_hitreco__20240508T050351Z_reco2_reco_data_2024-11-22T_233612Z.root"
23-Nov-2024 01:15:55 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/ff/df/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_288_20231203T123133Z_gen_g4_detsim_hitreco__20240508T050351Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.123857       22.9549       290.783       9.7292        40.5926        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00155051     0.015831      0.0324149     0.0190711    0.00883427       100    
reco:linecluster:LineCluster                          0.00220808     0.0918236      2.82822      0.0189848     0.332542        100    
reco:trajcluster:TrajCluster                          0.00118169     0.540847       19.7062      0.0107973      2.3945         100    
reco:pandora:StandardPandora                           0.0133107      1.8677        21.8038       1.26722       2.63925        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000150483   0.00519891     0.0658677    0.00137432     0.0104521       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000204034    0.0100345      0.13421     0.00282717     0.0191328       100    
reco:pandoracalo:Calorimetry                          0.000108842    0.0114671     0.304981     0.000903617    0.0373176       100    
reco:pandorapid:Chi2ParticleID                        4.3541e-05    0.000165389   0.00193734    0.000118503   0.000206546      100    
reco:trkshowersplit:TrackShowerHits                   0.000111352    0.0141037     0.354629     0.000710322    0.046325        100    
reco:pmtrack:PMAlgTrackMaker                           0.0014267      1.7014        31.9495      0.157937       4.53103        100    
reco:pmtrackcalo:Calorimetry                          7.2221e-05    0.00674796      0.16645     0.000847295    0.0204432       100    
reco:pmtrackpid:Chi2ParticleID                         2.97e-05     0.000264449    0.0021512    0.000145817   0.000350798      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000200813     1.62862       32.3364      0.328743       4.15545        100    
reco:pmtrajfitcalo:Calorimetry                        6.7641e-05     0.0118476     0.299363     0.00141359     0.0367657       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8961e-05    0.000178537   0.000935317   0.000141433   0.000139979      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00120158      2.2723        47.6958      0.271869       6.51383        100    
reco:pmtracktccalo:Calorimetry                        6.9801e-05    0.00531855     0.0646453    0.000858801    0.0124051       100    
reco:pmtracktcpid:Chi2ParticleID                       2.977e-05    0.000253875    0.0016128    0.000151138   0.000301983      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000185233     1.22277       15.1139      0.274099       2.39371        100    
reco:pmtrajfittccalo:Calorimetry                      6.8821e-05     0.0107305     0.281662     0.00112567     0.0336169       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7691e-05    0.000349519   0.00365498    0.000139568   0.000513134      100    
reco:blurredcluster:BlurredClustering                 0.000168923     1.10226       7.95448      0.452143       1.58989        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000126002     9.65865       109.424       3.17188       16.9725        100    
reco:emshower:EMShower                                0.000183384    0.0283178     0.309605     0.00480535     0.0531387       100    
reco:cvnmap:CVNMapper                                 2.3101e-05    0.00968134     0.0479924    0.00745482     0.0103518       100    
reco:cvneva:CVNEvaluator                              1.7871e-05     0.877367       2.16706       1.21454       0.59672        100    
reco:energyrecnumu:EnergyReco                         0.000310755    0.0316399     0.897774     0.00346467     0.107207        100    
reco:energyrecnue:EnergyReco                          0.000149332   0.00175134     0.0182204    0.000608791   0.00321048       100    
reco:energyrecnc:EnergyReco                           0.00209933    0.00410368     0.0219949    0.00283398    0.00355012       100    
reco:energyrecnumurange:EnergyReco                    0.000134302     0.02964      0.873912     0.00256447     0.104828        100    
reco:energyrecnumumcs:EnergyReco                      0.00208509     0.0313721     0.880171     0.00349931     0.105304        100    
reco:opdec:Deconvolution                              0.00668798     0.192487      0.390731      0.192292      0.0850725       100    
reco:ophitspe:OpHitFinderDeco                          0.0275563      1.31203       2.63471       1.29347      0.599345        100    
reco:opflash:OpFlashFinder                            0.000197083   0.00135681     0.0046097    0.00128882    0.000684258      100    
reco:rns:RandomNumberSaver                            2.1171e-05    3.37935e-05   0.000317156   2.62155e-05   3.0737e-05       100    
reco:anglereconue:NuAngularReco                       0.000164843    0.0012612     0.0157933    0.000389352   0.00265018       100    
reco:anglereconumu:NuAngularReco                      7.5251e-05    0.00107448     0.0157579    0.000220434    0.0026526       100    
reco:anglereconuepfps:NuAngularReco                   0.000132222   0.00122667     0.0153755    0.000347436   0.00265914       100    
reco:anglereconumupfps:NuAngularReco                  0.000110102   0.00115618     0.0156443    0.00028882    0.00269643       100    
[art]:TriggerResults:TriggerResultInserter             9.841e-06    1.28804e-05   5.1391e-05    1.17205e-05   5.20872e-06      100    
end_path:out1:RootOutput                               3.49e-06      6.249e-06    3.2341e-05      4.7e-06     4.18661e-06      100    
end_path:out1:RootOutput(write)                        0.0190503     0.249446      0.898287      0.257887      0.164187        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39379.4 MB
  Peak resident set size usage (VmHWM): 2253.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_288_20231203T123133Z_gen_g4_detsim_hitreco__20240508T050351Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_288_20231203T123133Z_gen_g4_detsim_hitreco__20240508T050351Z_reco2_reco_2024-11-22T_233612Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_288_20231203T123133Z_gen_g4_detsim_hitreco__20240508T050351Z_reco2_reco_data_2024-11-22T_233612Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work