Jobsub ID 108742.27@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.27@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:48:29
From worker node	Hostname	b9g37p7399.cern.ch
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:31:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_187_20231207T122336Z_gen_g4_detsim_hitreco__20240510T054324Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4620s)
	CPU time	1h (4472s = 96%)
Outputting started	2024-11-23 05:48:11
Output files
Finished	2024-11-23 05:48:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

uneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:46:33 CET run: 6456061 subRun: 1 event: 18800
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:46:33 CET run: 6456061 subRun: 1 event: 18800
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:46:33 CET run: 6456061 subRun: 1 event: 18800
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:46:34 CET run: 6456061 subRun: 1 event: 18800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:46:34 CET run: 6456061 subRun: 1 event: 18800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:48:03 CET run: 6456061 subRun: 1 event: 18800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:48:03 CET run: 6456061 subRun: 1 event: 18800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 3219, 3284, 2087
minwire 0: 490
minwire 1: 1101
minwire 2: 239
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00695066, 0.973606, 0.0194432, 
Output 1: 0.110305, 0.328598, 0.332626, 0.228471, 
Output 2: 0.0471477, 0.952852, 

841
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3309!
23-Nov-2024 06:48:10 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_187_20231207T122336Z_gen_g4_detsim_hitreco__20240510T054324Z_reco2_reco_data_2024-11-23T_043115Z.root"
23-Nov-2024 06:48:10 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/4f/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_187_20231207T122336Z_gen_g4_detsim_hitreco__20240510T054324Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.87257       43.571        384.044       15.1591       75.505         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0172497     0.047426      0.298209      0.0442051     0.0302049       100    
reco:linecluster:LineCluster                           0.0960467     0.576238       14.6315      0.202304       1.7866         100    
reco:trajcluster:TrajCluster                           0.0486439      1.98085       49.2747      0.0808168      7.0606         100    
reco:pandora:StandardPandora                           0.0751706      3.06328        20.07        2.15299       2.97474        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000173601    0.0109396     0.111121     0.00285841     0.0208037       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000242524    0.0219213     0.547988     0.00493453     0.0581487       100    
reco:pandoracalo:Calorimetry                          0.000144452    0.0367831     0.669406     0.00235992     0.107538        100    
reco:pandorapid:Chi2ParticleID                        5.7058e-05    0.000267719   0.00200547    0.000221911   0.000224349      100    
reco:trkshowersplit:TrackShowerHits                   0.000136114    0.0293134     0.516762     0.00149052     0.0882607       100    
reco:pmtrack:PMAlgTrackMaker                          0.00158143      3.78574       67.1409      0.421109       10.3371        100    
reco:pmtrackcalo:Calorimetry                          0.000112735    0.0147792     0.167084     0.00193158     0.033541        100    
reco:pmtrackpid:Chi2ParticleID                         4.559e-05    0.000440453   0.00288751    0.000253712   0.000544078      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000267443     2.85985       25.9167      0.515685       5.65727        100    
reco:pmtrajfitcalo:Calorimetry                        9.8032e-05     0.0379776      0.61635     0.00255889     0.107805        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.1822e-05    0.000302025   0.00146993    0.000226997   0.000240822      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00140889      4.56443       61.325       0.445884       11.6918        100    
reco:pmtracktccalo:Calorimetry                        0.000104767    0.0111537     0.108887     0.00202257     0.0226114       100    
reco:pmtracktcpid:Chi2ParticleID                      4.4082e-05    0.000422664   0.00251736    0.000262707   0.000496524      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000241532     2.44212       21.5036      0.642665       4.63072        100    
reco:pmtrajfittccalo:Calorimetry                      9.9772e-05     0.0260487     0.423625     0.00212272     0.069331        100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.218e-05    0.000585039    0.0051604    0.000264302   0.000861078      100    
reco:blurredcluster:BlurredClustering                 0.000147503     1.53214       9.48907      0.785442       1.88933        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000292421     18.3055       136.699       5.2771        31.1157        100    
reco:emshower:EMShower                                0.000351547    0.0377491     0.296722     0.00604742     0.0626069       100    
reco:cvnmap:CVNMapper                                 3.6469e-05     0.0209363     0.0924961     0.0226111     0.0162212       100    
reco:cvneva:CVNEvaluator                              2.4213e-05     0.862421       1.75076      0.991223      0.554965        100    
reco:energyrecnumu:EnergyReco                         0.000443939    0.0910805      1.64376     0.00652067     0.256026        100    
reco:energyrecnue:EnergyReco                          0.000208139   0.00366023     0.0379746    0.00102684    0.00650511       100    
reco:energyrecnc:EnergyReco                           0.00264647    0.00646937     0.0398493    0.00391142    0.00695733       100    
reco:energyrecnumurange:EnergyReco                    0.000214457    0.0876897      1.97553     0.00385198     0.268209        100    
reco:energyrecnumumcs:EnergyReco                      0.00267538     0.0894649      1.93242      0.0050874     0.266015        100    
reco:opdec:Deconvolution                               0.0901177     0.397365      0.824779      0.396672      0.137213        100    
reco:ophitspe:OpHitFinderDeco                          0.404336       2.24848       4.2408        2.16586      0.864566        100    
reco:opflash:OpFlashFinder                            0.000543864   0.00230042    0.00704516    0.00210894    0.00117381       100    
reco:rns:RandomNumberSaver                            2.2036e-05    6.24048e-05   0.000915741   4.45935e-05   9.55211e-05      100    
reco:anglereconue:NuAngularReco                       0.000185745   0.00278074     0.0271904    0.000653307   0.00532217       100    
reco:anglereconumu:NuAngularReco                      0.000112008   0.00246416     0.0255816    0.000386682   0.00523661       100    
reco:anglereconuepfps:NuAngularReco                   0.000164087   0.00259351     0.025592     0.00057049    0.00499405       100    
reco:anglereconumupfps:NuAngularReco                  0.000148414   0.00249284     0.0256391    0.000483333   0.00497581       100    
[art]:TriggerResults:TriggerResultInserter            1.3053e-05    3.17496e-05   0.000801296   2.2045e-05    7.79758e-05      100    
end_path:out1:RootOutput                               3.26e-06     9.54729e-06   3.8089e-05    8.8765e-06    5.48446e-06      100    
end_path:out1:RootOutput(write)                        0.0303661     0.361447       1.22184      0.354993      0.234872        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9441.42 MB
  Peak resident set size usage (VmHWM): 2239.62 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_187_20231207T122336Z_gen_g4_detsim_hitreco__20240510T054324Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_187_20231207T122336Z_gen_g4_detsim_hitreco__20240510T054324Z_reco2_reco_2024-11-23T_043115Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_187_20231207T122336Z_gen_g4_detsim_hitreco__20240510T054324Z_reco2_reco_data_2024-11-23T_043115Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work