Jobsub ID 108752.59@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108752.59@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:46:20
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 05:33:44
From worker node	Hostname	b9p09p0990.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:49:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_323_20231203T040635Z_gen_g4_detsim_hitreco__20240508T044835Z_reco2.root
Jobscript	Exit code	0
	Real time	44m (2651s)
	CPU time	45m (2735s = 103%)
Outputting started	2024-11-23 05:33:24
Output files
Finished	2024-11-23 05:33:44
List job events Wrapper job log
Jobscript log (last 10,000 characters)

G
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:33:14 CET run: 6407968 subRun: 1 event: 32400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:33:14 CET run: 6407968 subRun: 1 event: 32400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:14 CET run: 6407968 subRun: 1 event: 32400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:14 CET run: 6407968 subRun: 1 event: 32400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:15 CET run: 6407968 subRun: 1 event: 32400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:15 CET run: 6407968 subRun: 1 event: 32400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:33:15 CET run: 6407968 subRun: 1 event: 32400
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 165, 85, 89
minwire 0: 2442
minwire 1: 651
minwire 2: 2408
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00698151, 0.000522069, 0.992496, 
Output 1: 0.00170174, 0.997618, 0.000675477, 4.57505e-06, 
Output 2: 0.992436, 0.00756411, 

445
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1369!
23-Nov-2024 06:33:23 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_323_20231203T040635Z_gen_g4_detsim_hitreco__20240508T044835Z_reco2_reco_data_2024-11-23T_044918Z.root"
23-Nov-2024 06:33:23 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/14/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_323_20231203T040635Z_gen_g4_detsim_hitreco__20240508T044835Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.13903       24.5059       488.387       10.071        60.0726        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0170823     0.0399953     0.0969033     0.0401967     0.0121232       100    
reco:linecluster:LineCluster                           0.149954      0.312097       5.72971      0.179811       0.75863        100    
reco:trajcluster:TrajCluster                           0.0478273     0.650296       21.3536      0.0609508      2.85277        100    
reco:pandora:StandardPandora                           0.859344       2.32163       63.8617       1.3112        6.39072        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000128142   0.00596398     0.140766     0.00148963     0.0176539       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000206411   0.00842805     0.0750918    0.00301618     0.0153069       100    
reco:pandoracalo:Calorimetry                          0.000109151    0.0180156     0.596094     0.00114799     0.0726779       100    
reco:pandorapid:Chi2ParticleID                         3.429e-05    0.000164792   0.00118767    0.000123316   0.000172396      100    
reco:trkshowersplit:TrackShowerHits                   0.000119191    0.0239493     0.898784     0.000719791    0.107901        100    
reco:pmtrack:PMAlgTrackMaker                           0.0010037      2.07166       59.8168      0.205773       7.77977        100    
reco:pmtrackcalo:Calorimetry                          5.8401e-05     0.0108924     0.439104     0.000852682    0.0485102       100    
reco:pmtrackpid:Chi2ParticleID                         2.383e-05    0.000287315   0.00432903    0.000136282   0.000560756      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000178901     1.6478        32.7936      0.322708       4.57258        100    
reco:pmtrajfitcalo:Calorimetry                         5.604e-05     0.0174802     0.613206     0.00131706     0.0718714       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.613e-05    0.00018878    0.00173762    0.000132841   0.000220284      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000861307     2.77864       74.6392      0.193041       10.4547        100    
reco:pmtracktccalo:Calorimetry                         5.866e-05    0.00668634     0.267465     0.000864602    0.0286997       100    
reco:pmtracktcpid:Chi2ParticleID                       2.331e-05    0.000236901   0.00236123    0.000145686   0.000355118      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000152841     1.44608       41.0524      0.270943       4.66337        100    
reco:pmtrajfittccalo:Calorimetry                      5.4261e-05     0.0144293     0.675383     0.00118934     0.0708414       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.461e-05    0.000310358   0.00367044    0.000165526   0.000504025      100    
reco:blurredcluster:BlurredClustering                 0.00368946     0.960306       10.4647      0.500108       1.43559        100    
reco:emtrkmichelid:EmTrackMichelId                     0.169008       8.75115       165.758       2.5996        21.3871        100    
reco:emshower:EMShower                                0.000278412    0.0278902     0.479597     0.00368205     0.065971        100    
reco:cvnmap:CVNMapper                                 2.5321e-05     0.0120609     0.0682179     0.0142143     0.0108228       100    
reco:cvneva:CVNEvaluator                               1.583e-05     0.698789       5.62619      0.814031      0.675016        100    
reco:energyrecnumu:EnergyReco                         0.000322643    0.0421737      1.32496     0.00386444     0.185618        100    
reco:energyrecnue:EnergyReco                          0.000110971   0.00197664     0.0537065    0.00052308    0.00596594       100    
reco:energyrecnc:EnergyReco                           0.00152954     0.0043714     0.0620259     0.0031536    0.00665581       100    
reco:energyrecnumurange:EnergyReco                    0.000146091    0.0389003      1.30072     0.00276422     0.177959        100    
reco:energyrecnumumcs:EnergyReco                       0.0015739     0.0405286      1.29111     0.00338286     0.178293        100    
reco:opdec:Deconvolution                               0.0732383     0.274164      0.518497      0.280091      0.0842867       100    
reco:ophitspe:OpHitFinderDeco                          0.383413       2.02996       4.54733       2.05251      0.890238        100    
reco:opflash:OpFlashFinder                            0.000297752   0.00146637    0.00341793    0.00142726    0.000635964      100    
reco:rns:RandomNumberSaver                             1.716e-05    4.20477e-05   0.000318453   3.81755e-05   2.93539e-05      100    
reco:anglereconue:NuAngularReco                       0.000169421   0.00155493     0.0441829    0.000429099   0.00500798       100    
reco:anglereconumu:NuAngularReco                      6.1741e-05    0.00130522     0.0438888    0.000214342   0.00496285       100    
reco:anglereconuepfps:NuAngularReco                   0.000120801   0.00145655     0.0442664    0.000361043   0.00498721       100    
reco:anglereconumupfps:NuAngularReco                  9.0851e-05    0.00136219     0.0440588    0.000275067   0.00499824       100    
[art]:TriggerResults:TriggerResultInserter             8.26e-06     1.41639e-05   4.8241e-05     1.311e-05    5.50193e-06      100    
end_path:out1:RootOutput                               2.52e-06     7.95983e-06   4.5261e-05      7.3e-06     4.88083e-06      100    
end_path:out1:RootOutput(write)                        0.024033       0.2392        1.32469      0.242879       0.1837         100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21837.8 MB
  Peak resident set size usage (VmHWM): 2401.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_323_20231203T040635Z_gen_g4_detsim_hitreco__20240508T044835Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_323_20231203T040635Z_gen_g4_detsim_hitreco__20240508T044835Z_reco2_reco_2024-11-23T_044918Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_323_20231203T040635Z_gen_g4_detsim_hitreco__20240508T044835Z_reco2_reco_data_2024-11-23T_044918Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work