Jobsub ID 108830.152@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108830.152@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:33:22
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 07:23:10
From worker node	Hostname	tds405.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:34:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1104_20231203T155620Z_gen_g4_detsim_hitreco__20240508T073137Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6497s)
	CPU time	1h (5464s = 84%)
Outputting started	2024-11-23 07:22:50
Output files
Finished	2024-11-23 07:23:10
List job events Wrapper job log
Jobscript log (last 10,000 characters)

WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:22:35 CET run: 74511302 subRun: 1 event: 110500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:22:35 CET run: 74511302 subRun: 1 event: 110500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:22:35 CET run: 74511302 subRun: 1 event: 110500
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:22:35 CET run: 74511302 subRun: 1 event: 110500
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:22:35 CET run: 74511302 subRun: 1 event: 110500
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:22:41 CET run: 74511302 subRun: 1 event: 110500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:22:41 CET run: 74511302 subRun: 1 event: 110500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 205, 158, 154
minwire 0: 1624
minwire 1: 1779
minwire 2: 1021
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00422413, 0.995468, 0.000307473, 
Output 1: 0.257085, 0.739528, 0.00335172, 3.57179e-05, 
Output 2: 0.997245, 0.00275493, 

594
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2066!
23-Nov-2024 08:22:48 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1104_20231203T155620Z_gen_g4_detsim_hitreco__20240508T073137Z_reco2_reco_data_2024-11-23T_053438Z.root"
23-Nov-2024 08:22:48 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/51/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1104_20231203T155620Z_gen_g4_detsim_hitreco__20240508T073137Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.4381        56.6938       899.357       17.5123       136.039        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0462727     0.0876237     0.147492      0.0925701     0.0263094       100    
reco:linecluster:LineCluster                           0.268316       1.09555       24.6526      0.449569       2.93181        100    
reco:trajcluster:TrajCluster                           0.134653       2.53818       88.3812      0.156057       11.1945        100    
reco:pandora:StandardPandora                           0.145357       3.86847       75.019        1.79402       9.37898        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000163863    0.013173      0.329055     0.00243531     0.0383158       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000254656    0.0214869     0.262974     0.00544159     0.0435611       100    
reco:pandoracalo:Calorimetry                          0.000128402    0.0429258      1.13966     0.00206298     0.161959        100    
reco:pandorapid:Chi2ParticleID                        5.2411e-05    0.000264708   0.00310086    0.000186729   0.000342095      100    
reco:trkshowersplit:TrackShowerHits                   7.2872e-05     0.0551231      1.65204     0.00155764     0.218354        100    
reco:pmtrack:PMAlgTrackMaker                          0.00184197      5.93031       142.798      0.584411       19.6575        100    
reco:pmtrackcalo:Calorimetry                          0.000107023    0.0268021     0.754882     0.00169254     0.0991218       100    
reco:pmtrackpid:Chi2ParticleID                        4.2711e-05    0.000527206   0.00584276    0.000210544   0.00095176       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000265446     4.35034       89.5803      0.679137       11.8833        100    
reco:pmtrajfitcalo:Calorimetry                        9.2542e-05     0.0445599      1.14018     0.00261246     0.164064        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.718e-05    0.000470973    0.0184119    0.000222845   0.00181966       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00153381      7.73914       184.501       0.55799       26.0832        100    
reco:pmtracktccalo:Calorimetry                        9.4692e-05     0.0156151     0.276586     0.00195107     0.0435727       100    
reco:pmtracktcpid:Chi2ParticleID                       4.008e-05    0.000462477    0.0033889    0.000229315   0.000674237      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000247565     3.76684       81.2584      0.724167       10.445         100    
reco:pmtrajfittccalo:Calorimetry                      8.8272e-05     0.0375052     0.933613     0.00223274     0.134498        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.7301e-05    0.000660191   0.00690365    0.000259001   0.00113559       100    
reco:blurredcluster:BlurredClustering                 0.000109892     1.63979       14.7206      0.701145       2.47715        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000137773     20.2533       283.213       5.52732       42.2751        100    
reco:emshower:EMShower                                0.000198694    0.0624582      0.62925      0.011796      0.111605        100    
reco:cvnmap:CVNMapper                                  2.36e-05      0.0141702     0.0669667    0.00938594     0.0144443       100    
reco:cvneva:CVNEvaluator                               2.187e-05      1.0106        1.84099       1.27061      0.558968        100    
reco:energyrecnumu:EnergyReco                         0.000367698    0.115245        6.468      0.00884163     0.647668        100    
reco:energyrecnue:EnergyReco                          0.000203385   0.00464905     0.103756     0.000904879    0.0124695       100    
reco:energyrecnc:EnergyReco                           0.00298719    0.00868555     0.112664      0.0041712     0.0134203       100    
reco:energyrecnumurange:EnergyReco                    0.000242725    0.108365       6.45793     0.00410943     0.646577        100    
reco:energyrecnumumcs:EnergyReco                      0.00146253     0.107929       6.19152     0.00873599      0.61993        100    
reco:opdec:Deconvolution                                0.14478       0.58537       2.15466      0.552448      0.241653        100    
reco:ophitspe:OpHitFinderDeco                          0.473398       2.74515       4.94581       2.72456      0.965175        100    
reco:opflash:OpFlashFinder                            0.000285006   0.00185618      0.00521     0.00173951    0.00085362       100    
reco:rns:RandomNumberSaver                            2.0661e-05    4.26439e-05   0.000379738   3.4896e-05    3.68908e-05      100    
reco:anglereconue:NuAngularReco                       0.000206315     0.00364      0.0890056    0.000615668    0.0107649       100    
reco:anglereconumu:NuAngularReco                      9.6062e-05    0.00303569     0.0877828    0.000353793    0.0102388       100    
reco:anglereconuepfps:NuAngularReco                   0.000157743   0.00322909     0.0878994    0.000485286    0.0102634       100    
reco:anglereconumupfps:NuAngularReco                  0.000140113   0.00315496     0.0884228    0.000441829    0.0103074       100    
[art]:TriggerResults:TriggerResultInserter             1.048e-05    1.62944e-05   7.0091e-05    1.4715e-05    7.78341e-06      100    
end_path:out1:RootOutput                               3.29e-06     6.65798e-06   3.4681e-05     5.27e-06     4.18449e-06      100    
end_path:out1:RootOutput(write)                        0.0202752     0.384278       1.93037      0.344548       0.29967        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21691.1 MB
  Peak resident set size usage (VmHWM): 2539.62 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1104_20231203T155620Z_gen_g4_detsim_hitreco__20240508T073137Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1104_20231203T155620Z_gen_g4_detsim_hitreco__20240508T073137Z_reco2_reco_2024-11-23T_053438Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1104_20231203T155620Z_gen_g4_detsim_hitreco__20240508T073137Z_reco2_reco_data_2024-11-23T_053438Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work