Jobsub ID 108740.76@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108740.76@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:25:45
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 06:11:34
From worker node	Hostname	tds434.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:33:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_162_20231207T070812Z_gen_g4_detsim_hitreco__20240510T062549Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5877s)
	CPU time	1h (5318s = 90%)
Outputting started	2024-11-23 06:11:13
Output files
Finished	2024-11-23 06:11:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:50 CET run: 6452759 subRun: 1 event: 16300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:11:05 CET run: 6452759 subRun: 1 event: 16300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 702, 704, 432
minwire 0: 122
minwire 1: 2096
minwire 2: 107
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00485211, 0.994619, 0.000529241, 
Output 1: 0.834892, 0.154802, 0.00930912, 0.000996778, 
Output 2: 0.92171, 0.0782895, 

495
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1540!
23-Nov-2024 07:11:11 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_162_20231207T070812Z_gen_g4_detsim_hitreco__20240510T062549Z_reco2_reco_data_2024-11-23T_043321Z.root"
23-Nov-2024 07:11:11 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/59/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_162_20231207T070812Z_gen_g4_detsim_hitreco__20240510T062549Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.40063       54.2769       1171.79       12.4614       171.472        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0456722     0.0859382     0.218311      0.0831657     0.0271229       100    
reco:linecluster:LineCluster                           0.268649       1.59571       64.445       0.439238       7.39247        100    
reco:trajcluster:TrajCluster                           0.134491       5.38125       211.435      0.149386       28.2916        100    
reco:pandora:StandardPandora                           0.147504       4.12874       84.4752       1.7917         11.32         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000150983    0.0125805     0.360619     0.00153134     0.0454822       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000236265    0.0227102     0.630706     0.00290514     0.0802597       100    
reco:pandoracalo:Calorimetry                          9.9342e-05     0.0492073      1.77445     0.00121643     0.224215        100    
reco:pandorapid:Chi2ParticleID                         4.011e-05    0.000213105   0.00193244    0.000140962   0.000274192      100    
reco:trkshowersplit:TrackShowerHits                   7.6631e-05     0.0648354      2.59662     0.000870067     0.32106        100    
reco:pmtrack:PMAlgTrackMaker                           0.0013181      5.15822       127.422      0.271573       20.2267        100    
reco:pmtrackcalo:Calorimetry                          7.4511e-05     0.0243962     0.954602     0.00112443     0.110434        100    
reco:pmtrackpid:Chi2ParticleID                         2.882e-05    0.000393906   0.00531212    0.000162963   0.000729238      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000243074     5.85502       236.796       0.3842        26.1643        100    
reco:pmtrajfitcalo:Calorimetry                        8.7572e-05     0.0461246      1.66143     0.00165963     0.203388        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5531e-05    0.000247682   0.00133084    0.000171758   0.000231804      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00135147      6.93853       187.927      0.337665       28.3965        100    
reco:pmtracktccalo:Calorimetry                        7.8872e-05     0.0117534      0.29925     0.00113439     0.0411671       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6341e-05    0.00035251    0.00309834    0.000179578   0.000518303      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000221855     3.02081       61.9782      0.400994       9.75034        100    
reco:pmtrajfittccalo:Calorimetry                      8.3472e-05     0.0338271     0.962262     0.00144977     0.136353        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.982e-05    0.000549976    0.0076445    0.000186664   0.00111375       100    
reco:blurredcluster:BlurredClustering                 0.000134232     1.19257       8.92793      0.768481       1.61091        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000143743     16.0257       243.238       3.51293       41.6108        100    
reco:emshower:EMShower                                0.000213414    0.0464987      0.7888       0.0066934     0.113669        100    
reco:cvnmap:CVNMapper                                  2.269e-05    0.00814025     0.0560631    0.00760633    0.00849457       100    
reco:cvneva:CVNEvaluator                              1.6611e-05     0.789594       1.45211       1.06446      0.493412        100    
reco:energyrecnumu:EnergyReco                         0.000421538    0.155133       10.1041     0.00523942      1.02236        100    
reco:energyrecnue:EnergyReco                          0.000154113   0.00369835     0.0814728    0.000639838    0.0120232       100    
reco:energyrecnc:EnergyReco                           0.00229359    0.00686545     0.110987      0.0036535     0.0140352       100    
reco:energyrecnumurange:EnergyReco                    0.000245155    0.136755       8.52347     0.00344347     0.868855        100    
reco:energyrecnumumcs:EnergyReco                      0.00227745     0.127283       7.45078      0.0041866     0.764844        100    
reco:opdec:Deconvolution                               0.140678      0.517074      0.801638      0.527663      0.127524        100    
reco:ophitspe:OpHitFinderDeco                          0.402136       2.48604       4.86345       2.46058      0.955807        100    
reco:opflash:OpFlashFinder                            0.000164874   0.00160725    0.00584166    0.00147459    0.000857145      100    
reco:rns:RandomNumberSaver                             1.797e-05    3.47437e-05   0.000395077    2.598e-05    3.77119e-05      100    
reco:anglereconue:NuAngularReco                       0.000180634   0.00305765     0.0923053    0.000388938    0.0119963       100    
reco:anglereconumu:NuAngularReco                      7.3651e-05    0.00288417     0.0920876    0.000268631    0.0118356       100    
reco:anglereconuepfps:NuAngularReco                   0.000122402   0.00305209     0.0921042    0.000410613    0.0118905       100    
reco:anglereconumupfps:NuAngularReco                  0.000107833   0.00302211     0.0910208    0.000348691    0.012034        100    
[art]:TriggerResults:TriggerResultInserter            1.0001e-05    1.53411e-05   8.6352e-05     1.397e-05    8.74201e-06      100    
end_path:out1:RootOutput                               3.001e-06    5.61363e-06   3.7471e-05     3.825e-06    4.3277e-06       100    
end_path:out1:RootOutput(write)                        0.0174411     0.334186       2.0305       0.308967      0.314124        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21567.5 MB
  Peak resident set size usage (VmHWM): 2427.32 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_162_20231207T070812Z_gen_g4_detsim_hitreco__20240510T062549Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_162_20231207T070812Z_gen_g4_detsim_hitreco__20240510T062549Z_reco2_reco_2024-11-23T_043321Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_162_20231207T070812Z_gen_g4_detsim_hitreco__20240510T062549Z_reco2_reco_data_2024-11-23T_043321Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work