Jobsub ID 108787.59@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.59@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug_multicore
Last heartbeat	2024-11-23 06:15:25
From worker node	Hostname	wn-choc-009.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_76_20231204T100341Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root
Jobscript	Exit code	0
	Real time	55m (3308s)
	CPU time	51m (3118s = 94%)
Outputting started	2024-11-23 06:15:10
Output files
Finished	2024-11-23 06:15:25
List job events Wrapper job log
Jobscript log (last 10,000 characters)

inModule  23-Nov-2024 07:15:02 CET run: 74517726 subRun: 1 event: 7700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:02 CET run: 74517726 subRun: 1 event: 7700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:02 CET run: 74517726 subRun: 1 event: 7700
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:02 CET run: 74517726 subRun: 1 event: 7700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:02 CET run: 74517726 subRun: 1 event: 7700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:02 CET run: 74517726 subRun: 1 event: 7700
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:15:06 CET run: 74517726 subRun: 1 event: 7700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:15:06 CET run: 74517726 subRun: 1 event: 7700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 122, 153, 108
minwire 0: 874
minwire 1: 2452
minwire 2: 286
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.013551, 0.985579, 0.000869793, 
Output 1: 0.44686, 0.542028, 0.0109336, 0.000177667, 
Output 2: 0.992231, 0.00776868, 

461
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1412!
23-Nov-2024 07:15:09 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_76_20231204T100341Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2_reco_data_2024-11-23T_052008Z.root"
23-Nov-2024 07:15:10 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f5/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_76_20231204T100341Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.874474       30.813        456.782       9.63069       63.6767        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0186903     0.0416563     0.151227      0.0363058     0.022703        100    
reco:linecluster:LineCluster                            0.13703      0.365054       7.02627      0.167264      0.971346        100    
reco:trajcluster:TrajCluster                           0.0433959      1.00367       29.6713      0.0536038      3.88569        100    
reco:pandora:StandardPandora                           0.0574405      2.58413       42.2768       1.51346       5.05265        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000147079   0.00755118     0.141557     0.00145124     0.0189593       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000206922    0.0144871     0.193129     0.00263942     0.0278344       100    
reco:pandoracalo:Calorimetry                          0.000116096    0.0251094     0.659397     0.00103686     0.0904677       100    
reco:pandorapid:Chi2ParticleID                        4.8004e-05    0.000166289   0.000723472   0.000124086   0.000137031      100    
reco:trkshowersplit:TrackShowerHits                   0.000120483    0.0240478     0.634459     0.000643991    0.0889408       100    
reco:pmtrack:PMAlgTrackMaker                          0.00138984      2.43084       57.5191      0.210373       7.49174        100    
reco:pmtrackcalo:Calorimetry                          9.5717e-05     0.0154744     0.554313     0.000702654    0.062366        100    
reco:pmtrackpid:Chi2ParticleID                        4.0108e-05    0.000306502   0.00375992    0.000136333   0.000524357      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000227219     2.11302       27.5059      0.290249       4.83161        100    
reco:pmtrajfitcalo:Calorimetry                        9.0026e-05     0.0258755     0.650134     0.00123232     0.0912351       100    
reco:pmtrajfitpid:Chi2ParticleID                      4.0016e-05    0.00018093    0.000822281   0.000141797   0.000151568      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00126613      3.19022       74.7239      0.200185       9.70284        100    
reco:pmtracktccalo:Calorimetry                        9.5714e-05    0.00945125     0.259913     0.00082062     0.0303193       100    
reco:pmtracktcpid:Chi2ParticleID                      3.9547e-05    0.000285723   0.00292335    0.000144236   0.000418718      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000215893     1.76384       33.1252      0.233923       4.42141        100    
reco:pmtrajfittccalo:Calorimetry                      9.0722e-05     0.0219606     0.669993     0.00096103     0.0832881       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9963e-05    0.000373307   0.00420704    0.000152606   0.000652467      100    
reco:blurredcluster:BlurredClustering                 8.4754e-05      1.27773       9.48442      0.441844       1.86539        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0298885      12.8458       165.153       2.9864        25.721         100    
reco:emshower:EMShower                                0.000272818    0.0336319     0.391899     0.00966463     0.0592966       100    
reco:cvnmap:CVNMapper                                 2.5364e-05     0.0122405     0.0490393     0.0113794     0.0108146       100    
reco:cvneva:CVNEvaluator                              1.9999e-05      0.60573       1.07098      0.820108      0.389537        100    
reco:energyrecnumu:EnergyReco                         0.000327135    0.0422856      1.1748      0.00393955      0.13662        100    
reco:energyrecnue:EnergyReco                          0.000188063   0.00273721     0.0460008    0.00050534    0.00662769       100    
reco:energyrecnc:EnergyReco                           0.00234885     0.0050613     0.0499511     0.0028553    0.00689651       100    
reco:energyrecnumurange:EnergyReco                    0.000181481    0.0406885      1.17582     0.00270008     0.136526        100    
reco:energyrecnumumcs:EnergyReco                      0.00229383     0.0421956      1.17752     0.00351918     0.136467        100    
reco:opdec:Deconvolution                               0.0911485     0.306766      0.526282      0.306798      0.0924367       100    
reco:ophitspe:OpHitFinderDeco                          0.436134       1.68805       4.0281         1.639       0.665036        100    
reco:opflash:OpFlashFinder                            0.000322489   0.00141463    0.00319093    0.00131961    0.000632893      100    
reco:rns:RandomNumberSaver                            1.7363e-05    2.78733e-05   0.000252235   2.4741e-05    2.34628e-05      100    
reco:anglereconue:NuAngularReco                       0.000119759   0.00182019     0.0347274    0.00030524    0.00491602       100    
reco:anglereconumu:NuAngularReco                      9.2842e-05    0.00169858     0.0343591    0.00024391    0.00488629       100    
reco:anglereconuepfps:NuAngularReco                   0.000136016   0.00181298     0.0346377    0.000345918   0.00485005       100    
reco:anglereconumupfps:NuAngularReco                  0.000136153   0.00176814     0.0341604    0.000328889   0.00482143       100    
[art]:TriggerResults:TriggerResultInserter            1.1869e-05    1.38185e-05   4.2474e-05    1.2968e-05    3.24797e-06      100    
end_path:out1:RootOutput                               2.672e-06    4.46252e-06   1.7818e-05     3.112e-06    2.28643e-06      100    
end_path:out1:RootOutput(write)                         0.02983      0.262258       1.11717      0.263092      0.182524        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7403.38 MB
  Peak resident set size usage (VmHWM): 2502.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_76_20231204T100341Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_76_20231204T100341Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2_reco_2024-11-23T_052008Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_76_20231204T100341Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2_reco_data_2024-11-23T_052008Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work