Jobsub ID 108738.124@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108738.124@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:22:32
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:39:37
From worker node	Hostname	n259.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:30:57
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_696_20231207T084620Z_gen_g4_detsim_hitreco__20240510T023848Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4102s)
	CPU time	58m (3487s = 85%)
Outputting started	2024-11-23 05:39:20
Output files
Finished	2024-11-23 05:39:37
List job events Wrapper job log
Jobscript log (last 10,000 characters)

23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:39:09 GMT run: 6452759 subRun: 1 event: 69700
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 182, 169, 136
minwire 0: 525
minwire 1: 1942
minwire 2: 361
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.335923, 0.660016, 0.00406098, 
Output 1: 0.878725, 0.113913, 0.00654595, 0.000815837, 
Output 2: 0.991804, 0.00819598, 

479
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1582!
23-Nov-2024 05:39:19 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_696_20231207T084620Z_gen_g4_detsim_hitreco__20240510T023848Z_reco2_reco_data_2024-11-23T_043105Z.root"
23-Nov-2024 05:39:19 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/fc/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_696_20231207T084620Z_gen_g4_detsim_hitreco__20240510T023848Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                               1.098        35.3953       884.907       10.8138       110.584        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0346534     0.0738736      0.18167      0.0792577     0.0257525       100    
reco:linecluster:LineCluster                           0.240162      0.925786       25.2657      0.364442       3.36058        100    
reco:trajcluster:TrajCluster                            0.10128       1.13132       53.5824      0.112839       5.86894        100    
reco:pandora:StandardPandora                           0.113422       5.28941       231.09        1.44042       26.1328        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000167698    0.0067561     0.218958     0.00136318     0.0241853       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000242049    0.0106396     0.0787765    0.00268175     0.0172539       100    
reco:pandoracalo:Calorimetry                          0.000125217    0.0316079      1.32743     0.00088354     0.165619        100    
reco:pandorapid:Chi2ParticleID                        4.9434e-05    0.000186624    0.0014403    0.000135156   0.000186395      100    
reco:trkshowersplit:TrackShowerHits                   0.000123554    0.0342692      1.53607     0.000665622    0.178675        100    
reco:pmtrack:PMAlgTrackMaker                          0.00104354      3.16988       95.4386       0.17442       12.9993        100    
reco:pmtrackcalo:Calorimetry                          6.7157e-05     0.0129533     0.593943     0.000941977    0.0648984       100    
reco:pmtrackpid:Chi2ParticleID                        2.7081e-05    0.000314974   0.00379276    0.000154553   0.000583747      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000268559     2.16685       44.369       0.339001       6.94131        100    
reco:pmtrajfitcalo:Calorimetry                        8.5732e-05     0.0261503     0.960699     0.00140935     0.124068        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.0006e-05    0.000210431   0.00109616    0.000175213   0.00017131       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000904425     3.98183       116.309      0.185791       16.8595        100    
reco:pmtracktccalo:Calorimetry                        6.5714e-05    0.00786499     0.330203     0.00109934     0.0351618       100    
reco:pmtracktcpid:Chi2ParticleID                      2.6841e-05    0.000310236   0.00439128    0.000176249   0.00055043       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000167969     2.13003       75.4831      0.309401       8.44796        100    
reco:pmtrajfittccalo:Calorimetry                      6.2338e-05     0.0228288      1.06573     0.00125632     0.119884        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5047e-05    0.000424752   0.00743337    0.000189604   0.000944277      100    
reco:blurredcluster:BlurredClustering                 8.8758e-05      1.20023       9.17339      0.538536       1.66325        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000139685     11.2094       237.213       2.87996       30.1387        100    
reco:emshower:EMShower                                0.000218294    0.0261193     0.413427     0.00822533     0.0537414       100    
reco:cvnmap:CVNMapper                                 2.5488e-05    0.00807127     0.057859     0.00645567    0.00885751       100    
reco:cvneva:CVNEvaluator                              1.8235e-05     0.637035       1.03276      0.921672      0.439329        100    
reco:energyrecnumu:EnergyReco                         0.000442959    0.0229953     0.403385      0.0051133     0.0590831       100    
reco:energyrecnue:EnergyReco                          0.000161226   0.00250433     0.100467     0.000529274    0.0103969       100    
reco:energyrecnc:EnergyReco                           0.00202291    0.00557997     0.104427      0.0036232     0.0105382       100    
reco:energyrecnumurange:EnergyReco                    0.000236569    0.0207657     0.399178     0.00343259     0.0585375       100    
reco:energyrecnumumcs:EnergyReco                      0.00132964     0.0224035     0.402995     0.00377678     0.058447        100    
reco:opdec:Deconvolution                               0.137345      0.418025      0.768915      0.406199      0.120618        100    
reco:ophitspe:OpHitFinderDeco                          0.303814       2.51572       5.22067       2.31268       1.05649        100    
reco:opflash:OpFlashFinder                            0.000140206   0.00158587    0.00484141    0.00141659    0.00080328       100    
reco:rns:RandomNumberSaver                            1.8285e-05    3.81047e-05   0.000324927   3.1179e-05    3.05484e-05      100    
reco:anglereconue:NuAngularReco                       0.000164121   0.00196461     0.0884653    0.000398731   0.00904408       100    
reco:anglereconumu:NuAngularReco                      8.1985e-05     0.0017449     0.087417     0.000249317   0.00893793       100    
reco:anglereconuepfps:NuAngularReco                   0.000160665    0.0018642     0.0862497    0.000342093   0.00881985       100    
reco:anglereconumupfps:NuAngularReco                  0.000138042   0.00181201     0.0863518    0.00032418    0.00884197       100    
[art]:TriggerResults:TriggerResultInserter             8.897e-06    1.5054e-05    4.3553e-05    1.4237e-05    4.02869e-06      100    
end_path:out1:RootOutput                               2.595e-06    5.97669e-06   2.3184e-05    4.4235e-06    3.40288e-06      100    
end_path:out1:RootOutput(write)                        0.0143281     0.271605       1.58001      0.282156      0.225808        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38550.1 MB
  Peak resident set size usage (VmHWM): 2490.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_696_20231207T084620Z_gen_g4_detsim_hitreco__20240510T023848Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_696_20231207T084620Z_gen_g4_detsim_hitreco__20240510T023848Z_reco2_reco_2024-11-23T_043105Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_696_20231207T084620Z_gen_g4_detsim_hitreco__20240510T023848Z_reco2_reco_data_2024-11-23T_043105Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work