Jobsub ID 108830.71@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108830.71@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:33:22
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 06:35:59
From worker node	Hostname	heplnc149.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:35:01
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1020_20231202T183134Z_gen_g4_detsim_hitreco__20240508T061446Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3637s)
	CPU time	57m (3435s = 94%)
Outputting started	2024-11-23 06:35:38
Output files
Finished	2024-11-23 06:35:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

re
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:25 GMT run: 74506302 subRun: 1 event: 102100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:25 GMT run: 74506302 subRun: 1 event: 102100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:25 GMT run: 74506302 subRun: 1 event: 102100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:25 GMT run: 74506302 subRun: 1 event: 102100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:25 GMT run: 74506302 subRun: 1 event: 102100
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:35:32 GMT run: 74506302 subRun: 1 event: 102100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:35:32 GMT run: 74506302 subRun: 1 event: 102100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 165, 172, 173
minwire 0: 113
minwire 1: 2309
minwire 2: 8
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.625331, 0.322463, 0.0522064, 
Output 1: 0.166406, 0.702522, 0.12733, 0.00374164, 
Output 2: 0.933537, 0.0664634, 

341
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 954!
23-Nov-2024 06:35:37 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1020_20231202T183134Z_gen_g4_detsim_hitreco__20240508T061446Z_reco2_reco_data_2024-11-23T_053507Z.root"
23-Nov-2024 06:35:37 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a3/9c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1020_20231202T183134Z_gen_g4_detsim_hitreco__20240508T061446Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.16316       33.6284       477.647       14.7611       67.0012        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0235671     0.0508643     0.0856041     0.0532708     0.0131576       100    
reco:linecluster:LineCluster                           0.203503      0.505269       15.9878       0.2339        1.78272        100    
reco:trajcluster:TrajCluster                           0.0653425     0.913825       34.6719      0.0838232      4.09571        100    
reco:pandora:StandardPandora                            1.49986       2.61362       22.1629       1.98354       2.6921         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00019634    0.00654073     0.0995218    0.00255791     0.012761        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000278195    0.0146993     0.222021     0.00379928     0.0303441       100    
reco:pandoracalo:Calorimetry                          0.000187855    0.0160401     0.454214     0.00167983     0.0603147       100    
reco:pandorapid:Chi2ParticleID                        6.0915e-05    0.000287241   0.00272748    0.000201526   0.000318519      100    
reco:trkshowersplit:TrackShowerHits                   0.000204626    0.0211344     0.728269     0.00127907     0.0895853       100    
reco:pmtrack:PMAlgTrackMaker                          0.00188792      2.73582       74.8675      0.241932       9.08598        100    
reco:pmtrackcalo:Calorimetry                          9.2044e-05    0.00816558     0.100225     0.00147648     0.0189329       100    
reco:pmtrackpid:Chi2ParticleID                        3.7211e-05    0.000383517   0.00263294    0.000216945   0.000471839      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000306889     2.88895       82.9242      0.344783       9.90194        100    
reco:pmtrajfitcalo:Calorimetry                        8.7014e-05     0.0170381     0.495437     0.00218701     0.0625586       100    
reco:pmtrajfitpid:Chi2ParticleID                      4.1989e-05    0.000288561    0.0010656    0.000231897   0.000197649      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00138777      3.18707       69.1573      0.272192       10.2776        100    
reco:pmtracktccalo:Calorimetry                        8.7836e-05    0.00611027     0.0576941    0.00156263     0.012083        100    
reco:pmtracktcpid:Chi2ParticleID                      4.2229e-05    0.000375034    0.0023715    0.000216764   0.000432971      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000238199     1.56793       26.756       0.379813       3.82904        100    
reco:pmtrajfittccalo:Calorimetry                      0.000108364    0.0133016     0.330056     0.00182432     0.0440256       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9805e-05    0.000482814   0.00502324    0.000274879   0.000734069      100    
reco:blurredcluster:BlurredClustering                 0.00601253      1.14957       6.02819      0.521254       1.45815        100    
reco:emtrkmichelid:EmTrackMichelId                     0.354642       13.8388       166.276       5.18315       25.697         100    
reco:emshower:EMShower                                0.000359348    0.0379697     0.241098     0.00780107     0.0546693       100    
reco:cvnmap:CVNMapper                                 3.4615e-05     0.0175531     0.0523477     0.017965      0.0117148       100    
reco:cvneva:CVNEvaluator                               2.635e-05      1.1419        2.17679       1.32636      0.626984        100    
reco:energyrecnumu:EnergyReco                         0.000501236    0.0383174     0.865838     0.00715066     0.105302        100    
reco:energyrecnue:EnergyReco                          0.000190529   0.00192739     0.0174979    0.000773404   0.00294719       100    
reco:energyrecnc:EnergyReco                           0.00219475    0.00475771     0.0223956    0.00377971    0.00312363       100    
reco:energyrecnumurange:EnergyReco                    0.000226888    0.0372383     0.869453     0.00359598     0.111266        100    
reco:energyrecnumumcs:EnergyReco                      0.00212582     0.0385543     0.840834     0.00511766     0.107369        100    
reco:opdec:Deconvolution                               0.118227      0.356067      0.534105      0.354445      0.0861905       100    
reco:ophitspe:OpHitFinderDeco                          0.362315       2.08883       3.64194       2.03721      0.732869        100    
reco:opflash:OpFlashFinder                            0.000549116   0.00225812     0.0143963    0.00211765    0.00149306       100    
reco:rns:RandomNumberSaver                            3.1169e-05    7.3002e-05    0.000545699   6.13105e-05   5.38376e-05      100    
reco:anglereconue:NuAngularReco                       0.000218572   0.00149645     0.0165568    0.00067329    0.00234194       100    
reco:anglereconumu:NuAngularReco                      8.4279e-05    0.00106954     0.0149103    0.000297957   0.00225935       100    
reco:anglereconuepfps:NuAngularReco                   0.000149772   0.00136479     0.0168453    0.000536583   0.00246928       100    
reco:anglereconumupfps:NuAngularReco                  0.00012248    0.00122316     0.0165147    0.000383519   0.00257652       100    
[art]:TriggerResults:TriggerResultInserter            1.2193e-05    1.9486e-05    8.7244e-05    1.6927e-05    8.84202e-06      100    
end_path:out1:RootOutput                               4.619e-06    1.09811e-05   4.3702e-05     9.518e-06    5.8122e-06       100    
end_path:out1:RootOutput(write)                        0.0273615     0.297618      0.929818      0.302838      0.163025        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21460.2 MB
  Peak resident set size usage (VmHWM): 2039.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1020_20231202T183134Z_gen_g4_detsim_hitreco__20240508T061446Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1020_20231202T183134Z_gen_g4_detsim_hitreco__20240508T061446Z_reco2_reco_2024-11-23T_053507Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1020_20231202T183134Z_gen_g4_detsim_hitreco__20240508T061446Z_reco2_reco_data_2024-11-23T_053507Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work