Jobsub ID 108712.91@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108712.91@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:22:55
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 05:05:39
From worker node	Hostname	heplnc154.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:28:18
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_543_20231207T145342Z_gen_g4_detsim_hitreco__20240510T062057Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5820s)
	CPU time	1h (6001s = 103%)
Outputting started	2024-11-23 05:05:18
Output files
Finished	2024-11-23 05:05:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

tter:pmtrajfittc@BeginModule  23-Nov-2024 05:04:59 GMT run: 6457551 subRun: 1 event: 54400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:04:59 GMT run: 6457551 subRun: 1 event: 54400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:04:59 GMT run: 6457551 subRun: 1 event: 54400
Cannot attach back to inner node of other track.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:04:59 GMT run: 6457551 subRun: 1 event: 54400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:04:59 GMT run: 6457551 subRun: 1 event: 54400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:05:11 GMT run: 6457551 subRun: 1 event: 54400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:05:11 GMT run: 6457551 subRun: 1 event: 54400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 409, 545, 223
minwire 0: 2428
minwire 1: 0
minwire 2: 2702
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0118767, 0.986585, 0.00153853, 
Output 1: 0.0422885, 0.147425, 0.223214, 0.587072, 
Output 2: 0.964208, 0.0357917, 

486
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1556!
23-Nov-2024 05:05:17 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_543_20231207T145342Z_gen_g4_detsim_hitreco__20240510T062057Z_reco2_reco_data_2024-11-23T_032824Z.root"
23-Nov-2024 05:05:17 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/52/49/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_543_20231207T145342Z_gen_g4_detsim_hitreco__20240510T062057Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.65941       55.4902       993.508       10.732        154.07         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0053777     0.140695       0.87843      0.0444852     0.185059        100    
reco:linecluster:LineCluster                           0.0446325      1.18671       21.2516      0.0960962      3.41665        100    
reco:trajcluster:TrajCluster                           0.0116407      2.29409       77.2852      0.0232457      9.85947        100    
reco:pandora:StandardPandora                            1.01445       7.18075       264.534       1.4103        30.536         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000173074    0.0126002     0.273122     0.00172382     0.0369486       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000246433    0.018288      0.315795     0.00309087     0.0423958       100    
reco:pandoracalo:Calorimetry                          0.000129493    0.0604879      1.28223      0.0010896     0.225554        100    
reco:pandorapid:Chi2ParticleID                        5.0544e-05    0.000249799   0.00273361    0.000158833   0.00031909       100    
reco:trkshowersplit:TrackShowerHits                   0.000129824    0.0725085      1.46928     0.000792708    0.266184        100    
reco:pmtrack:PMAlgTrackMaker                          0.00132459      6.11051       157.836      0.183036       21.8518        100    
reco:pmtrackcalo:Calorimetry                          7.3849e-05     0.0318263      1.06158     0.000878284    0.125412        100    
reco:pmtrackpid:Chi2ParticleID                        2.8243e-05    0.000477229   0.00563809    0.000170625   0.000897569      100    
reco:pmtrajfit:PMAlgTrajFitter                         0.0002342      4.32348       86.2306      0.199902       13.4973        100    
reco:pmtrajfitcalo:Calorimetry                        6.7928e-05     0.0567102      1.30908     0.00143114     0.211599        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9345e-05     0.0002901     0.0024545    0.000172539   0.000358949      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102902      7.53364       175.112      0.185688       26.0935        100    
reco:pmtracktccalo:Calorimetry                        7.5202e-05     0.0162556     0.447277     0.000941732    0.0545658       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9856e-05     0.0004029    0.00430771    0.000171882   0.00065989       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000176301     3.48202       80.7734      0.278385       11.1225        100    
reco:pmtrajfittccalo:Calorimetry                      9.2494e-05     0.0448571      1.39645     0.00127482     0.178706        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6208e-05    0.000590751   0.00808485    0.000182197   0.00120608       100    
reco:blurredcluster:BlurredClustering                 0.00765278      1.46671       12.0885      0.661102       2.13703        100    
reco:emtrkmichelid:EmTrackMichelId                     0.238143       16.7606       269.077       2.72126       40.4069        100    
reco:emshower:EMShower                                0.000307308    0.0466315     0.654347     0.00937545     0.0997275       100    
reco:cvnmap:CVNMapper                                 2.8804e-05     0.0125336     0.0624626     0.0123694     0.0115401       100    
reco:cvneva:CVNEvaluator                              1.6781e-05     0.958355       1.64772       1.31182      0.623606        100    
reco:energyrecnumu:EnergyReco                         0.000390022    0.101807       5.48266     0.00498958     0.562326        100    
reco:energyrecnue:EnergyReco                          0.000142788   0.00391247     0.0899204    0.000615671    0.0113251       100    
reco:energyrecnc:EnergyReco                           0.00178376    0.00688825     0.0990021    0.00366054     0.0124711       100    
reco:energyrecnumurange:EnergyReco                    0.000169708    0.106446       6.22678     0.00319871     0.633785        100    
reco:energyrecnumumcs:EnergyReco                      0.00172073     0.101859       5.57691     0.00382854     0.570535        100    
reco:opdec:Deconvolution                               0.0301809     0.476192       2.80245      0.252358      0.493494        100    
reco:ophitspe:OpHitFinderDeco                          0.224646       2.54059       6.01912       2.12592       1.31146        100    
reco:opflash:OpFlashFinder                            0.000377418   0.00175134    0.00800114     0.0014702    0.00100393       100    
reco:rns:RandomNumberSaver                            2.0999e-05    5.31951e-05   0.000570772   4.83455e-05   5.44346e-05      100    
reco:anglereconue:NuAngularReco                       0.000174728    0.0033162     0.0838998    0.000438849    0.0106239       100    
reco:anglereconumu:NuAngularReco                      8.7845e-05    0.00301282     0.0803954    0.000257404    0.010277        100    
reco:anglereconuepfps:NuAngularReco                   0.000139542   0.00320726     0.0804428    0.000422779    0.0103089       100    
reco:anglereconumupfps:NuAngularReco                  0.000124393   0.00309252     0.080141     0.000358182    0.0102702       100    
[art]:TriggerResults:TriggerResultInserter            1.0079e-05    1.84359e-05   9.5339e-05    1.7483e-05    9.42214e-06      100    
end_path:out1:RootOutput                               3.286e-06    7.10985e-06   3.0968e-05    7.0885e-06    3.77186e-06      100    
end_path:out1:RootOutput(write)                        0.0191464     0.323235       1.99931      0.285447      0.309979        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39589.3 MB
  Peak resident set size usage (VmHWM): 2407.42 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_543_20231207T145342Z_gen_g4_detsim_hitreco__20240510T062057Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_543_20231207T145342Z_gen_g4_detsim_hitreco__20240510T062057Z_reco2_reco_2024-11-23T_032824Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_543_20231207T145342Z_gen_g4_detsim_hitreco__20240510T062057Z_reco2_reco_data_2024-11-23T_032824Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work