Jobsub ID 108725.102@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108725.102@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:51:13
Site	UK_Brunel
Entry	CMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat	2024-11-23 04:54:59
From worker node	Hostname	wn-a2-26-01.brunel.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:00:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_296_20231201T105759Z_gen_g4_detsim_hitreco__20240507T203249Z_reco2.root
Jobscript	Exit code	0
	Real time	54m (3267s)
	CPU time	49m (2971s = 90%)
Outputting started	2024-11-23 04:54:45
Output files
Finished	2024-11-23 04:54:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

lure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:54:37 GMT run: 6370891 subRun: 1 event: 29700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:54:37 GMT run: 6370891 subRun: 1 event: 29700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:54:37 GMT run: 6370891 subRun: 1 event: 29700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:54:37 GMT run: 6370891 subRun: 1 event: 29700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:54:40 GMT run: 6370891 subRun: 1 event: 29700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 56, 57, 48
minwire 0: 2096
minwire 1: 701
minwire 2: 1789
Used alternate method to get min and max wires due to vertex determination failure: 205, 404
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 502, 701
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 186, 385
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.990564, 0.000338654, 0.00909686, 
Output 1: 0.557683, 0.37545, 0.0618261, 0.0050414, 
Output 2: 0.993699, 0.00630101, 

507
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1831!
23-Nov-2024 04:54:44 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_296_20231201T105759Z_gen_g4_detsim_hitreco__20240507T203249Z_reco2_reco_data_2024-11-23T_040020Z.root"
23-Nov-2024 04:54:44 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/4c/df/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_296_20231201T105759Z_gen_g4_detsim_hitreco__20240507T203249Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.75593       29.0134       621.068       10.5573       75.1408        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0223735     0.0463785     0.0878167     0.0490013     0.0137271       100    
reco:linecluster:LineCluster                           0.130426      0.505349       12.9522      0.236964       1.61612        100    
reco:trajcluster:TrajCluster                           0.0643375      1.13877       37.1143      0.0807452      5.35508        100    
reco:pandora:StandardPandora                           0.973776       2.48544       66.2515       1.37042       6.80677        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000137237   0.00703832     0.264065     0.00191774     0.0271753       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000188152    0.0110289     0.0919841    0.00374118     0.0189257       100    
reco:pandoracalo:Calorimetry                          0.000105172    0.031413        1.237      0.00142553     0.148557        100    
reco:pandorapid:Chi2ParticleID                        4.3639e-05    0.000152995   0.000829166   0.000130002   0.000122185      100    
reco:trkshowersplit:TrackShowerHits                   0.000106446    0.0199251     0.663033     0.00080139     0.0893363       100    
reco:pmtrack:PMAlgTrackMaker                          0.00127633      2.55227       76.045       0.247637       9.92245        100    
reco:pmtrackcalo:Calorimetry                          8.7522e-05     0.0139107     0.669779     0.00116727     0.0704778       100    
reco:pmtrackpid:Chi2ParticleID                        3.6994e-05    0.000282361   0.00437236     0.0001462    0.000524242      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000209511     2.36326       53.2822      0.361312       7.72624        100    
reco:pmtrajfitcalo:Calorimetry                        7.9989e-05     0.0316218      1.24224     0.00173729     0.148319        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6723e-05    0.000176157   0.00171553    0.000144385   0.000187407      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113238      2.82772       87.7424      0.430677       10.7355        100    
reco:pmtracktccalo:Calorimetry                        8.7267e-05    0.00935067     0.399628     0.00127117     0.0420726       100    
reco:pmtracktcpid:Chi2ParticleID                      3.7008e-05    0.000263936   0.00337592    0.000157166   0.000408395      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000192341     1.77791       50.4622      0.342071       5.72224        100    
reco:pmtrajfittccalo:Calorimetry                      8.3675e-05     0.0208586      1.01003     0.00138936     0.106986        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6678e-05    0.000347792   0.00546544    0.000152188   0.00067329       100    
reco:blurredcluster:BlurredClustering                 0.000359397    0.919231       5.92338       0.34337       1.18432        100    
reco:emtrkmichelid:EmTrackMichelId                    0.00108622      11.5418       217.791       3.80859       26.6113        100    
reco:emshower:EMShower                                0.000261806    0.0269464     0.533072     0.00265059     0.0609997       100    
reco:cvnmap:CVNMapper                                 2.6808e-05     0.0107843     0.0381564     0.0101229    0.00861443       100    
reco:cvneva:CVNEvaluator                              2.0432e-05      0.54948       1.46245      0.735062      0.350883        100    
reco:energyrecnumu:EnergyReco                         0.000293827    0.032867       1.18164     0.00404705     0.126185        100    
reco:energyrecnue:EnergyReco                          0.000169664   0.00229368     0.0748596    0.000670886   0.00778626       100    
reco:energyrecnc:EnergyReco                           0.00223233    0.00439605     0.0800469    0.00272374    0.00816323       100    
reco:energyrecnumurange:EnergyReco                    0.000174426    0.0312085      1.1679       0.0024508     0.124691        100    
reco:energyrecnumumcs:EnergyReco                      0.000951003    0.0329629      1.18386     0.00357739     0.126096        100    
reco:opdec:Deconvolution                               0.0841211     0.341269      0.576667      0.346767      0.0986826       100    
reco:ophitspe:OpHitFinderDeco                          0.301254       1.4324        2.6238        1.52313      0.510757        100    
reco:opflash:OpFlashFinder                            0.000296183   0.00146625    0.00459379    0.00146669    0.000666503      100    
reco:rns:RandomNumberSaver                             1.852e-05    3.07146e-05   0.000285917   2.62845e-05   2.72448e-05      100    
reco:anglereconue:NuAngularReco                       0.000147186   0.00170142     0.0694163    0.000400853   0.00711631       100    
reco:anglereconumu:NuAngularReco                      8.7666e-05     0.0015367     0.0684777    0.00024836    0.00702117       100    
reco:anglereconuepfps:NuAngularReco                   0.000130127   0.00166428     0.0685006    0.000400451   0.00701684       100    
reco:anglereconumupfps:NuAngularReco                  0.000125476   0.00161885     0.0685509    0.000317035   0.00702682       100    
[art]:TriggerResults:TriggerResultInserter            1.1012e-05    1.31416e-05   4.4992e-05    1.20685e-05   3.86816e-06      100    
end_path:out1:RootOutput                               2.689e-06    4.71512e-06   1.3617e-05     3.559e-06    2.33316e-06      100    
end_path:out1:RootOutput(write)                        0.0235251     0.234572       1.26696      0.246363      0.172599        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8278.51 MB
  Peak resident set size usage (VmHWM): 2190.59 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_296_20231201T105759Z_gen_g4_detsim_hitreco__20240507T203249Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_296_20231201T105759Z_gen_g4_detsim_hitreco__20240507T203249Z_reco2_reco_2024-11-23T_040020Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_296_20231201T105759Z_gen_g4_detsim_hitreco__20240507T203249Z_reco2_reco_data_2024-11-23T_040020Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work