Jobsub ID 301701.23@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301701.23@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:24:39
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 05:31:01
From worker node	Hostname	n244.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:26:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1108_20231203T065228Z_gen_g4_detsim_hitreco__20240508T042829Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3877s)
	CPU time	57m (3464s = 89%)
Outputting started	2024-11-23 05:30:46
Output files
Finished	2024-11-23 05:31:01
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:37 GMT run: 6407968 subRun: 1 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:38 GMT run: 6407968 subRun: 1 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:38 GMT run: 6407968 subRun: 1 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:38 GMT run: 6407968 subRun: 1 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:38 GMT run: 6407968 subRun: 1 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:30:41 GMT run: 6407968 subRun: 1 event: 110900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 90, 105, 77
minwire 0: 1750
minwire 1: 682
minwire 2: 1914
Used alternate method to get min and max wires due to vertex determination failure: 1199, 1398
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 792, 991
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.971252, 0.00673801, 0.0220095, 
Output 1: 0.918246, 0.0798643, 0.00165303, 0.000236362, 
Output 2: 0.781575, 0.218425, 

430
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1273!
23-Nov-2024 05:30:45 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1108_20231203T065228Z_gen_g4_detsim_hitreco__20240508T042829Z_reco2_reco_data_2024-11-23T_042612Z.root"
23-Nov-2024 05:30:45 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c8/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1108_20231203T065228Z_gen_g4_detsim_hitreco__20240508T042829Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.577856       34.6505       703.301       8.9706        89.7315        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179182     0.0426048     0.0731601     0.0422278     0.0141435       100    
reco:linecluster:LineCluster                           0.100849       1.19562       51.3914       0.18574       6.63293        100    
reco:trajcluster:TrajCluster                           0.0507944      1.47743       55.0469      0.0597645      7.40145        100    
reco:pandora:StandardPandora                           0.0610714      2.49947       38.1799       1.23959       5.36352        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000116771    0.0065642     0.0853119    0.00174181     0.014745        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00016408     0.0109399      0.0844      0.00271863     0.0187493       100    
reco:pandoracalo:Calorimetry                          9.6693e-05     0.0273883     0.634311     0.000930194    0.100811        100    
reco:pandorapid:Chi2ParticleID                        3.5086e-05    0.000173588   0.00182242    0.000135371   0.000198589      100    
reco:trkshowersplit:TrackShowerHits                   9.7424e-05     0.0357134      1.01398     0.000702062    0.142938        100    
reco:pmtrack:PMAlgTrackMaker                          0.00123423      3.90216       135.025      0.191069       16.1931        100    
reco:pmtrackcalo:Calorimetry                          8.0602e-05     0.0125954     0.396336     0.00090129     0.0463493       100    
reco:pmtrackpid:Chi2ParticleID                         3.17e-05     0.000340027   0.00429994    0.000149888   0.000583411      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166214     2.90315       99.8862      0.282732       11.3276        100    
reco:pmtrajfitcalo:Calorimetry                        5.8521e-05     0.0256512     0.569748     0.00136801     0.0873168       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.2853e-05    0.00020211    0.000866839   0.00016657    0.000163767      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113461      4.0264        91.6914      0.240176       13.2215        100    
reco:pmtracktccalo:Calorimetry                        7.2958e-05    0.00798154     0.154943     0.00109797     0.0228133       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9706e-05    0.000313439   0.00259139    0.000175061   0.000434222      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000181944     1.8629        42.2045      0.265893       5.35518        100    
reco:pmtrajfittccalo:Calorimetry                      8.5852e-05     0.021087      0.620323     0.00124239     0.0759559       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8874e-05    0.000426928   0.00485479    0.00017429    0.000739587      100    
reco:blurredcluster:BlurredClustering                 7.6494e-05      1.12235       10.2893      0.462393       1.76979        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000132861     12.2446       167.343       2.6829        25.8465        100    
reco:emshower:EMShower                                0.000211039    0.0340269     0.418332     0.00701575     0.0627396       100    
reco:cvnmap:CVNMapper                                 2.3024e-05    0.00938738     0.049945     0.00721095    0.00946188       100    
reco:cvneva:CVNEvaluator                               1.594e-05     0.610428       1.23068      0.837343      0.385269        100    
reco:energyrecnumu:EnergyReco                         0.000276633    0.0419446     0.811409     0.00523893     0.129862        100    
reco:energyrecnue:EnergyReco                          0.000128062   0.00229498     0.0388077    0.000566576   0.00527251       100    
reco:energyrecnc:EnergyReco                           0.00147462    0.00472089     0.0429389    0.00314567    0.00571185       100    
reco:energyrecnumurange:EnergyReco                    0.000131188    0.0391458     0.817143     0.00288089     0.128912        100    
reco:energyrecnumumcs:EnergyReco                       0.0017199     0.0410213     0.807052     0.00355027     0.129419        100    
reco:opdec:Deconvolution                               0.0860348     0.274498      0.560683      0.264967      0.0961561       100    
reco:ophitspe:OpHitFinderDeco                          0.151699       1.91494       4.50203       1.74141      0.950356        100    
reco:opflash:OpFlashFinder                            0.000129275   0.00143222     0.0071501    0.00126855    0.000838094      100    
reco:rns:RandomNumberSaver                            1.6732e-05    3.75247e-05   0.000359099   3.05375e-05   3.45955e-05      100    
reco:anglereconue:NuAngularReco                       0.000141648   0.00166613     0.0324149    0.000415596   0.00430692       100    
reco:anglereconumu:NuAngularReco                      7.5814e-05    0.00144281     0.0296538    0.000268748   0.00404668       100    
reco:anglereconuepfps:NuAngularReco                   0.000107573   0.00160228     0.0353347    0.00040159     0.0044648       100    
reco:anglereconumupfps:NuAngularReco                  8.9429e-05    0.00156682     0.0342236    0.000330826   0.00443052       100    
[art]:TriggerResults:TriggerResultInserter             8.886e-06    1.42721e-05   8.3508e-05    1.28745e-05   7.72968e-06      100    
end_path:out1:RootOutput                               2.625e-06    5.75062e-06   2.3525e-05    4.6385e-06    3.08845e-06      100    
end_path:out1:RootOutput(write)                       0.00961123     0.242234      0.908357      0.237225      0.183136        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38501.7 MB
  Peak resident set size usage (VmHWM): 2431.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1108_20231203T065228Z_gen_g4_detsim_hitreco__20240508T042829Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1108_20231203T065228Z_gen_g4_detsim_hitreco__20240508T042829Z_reco2_reco_2024-11-23T_042612Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1108_20231203T065228Z_gen_g4_detsim_hitreco__20240508T042829Z_reco2_reco_data_2024-11-23T_042612Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work