Jobsub ID 301666.75@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301666.75@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:09:01
Site	UK_Brunel
Entry	CMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat	2024-11-23 04:17:15
From worker node	Hostname	wn-a2-26-01.brunel.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:12:09
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_839_20231204T083121Z_gen_g4_detsim_hitreco__20240509T194559Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3892s)
	CPU time	58m (3532s = 90%)
Outputting started	2024-11-23 04:17:01
Output files
Finished	2024-11-23 04:17:15
List job events Wrapper job log
Jobscript log (last 10,000 characters)

000
1st wire C:0 T:12 P:2 W:771 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:50 GMT run: 6422672 subRun: 1 event: 84000
1st wire C:0 T:12 P:2 W:736 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:50 GMT run: 6422672 subRun: 1 event: 84000
1st wire C:0 T:12 P:2 W:517 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:50 GMT run: 6422672 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:50 GMT run: 6422672 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:16:56 GMT run: 6422672 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:16:56 GMT run: 6422672 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 213, 248, 193
minwire 0: 1690
minwire 1: 1499
minwire 2: 1496
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00187113, 0.997733, 0.000396219, 
Output 1: 0.0263225, 0.393085, 0.562821, 0.0177715, 
Output 2: 0.997863, 0.0021367, 

630
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2392!
23-Nov-2024 04:17:00 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_839_20231204T083121Z_gen_g4_detsim_hitreco__20240509T194559Z_reco2_reco_data_2024-11-23T_031212Z.root"
23-Nov-2024 04:17:00 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/07/4c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_839_20231204T083121Z_gen_g4_detsim_hitreco__20240509T194559Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.867441       34.6601       613.467       9.4408        88.7457        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0218106     0.0621907     0.592605      0.0554745     0.0596771       100    
reco:linecluster:LineCluster                           0.126916      0.593724       18.3363      0.245681       1.93257        100    
reco:trajcluster:TrajCluster                           0.0635812      2.70199       126.95       0.0750379      14.3798        100    
reco:pandora:StandardPandora                           0.0746345      2.33614       30.8303       1.37023       4.35121        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000144755   0.00716458     0.168805     0.00143734     0.0213303       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00019878     0.0220595     0.537466     0.00282267     0.0777261       100    
reco:pandoracalo:Calorimetry                          0.000109876    0.0347012      1.04401     0.00101886     0.137652        100    
reco:pandorapid:Chi2ParticleID                        4.3423e-05    0.000158311   0.00130749    0.000122111   0.000152116      100    
reco:trkshowersplit:TrackShowerHits                   7.3898e-05     0.0319882     0.910957     0.000693226    0.136839        100    
reco:pmtrack:PMAlgTrackMaker                          0.00130801      3.2613        80.5964      0.162084       11.2069        100    
reco:pmtrackcalo:Calorimetry                          8.8758e-05     0.0186025     0.728485     0.000783205    0.0822694       100    
reco:pmtrackpid:Chi2ParticleID                         3.557e-05    0.000304408   0.00380061    0.00013894    0.00056919       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000226596     2.69759       49.4883      0.255196       7.22975        100    
reco:pmtrajfitcalo:Calorimetry                        8.3311e-05     0.0355203      1.06416     0.00132515     0.141001        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.637e-05    0.000183612   0.00094609    0.000144346   0.000147269      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00114521      4.53421       117.04       0.258115       16.1566        100    
reco:pmtracktccalo:Calorimetry                        8.5902e-05     0.0130186     0.576328     0.00081979     0.0603729       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5872e-05    0.000278076   0.00295449    0.000142102   0.000419165      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.00019607      1.88095       37.3993      0.284393       4.99596        100    
reco:pmtrajfittccalo:Calorimetry                       8.348e-05     0.0240695     0.779284     0.00103694     0.0970656       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5233e-05    0.000371615   0.00429251    0.000157217   0.000659687      100    
reco:blurredcluster:BlurredClustering                 8.2167e-05      1.01033       7.85618       0.42822       1.45286        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000120741     12.6592       190.381       3.01975       28.668         100    
reco:emshower:EMShower                                0.000179686    0.0357972     0.474953     0.00699765     0.0706179       100    
reco:cvnmap:CVNMapper                                 2.3442e-05     0.012913      0.0453894     0.0107388     0.0105409       100    
reco:cvneva:CVNEvaluator                              1.7808e-05     0.565418      0.990918      0.745231      0.346841        100    
reco:energyrecnumu:EnergyReco                         0.000306965    0.045695       1.08638      0.003798      0.157593        100    
reco:energyrecnue:EnergyReco                          0.000178236   0.00251945     0.0544687    0.000562707   0.00713262       100    
reco:energyrecnc:EnergyReco                           0.00228421    0.00466969     0.0582553    0.00267606    0.00749343       100    
reco:energyrecnumurange:EnergyReco                    0.000174015    0.0436167      1.09477      0.0024516     0.157018        100    
reco:energyrecnumumcs:EnergyReco                      0.000700522    0.045493       1.11229     0.00341744     0.158593        100    
reco:opdec:Deconvolution                               0.0882303     0.346766       1.30922      0.352494      0.149508        100    
reco:ophitspe:OpHitFinderDeco                          0.193547       1.37888       2.81996       1.40565      0.558583        100    
reco:opflash:OpFlashFinder                            0.000120885   0.00136691    0.00339854    0.00133633    0.000634531      100    
reco:rns:RandomNumberSaver                            1.8276e-05    2.83192e-05   0.000271057   2.3992e-05    2.51894e-05      100    
reco:anglereconue:NuAngularReco                       0.000147099   0.00181172     0.0472947    0.000328608   0.00600847       100    
reco:anglereconumu:NuAngularReco                      8.8472e-05    0.00169332     0.0469914    0.000248731   0.00598373       100    
reco:anglereconuepfps:NuAngularReco                   0.000135937    0.0018256     0.047261      0.0003565    0.00601541       100    
reco:anglereconumupfps:NuAngularReco                  0.000126605   0.00177351     0.047298     0.00032351    0.00599946       100    
[art]:TriggerResults:TriggerResultInserter            1.0281e-05    1.27882e-05   5.4985e-05    1.16355e-05   4.92304e-06      100    
end_path:out1:RootOutput                               2.421e-06    4.22077e-06   1.5246e-05     3.153e-06    2.21457e-06      100    
end_path:out1:RootOutput(write)                       0.00938537     0.242226       1.25077      0.233016      0.189855        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8433.51 MB
  Peak resident set size usage (VmHWM): 2340.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_839_20231204T083121Z_gen_g4_detsim_hitreco__20240509T194559Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_839_20231204T083121Z_gen_g4_detsim_hitreco__20240509T194559Z_reco2_reco_2024-11-23T_031212Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_839_20231204T083121Z_gen_g4_detsim_hitreco__20240509T194559Z_reco2_reco_data_2024-11-23T_031212Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work