Jobsub ID 108861.37@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108861.37@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:33:18
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 07:30:42
From worker node	Hostname	comp05-21
	cpuinfo	Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:34:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_376_20231207T170202Z_gen_g4_detsim_hitreco__20240510T052228Z_reco2.root
Jobscript	Exit code	0
	Real time	56m (3380s)
	CPU time	51m (3075s = 90%)
Outputting started	2024-11-23 07:30:31
Output files
Finished	2024-11-23 07:30:42
List job events Wrapper job log
Jobscript log (last 10,000 characters)

nModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:30:26 GMT run: 50640579 subRun: 1 event: 37700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 807, 925, 880
minwire 0: 1409
minwire 1: 681
minwire 2: 1534
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00556924, 0.992773, 0.00165736, 
Output 1: 0.00521744, 0.0241335, 0.0550892, 0.91556, 
Output 2: 0.558999, 0.441001, 

743
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2715!
23-Nov-2024 07:30:30 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_376_20231207T170202Z_gen_g4_detsim_hitreco__20240510T052228Z_reco2_reco_data_2024-11-23T_063412Z.root"
23-Nov-2024 07:30:30 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/1e/4c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_376_20231207T170202Z_gen_g4_detsim_hitreco__20240510T052228Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.847035       31.1549       1695.41       7.72155       169.076        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0286075     0.0572851     0.170943      0.0623615     0.0205151       100    
reco:linecluster:LineCluster                           0.166447      0.621618        34.54       0.266938       3.40972        100    
reco:trajcluster:TrajCluster                           0.0809076      2.86709       258.479      0.0864991      25.7169        100    
reco:pandora:StandardPandora                           0.0918813      7.64713       639.042       1.12356       63.4625        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139696   0.00695793     0.403932     0.000938452    0.0405102       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174964   0.00964361     0.214085     0.00165929     0.0308646       100    
reco:pandoracalo:Calorimetry                          8.9959e-05     0.0310286      2.57737     0.000578127    0.256764        100    
reco:pandorapid:Chi2ParticleID                        3.4709e-05    0.00013449    0.00126019    0.000100257   0.00015772       100    
reco:trkshowersplit:TrackShowerHits                   7.5189e-05     0.0228262      1.7786      0.000406454     0.17795        100    
reco:pmtrack:PMAlgTrackMaker                          0.00121868      2.29369       158.979      0.0726966      15.8814        100    
reco:pmtrackcalo:Calorimetry                          7.5132e-05     0.0140713      1.08213     0.000500025    0.108047        100    
reco:pmtrackpid:Chi2ParticleID                        3.0366e-05    0.000196606   0.00464557     0.000102     0.000483546      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000175087     1.21319       48.8952      0.114987       5.37101        100    
reco:pmtrajfitcalo:Calorimetry                        5.7784e-05     0.0325056      2.6594      0.000722477    0.265068        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4154e-05    0.000142648   0.00107209    0.000109318   0.000135829      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102516      2.27795       151.961      0.0683802      15.2262        100    
reco:pmtracktccalo:Calorimetry                        6.9746e-05    0.00774978     0.477729     0.000577266    0.0480145       100    
reco:pmtracktcpid:Chi2ParticleID                      2.6982e-05    0.000184861   0.00287634    0.000114722   0.000322849      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000142675     1.12953       58.5379      0.0916775      5.93415        100    
reco:pmtrajfittccalo:Calorimetry                       5.57e-05      0.014715       1.06647     0.000628822    0.106704        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.2876e-05    0.000226141   0.00460787    0.00011296    0.000496751      100    
reco:blurredcluster:BlurredClustering                 5.3018e-05      1.05153       11.9942      0.424321       1.71395        100    
reco:emtrkmichelid:EmTrackMichelId                     8.867e-05      9.1444        288.043       2.10585       31.0069        100    
reco:emshower:EMShower                                0.000132706    0.0227256     0.565912     0.00485428     0.0636977       100    
reco:cvnmap:CVNMapper                                 1.5173e-05     0.0116415     0.0467582     0.011985      0.0101043       100    
reco:cvneva:CVNEvaluator                               1.36e-05      0.473271       1.11876      0.689744      0.350738        100    
reco:energyrecnumu:EnergyReco                         0.000257893    0.108385       9.64523     0.00397957     0.958865        100    
reco:energyrecnue:EnergyReco                          0.000133301   0.00232763      0.13035     0.000308012    0.0130828       100    
reco:energyrecnc:EnergyReco                           0.00170848    0.00397352     0.132205     0.00205971     0.0131119       100    
reco:energyrecnumurange:EnergyReco                    0.000162882    0.100817       9.11924     0.00180848     0.906744        100    
reco:energyrecnumumcs:EnergyReco                      0.00167361     0.101131       9.03296     0.00230684      0.89805        100    
reco:opdec:Deconvolution                               0.0850182     0.335184       0.60381      0.339643      0.0975119       100    
reco:ophitspe:OpHitFinderDeco                          0.242784       1.35407       2.62158       1.28884      0.524537        100    
reco:opflash:OpFlashFinder                            0.000126988   0.00115415    0.00325088    0.00107535    0.000530891      100    
reco:rns:RandomNumberSaver                            1.6224e-05    2.84281e-05   0.00019518    2.74615e-05   1.84349e-05      100    
reco:anglereconue:NuAngularReco                       0.000138575   0.00195392     0.119909     0.000264676    0.012008        100    
reco:anglereconumu:NuAngularReco                      6.5925e-05     0.0018351     0.122354     0.000123982    0.0122572       100    
reco:anglereconuepfps:NuAngularReco                   0.000113197   0.00193067     0.120997     0.000229218    0.0121196       100    
reco:anglereconumupfps:NuAngularReco                   9.253e-05    0.00189735     0.122743     0.000182748    0.0122947       100    
[art]:TriggerResults:TriggerResultInserter             8.645e-06    1.08776e-05   3.8055e-05    1.02165e-05   3.10665e-06      100    
end_path:out1:RootOutput                               2.434e-06    5.43719e-06   2.0283e-05     6.02e-06     2.77066e-06      100    
end_path:out1:RootOutput(write)                       0.00570759     0.187265       1.53863      0.195561      0.176883        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12973.8 MB
  Peak resident set size usage (VmHWM): 2589.75 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_376_20231207T170202Z_gen_g4_detsim_hitreco__20240510T052228Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_376_20231207T170202Z_gen_g4_detsim_hitreco__20240510T052228Z_reco2_reco_2024-11-23T_063412Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_376_20231207T170202Z_gen_g4_detsim_hitreco__20240510T052228Z_reco2_reco_data_2024-11-23T_063412Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work