Jobsub ID 301741.14@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301741.14@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:52:22
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 06:01:00
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:54:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_250_20231203T144453Z_gen_g4_detsim_hitreco__20240508T053745Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3999s)
	CPU time	54m (3291s = 82%)
Outputting started	2024-11-23 06:00:44
Output files
Finished	2024-11-23 06:01:00
List job events Wrapper job log
Jobscript log (last 10,000 characters)

113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:33 GMT run: 6419113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:33 GMT run: 6419113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:33 GMT run: 6419113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:33 GMT run: 6419113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:33 GMT run: 6419113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:33 GMT run: 6419113 subRun: 1 event: 25100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 263, 226, 211
minwire 0: 657
minwire 1: 1806
minwire 2: 277
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.66731, 0.00229873, 0.330391, 
Output 1: 0.323332, 0.647242, 0.0273351, 0.00209098, 
Output 2: 0.0804537, 0.919546, 

524
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1644!
23-Nov-2024 06:00:43 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_250_20231203T144453Z_gen_g4_detsim_hitreco__20240508T053745Z_reco2_reco_data_2024-11-23T_045409Z.root"
23-Nov-2024 06:00:43 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/7c/26/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_250_20231203T144453Z_gen_g4_detsim_hitreco__20240508T053745Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.14662       33.4174       576.865       11.9858       76.558         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0352472     0.0920317      1.29421      0.0786013     0.129543        100    
reco:linecluster:LineCluster                           0.203133      0.720363       12.7975      0.381857       1.67422        100    
reco:trajcluster:TrajCluster                            0.10185       1.15547       35.1935      0.121138       4.79503        100    
reco:pandora:StandardPandora                           0.114707       2.47682       42.4669       1.48916       5.11854        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000167496   0.00728544     0.140617     0.00176458     0.0182183       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00021233     0.0133435     0.121259     0.00365503     0.0243714       100    
reco:pandoracalo:Calorimetry                          0.000130897    0.0232724     0.854927     0.00123783     0.0985139       100    
reco:pandorapid:Chi2ParticleID                        5.0294e-05    0.00023087     0.0020088    0.000166724   0.000256203      100    
reco:trkshowersplit:TrackShowerHits                   8.8357e-05     0.0342583      0.94865     0.000918763     0.13641        100    
reco:pmtrack:PMAlgTrackMaker                          0.00117387      3.18372       79.6571      0.261084       10.4904        100    
reco:pmtrackcalo:Calorimetry                           7.384e-05     0.0128788     0.369854     0.00129489     0.0457327       100    
reco:pmtrackpid:Chi2ParticleID                        3.0658e-05    0.00036579    0.00406232    0.000190694   0.000578335      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000242567     2.40381       69.5856      0.393625       8.22644        100    
reco:pmtrajfitcalo:Calorimetry                        7.5913e-05     0.0240248      0.85654     0.00187572     0.0985646       100    
reco:pmtrajfitpid:Chi2ParticleID                       3.188e-05    0.000247841   0.00140585    0.000190955   0.000216941      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000996037     4.0782        134.892      0.257726       15.4366        100    
reco:pmtracktccalo:Calorimetry                        7.4671e-05    0.00812599     0.148979     0.00122251     0.0232548       100    
reco:pmtracktcpid:Chi2ParticleID                      3.0677e-05    0.000332545   0.00275643    0.000185565   0.000447151      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000202351     1.96562       43.7251      0.410365       5.29572        100    
reco:pmtrajfittccalo:Calorimetry                       7.966e-05     0.0196985     0.613015     0.00160737     0.0738937       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.1169e-05    0.000460753   0.00516743    0.000210978   0.000746259      100    
reco:blurredcluster:BlurredClustering                 9.9387e-05      1.03867       7.25413      0.367471       1.62068        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000140415     12.3738       173.492       3.51083       25.626         100    
reco:emshower:EMShower                                0.000242818    0.0388136     0.578513      0.0065969     0.0766781       100    
reco:cvnmap:CVNMapper                                 2.9876e-05     0.0121005     0.0582235    0.00934431     0.0124068       100    
reco:cvneva:CVNEvaluator                              1.9196e-05      0.58758       1.05104      0.791646      0.412713        100    
reco:energyrecnumu:EnergyReco                         0.000298062    0.0359369      0.73766     0.00489676     0.104734        100    
reco:energyrecnue:EnergyReco                          0.000155364   0.00239666     0.0470376    0.00059936    0.00607336       100    
reco:energyrecnc:EnergyReco                           0.00201084     0.0052127     0.0518986     0.0034968    0.00632513       100    
reco:energyrecnumurange:EnergyReco                    0.000140294    0.0336808     0.729127      0.0031045     0.103915        100    
reco:energyrecnumumcs:EnergyReco                      0.00162544     0.0355508     0.730709     0.00469034     0.103447        100    
reco:opdec:Deconvolution                               0.136311      0.513372       3.71131       0.4346       0.493551        100    
reco:ophitspe:OpHitFinderDeco                          0.304135       2.24758       4.22736       2.13268      0.939069        100    
reco:opflash:OpFlashFinder                            0.000143541   0.00159562    0.00637723    0.00140522    0.000891419      100    
reco:rns:RandomNumberSaver                            1.9567e-05    4.45975e-05   0.000402238   3.7916e-05    3.88142e-05      100    
reco:anglereconue:NuAngularReco                       0.000191742   0.00195633     0.0430653    0.000473628   0.00540104       100    
reco:anglereconumu:NuAngularReco                      8.7215e-05    0.00167834     0.040243     0.00031295    0.00507621       100    
reco:anglereconuepfps:NuAngularReco                   0.000152738   0.00184594     0.0409743    0.000455594   0.00514159       100    
reco:anglereconumupfps:NuAngularReco                  0.000136317    0.0017557     0.0399375    0.000378684   0.00508676       100    
[art]:TriggerResults:TriggerResultInserter             8.716e-06    1.56175e-05    5.296e-05    1.4818e-05    5.34602e-06      100    
end_path:out1:RootOutput                               3.206e-06    7.98033e-06   3.1049e-05     6.658e-06    4.99375e-06      100    
end_path:out1:RootOutput(write)                       0.00975294      0.26089       1.24137      0.258758      0.205776        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38424 MB
  Peak resident set size usage (VmHWM): 2357.78 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_250_20231203T144453Z_gen_g4_detsim_hitreco__20240508T053745Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_250_20231203T144453Z_gen_g4_detsim_hitreco__20240508T053745Z_reco2_reco_2024-11-23T_045409Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_250_20231203T144453Z_gen_g4_detsim_hitreco__20240508T053745Z_reco2_reco_data_2024-11-23T_045409Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work