Jobsub ID 301680.72@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301680.72@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:42:17
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 04:36:04
From worker node	Hostname	n259.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:43:52
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_338_20231202T072912Z_gen_g4_detsim_hitreco__20240507T195355Z_reco2.root
Jobscript	Exit code	0
	Real time	51m (3115s)
	CPU time	45m (2732s = 87%)
Outputting started	2024-11-23 04:35:47
Output files
Finished	2024-11-23 04:36:04
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ule  23-Nov-2024 04:35:39 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:39 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:39 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:39 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:35:42 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:35:42 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:35:42 GMT run: 50546594 subRun: 1 event: 33900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 90, 122, 80
minwire 0: 2009
minwire 1: 237
minwire 2: 2161
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.329546, 0.566822, 0.103632, 
Output 1: 0.124455, 0.416382, 0.42325, 0.0359139, 
Output 2: 0.842007, 0.157993, 

440
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1348!
23-Nov-2024 04:35:46 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_338_20231202T072912Z_gen_g4_detsim_hitreco__20240507T195355Z_reco2_reco_data_2024-11-23T_034356Z.root"
23-Nov-2024 04:35:46 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c8/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_338_20231202T072912Z_gen_g4_detsim_hitreco__20240507T195355Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.29763       27.0271       358.202       10.4619       50.1978        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0176415     0.0292181     0.0449302     0.023129     0.00884932       100    
reco:linecluster:LineCluster                           0.173527      0.288907       2.90809      0.199472      0.326251        100    
reco:trajcluster:TrajCluster                           0.0503334     0.543534       17.789       0.0617706      2.06668        100    
reco:pandora:StandardPandora                           0.0633636      1.79027        19.78        1.28299       2.35374        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000149974   0.00702432     0.128872      0.0022343     0.0165084       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000208795    0.0154985     0.319097     0.00386181     0.0386295       100    
reco:pandoracalo:Calorimetry                          0.00012649     0.0142267     0.361134      0.0014666     0.0461727       100    
reco:pandorapid:Chi2ParticleID                        4.6207e-05    0.000232474   0.00341399    0.000174901   0.000341548      100    
reco:trkshowersplit:TrackShowerHits                   0.00011076     0.0212718     0.646022     0.000890288    0.076906        100    
reco:pmtrack:PMAlgTrackMaker                          0.00138694      2.34473       38.3426      0.236574       6.15616        100    
reco:pmtrackcalo:Calorimetry                          7.8038e-05     0.0104467     0.206836     0.00146492     0.0308211       100    
reco:pmtrackpid:Chi2ParticleID                        2.9836e-05    0.000336904   0.00257848    0.000203611   0.000441334      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000193467     2.07378       56.2977      0.372104       6.76031        100    
reco:pmtrajfitcalo:Calorimetry                        7.2247e-05     0.0154418      0.39394     0.00207131     0.0489851       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9707e-05    0.000260601   0.000992441   0.000210078   0.000188054      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00110127      2.84026       69.1789      0.354772       8.67606        100    
reco:pmtracktccalo:Calorimetry                        7.4551e-05    0.00808602     0.134161     0.00132293     0.0216776       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2251e-05    0.000327404   0.00216567    0.000207248   0.000366774      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000188216     1.52222       21.7129      0.401722       3.28145        100    
reco:pmtrajfittccalo:Calorimetry                      7.6005e-05     0.0126336     0.311446     0.00156134     0.0400516       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.9696e-05    0.000397676   0.00311224    0.000221139   0.000525905      100    
reco:blurredcluster:BlurredClustering                 8.3288e-05      1.17889       7.4546       0.594597       1.47284        100    
reco:emtrkmichelid:EmTrackMichelId                     0.031853       10.8379       118.947       3.18447       19.552         100    
reco:emshower:EMShower                                0.000323323    0.0356952     0.272903     0.00693855     0.0555011       100    
reco:cvnmap:CVNMapper                                 2.5218e-05     0.0147685     0.0432141     0.0173219    0.00908402       100    
reco:cvneva:CVNEvaluator                              1.9367e-05     0.677013       1.03289      0.832146      0.341435        100    
reco:energyrecnumu:EnergyReco                         0.000397493    0.0342011      0.55246     0.00727538     0.0840018       100    
reco:energyrecnue:EnergyReco                          0.000175603   0.00250515     0.0438166    0.000674279   0.00586655       100    
reco:energyrecnc:EnergyReco                           0.00255717    0.00513363     0.0473055    0.00331241    0.00606409       100    
reco:energyrecnumurange:EnergyReco                    0.00019002     0.0284817     0.521119     0.00309905     0.0713519       100    
reco:energyrecnumumcs:EnergyReco                      0.00211157     0.0297477     0.521643      0.0047428     0.0695345       100    
reco:opdec:Deconvolution                               0.0559139     0.307437      0.568195      0.316165      0.101292        100    
reco:ophitspe:OpHitFinderDeco                          0.198115       2.05918       3.6199        2.19639      0.788451        100    
reco:opflash:OpFlashFinder                            0.000226169   0.00179891    0.00781698     0.0017377    0.000869573      100    
reco:rns:RandomNumberSaver                             2.629e-05    5.47892e-05   0.000429394   5.1122e-05    3.90837e-05      100    
reco:anglereconue:NuAngularReco                       0.000188107   0.00195368     0.0395192    0.000599211   0.00489977       100    
reco:anglereconumu:NuAngularReco                      9.4679e-05    0.00168465     0.0372816    0.000356781   0.00483148       100    
reco:anglereconuepfps:NuAngularReco                   0.000147319   0.00185624     0.0424672    0.000524745   0.00509361       100    
reco:anglereconumupfps:NuAngularReco                  0.00012636    0.00174598     0.0390418    0.000431488   0.00484063       100    
[art]:TriggerResults:TriggerResultInserter            1.1291e-05    1.64929e-05   8.5722e-05    1.4828e-05    8.38629e-06      100    
end_path:out1:RootOutput                               3.667e-06    8.05745e-06   3.2031e-05    7.8545e-06    3.31067e-06      100    
end_path:out1:RootOutput(write)                        0.0126371     0.265516       0.84681      0.265936      0.155767        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38253.9 MB
  Peak resident set size usage (VmHWM): 2187.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_338_20231202T072912Z_gen_g4_detsim_hitreco__20240507T195355Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_338_20231202T072912Z_gen_g4_detsim_hitreco__20240507T195355Z_reco2_reco_2024-11-23T_034356Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_338_20231202T072912Z_gen_g4_detsim_hitreco__20240507T195355Z_reco2_reco_data_2024-11-23T_034356Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work