Jobsub ID 108715.52@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108715.52@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:31:31
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:13:23
From worker node	Hostname	n257.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:39:03
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_281_20231201T105553Z_gen_g4_detsim_hitreco__20240507T193244Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5643s)
	CPU time	1h (5196s = 92%)
Outputting started	2024-11-23 05:13:06
Output files
Finished	2024-11-23 05:13:23
List job events Wrapper job log
Jobscript log (last 10,000 characters)

4481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:44 GMT run: 74481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:44 GMT run: 74481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:44 GMT run: 74481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:44 GMT run: 74481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:44 GMT run: 74481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:44 GMT run: 74481105 subRun: 1 event: 28200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 808, 609, 530
minwire 0: 894
minwire 1: 1170
minwire 2: 1003
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0302041, 0.00916696, 0.960629, 
Output 1: 0.355763, 0.471865, 0.131944, 0.0404283, 
Output 2: 0.220712, 0.779288, 

641
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2392!
23-Nov-2024 05:13:05 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_281_20231201T105553Z_gen_g4_detsim_hitreco__20240507T193244Z_reco2_reco_data_2024-11-23T_033907Z.root"
23-Nov-2024 05:13:05 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/27/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_281_20231201T105553Z_gen_g4_detsim_hitreco__20240507T193244Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.60483       51.8352       1807.15       9.88562       198.877        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0183087     0.0434169     0.0671176     0.0419978     0.0142191       100    
reco:linecluster:LineCluster                           0.177776      0.948412       28.725       0.188863       3.69983        100    
reco:trajcluster:TrajCluster                           0.0511451      1.4395        61.4069      0.0627048      6.85556        100    
reco:pandora:StandardPandora                            1.04464       11.5223       842.799       1.35714       84.0824        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000147881   0.00830438      0.20593     0.00184901     0.0245336       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000214066    0.016839      0.491737      0.0033973     0.0530948       100    
reco:pandoracalo:Calorimetry                          0.000115108    0.0576443      2.3265      0.00136668     0.274455        100    
reco:pandorapid:Chi2ParticleID                        4.6558e-05    0.000210227   0.00156208    0.000142035   0.000230829      100    
reco:trkshowersplit:TrackShowerHits                   0.000109157    0.0524181      2.10444     0.000978158    0.241698        100    
reco:pmtrack:PMAlgTrackMaker                          0.00136851      4.62771       140.676      0.210937       18.1128        100    
reco:pmtrackcalo:Calorimetry                          8.4761e-05     0.0213811     0.923603     0.000945907     0.10193        100    
reco:pmtrackpid:Chi2ParticleID                        3.5337e-05    0.000382517    0.0055963    0.000163655   0.00080964       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000196413     4.61035       156.628      0.359742       19.6362        100    
reco:pmtrajfitcalo:Calorimetry                        7.5574e-05     0.0622933      2.45327     0.00164052     0.292405        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.1069e-05    0.000215857   0.00126089    0.000158902   0.000196659      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00112714      5.79794       192.128      0.268272       24.0766        100    
reco:pmtracktccalo:Calorimetry                         7.948e-05     0.0109646      0.36681      0.0008184     0.0414926       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3294e-05    0.000327614   0.00430597    0.000168335   0.000575468      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000202284     3.30209       101.988      0.283243       12.4503        100    
reco:pmtrajfittccalo:Calorimetry                      8.0823e-05     0.0383223      1.51125      0.001315      0.177989        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7953e-05    0.000490163   0.00767466    0.000169442   0.00110122       100    
reco:blurredcluster:BlurredClustering                  0.0114451      1.21488       13.4593       0.47101       2.21455        100    
reco:emtrkmichelid:EmTrackMichelId                     0.321557       14.4217       249.678       3.06011       37.2676        100    
reco:emshower:EMShower                                0.000296904    0.0327849     0.196373     0.00695928     0.0498048       100    
reco:cvnmap:CVNMapper                                 2.5979e-05     0.0129508     0.0614064     0.0125857     0.0124823       100    
reco:cvneva:CVNEvaluator                              1.8324e-05      0.63183       1.19073      0.853138      0.419904        100    
reco:energyrecnumu:EnergyReco                         0.000462389    0.0643707      2.04642     0.00492586     0.254563        100    
reco:energyrecnue:EnergyReco                          0.000181064   0.00291523     0.0765667    0.000568921   0.00891878       100    
reco:energyrecnc:EnergyReco                             0.00248     0.00564614     0.080702     0.00335472    0.00933893       100    
reco:energyrecnumurange:EnergyReco                    0.000211973    0.0603556      2.02232     0.00287808      0.24982        100    
reco:energyrecnumumcs:EnergyReco                      0.000960695    0.0614165      2.01321     0.00361431     0.248148        100    
reco:opdec:Deconvolution                               0.0892518     0.297202      0.604946       0.29628      0.107226        100    
reco:ophitspe:OpHitFinderDeco                          0.335344       2.17749       4.99126       2.07777      0.981297        100    
reco:opflash:OpFlashFinder                            0.000465624   0.00170217     0.0050677    0.00157382    0.000803135      100    
reco:rns:RandomNumberSaver                             2.13e-05     5.28171e-05   0.000469843   4.05625e-05   6.50057e-05      100    
reco:anglereconue:NuAngularReco                       0.000176806   0.00213388     0.0620365    0.000455876   0.00707135       100    
reco:anglereconumu:NuAngularReco                      8.3128e-05    0.00185081     0.0603215    0.000233168    0.0069226       100    
reco:anglereconuepfps:NuAngularReco                   0.000146888   0.00206989     0.063078     0.00041066    0.00715845       100    
reco:anglereconumupfps:NuAngularReco                  0.000120549   0.00192509     0.060103     0.000292275   0.00690262       100    
[art]:TriggerResults:TriggerResultInserter            1.0079e-05    1.55798e-05   6.4712e-05    1.43925e-05   6.07219e-06      100    
end_path:out1:RootOutput                               2.956e-06    7.46848e-06   3.3133e-05    7.5095e-06    3.40153e-06      100    
end_path:out1:RootOutput(write)                        0.0212806     0.278121       1.71055      0.264712       0.25201        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38713.7 MB
  Peak resident set size usage (VmHWM): 2643.77 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_281_20231201T105553Z_gen_g4_detsim_hitreco__20240507T193244Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_281_20231201T105553Z_gen_g4_detsim_hitreco__20240507T193244Z_reco2_reco_2024-11-23T_033907Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_281_20231201T105553Z_gen_g4_detsim_hitreco__20240507T193244Z_reco2_reco_data_2024-11-23T_033907Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work