Jobsub ID 108779.46@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.46@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:08:27
From worker node	Hostname	comp21-02
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:04:52
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_232_20231202T180405Z_gen_g4_detsim_hitreco__20240508T063333Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3800s)
	CPU time	56m (3416s = 89%)
Outputting started	2024-11-23 06:08:12
Output files
Finished	2024-11-23 06:08:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:58 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:58 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:58 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:58 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:59 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:59 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:59 GMT run: 74506302 subRun: 1 event: 23300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 221, 302, 259
minwire 0: 374
minwire 1: 1405
minwire 2: 277
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.115422, 0.0017988, 0.882779, 
Output 1: 0.0046842, 0.0592937, 0.369495, 0.566527, 
Output 2: 0.801365, 0.198635, 

578
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1841!
23-Nov-2024 06:08:10 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_232_20231202T180405Z_gen_g4_detsim_hitreco__20240508T063333Z_reco2_reco_data_2024-11-23T_050455Z.root"
23-Nov-2024 06:08:10 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/18/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_232_20231202T180405Z_gen_g4_detsim_hitreco__20240508T063333Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.392029       34.1611       735.987       8.55972       90.8845        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0105692     0.0440785     0.348463      0.0259304     0.0552354       100    
reco:linecluster:LineCluster                           0.0641277     0.672249       25.9108      0.142571       2.83628        100    
reco:trajcluster:TrajCluster                           0.0290636      1.33145       53.0368      0.0454654      6.1344         100    
reco:pandora:StandardPandora                           0.0485002      3.24149       112.94        1.09506       11.9248        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00013026    0.00822603     0.211598     0.00130287     0.0247172       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000170829    0.0101642     0.125086     0.00217687     0.0198351       100    
reco:pandoracalo:Calorimetry                          9.4636e-05     0.0327967     0.945938     0.000755707    0.131843        100    
reco:pandorapid:Chi2ParticleID                        3.5065e-05    0.000146397   0.00119795    0.000104944   0.000149217      100    
reco:trkshowersplit:TrackShowerHits                   8.9259e-05     0.0283273      1.29807     0.000566076    0.139482        100    
reco:pmtrack:PMAlgTrackMaker                          0.00117057      2.76976       90.8472      0.108239       10.7819        100    
reco:pmtrackcalo:Calorimetry                          7.7419e-05     0.0140896     0.438496     0.000576702    0.0527549       100    
reco:pmtrackpid:Chi2ParticleID                        3.0071e-05    0.000257586   0.00295614    0.000107977   0.000439332      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000189243     2.26939       66.593       0.159416       8.20735        100    
reco:pmtrajfitcalo:Calorimetry                        6.3103e-05     0.0337649     0.979928     0.00088485     0.136153        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6398e-05    0.000168111   0.000918069   0.000113663   0.000164296      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102412      3.31984       87.6397      0.124926       11.7956        100    
reco:pmtracktccalo:Calorimetry                        6.9332e-05    0.00915238     0.271625     0.000604631     0.03137        100    
reco:pmtracktcpid:Chi2ParticleID                      2.6989e-05    0.000240055   0.00243175    0.000107271   0.000358454      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000163135     1.93337       55.2386      0.224965       6.50405        100    
reco:pmtrajfittccalo:Calorimetry                      5.8523e-05     0.0262748     0.941148     0.000774036    0.108018        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5598e-05    0.000345339   0.00437791    0.00012388    0.00064119       100    
reco:blurredcluster:BlurredClustering                 6.3341e-05      1.06033       8.99547      0.587645       1.50011        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000122609     14.1563       220.826       2.68931       32.3686        100    
reco:emshower:EMShower                                0.000161814    0.0272347     0.307852     0.00376431     0.0528744       100    
reco:cvnmap:CVNMapper                                 1.9842e-05     0.0118674     0.0415465     0.0123897    0.00970966       100    
reco:cvneva:CVNEvaluator                               1.478e-05     0.519017      0.828285      0.726583      0.340698        100    
reco:energyrecnumu:EnergyReco                         0.000254494    0.0353616     0.890345     0.00390083     0.130632        100    
reco:energyrecnue:EnergyReco                          0.000128281   0.00262536     0.0646084    0.000524448   0.00763622       100    
reco:energyrecnc:EnergyReco                           0.00168068    0.00422307     0.0662186    0.00209328    0.00769569       100    
reco:energyrecnumurange:EnergyReco                    0.000184723    0.0334168     0.864626     0.00186371     0.130118        100    
reco:energyrecnumumcs:EnergyReco                       0.0016585     0.0343924     0.866353     0.00270084     0.129963        100    
reco:opdec:Deconvolution                               0.0370858     0.289148       1.41351      0.264708      0.191965        100    
reco:ophitspe:OpHitFinderDeco                          0.0844187      2.0285        3.76861       2.11297      0.872017        100    
reco:opflash:OpFlashFinder                            0.000149791   0.00133106     0.0144126    0.00118176    0.00142688       100    
reco:rns:RandomNumberSaver                            1.7859e-05    3.0519e-05    0.000269526   2.4466e-05    2.73505e-05      100    
reco:anglereconue:NuAngularReco                       0.000137624   0.00213934     0.0594972    0.000323287   0.00682523       100    
reco:anglereconumu:NuAngularReco                      6.4876e-05    0.00192263     0.0572586    0.000191595    0.0065864       100    
reco:anglereconuepfps:NuAngularReco                   0.000107869   0.00202314     0.0570379    0.000287983   0.00656391       100    
reco:anglereconumupfps:NuAngularReco                  9.1103e-05    0.00196736     0.0569968    0.000248387   0.00657476       100    
[art]:TriggerResults:TriggerResultInserter             8.548e-06    1.14424e-05   3.8108e-05    1.02465e-05   4.1409e-06       100    
end_path:out1:RootOutput                                2.5e-06     4.90664e-06   1.9992e-05     3.562e-06    2.85342e-06      100    
end_path:out1:RootOutput(write)                       0.00591395      0.20205       1.07731      0.196242      0.167683        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9822.89 MB
  Peak resident set size usage (VmHWM): 2667.11 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_232_20231202T180405Z_gen_g4_detsim_hitreco__20240508T063333Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_232_20231202T180405Z_gen_g4_detsim_hitreco__20240508T063333Z_reco2_reco_2024-11-23T_050455Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_232_20231202T180405Z_gen_g4_detsim_hitreco__20240508T063333Z_reco2_reco_data_2024-11-23T_050455Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work