Jobsub ID 108719.136@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108719.136@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:36:55
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 04:51:04
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:43:00
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_527_20231208T100319Z_gen_g4_detsim_hitreco__20240510T065019Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4068s)
	CPU time	59m (3584s = 88%)
Outputting started	2024-11-23 04:50:49
Output files
Finished	2024-11-23 04:51:04
List job events Wrapper job log
Jobscript log (last 10,000 characters)

-----------------------
Found: 2212 with KE=0.0257631 GeV, E=0.964035 GeV, p2=0.0490093
NCQEL_P (Nuance 1002, Nu 1, CR 0)
IsCorrectNu (NNuMatches: 1) 
Parent Neutrino: -14
Neutrino Energy: 0.267749
Neutrino Momentum:   x: 0.0693107  y: 0.257488  z: 0.0241991 length: 0.267749
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.964035, Dist. 0.69865, nMCHits 23 (14, 7, 2)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 7 (4, 3, 0), nPfoHits 7 (4, 3, 0)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.0257631 GeV, E=0.964035 GeV, p2=0.0490093
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:50:45 GMT run: 6472938 subRun: 1 event: 52800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 04:50:45 GMT run: 6472938 subRun: 1 event: 52800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 04:50:45 GMT run: 6472938 subRun: 1 event: 52800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:50:45 GMT run: 6472938 subRun: 1 event: 52800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:50:45 GMT run: 6472938 subRun: 1 event: 52800
Comparing two wires in the same plane: return failure
%MSG
130
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 236!
23-Nov-2024 04:50:47 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_527_20231208T100319Z_gen_g4_detsim_hitreco__20240510T065019Z_reco2_reco_data_2024-11-23T_034305Z.root"
23-Nov-2024 04:50:47 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/82/ca/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_527_20231208T100319Z_gen_g4_detsim_hitreco__20240510T065019Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.53157       35.7396       824.959       11.0455       94.5011        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179244     0.0394479     0.069007      0.0380386     0.0132017       100    
reco:linecluster:LineCluster                           0.163136       0.80239       49.843        0.19892       4.9438         100    
reco:trajcluster:TrajCluster                            0.05136       1.41765       92.3748      0.0658265      9.29266        100    
reco:pandora:StandardPandora                            0.87796       2.59672       48.9984       1.27296       6.08845        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000142278   0.00772888     0.127189     0.00195486     0.0186687       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000200478    0.0187988     0.410228     0.00446403     0.048428        100    
reco:pandoracalo:Calorimetry                          0.000120027    0.0294002      1.20466      0.0017971     0.129854        100    
reco:pandorapid:Chi2ParticleID                        4.1087e-05    0.000222173   0.00240549    0.000169279   0.000261795      100    
reco:trkshowersplit:TrackShowerHits                   0.00011707     0.0351871      1.46784     0.00110286     0.161105        100    
reco:pmtrack:PMAlgTrackMaker                           0.0010462      4.03539       158.406      0.289872        17.19         100    
reco:pmtrackcalo:Calorimetry                           7.463e-05     0.0149161     0.505393     0.00133142     0.0607236       100    
reco:pmtrackpid:Chi2ParticleID                        2.9997e-05    0.000385002   0.00444165    0.000182574   0.000639325      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000186291     3.32261       136.725      0.402803       14.3572        100    
reco:pmtrajfitcalo:Calorimetry                         5.84e-05      0.0298218      1.17569     0.00237617     0.127381        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4185e-05    0.000255235   0.000948617   0.00021769    0.000194979      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000762888     4.06971       115.784      0.267793       14.2773        100    
reco:pmtracktccalo:Calorimetry                        6.6806e-05    0.00775229     0.189337     0.00131933     0.0237046       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4747e-05    0.000328204   0.00275277    0.000176347   0.000417767      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000147027     1.88551       38.355       0.324368       5.01449        100    
reco:pmtrajfittccalo:Calorimetry                      5.4172e-05     0.0176482     0.487628     0.00155186     0.060883        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.2413e-05    0.000445791    0.0045086    0.00021756    0.000653012      100    
reco:blurredcluster:BlurredClustering                 0.00252897      1.05642       7.14256       0.48532       1.3841         100    
reco:emtrkmichelid:EmTrackMichelId                     0.232121       12.6297        165.1        3.55523       25.1854        100    
reco:emshower:EMShower                                0.000257906    0.0312971     0.233958      0.0061155     0.0499999       100    
reco:cvnmap:CVNMapper                                 3.5838e-05     0.0114428     0.0400669     0.0111919    0.00893252       100    
reco:cvneva:CVNEvaluator                              1.9898e-05     0.686379       1.20504      0.856913      0.378305        100    
reco:energyrecnumu:EnergyReco                         0.000439208    0.0365801     0.855913     0.00559022     0.121127        100    
reco:energyrecnue:EnergyReco                          0.000136528   0.00215301     0.0319533    0.000624657   0.00483652       100    
reco:energyrecnc:EnergyReco                           0.00144951    0.00468604     0.0356569    0.00315143     0.0049895       100    
reco:energyrecnumurange:EnergyReco                    0.000193575     0.03396      0.860993     0.00293087      0.1207         100    
reco:energyrecnumumcs:EnergyReco                      0.00140686     0.0355504     0.873501      0.0037484     0.119599        100    
reco:opdec:Deconvolution                               0.0890605      0.31133      0.499157      0.318626      0.0958484       100    
reco:ophitspe:OpHitFinderDeco                          0.343327       2.28445       4.1847        2.31058      0.911414        100    
reco:opflash:OpFlashFinder                            0.000344469   0.00181734    0.00932326    0.00170574    0.00102319       100    
reco:rns:RandomNumberSaver                             2.124e-05    5.60068e-05   0.000835004   5.06505e-05   7.97926e-05      100    
reco:anglereconue:NuAngularReco                       0.000158939   0.00179789     0.0320807    0.000587448   0.00408763       100    
reco:anglereconumu:NuAngularReco                      9.2193e-05    0.00152045     0.0317621    0.000338017   0.00404737       100    
reco:anglereconuepfps:NuAngularReco                   0.000120117   0.00170912     0.0318133    0.000484258   0.00408244       100    
reco:anglereconumupfps:NuAngularReco                  9.3426e-05    0.00159619     0.0316034    0.000429549   0.00405721       100    
[art]:TriggerResults:TriggerResultInserter             9.248e-06    1.59489e-05   9.0912e-05    1.45075e-05   8.0551e-06       100    
end_path:out1:RootOutput                               3.256e-06    7.67423e-06   3.3423e-05     7.679e-06    3.99913e-06      100    
end_path:out1:RootOutput(write)                        0.0205126     0.272531       1.09586       0.26744      0.174087        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38438.5 MB
  Peak resident set size usage (VmHWM): 2383.91 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_527_20231208T100319Z_gen_g4_detsim_hitreco__20240510T065019Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_527_20231208T100319Z_gen_g4_detsim_hitreco__20240510T065019Z_reco2_reco_2024-11-23T_034305Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_527_20231208T100319Z_gen_g4_detsim_hitreco__20240510T065019Z_reco2_reco_data_2024-11-23T_034305Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work