Jobsub ID 301751.90@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301751.90@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:14:46
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:35:19
From worker node	Hostname	comp02-24
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:16:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_657_20231208T093703Z_gen_g4_detsim_hitreco__20240510T064330Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4734s)
	CPU time	1h (4532s = 95%)
Outputting started	2024-11-23 06:35:06
Output files
Finished	2024-11-23 06:35:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

z: -0.0228245 length: 0.133692
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.962818, Dist. 0.651006, nMCHits 17 (9, 6, 2)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 11, nMatchedHits 6 (3, 3, 0), nPfoHits 6 (3, 3, 0)

------------------------------------------------------------------------------------------------

---INTERPRETED-MATCHING-OUTPUT------------------(my version)-----------------------------------------
Found: 2212 with KE=0.0245462 GeV, E=0.962818 GeV, p2=0.0466646
NCQEL_P (Nuance 1002, Nu 1, CR 0)
IsCorrectNu (NNuMatches: 1) 
Parent Neutrino: -16
Neutrino Energy: 0.133691
Neutrino Momentum:   x: 0.109089  y: -0.0738392  z: -0.0228245 length: 0.133692
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.962818, Dist. 0.651006, nMCHits 17 (9, 6, 2)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 11, nMatchedHits 6 (3, 3, 0), nPfoHits 6 (3, 3, 0)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.0245462 GeV, E=0.962818 GeV, p2=0.0466646
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:35:03 GMT run: 6472782 subRun: 1 event: 65800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:03 GMT run: 6472782 subRun: 1 event: 65800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:35:03 GMT run: 6472782 subRun: 1 event: 65800
Comparing two wires in the same plane: return failure
%MSG
207
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 404!
23-Nov-2024 06:35:04 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_657_20231208T093703Z_gen_g4_detsim_hitreco__20240510T064330Z_reco2_reco_data_2024-11-23T_051614Z.root"
23-Nov-2024 06:35:04 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/59/83/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_657_20231208T093703Z_gen_g4_detsim_hitreco__20240510T064330Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.12542       45.0892       1136.09       8.4645        145.356        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0105418     0.0510039      0.68505      0.0260761     0.0861934       100    
reco:linecluster:LineCluster                           0.0553902      1.33361       61.2938      0.123672       6.86745        100    
reco:trajcluster:TrajCluster                           0.0277778      3.70596       150.266      0.0366476      19.1104        100    
reco:pandora:StandardPandora                           0.463008       4.90823       166.491       1.45781       18.4808        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130634    0.0106662     0.242383     0.00122064     0.0340322       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174048    0.0143631     0.376862      0.0023657     0.0431902       100    
reco:pandoracalo:Calorimetry                          9.8599e-05     0.0609865      2.47345     0.000777224    0.286091        100    
reco:pandorapid:Chi2ParticleID                        4.1026e-05    0.00016729    0.00103368    0.000114524   0.000169097      100    
reco:trkshowersplit:TrackShowerHits                   8.8786e-05     0.0636661      2.20778     0.000538342    0.303257        100    
reco:pmtrack:PMAlgTrackMaker                          0.00120874      3.85947       114.575      0.167474       14.8634        100    
reco:pmtrackcalo:Calorimetry                           8.37e-05      0.0265394      1.05021     0.000694996    0.120924        100    
reco:pmtrackpid:Chi2ParticleID                        3.3402e-05    0.000325399   0.00445488     0.0001312    0.00066901       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000206442     4.47771       146.796      0.217319       18.5438        100    
reco:pmtrajfitcalo:Calorimetry                        7.4064e-05     0.0605417      2.2836      0.000872597     0.27301        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2718e-05    0.000185344   0.00113847    0.000124633   0.000198249      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00110564      4.62598       130.262      0.156663       17.7917        100    
reco:pmtracktccalo:Calorimetry                        8.0261e-05     0.0155368     0.411109     0.000703876    0.0572198       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3708e-05    0.000303982   0.00329963    0.000128685   0.000558942      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000192213     2.62572       69.6936      0.189806       9.19189        100    
reco:pmtrajfittccalo:Calorimetry                      7.6062e-05     0.0453169      1.61697     0.00100327     0.198249        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.1907e-05    0.000450741    0.0065356    0.00016044    0.000987295      100    
reco:blurredcluster:BlurredClustering                 7.6119e-05      1.19721       11.7482      0.435943       1.9866         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000152225     15.3609       267.654       2.53856       39.9382        100    
reco:emshower:EMShower                                0.000172817    0.0279514     0.346932     0.00261234     0.0667216       100    
reco:cvnmap:CVNMapper                                 2.0828e-05     0.011471      0.0440491     0.0109773     0.0103785       100    
reco:cvneva:CVNEvaluator                              1.7259e-05     0.542508       1.04989      0.716562      0.379993        100    
reco:energyrecnumu:EnergyReco                         0.000275939    0.0337113     0.760441     0.00407268     0.0984274       100    
reco:energyrecnue:EnergyReco                           0.0001588    0.00362593     0.0702379    0.000515092    0.010561        100    
reco:energyrecnc:EnergyReco                            0.002082     0.00578483     0.0762917    0.00273835     0.0110292       100    
reco:energyrecnumurange:EnergyReco                    0.000156678    0.0319711     0.763728     0.00253812     0.0979571       100    
reco:energyrecnumumcs:EnergyReco                      0.00127042     0.0332214     0.754051      0.0030343     0.0969893       100    
reco:opdec:Deconvolution                               0.085397      0.315289       2.36575      0.244953      0.288059        100    
reco:ophitspe:OpHitFinderDeco                          0.268554       1.3772        3.25156       1.30007      0.611273        100    
reco:opflash:OpFlashFinder                            0.000135587   0.00125104    0.00379596    0.00116832    0.000655896      100    
reco:rns:RandomNumberSaver                            1.5668e-05    2.63254e-05   0.000249998   2.2065e-05    2.36592e-05      100    
reco:anglereconue:NuAngularReco                       0.000101772   0.00291967     0.0608013    0.000271193    0.0094689       100    
reco:anglereconumu:NuAngularReco                      7.9626e-05    0.00279763     0.0607266    0.000186995   0.00955431       100    
reco:anglereconuepfps:NuAngularReco                   0.000117903   0.00285249     0.0604152    0.000286948   0.00928428       100    
reco:anglereconumupfps:NuAngularReco                  0.000115059   0.00277871     0.0600588    0.000262844   0.00912106       100    
[art]:TriggerResults:TriggerResultInserter             9.31e-06     1.26503e-05   4.3373e-05    1.11395e-05   5.9378e-06       100    
end_path:out1:RootOutput                               2.373e-06    4.2983e-06    2.2675e-05     3.184e-06    2.97121e-06      100    
end_path:out1:RootOutput(write)                       0.00832109     0.247455       1.4592       0.238695      0.228627        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7120.88 MB
  Peak resident set size usage (VmHWM): 3134.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_657_20231208T093703Z_gen_g4_detsim_hitreco__20240510T064330Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_657_20231208T093703Z_gen_g4_detsim_hitreco__20240510T064330Z_reco2_reco_2024-11-23T_051614Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_657_20231208T093703Z_gen_g4_detsim_hitreco__20240510T064330Z_reco2_reco_data_2024-11-23T_051614Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work