Jobsub ID 301838.4@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301838.4@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:17:28
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 07:23:10
From worker node	Hostname	comp08-10
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:18:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_989_20231205T045230Z_gen_g4_detsim_hitreco__20240510T021547Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3868s)
	CPU time	59m (3543s = 91%)
Outputting started	2024-11-23 07:22:54
Output files
Finished	2024-11-23 07:23:10
List job events Wrapper job log
Jobscript log (last 10,000 characters)

imaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 0.963555, Dist. 0.696985, nMCHits 20 (12, 6, 2)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 6 (3, 3, 0), nPfoHits 6 (3, 3, 0)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.0252833 GeV, E=0.963555 GeV, p2=0.0480846
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:22:50 GMT run: 50601160 subRun: 1 event: 99000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:22:50 GMT run: 50601160 subRun: 1 event: 99000
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:22:50 GMT run: 50601160 subRun: 1 event: 99000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:22:50 GMT run: 50601160 subRun: 1 event: 99000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:22:50 GMT run: 50601160 subRun: 1 event: 99000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:22:50 GMT run: 50601160 subRun: 1 event: 99000
Comparing two wires in the same plane: return failure
%MSG
126
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 315!
23-Nov-2024 07:22:52 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_989_20231205T045230Z_gen_g4_detsim_hitreco__20240510T021547Z_reco2_reco_data_2024-11-23T_061830Z.root"
23-Nov-2024 07:22:52 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/18/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_989_20231205T045230Z_gen_g4_detsim_hitreco__20240510T021547Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.27687       35.693        497.826       11.0431       76.281         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0290432     0.0640431     0.530803      0.0563573     0.0495795       100    
reco:linecluster:LineCluster                           0.263447      0.622411       11.1681      0.297491       1.40954        100    
reco:trajcluster:TrajCluster                           0.0826651      1.48445       25.6323      0.106175       4.88904        100    
reco:pandora:StandardPandora                            1.03211       2.57508       50.244        1.16889       6.33133        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139059   0.00932432     0.182007     0.00212099     0.0259044       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000171596    0.0134067     0.352603      0.0040504     0.0400401       100    
reco:pandoracalo:Calorimetry                          0.000101991    0.0348573      1.17353      0.001176      0.143774        100    
reco:pandorapid:Chi2ParticleID                         3.845e-05    0.000163783    0.0010782    0.000121044   0.000148207      100    
reco:trkshowersplit:TrackShowerHits                   9.0699e-05     0.0305967      0.60238     0.000821812     0.10377        100    
reco:pmtrack:PMAlgTrackMaker                          0.00122336      2.83846       56.656       0.227936       8.36479        100    
reco:pmtrackcalo:Calorimetry                           8.244e-05     0.0157933     0.352318     0.00101044     0.052812        100    
reco:pmtrackpid:Chi2ParticleID                        3.2144e-05    0.00029897    0.00289342    0.000121747   0.000481184      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166654     2.59124       60.4828      0.281489       7.67309        100    
reco:pmtrajfitcalo:Calorimetry                        6.2941e-05     0.0346015      1.17197     0.00151208     0.141089        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5253e-05    0.000190916   0.00118447    0.000129941   0.000196453      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0010062      3.22016       55.6881      0.227452       9.20234        100    
reco:pmtracktccalo:Calorimetry                         6.908e-05     0.0108133     0.196433     0.00112337     0.0335019       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7217e-05    0.000276517   0.00257875    0.000132418   0.00041842       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000157475     1.80089       34.1699      0.274688       4.80268        100    
reco:pmtrajfittccalo:Calorimetry                      5.8077e-05     0.0261228     0.740453     0.00118583     0.0986245       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4045e-05    0.000373808   0.00399534    0.000144146   0.000640336      100    
reco:blurredcluster:BlurredClustering                 0.00544728      1.15265       10.1994      0.496781       1.71521        100    
reco:emtrkmichelid:EmTrackMichelId                      0.24785       15.5263       186.712       3.6823        32.2673        100    
reco:emshower:EMShower                                0.000233805    0.0295388     0.395338     0.00581637     0.0639483       100    
reco:cvnmap:CVNMapper                                  2.001e-05     0.0150597     0.044474      0.0150373     0.0104719       100    
reco:cvneva:CVNEvaluator                              1.5402e-05     0.613303       2.76917      0.749435      0.396939        100    
reco:energyrecnumu:EnergyReco                         0.000290578    0.0294572     0.693732     0.00743991     0.0816394       100    
reco:energyrecnue:EnergyReco                          0.000148226   0.00289264     0.0548145    0.000582166   0.00842494       100    
reco:energyrecnc:EnergyReco                           0.00172592    0.00451139     0.0557508    0.00222065    0.00836541       100    
reco:energyrecnumurange:EnergyReco                    0.000212375    0.0260869     0.676026     0.00203166     0.0804226       100    
reco:energyrecnumumcs:EnergyReco                      0.000930772    0.0272945     0.681977      0.0030639     0.0806364       100    
reco:opdec:Deconvolution                               0.128988      0.388251      0.901612       0.37766      0.116586        100    
reco:ophitspe:OpHitFinderDeco                          0.393108       2.27085       4.20999       2.28012      0.853896        100    
reco:opflash:OpFlashFinder                            0.000282423   0.00136506    0.00424434    0.00131692    0.000610252      100    
reco:rns:RandomNumberSaver                            1.8503e-05    3.18806e-05   0.000274267   2.6979e-05    2.5719e-05       100    
reco:anglereconue:NuAngularReco                       0.000151547   0.00232857     0.0476126    0.000358418   0.00725517       100    
reco:anglereconumu:NuAngularReco                      6.8765e-05    0.00207636     0.045834     0.000214626   0.00702523       100    
reco:anglereconuepfps:NuAngularReco                   0.000116096   0.00220397     0.0467474     0.0003131    0.00706476       100    
reco:anglereconumupfps:NuAngularReco                  9.3313e-05    0.00215886     0.0471431    0.00028755     0.0070908       100    
[art]:TriggerResults:TriggerResultInserter             8.134e-06    1.03904e-05    4.251e-05    9.8405e-06    3.47969e-06      100    
end_path:out1:RootOutput                               2.554e-06    5.54748e-06   2.5686e-05    4.0025e-06    3.37373e-06      100    
end_path:out1:RootOutput(write)                        0.0107958     0.221264      0.976923      0.203655      0.167846        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9733.21 MB
  Peak resident set size usage (VmHWM): 2586.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_989_20231205T045230Z_gen_g4_detsim_hitreco__20240510T021547Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_989_20231205T045230Z_gen_g4_detsim_hitreco__20240510T021547Z_reco2_reco_2024-11-23T_061830Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_989_20231205T045230Z_gen_g4_detsim_hitreco__20240510T021547Z_reco2_reco_data_2024-11-23T_061830Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work