Jobsub ID 108697.28@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108697.28@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:57:06
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 06:21:43
From worker node	Hostname	n251.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:05:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_183_20231028T042442Z_gen_g4_detsim_hitreco__20240503T051012Z_reco2.root
Jobscript	Exit code	0
	Real time	3h (11772s)
	CPU time	2h (10796s = 91%)
Outputting started	2024-11-23 06:21:26
Output files
Finished	2024-11-23 06:21:43
List job events Wrapper job log
Jobscript log (last 10,000 characters)

rino: -14
Neutrino Energy: 2.16486
Neutrino Momentum:   x: -0.39254  y: 1.4411  z: -1.56711 length: 2.16487
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 1.09064, Dist. 16.1586, nMCHits 45 (16, 22, 7)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 45 (16, 22, 7), nPfoHits 45 (16, 22, 7)

------------------------------------------------------------------------------------------------

---INTERPRETED-MATCHING-OUTPUT------------------(my version)-----------------------------------------
Found: 2212 with KE=0.152368 GeV, E=1.09064 GeV, p2=0.309142
NCQEL_P (Nuance 1002, Nu 1, CR 0)
IsCorrectNu (NNuMatches: 1) 
Parent Neutrino: -14
Neutrino Energy: 2.16486
Neutrino Momentum:   x: -0.39254  y: 1.4411  z: -1.56711 length: 2.16487
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 1.09064, Dist. 16.1586, nMCHits 45 (16, 22, 7)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 45 (16, 22, 7), nPfoHits 45 (16, 22, 7)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.152368 GeV, E=1.09064 GeV, p2=0.309142
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:21:21 GMT run: 1794 subRun: 1 event: 36800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:21:21 GMT run: 1794 subRun: 1 event: 36800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:21:21 GMT run: 1794 subRun: 1 event: 36800
Comparing two wires in the same plane: return failure
%MSG
282
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 678!
23-Nov-2024 06:21:25 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_183_20231028T042442Z_gen_g4_detsim_hitreco__20240503T051012Z_reco2_reco_data_2024-11-23T_030520Z.root"
23-Nov-2024 06:21:25 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ab/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_183_20231028T042442Z_gen_g4_detsim_hitreco__20240503T051012Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.657764       54.2254       1252.13       11.2014       152.682        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0181345     0.0406149      0.10316      0.0380014     0.0138909       200    
reco:linecluster:LineCluster                           0.102528       1.01802       55.7124       0.19193       4.5011         200    
reco:trajcluster:TrajCluster                           0.0510093      3.00332       153.017      0.061076       14.4402        200    
reco:pandora:StandardPandora                           0.0585231      5.72513       319.888       1.47553       25.6335        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136479    0.0120011     0.285017     0.00175387     0.036458        200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000188598    0.0139804     0.233956     0.00319663     0.0307951       200    
reco:pandoracalo:Calorimetry                          0.000103557    0.0502722      1.59694     0.00115608     0.180815        200    
reco:pandorapid:Chi2ParticleID                        3.9435e-05    0.000226112   0.00298752    0.000159247   0.000286814      200    
reco:trkshowersplit:TrackShowerHits                   7.4271e-05     0.0771972      3.23901     0.00081419     0.311278        200    
reco:pmtrack:PMAlgTrackMaker                          0.00109802      5.93668       171.546      0.233383       20.2557        200    
reco:pmtrackcalo:Calorimetry                           6.866e-05     0.0277981      1.01795     0.00115287     0.112687        200    
reco:pmtrackpid:Chi2ParticleID                        2.7041e-05    0.000471679   0.00599505    0.000177337   0.000949615      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000173039     4.92311       180.459      0.268189       18.104         200    
reco:pmtrajfitcalo:Calorimetry                         6.317e-05     0.0516694      1.62917     0.00167668     0.184274        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.649e-05    0.000262475   0.00188673    0.000176149   0.000289364      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000935587     7.09403       200.345      0.232562       23.9926        200    
reco:pmtracktccalo:Calorimetry                        6.7839e-05     0.0152425     0.561086     0.00119565     0.0557087       200    
reco:pmtracktcpid:Chi2ParticleID                      2.6821e-05    0.000421513   0.00487024    0.000188247   0.000712315      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000163911     3.20722       68.8501       0.30117       9.33627        200    
reco:pmtrajfittccalo:Calorimetry                      6.3771e-05     0.0411607      1.06297     0.00138992     0.147013        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4948e-05    0.000608909   0.00706059    0.000209303   0.00120622       200    
reco:blurredcluster:BlurredClustering                 6.6656e-05      1.3597        11.6481      0.601637       2.16896        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000109117     17.5846       258.336       3.25336       42.1832        200    
reco:emshower:EMShower                                0.000153571    0.0413091     0.598118     0.00444658     0.0827715       200    
reco:cvnmap:CVNMapper                                 1.8906e-05     0.0111085     0.0619886    0.00785294     0.0119945       200    
reco:cvneva:CVNEvaluator                              1.8094e-05     0.697379       1.31087      0.901632      0.471836        200    
reco:energyrecnumu:EnergyReco                         0.000362648    0.127786       8.04101     0.00605171     0.657798        200    
reco:energyrecnue:EnergyReco                          0.000138653   0.00414657     0.0864262    0.000692394    0.0121414       200    
reco:energyrecnc:EnergyReco                           0.00201856    0.00697171     0.0940792    0.00389806     0.0127648       200    
reco:energyrecnumurange:EnergyReco                    0.000150465    0.124157       8.05138     0.00329129     0.654534        200    
reco:energyrecnumumcs:EnergyReco                      0.00155252     0.126657       8.06306      0.004324      0.658288        200    
reco:opdec:Deconvolution                               0.0656485     0.311872      0.653855      0.301919      0.121299        200    
reco:ophitspe:OpHitFinderDeco                          0.240158       2.26934       5.36167       2.01117       1.2576         200    
reco:opflash:OpFlashFinder                            0.000280432   0.00168174    0.00690566    0.00151485    0.000950684      200    
reco:rns:RandomNumberSaver                            2.0559e-05    4.00712e-05   0.000451387    3.449e-05    3.17531e-05      200    
reco:anglereconue:NuAngularReco                       0.00017356    0.00324611     0.0815435    0.00042422     0.0108213       200    
reco:anglereconumu:NuAngularReco                      7.1696e-05    0.00296548     0.0744707    0.000287832    0.0104136       200    
reco:anglereconuepfps:NuAngularReco                   0.000119857   0.00310352     0.0761401    0.000406491    0.0104939       200    
reco:anglereconumupfps:NuAngularReco                  0.000100591   0.00305224     0.0757588    0.000368098    0.0104609       200    
[art]:TriggerResults:TriggerResultInserter             9.347e-06    1.54971e-05   7.0123e-05    1.4553e-05    5.96656e-06      200    
end_path:out1:RootOutput                               3.106e-06    6.47225e-06   2.9285e-05    4.9945e-06    3.57507e-06      200    
end_path:out1:RootOutput(write)                        0.0134201     0.304451       1.74107      0.271858      0.297598        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38576 MB
  Peak resident set size usage (VmHWM): 2524.93 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_183_20231028T042442Z_gen_g4_detsim_hitreco__20240503T051012Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_183_20231028T042442Z_gen_g4_detsim_hitreco__20240503T051012Z_reco2_reco_2024-11-23T_030520Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_183_20231028T042442Z_gen_g4_detsim_hitreco__20240503T051012Z_reco2_reco_data_2024-11-23T_030520Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work