Jobsub ID 301626.190@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301626.190@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:40:29
Site	US_UConn-HPC
Entry	GLUEX_US_UConn-HPC_osgce
Last heartbeat	2024-11-23 03:03:40
From worker node	Hostname	cn425
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:46:27
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_49_20231201T181004Z_gen_g4_detsim_hitreco__20240507T212907Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4620s)
	CPU time	1h (4345s = 94%)
Outputting started	2024-11-23 03:03:27
Output files
Finished	2024-11-23 03:03:40
List job events Wrapper job log
Jobscript log (last 10,000 characters)

, CR 0, PDG 13, nMatchedHits 44 (8, 23, 13), nPfoHits 44 (8, 23, 13)

------------------------------------------------------------------------------------------------

---INTERPRETED-MATCHING-OUTPUT------------------(my version)-----------------------------------------
Found: 2212 with KE=0.147118 GeV, E=1.08539 GeV, p2=0.297717
NCQEL_P (Nuance 1002, Nu 1, CR 0)
IsCorrectNu (NNuMatches: 1) 
Parent Neutrino: -14
Neutrino Energy: 0.716587
Neutrino Momentum:   x: 0.3144  y: 0.637645  z: -0.0897753 length: 0.716588
PrimaryId 1, Nu 1, CR 0, MCPDG 2212, Energy 1.08539, Dist. 15.4309, nMCHits 44 (8, 23, 13)
-MatchedPfoId 1, Nu 1 [NuId: 1], CR 0, PDG 13, nMatchedHits 44 (8, 23, 13), nPfoHits 44 (8, 23, 13)

---SUMMARY--------------------------------------------------------------------------------------
#CorrectNu: 1/1, Fraction: 1
------------------------------------------------------------------------------------------------

Found: 2212 with KE=0.147118 GeV, E=1.08539 GeV, p2=0.297717
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  22-Nov-2024 22:03:22 EST run: 6389847 subRun: 1 event: 5000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 22:03:22 EST run: 6389847 subRun: 1 event: 5000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 22:03:22 EST run: 6389847 subRun: 1 event: 5000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:03:22 EST run: 6389847 subRun: 1 event: 5000
1st wire C:0 T:1 P:2 W:915 does not exist (max wire number: 480)
%MSG
342
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 912!
22-Nov-2024 22:03:25 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_49_20231201T181004Z_gen_g4_detsim_hitreco__20240507T212907Z_reco2_reco_data_2024-11-23T_014631Z.root"
22-Nov-2024 22:03:25 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/03/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_49_20231201T181004Z_gen_g4_detsim_hitreco__20240507T212907Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.22067       43.6971       649.236       12.2876       106.007        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0074746     0.0569407     0.137881      0.0540383     0.0289727       100    
reco:linecluster:LineCluster                           0.0682386     0.448054       18.3818      0.107027       1.96077        100    
reco:trajcluster:TrajCluster                           0.0207522      1.75544       64.4528      0.0363319      8.03276        100    
reco:pandora:StandardPandora                           0.998504       4.04135       126.84        1.58455       14.0764        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00015867    0.00859822     0.119144     0.00199932     0.0197624       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000214906    0.0181719     0.243055     0.00357184     0.0438417       100    
reco:pandoracalo:Calorimetry                          0.000106901    0.033298      0.955178     0.00147159     0.124599        100    
reco:pandorapid:Chi2ParticleID                        4.8131e-05    0.000206853   0.00165406    0.000137525   0.000207531      100    
reco:trkshowersplit:TrackShowerHits                   0.000120948    0.0303373     0.673476     0.00101923     0.107528        100    
reco:pmtrack:PMAlgTrackMaker                          0.00119022      3.6948        68.3933      0.261163       11.4917        100    
reco:pmtrackcalo:Calorimetry                           6.346e-05     0.0186865     0.474674     0.00114728     0.0650181       100    
reco:pmtrackpid:Chi2ParticleID                        2.6269e-05    0.000397232   0.00441801    0.000168979   0.000708283      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000202733     4.31079       124.035      0.441764       15.6854        100    
reco:pmtrajfitcalo:Calorimetry                         5.841e-05     0.0328897     0.910262     0.00204847     0.126693        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7031e-05    0.000225577   0.00135346    0.000160032   0.000209348      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102263      4.22113       118.501      0.280681       14.7969        100    
reco:pmtracktccalo:Calorimetry                        6.1786e-05     0.0109085     0.179442     0.00124764     0.0300404       100    
reco:pmtracktcpid:Chi2ParticleID                       2.653e-05    0.000338653   0.00288011    0.000167351   0.000505318      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000229524     2.1407        40.5653      0.383435       5.43893        100    
reco:pmtrajfittccalo:Calorimetry                      6.6175e-05     0.023117      0.631583     0.00164138     0.0798096       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4616e-05    0.000464416   0.00547301    0.000186798   0.000773888      100    
reco:blurredcluster:BlurredClustering                 0.00334482      1.4258        9.04446      0.781911       1.86261        100    
reco:emtrkmichelid:EmTrackMichelId                      0.27625       17.9306       190.295       4.99951       36.7964        100    
reco:emshower:EMShower                                0.000289067    0.0273177     0.271033      0.0028331     0.0481565       100    
reco:cvnmap:CVNMapper                                 2.0518e-05     0.0126344     0.0460193     0.0124888     0.0104972       100    
reco:cvneva:CVNEvaluator                               1.571e-05      1.12468       13.6854      0.902794       2.2616         100    
reco:energyrecnumu:EnergyReco                         0.000316588    0.051258       1.32197     0.00460008     0.169867        100    
reco:energyrecnue:EnergyReco                          0.00012177    0.00242328     0.0353442    0.000642904   0.00559106       100    
reco:energyrecnc:EnergyReco                           0.00184479    0.00484693     0.0372254     0.0030044    0.00615063       100    
reco:energyrecnumurange:EnergyReco                    0.000150895    0.0472052      1.28447     0.00245897     0.165705        100    
reco:energyrecnumumcs:EnergyReco                      0.00188283     0.0486192      1.30115     0.00367912      0.16668        100    
reco:opdec:Deconvolution                               0.0491026     0.377212       9.98927      0.243852      0.978938        100    
reco:ophitspe:OpHitFinderDeco                          0.228613       1.53397       3.24741       1.52953      0.760585        100    
reco:opflash:OpFlashFinder                            0.00051368    0.00160163    0.00586715    0.00143934    0.000849288      100    
reco:rns:RandomNumberSaver                            1.7773e-05    3.76454e-05   0.000455841   3.16095e-05   4.36384e-05      100    
reco:anglereconue:NuAngularReco                       0.000157398   0.00187649     0.0298357    0.000448943   0.00447623       100    
reco:anglereconumu:NuAngularReco                      6.2478e-05    0.00168655     0.0312036    0.000241126   0.00475497       100    
reco:anglereconuepfps:NuAngularReco                   0.000110639   0.00187838     0.0310108    0.000364995   0.00483107       100    
reco:anglereconumupfps:NuAngularReco                   8.985e-05    0.00172216     0.0307037    0.000308989   0.00454643       100    
[art]:TriggerResults:TriggerResultInserter             8.145e-06    1.4199e-05    6.6626e-05    1.29895e-05   6.4424e-06       100    
end_path:out1:RootOutput                               3.036e-06    6.49981e-06   2.6961e-05    4.9745e-06    4.07159e-06      100    
end_path:out1:RootOutput(write)                        0.0244219     0.252666       1.31913      0.230087      0.217825        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21296.8 MB
  Peak resident set size usage (VmHWM): 2411.87 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_49_20231201T181004Z_gen_g4_detsim_hitreco__20240507T212907Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_49_20231201T181004Z_gen_g4_detsim_hitreco__20240507T212907Z_reco2_reco_2024-11-23T_014631Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_49_20231201T181004Z_gen_g4_detsim_hitreco__20240507T212907Z_reco2_reco_data_2024-11-23T_014631Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work