Jobsub ID 108715.55@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108715.55@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:31:31
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-23 04:47:40
From worker node	Hostname	heplnc153.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:33:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_200_20231201T104644Z_gen_g4_detsim_hitreco__20240507T193332Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4433s)
	CPU time	1h (4887s = 110%)
Outputting started	2024-11-23 04:47:10
Output files
Finished	2024-11-23 04:47:40
List job events Wrapper job log
Jobscript log (last 10,000 characters)

pmtracktc@BeginModule  23-Nov-2024 04:46:49 GMT run: 74481105 subRun: 1 event: 20100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:46:49 GMT run: 74481105 subRun: 1 event: 20100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:46:49 GMT run: 74481105 subRun: 1 event: 20100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:46:49 GMT run: 74481105 subRun: 1 event: 20100
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:49 GMT run: 74481105 subRun: 1 event: 20100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:50 GMT run: 74481105 subRun: 1 event: 20100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:50 GMT run: 74481105 subRun: 1 event: 20100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:50 GMT run: 74481105 subRun: 1 event: 20100
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 296, 108, 229
minwire 0: 1217
minwire 1: 1088
minwire 2: 1370
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0135495, 0.00046152, 0.985989, 
Output 1: 0.0312776, 0.485727, 0.464803, 0.0181928, 
Output 2: 0.952563, 0.0474365, 

483
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1524!
23-Nov-2024 04:47:08 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_200_20231201T104644Z_gen_g4_detsim_hitreco__20240507T193332Z_reco2_reco_data_2024-11-23T_033325Z.root"
23-Nov-2024 04:47:08 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/5a/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_200_20231201T104644Z_gen_g4_detsim_hitreco__20240507T193332Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.63197       41.2066       251.851       19.0796       51.7496        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00568195     0.201135       1.82896      0.118361      0.247236        100    
reco:linecluster:LineCluster                           0.0441426     0.755806       6.00149      0.276127       1.12293        100    
reco:trajcluster:TrajCluster                           0.0127587     0.433203       8.29846      0.0383354      1.19125        100    
reco:pandora:StandardPandora                            1.15975       3.0677        23.4163       2.57398       2.44069        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000281798   0.00803269     0.0390096    0.00351999    0.00957383       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00058754     0.0148787     0.181829     0.00530898     0.0252305       100    
reco:pandoracalo:Calorimetry                          0.000217718    0.0173944     0.234451     0.00217555     0.0378393       100    
reco:pandorapid:Chi2ParticleID                         6.938e-05    0.000358827    0.0029691    0.000290208   0.00033541       100    
reco:trkshowersplit:TrackShowerHits                   0.000166311    0.0158767     0.208321     0.00144831     0.033679        100    
reco:pmtrack:PMAlgTrackMaker                          0.00152903      2.19575       26.8319      0.247812       4.66492        100    
reco:pmtrackcalo:Calorimetry                          0.000104435   0.00751732     0.088366     0.00185818     0.0129875       100    
reco:pmtrackpid:Chi2ParticleID                        4.5245e-05    0.000509325   0.00199182    0.000331882   0.00046646       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000468999     2.05761       22.5792      0.400063       3.61157        100    
reco:pmtrajfitcalo:Calorimetry                        0.000121377    0.0172927     0.214378     0.00293241     0.035935        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.7228e-05    0.000378174   0.00116667    0.000333034   0.00022807       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00142881       2.565        36.464       0.262731       5.90737        100    
reco:pmtracktccalo:Calorimetry                        0.000103454   0.00716537     0.114647     0.00215696     0.0140597       100    
reco:pmtracktcpid:Chi2ParticleID                       4.245e-05    0.000501563   0.00248714    0.000345182   0.000483521      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000270566     1.50798       11.0275      0.348463       2.30348        100    
reco:pmtrajfittccalo:Calorimetry                      0.000101049    0.0116014     0.121891     0.00232662     0.020727        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.1618e-05    0.000623612   0.00379787    0.000374987   0.000650445      100    
reco:blurredcluster:BlurredClustering                 0.00124281      1.31929       6.17648      0.664281       1.47809        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0027561      21.7917       164.351       7.50786       30.3855        100    
reco:emshower:EMShower                                0.00071068     0.0345365     0.244346     0.00880467     0.0553223       100    
reco:cvnmap:CVNMapper                                 6.4621e-05     0.0278799     0.134189      0.0310774     0.0228197       100    
reco:cvneva:CVNEvaluator                              3.5046e-05      1.4972        3.22604       1.93379       1.00295        100    
reco:energyrecnumu:EnergyReco                         0.000752479    0.0372598     0.830346     0.00978662     0.101485        100    
reco:energyrecnue:EnergyReco                          0.000244698   0.00228011     0.0150349    0.00128764    0.00261455       100    
reco:energyrecnc:EnergyReco                           0.00196687    0.00496256     0.0182365    0.00407935    0.00279509       100    
reco:energyrecnumurange:EnergyReco                    0.000300332    0.0297832     0.804948     0.00360807     0.0963449       100    
reco:energyrecnumumcs:EnergyReco                      0.00140712     0.0302317     0.676979     0.00460547     0.0863751       100    
reco:opdec:Deconvolution                               0.0332825     0.746876       2.82024      0.378991      0.695737        100    
reco:ophitspe:OpHitFinderDeco                          0.146375        2.47         7.56959       2.29258       1.34119        100    
reco:opflash:OpFlashFinder                            0.000786482   0.00271633    0.00749788    0.00224761    0.00148562       100    
reco:rns:RandomNumberSaver                            3.9403e-05    9.46014e-05   0.000666137   8.1397e-05    6.76816e-05      100    
reco:anglereconue:NuAngularReco                       0.000274554   0.00191081     0.0115635     0.0008705    0.00213919       100    
reco:anglereconumu:NuAngularReco                      0.000100438   0.00126333    0.00767322     0.0004381     0.0017275       100    
reco:anglereconuepfps:NuAngularReco                   0.000173315   0.00154129    0.00847498    0.000703687   0.00184273       100    
reco:anglereconumupfps:NuAngularReco                  0.000129663   0.00126362    0.00784145    0.000528269   0.00167782       100    
[art]:TriggerResults:TriggerResultInserter            1.5199e-05    2.99241e-05   9.9706e-05    2.5443e-05    1.45975e-05      100    
end_path:out1:RootOutput                               5.12e-06     1.43055e-05   7.1634e-05    1.2754e-05    8.0603e-06       100    
end_path:out1:RootOutput(write)                        0.036727      0.315302      0.807521      0.320884       0.19098        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39268.3 MB
  Peak resident set size usage (VmHWM): 2086.51 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_200_20231201T104644Z_gen_g4_detsim_hitreco__20240507T193332Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_200_20231201T104644Z_gen_g4_detsim_hitreco__20240507T193332Z_reco2_reco_2024-11-23T_033325Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_200_20231201T104644Z_gen_g4_detsim_hitreco__20240507T193332Z_reco2_reco_data_2024-11-23T_033325Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work