Jobsub ID 108851.32@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108851.32@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:03:18
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 07:14:31
From worker node	Hostname	heplnc147.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:04:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_92_20231207T082710Z_gen_g4_detsim_hitreco__20240510T042008Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4171s)
	CPU time	1h (3911s = 93%)
Outputting started	2024-11-23 07:13:59
Output files
Finished	2024-11-23 07:14:31
List job events Wrapper job log
Jobscript log (last 10,000 characters)

-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:57 GMT run: 6483936 subRun: 1 event: 9300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:13:50 GMT run: 6483936 subRun: 1 event: 9300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1398, 1335, 775
minwire 0: 2544
minwire 1: 621
minwire 2: 2364
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0707092, 0.761533, 0.167757, 
Output 1: 0.101715, 0.688627, 0.193687, 0.0159707, 
Output 2: 0.0686721, 0.931328, 

663
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2390!
23-Nov-2024 07:13:57 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_92_20231207T082710Z_gen_g4_detsim_hitreco__20240510T042008Z_reco2_reco_data_2024-11-23T_060451Z.root"
23-Nov-2024 07:13:57 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/5d/47/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_92_20231207T082710Z_gen_g4_detsim_hitreco__20240510T042008Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.02608       38.2898       702.709       15.8785       83.4888        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0262098     0.0553966     0.110161      0.0578333     0.0147859       100    
reco:linecluster:LineCluster                           0.159948      0.518703       17.2897      0.241835       1.80617        100    
reco:trajcluster:TrajCluster                           0.0657865      1.15009       37.1686      0.0899337      5.40492        100    
reco:pandora:StandardPandora                            1.49347       3.11045       43.9783       2.20388       4.56275        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000182184   0.00794978     0.0910116    0.00315936     0.0142626       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000286441    0.0149199     0.154676     0.00630573     0.0269586       100    
reco:pandoracalo:Calorimetry                          0.000149101    0.0217045     0.797671     0.00254449     0.0871955       100    
reco:pandorapid:Chi2ParticleID                        5.2789e-05    0.000313396   0.00420278    0.000221383   0.000454422      100    
reco:trkshowersplit:TrackShowerHits                   0.000182244    0.0223624     0.661515     0.00140553     0.0897766       100    
reco:pmtrack:PMAlgTrackMaker                          0.00166292      2.92397       79.2001      0.318385       9.9648         100    
reco:pmtrackcalo:Calorimetry                          8.4539e-05     0.0102856     0.332797     0.00172645     0.0359897       100    
reco:pmtrackpid:Chi2ParticleID                        3.5176e-05    0.000449187    0.0048455    0.000225416   0.000642595      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000238711     2.37398       51.0555      0.425888       6.33586        100    
reco:pmtrajfitcalo:Calorimetry                        7.1084e-05     0.0265824      1.07989     0.00282223     0.116942        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6519e-05    0.000317486   0.00143792    0.00023079    0.000269752      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00144621      3.63257       107.698      0.379186       13.4198        100    
reco:pmtracktccalo:Calorimetry                        8.0492e-05    0.00728441     0.155182     0.00184143     0.0193931       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6278e-05    0.000407413   0.00361465    0.000251775   0.000524659      100    
reco:pmtrajfittc:PMAlgTrajFitter                       0.0002596      1.93721       64.0072      0.407168       6.71373        100    
reco:pmtrajfittccalo:Calorimetry                      7.7196e-05     0.0271867      1.5525      0.00228883     0.156838        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.171e-05    0.000568878   0.00661284    0.000276447   0.00090521       100    
reco:blurredcluster:BlurredClustering                 0.000183377     1.29198       9.08369      0.726959       1.5481         100    
reco:emtrkmichelid:EmTrackMichelId                    0.00266806      17.1445       279.873       5.37663       34.7236        100    
reco:emshower:EMShower                                0.000375599    0.0315662     0.477717     0.00856585     0.0600401       100    
reco:cvnmap:CVNMapper                                 3.4084e-05     0.0212962     0.0638213     0.0255037     0.0162061       100    
reco:cvneva:CVNEvaluator                              2.4887e-05      1.1735        3.19928       1.42129      0.851452        100    
reco:energyrecnumu:EnergyReco                         0.000541462    0.0239981     0.361187     0.00843918     0.0523766       100    
reco:energyrecnue:EnergyReco                          0.000193835   0.00247699     0.0335948    0.000964095   0.00460311       100    
reco:energyrecnc:EnergyReco                           0.00295852    0.00531126     0.0377863    0.00403873     0.004469        100    
reco:energyrecnumurange:EnergyReco                    0.000283856    0.0206518     0.361778     0.00418318     0.0528278       100    
reco:energyrecnumumcs:EnergyReco                       0.0029378     0.0220657     0.386488      0.0050211     0.0541185       100    
reco:opdec:Deconvolution                               0.0710048     0.356846       0.96539      0.367347      0.121246        100    
reco:ophitspe:OpHitFinderDeco                          0.236355       2.0413        3.92064       2.14824      0.859676        100    
reco:opflash:OpFlashFinder                            0.000493922   0.00262244     0.0133248    0.00212821    0.00213417       100    
reco:rns:RandomNumberSaver                            3.0277e-05    0.000162421   0.00931249    5.9928e-05    0.000920278      100    
reco:anglereconue:NuAngularReco                       0.000231587   0.00188264     0.0236371    0.000698984    0.0036756       100    
reco:anglereconumu:NuAngularReco                      8.7776e-05    0.00134184     0.0222178    0.000324603   0.00335775       100    
reco:anglereconuepfps:NuAngularReco                   0.000184959    0.0018152     0.022366     0.000587303   0.00392924       100    
reco:anglereconumupfps:NuAngularReco                  0.000139934   0.00146895     0.0223828    0.000404428   0.00344754       100    
[art]:TriggerResults:TriggerResultInserter            1.1722e-05    6.46946e-05   0.00453227    1.7408e-05    0.000449088      100    
end_path:out1:RootOutput                               4.188e-06    1.35022e-05   9.9036e-05    1.0655e-05    1.06628e-05      100    
end_path:out1:RootOutput(write)                        0.0203166     0.298332       1.30506      0.317014      0.209523        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21730.3 MB
  Peak resident set size usage (VmHWM): 2292.36 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_92_20231207T082710Z_gen_g4_detsim_hitreco__20240510T042008Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_92_20231207T082710Z_gen_g4_detsim_hitreco__20240510T042008Z_reco2_reco_2024-11-23T_060451Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_92_20231207T082710Z_gen_g4_detsim_hitreco__20240510T042008Z_reco2_reco_data_2024-11-23T_060451Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work