Jobsub ID 108499.3@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108499.3@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 21:32:00
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2024-11-22 23:26:06
From worker node	Hostname	wn2210251.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 21:34:40
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_738_20231205T061550Z_gen_g4_detsim_hitreco__20240509T221818Z_reco2.root
Outputting started	2024-11-22 23:21:42
Output files
Finished	2024-11-22 23:26:06
List job events Wrapper job log
Jobscript log (last 10,000 characters)

un: 74521450 subRun: 1 event: 73900
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:43 GMT run: 74521450 subRun: 1 event: 73900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:43 GMT run: 74521450 subRun: 1 event: 73900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:43 GMT run: 74521450 subRun: 1 event: 73900
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:43 GMT run: 74521450 subRun: 1 event: 73900
Cannot attach back to inner node of other track.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:32 GMT run: 74521450 subRun: 1 event: 73900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:32 GMT run: 74521450 subRun: 1 event: 73900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:21:32 GMT run: 74521450 subRun: 1 event: 73900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 2382, 2161, 1683
minwire 0: 847
minwire 1: 543
minwire 2: 87
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0454156, 0.536159, 0.418426, 
Output 1: 0.26233, 0.410485, 0.207331, 0.119854, 
Output 2: 0.0795301, 0.92047, 

752
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2943!
22-Nov-2024 23:21:40 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_738_20231205T061550Z_gen_g4_detsim_hitreco__20240509T221818Z_reco2_reco_data_2024-11-22T_213447Z.root"
22-Nov-2024 23:21:40 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/82/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_738_20231205T061550Z_gen_g4_detsim_hitreco__20240509T221818Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.838374       61.9917       2517.9        13.4021       266.334        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0109815     0.0329535     0.079053      0.0338363     0.0118871       100    
reco:linecluster:LineCluster                           0.091245       1.47809       78.7716      0.106185       9.17992        100    
reco:trajcluster:TrajCluster                           0.027779       3.49171       212.028      0.0444819      23.0526        100    
reco:pandora:StandardPandora                           0.0501634      13.7431       962.449       1.57722       96.8745        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000185278    0.0115722      0.41761     0.00252937     0.0449261       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000256522    0.0189105     0.208201      0.0046115     0.0396555       100    
reco:pandoracalo:Calorimetry                          0.000163617    0.0528975      2.85845     0.00181351     0.303104        100    
reco:pandorapid:Chi2ParticleID                        5.7678e-05    0.000283496   0.00157187    0.000228344   0.000255721      100    
reco:trkshowersplit:TrackShowerHits                   0.000145723    0.0770773      3.10809     0.00139169     0.412824        100    
reco:pmtrack:PMAlgTrackMaker                          0.00166647      5.69343       276.056      0.354052       28.7082        100    
reco:pmtrackcalo:Calorimetry                          0.000104016    0.0295344      1.85191     0.00128598      0.18652        100    
reco:pmtrackpid:Chi2ParticleID                         4.791e-05    0.00047927     0.0082735    0.000243913   0.000942326      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000275086      5.026        150.127      0.429018       20.8715        100    
reco:pmtrajfitcalo:Calorimetry                        8.6943e-05     0.0541583      2.88496      0.0021617     0.306551        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.1417e-05    0.000295282   0.00189981    0.000247515   0.000257032      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00142664      6.85787       321.494      0.387824       34.1869        100    
reco:pmtracktccalo:Calorimetry                        9.6622e-05     0.0156371     0.676829     0.00141483     0.0716683       100    
reco:pmtracktcpid:Chi2ParticleID                      3.7019e-05    0.000431654   0.00524517    0.000253596   0.000661737      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000237797     3.23404       124.009      0.386895       13.4485        100    
reco:pmtrajfittccalo:Calorimetry                      8.5621e-05     0.0392507      2.0494      0.00162552     0.217223        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.749e-05    0.000614688    0.0106748    0.000248762   0.00129247       100    
reco:blurredcluster:BlurredClustering                 0.000214032     1.45226       15.3501      0.678833       2.2454         100    
reco:emtrkmichelid:EmTrackMichelId                     0.0477535      16.1145       335.663       3.88385       41.0911        100    
reco:emshower:EMShower                                0.000378651    0.0480676     0.940061     0.00871626     0.115367        100    
reco:cvnmap:CVNMapper                                 3.0648e-05     0.0193106     0.0613995     0.0175529     0.0161284       100    
reco:cvneva:CVNEvaluator                              2.4976e-05     0.943555       1.88893       1.15119      0.534354        100    
reco:energyrecnumu:EnergyReco                         0.000342703    0.121914       5.68955     0.00709992     0.603325        100    
reco:energyrecnue:EnergyReco                          0.000136927    0.003731      0.123231     0.000848899    0.0134112       100    
reco:energyrecnc:EnergyReco                           0.00202023    0.00726857     0.129326     0.00402982     0.0138415       100    
reco:energyrecnumurange:EnergyReco                    0.00015469     0.116025       5.56312     0.00378629     0.589447        100    
reco:energyrecnumumcs:EnergyReco                      0.000880594     0.11232       5.11244      0.0050008     0.547346        100    
reco:opdec:Deconvolution                               0.0485768     0.259337      0.631085      0.262607      0.105672        100    
reco:ophitspe:OpHitFinderDeco                          0.204881       2.53337       5.45046       2.53711       1.17424        100    
reco:opflash:OpFlashFinder                            0.000388599   0.00216797    0.00472598    0.00222712    0.000940863      100    
reco:rns:RandomNumberSaver                             2.672e-05    7.20132e-05    0.0004259    6.98015e-05   4.10688e-05      100    
reco:anglereconue:NuAngularReco                       0.000215154   0.00298635      0.10714     0.00075107     0.0115753       100    
reco:anglereconumu:NuAngularReco                      9.4207e-05     0.0025076      0.1037      0.000359871    0.0112291       100    
reco:anglereconuepfps:NuAngularReco                    0.0001705    0.00276885     0.103359     0.000597738    0.0111919       100    
reco:anglereconumupfps:NuAngularReco                  0.000137769   0.00261868     0.104204     0.000445476    0.0113228       100    
[art]:TriggerResults:TriggerResultInserter             1.048e-05    2.10185e-05   9.5439e-05    1.9376e-05    9.94115e-06      100    
end_path:out1:RootOutput                               3.376e-06    9.58696e-06   2.7381e-05     9.413e-06    3.18705e-06      100    
end_path:out1:RootOutput(write)                        0.0230227     0.385549       2.52225      0.371184      0.323342        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22110.2 MB
  Peak resident set size usage (VmHWM): 2678.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_738_20231205T061550Z_gen_g4_detsim_hitreco__20240509T221818Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_738_20231205T061550Z_gen_g4_detsim_hitreco__20240509T221818Z_reco2_reco_2024-11-22T_213447Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_738_20231205T061550Z_gen_g4_detsim_hitreco__20240509T221818Z_reco2_reco_data_2024-11-22T_213447Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work