Jobsub ID 108729.58@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108729.58@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:03:01
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 04:57:58
From worker node	Hostname	n259.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:09:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1169_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073127Z_reco2.root
Jobscript	Exit code	0
	Real time	47m (2876s)
	CPU time	38m (2324s = 80%)
Outputting started	2024-11-23 04:57:40
Output files
Finished	2024-11-23 04:57:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ring two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:26 GMT run: 50572032 subRun: 1 event: 117000
1st wire C:0 T:14 P:2 W:538 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:26 GMT run: 50572032 subRun: 1 event: 117000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:26 GMT run: 50572032 subRun: 1 event: 117000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:27 GMT run: 50572032 subRun: 1 event: 117000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:27 GMT run: 50572032 subRun: 1 event: 117000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:34 GMT run: 50572032 subRun: 1 event: 117000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 254, 266, 216
minwire 0: 1643
minwire 1: 190
minwire 2: 1553
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.170254, 0.828854, 0.000891299, 
Output 1: 0.978475, 0.0207262, 0.000662618, 0.000136229, 
Output 2: 0.979764, 0.020236, 

583
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2103!
23-Nov-2024 04:57:39 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1169_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073127Z_reco2_reco_data_2024-11-23T_040948Z.root"
23-Nov-2024 04:57:39 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/db/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1169_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073127Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.12545       23.4641       317.733       11.7433       42.3252        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0351313     0.0754954     0.128351      0.0798115     0.0212948       100    
reco:linecluster:LineCluster                           0.201811      0.469505       7.59947      0.368192      0.726854        100    
reco:trajcluster:TrajCluster                           0.101473      0.453441       13.1842      0.116952       1.66185        100    
reco:pandora:StandardPandora                           0.112114       1.8243        24.0234       1.3532        2.50186        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000173829   0.00484249     0.0748849    0.00173987     0.0105323       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000206401    0.0103262     0.109047     0.00373671     0.0183336       100    
reco:pandoracalo:Calorimetry                          0.000121981    0.0135385     0.500772     0.00138257     0.0554326       100    
reco:pandorapid:Chi2ParticleID                         4.719e-05    0.000202102   0.00306022    0.000159418   0.000302474      100    
reco:trkshowersplit:TrackShowerHits                   8.9249e-05     0.0150241     0.525432     0.00101251     0.0628755       100    
reco:pmtrack:PMAlgTrackMaker                          0.00146357      1.63751       35.3298      0.330255       4.95271        100    
reco:pmtrackcalo:Calorimetry                          9.2125e-05    0.00627104     0.166388     0.00117213     0.0202355       100    
reco:pmtrackpid:Chi2ParticleID                        3.6218e-05    0.000296161   0.00213486    0.000182691   0.00036079       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000216721     1.71671       38.5166      0.375008       5.05466        100    
reco:pmtrajfitcalo:Calorimetry                        7.8018e-05     0.0141873     0.499639     0.00197809     0.0556758       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3253e-05    0.000215227   0.000837988   0.000184154   0.000146044      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00122319      2.13503       52.6123      0.304782       6.89683        100    
reco:pmtracktccalo:Calorimetry                        8.1684e-05    0.00529728     0.133542     0.00123188     0.0148919       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3063e-05    0.00028978    0.00198628    0.000187836   0.000310986      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000222712     1.50516       37.8808      0.333487       4.24575        100    
reco:pmtrajfittccalo:Calorimetry                      8.2055e-05     0.0102639     0.278084      0.0015339     0.0354938       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3864e-05    0.000378606   0.00351691    0.00020301    0.000517522      100    
reco:blurredcluster:BlurredClustering                 8.6915e-05     0.906119       7.85305      0.581857       1.17934        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000139123     8.78452       94.653        3.6515        15.5627        100    
reco:emshower:EMShower                                0.000206502    0.0273205     0.196676     0.00832911     0.0383254       100    
reco:cvnmap:CVNMapper                                 2.8234e-05     0.0128088     0.0399452     0.016145     0.00880758       100    
reco:cvneva:CVNEvaluator                              2.2352e-05      0.69183      0.938269      0.852319      0.346791        100    
reco:energyrecnumu:EnergyReco                         0.000451936    0.0210089     0.216699     0.00544848     0.0437443       100    
reco:energyrecnue:EnergyReco                          0.000180412   0.00167262     0.0232113    0.000601751   0.00305961       100    
reco:energyrecnc:EnergyReco                            0.0026208     0.004266      0.0274565    0.00331431    0.00325935       100    
reco:energyrecnumurange:EnergyReco                    0.000230788    0.0186001     0.211471     0.00327018     0.0416016       100    
reco:energyrecnumumcs:EnergyReco                      0.00136583     0.0204563     0.211654     0.00447668     0.0418641       100    
reco:opdec:Deconvolution                               0.171961      0.420647      0.662664      0.424798      0.102151        100    
reco:ophitspe:OpHitFinderDeco                          0.302822       2.3811        3.93212       2.37464      0.839979        100    
reco:opflash:OpFlashFinder                            0.00013178     0.0016992    0.00772535    0.00162459    0.000882226      100    
reco:rns:RandomNumberSaver                            2.1591e-05    4.59925e-05   0.00054303    4.19645e-05   5.08538e-05      100    
reco:anglereconue:NuAngularReco                       0.000187024   0.00121244     0.0187116    0.000508083    0.0024954       100    
reco:anglereconumu:NuAngularReco                      9.7134e-05    0.000934523    0.0179936    0.000292269   0.00229979       100    
reco:anglereconuepfps:NuAngularReco                   0.000149444   0.00112012     0.0180368    0.000433422   0.00242307       100    
reco:anglereconumupfps:NuAngularReco                  0.000129075   0.00100765     0.0179759    0.000365768   0.00229903       100    
[art]:TriggerResults:TriggerResultInserter             1.053e-05    1.57072e-05   7.9291e-05    1.48185e-05   7.01114e-06      100    
end_path:out1:RootOutput                               3.266e-06    8.15511e-06   2.8173e-05     8.291e-06    3.42938e-06      100    
end_path:out1:RootOutput(write)                       0.00990539     0.267093      0.840824      0.274801      0.146063        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38309.7 MB
  Peak resident set size usage (VmHWM): 2250.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1169_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073127Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1169_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073127Z_reco2_reco_2024-11-23T_040948Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1169_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073127Z_reco2_reco_data_2024-11-23T_040948Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work