Jobsub ID 108778.6@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108778.6@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:03:03
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 05:58:32
From worker node	Hostname	b9p08p7421.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:04:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_214_20231207T200439Z_gen_g4_detsim_hitreco__20240510T041641Z_reco2.root
Jobscript	Exit code	0
	Real time	53m (3231s)
	CPU time	54m (3260s = 100%)
Outputting started	2024-11-23 05:58:12
Output files
Finished	2024-11-23 05:58:32
List job events Wrapper job log
Jobscript log (last 10,000 characters)

BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:00 CET run: 66050399 subRun: 1 event: 21500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 6853, 9947, 7283
minwire 0: 1152
minwire 1: 421
minwire 2: 866
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0450002, 0.509934, 0.445066, 
Output 1: 0.363172, 0.429283, 0.157997, 0.049548, 
Output 2: 0.111443, 0.888557, 

869
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3459!
23-Nov-2024 06:58:10 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_214_20231207T200439Z_gen_g4_detsim_hitreco__20240510T041641Z_reco2_reco_data_2024-11-23T_050427Z.root"
23-Nov-2024 06:58:10 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/54/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_214_20231207T200439Z_gen_g4_detsim_hitreco__20240510T041641Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.26753       30.2664       570.409       10.5725       71.2933        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.016934       0.04462       0.15881      0.0416228     0.022353        100    
reco:linecluster:LineCluster                            0.15211      0.336013       4.90441      0.177882      0.669361        100    
reco:trajcluster:TrajCluster                           0.0477334     0.972506       38.645       0.0636314      4.40774        100    
reco:pandora:StandardPandora                           0.916715       2.39882       57.6658       1.34096       5.91255        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00012832    0.00786099     0.196529     0.00201197     0.0226544       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000255712    0.015272      0.405609     0.00385356     0.0439213       100    
reco:pandoracalo:Calorimetry                          0.000101161    0.0239586     0.857364     0.00151001     0.100107        100    
reco:pandorapid:Chi2ParticleID                         4.055e-05    0.000197985   0.00112103    0.000157576   0.00016633       100    
reco:trkshowersplit:TrackShowerHits                   0.00011655     0.0319262      1.19434     0.000924475    0.134454        100    
reco:pmtrack:PMAlgTrackMaker                          0.00142603      2.91947       76.0928      0.252317       9.80109        100    
reco:pmtrackcalo:Calorimetry                          7.8511e-05     0.0163423     0.695147     0.00100453     0.0733287       100    
reco:pmtrackpid:Chi2ParticleID                         3.498e-05    0.000352893   0.00455187    0.000157506   0.00063706       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000220301      2.518        62.7769      0.429929       7.99875        100    
reco:pmtrajfitcalo:Calorimetry                        7.0471e-05     0.0220163      0.68832     0.00172243     0.0852903       100    
reco:pmtrajfitpid:Chi2ParticleID                       3.269e-05    0.000207672   0.00128014    0.000145876   0.000190141      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00106989      3.68979       102.165      0.330889       13.0835        100    
reco:pmtracktccalo:Calorimetry                        7.4451e-05    0.00904769     0.318847     0.00111603     0.0336593       100    
reco:pmtracktcpid:Chi2ParticleID                       3.635e-05    0.000304406   0.00385781    0.000161426   0.000467627      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000204841     1.89658       42.3471      0.395224       5.20269        100    
reco:pmtrajfittccalo:Calorimetry                      7.7241e-05     0.0188606     0.658242     0.00135546     0.0743669       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.358e-05    0.000405515   0.00555737    0.000161246   0.000723218      100    
reco:blurredcluster:BlurredClustering                  0.0254412     0.949967       6.6864       0.498291       1.30459        100    
reco:emtrkmichelid:EmTrackMichelId                     0.222215       10.7671       164.814       3.07045       22.9506        100    
reco:emshower:EMShower                                0.000272201    0.0388231     0.463889     0.00478293     0.0714012       100    
reco:cvnmap:CVNMapper                                  2.749e-05     0.0143076     0.0950793     0.0153211     0.0119216       100    
reco:cvneva:CVNEvaluator                               1.47e-05      0.744019       1.27721      0.895539      0.437796        100    
reco:energyrecnumu:EnergyReco                         0.000366582    0.065268       3.1918       0.0049209     0.326999        100    
reco:energyrecnue:EnergyReco                          0.000132701   0.00257654     0.0606076    0.000585724   0.00710851       100    
reco:energyrecnc:EnergyReco                           0.00169779    0.00513361     0.0646096    0.00344047    0.00736024       100    
reco:energyrecnumurange:EnergyReco                    0.000171551    0.0639033      3.23608     0.00305472     0.330939        100    
reco:energyrecnumumcs:EnergyReco                      0.000866295    0.0653523      3.24945      0.0042009     0.331649        100    
reco:opdec:Deconvolution                               0.103304      0.287426      0.499212      0.286749      0.0827868       100    
reco:ophitspe:OpHitFinderDeco                          0.293905       2.05712       4.42918       2.07368      0.863878        100    
reco:opflash:OpFlashFinder                            0.000317111   0.00168439    0.00636509    0.00158223    0.000886345      100    
reco:rns:RandomNumberSaver                             2.232e-05    4.87879e-05   0.000387232   4.38455e-05   3.57418e-05      100    
reco:anglereconue:NuAngularReco                       0.000166711   0.00188498     0.0328942    0.000550693   0.00443802       100    
reco:anglereconumu:NuAngularReco                       7.518e-05    0.00155598     0.0311859    0.000280292   0.00430317       100    
reco:anglereconuepfps:NuAngularReco                   0.000164961   0.00173707     0.0306005    0.000458337   0.00424671       100    
reco:anglereconumupfps:NuAngularReco                  0.000125411   0.00162794     0.0332599    0.000382267   0.00441805       100    
[art]:TriggerResults:TriggerResultInserter             8.73e-06     1.52322e-05   5.0251e-05     1.394e-05    5.90047e-06      100    
end_path:out1:RootOutput                               3.58e-06     8.17512e-06    2.684e-05     7.62e-06     3.39592e-06      100    
end_path:out1:RootOutput(write)                        0.022069      0.271896       1.21438      0.254522      0.197708        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21670.4 MB
  Peak resident set size usage (VmHWM): 2243.37 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_214_20231207T200439Z_gen_g4_detsim_hitreco__20240510T041641Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_214_20231207T200439Z_gen_g4_detsim_hitreco__20240510T041641Z_reco2_reco_2024-11-23T_050427Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_214_20231207T200439Z_gen_g4_detsim_hitreco__20240510T041641Z_reco2_reco_data_2024-11-23T_050427Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work