Jobsub ID 301743.4@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301743.4@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:55:33
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:40:49
From worker node	Hostname	b9p11p6612.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:57:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_4_20231207T135421Z_gen_g4_detsim_hitreco__20240510T061420Z_reco2.root
Jobscript	Exit code	0
	Real time	43m (2608s)
	CPU time	44m (2685s = 102%)
Outputting started	2024-11-23 05:40:31
Output files
Finished	2024-11-23 05:40:49
List job events Wrapper job log
Jobscript log (last 10,000 characters)

le  23-Nov-2024 06:40:22 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:22 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:22 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:22 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:22 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:22 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:40:25 CET run: 6457551 subRun: 1 event: 500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 76, 72, 69
minwire 0: 910
minwire 1: 1348
minwire 2: 984
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.973101, 0.01028, 0.0166187, 
Output 1: 0.0553764, 0.29162, 0.625065, 0.0279392, 
Output 2: 0.969282, 0.0307183, 

451
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1387!
23-Nov-2024 06:40:29 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_4_20231207T135421Z_gen_g4_detsim_hitreco__20240510T061420Z_reco2_reco_data_2024-11-23T_045708Z.root"
23-Nov-2024 06:40:29 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/6f/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_4_20231207T135421Z_gen_g4_detsim_hitreco__20240510T061420Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.58745       24.085        274.493       8.2465        48.2165        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0164815     0.0511896     0.708928      0.0416571     0.068125        100    
reco:linecluster:LineCluster                            0.09428      0.298664       4.52305      0.167836       0.58474        100    
reco:trajcluster:TrajCluster                            0.04731      0.595406       12.0518      0.0580262      1.97907        100    
reco:pandora:StandardPandora                           0.0583995      2.26444       57.4797       1.20817       6.02263        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000121341   0.00561954     0.0903258    0.00112991     0.0130152       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000160282    0.0131111     0.329283     0.00242363     0.037026        100    
reco:pandoracalo:Calorimetry                          9.7631e-05     0.0205622       0.433      0.000824962    0.0675365       100    
reco:pandorapid:Chi2ParticleID                         3.496e-05    0.000157251   0.00119574    0.000118717   0.000152561      100    
reco:trkshowersplit:TrackShowerHits                   8.7461e-05     0.0214729      0.37551     0.000580505    0.0695701       100    
reco:pmtrack:PMAlgTrackMaker                          0.00102158      2.14041       35.7636      0.128735       6.02511        100    
reco:pmtrackcalo:Calorimetry                          6.2661e-05     0.0089013     0.196065     0.000668047    0.0275607       100    
reco:pmtrackpid:Chi2ParticleID                         2.456e-05    0.00023533    0.00224519    0.000122871   0.000358262      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000167732     1.70519       37.5273      0.165151       4.78988        100    
reco:pmtrajfitcalo:Calorimetry                         5.68e-05      0.0172248     0.341696     0.000997134    0.0541714       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.397e-05    0.000152516   0.000566675   0.000115726   0.00011272       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000850048     2.76319       42.8358      0.171518       8.0437         100    
reco:pmtracktccalo:Calorimetry                        5.8361e-05    0.00635362     0.127155     0.000720046    0.018582        100    
reco:pmtracktcpid:Chi2ParticleID                       2.592e-05    0.000241374   0.00175639    0.000127086   0.000340758      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000171141     1.36321       16.4996      0.187691       3.12947        100    
reco:pmtrajfittccalo:Calorimetry                       5.976e-05     0.0131801     0.244285     0.000818348    0.0400527       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.432e-05    0.00031845    0.00289034    0.000129161   0.000540125      100    
reco:blurredcluster:BlurredClustering                 0.000116461    0.879411       9.19313      0.333758       1.40136        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000116721     8.9479        98.8517       2.12064       18.0273        100    
reco:emshower:EMShower                                0.000177532    0.0261956     0.260315     0.00504725     0.0488341       100    
reco:cvnmap:CVNMapper                                  2.057e-05     0.0102903     0.0403857     0.0122156    0.00881421       100    
reco:cvneva:CVNEvaluator                               1.491e-05      0.59295       1.19075      0.771593      0.419675        100    
reco:energyrecnumu:EnergyReco                         0.000242232    0.0330243      1.28531     0.00349975     0.134874        100    
reco:energyrecnue:EnergyReco                          0.000116701   0.00210107     0.0293239    0.000451274   0.00442005       100    
reco:energyrecnc:EnergyReco                            0.0015463    0.00436778     0.0347143    0.00297965     0.0049301       100    
reco:energyrecnumurange:EnergyReco                    0.000128281    0.0301509      1.22502     0.00218819     0.128804        100    
reco:energyrecnumumcs:EnergyReco                      0.000964179    0.0306681      1.18174     0.00343789     0.124347        100    
reco:opdec:Deconvolution                               0.065593      0.263454      0.480409       0.26361      0.0822643       100    
reco:ophitspe:OpHitFinderDeco                          0.183417       1.74341       3.82143       1.71514      0.751056        100    
reco:opflash:OpFlashFinder                            0.000315123   0.00138432     0.0033085    0.00131327    0.000577339      100    
reco:rns:RandomNumberSaver                             1.514e-05    3.34962e-05   0.000274373   3.0385e-05    2.54408e-05      100    
reco:anglereconue:NuAngularReco                       0.000142081   0.00138486     0.0228562    0.000371149   0.00314574       100    
reco:anglereconumu:NuAngularReco                      7.1241e-05    0.00115737     0.0212595    0.000182006   0.00297378       100    
reco:anglereconuepfps:NuAngularReco                   0.000115151   0.00131099     0.0199821    0.000332453    0.0029816       100    
reco:anglereconumupfps:NuAngularReco                  0.000100031   0.00126549     0.0184946    0.000248232   0.00304731       100    
[art]:TriggerResults:TriggerResultInserter             8.03e-06     1.27048e-05    5.011e-05    1.1965e-05    4.94199e-06      100    
end_path:out1:RootOutput                               2.38e-06     5.61275e-06    2.108e-05     5.275e-06    2.50573e-06      100    
end_path:out1:RootOutput(write)                        0.0128008     0.223556      0.801713      0.219011      0.170993        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21820.3 MB
  Peak resident set size usage (VmHWM): 2392.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_4_20231207T135421Z_gen_g4_detsim_hitreco__20240510T061420Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_4_20231207T135421Z_gen_g4_detsim_hitreco__20240510T061420Z_reco2_reco_2024-11-23T_045708Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_4_20231207T135421Z_gen_g4_detsim_hitreco__20240510T061420Z_reco2_reco_data_2024-11-23T_045708Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work