Jobsub ID 108829.61@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108829.61@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:32:18
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 06:29:27
From worker node	Hostname	b9p09p9768.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:34:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1024_20231203T071307Z_gen_g4_detsim_hitreco__20240508T072823Z_reco2.root
Jobscript	Exit code	0
	Real time	54m (3281s)
	CPU time	56m (3370s = 102%)
Outputting started	2024-11-23 06:29:09
Output files
Finished	2024-11-23 06:29:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

MAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:29:00 CET run: 50572032 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:29:00 CET run: 50572032 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:29:00 CET run: 50572032 subRun: 1 event: 102500
1st wire C:0 T:5 P:2 W:731 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:29:00 CET run: 50572032 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:29:00 CET run: 50572032 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:29:03 CET run: 50572032 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 145, 76, 107
minwire 0: 636
minwire 1: 1687
minwire 2: 34
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 603, 802
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.224402, 0.00280223, 0.772796, 
Output 1: 0.991415, 0.00810022, 0.00042743, 5.72881e-05, 
Output 2: 0.804776, 0.195224, 

522
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1723!
23-Nov-2024 07:29:08 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1024_20231203T071307Z_gen_g4_detsim_hitreco__20240508T072823Z_reco2_reco_data_2024-11-23T_053434Z.root"
23-Nov-2024 07:29:08 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f1/75/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1024_20231203T071307Z_gen_g4_detsim_hitreco__20240508T072823Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.522485       30.7909       619.933       7.71092       87.222         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0171642     0.0413129     0.115049      0.0408554     0.014821        100    
reco:linecluster:LineCluster                           0.0932948     0.696066       31.0912      0.174084       3.41682        100    
reco:trajcluster:TrajCluster                           0.0473468      1.24958       55.1711      0.0580542      6.20939        100    
reco:pandora:StandardPandora                           0.0571302      3.58966       118.144       1.2122        13.4879        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000123611   0.00648384     0.113714     0.00122776     0.0176375       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000156871    0.0105083     0.111452     0.00244469     0.019687        100    
reco:pandoracalo:Calorimetry                          8.8121e-05     0.0348662      1.37061     0.000779401    0.169131        100    
reco:pandorapid:Chi2ParticleID                         3.351e-05    0.000163474   0.00126781    0.000128386   0.000154425      100    
reco:trkshowersplit:TrackShowerHits                   7.5201e-05     0.0330532      1.03887     0.000631835    0.144236        100    
reco:pmtrack:PMAlgTrackMaker                          0.000938437     3.21141       93.5495      0.165057       13.291         100    
reco:pmtrackcalo:Calorimetry                          6.0071e-05     0.0118096     0.410596     0.000816096    0.0487579       100    
reco:pmtrackpid:Chi2ParticleID                         2.53e-05     0.000281517    0.0042998    0.000136146   0.000538094      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000168301     2.31989       81.9194      0.184969       9.13517        100    
reco:pmtrajfitcalo:Calorimetry                        5.8711e-05     0.0331603      1.24899     0.000977177    0.155945        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.727e-05    0.000172615   0.000931568   0.000134701   0.000144326      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000804706     3.43039       85.7092      0.138678       11.9919        100    
reco:pmtracktccalo:Calorimetry                        6.1221e-05    0.00741675     0.192793     0.000801181    0.0237997       100    
reco:pmtracktcpid:Chi2ParticleID                       2.444e-05    0.000254869   0.00221066    0.000133851   0.000382467      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000154471     2.07004       46.9777      0.203835       6.87746        100    
reco:pmtrajfittccalo:Calorimetry                      5.6521e-05     0.0197786     0.632853     0.000825962    0.0812696       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4571e-05    0.000344943   0.00432442    0.000134661   0.000633442      100    
reco:blurredcluster:BlurredClustering                 0.000110601    0.902266       7.09166       0.42944       1.3198         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000122961     10.1478       156.711       1.80942       23.9296        100    
reco:emshower:EMShower                                0.000173332    0.0284624     0.590708      0.0050477     0.0674869       100    
reco:cvnmap:CVNMapper                                  2.086e-05     0.0100514     0.0452922     0.0111012    0.00944588       100    
reco:cvneva:CVNEvaluator                               1.564e-05     0.573267       1.23355      0.767157       0.42265        100    
reco:energyrecnumu:EnergyReco                         0.000273632    0.0225652     0.638561     0.00345741     0.0720832       100    
reco:energyrecnue:EnergyReco                          0.000119481   0.00231609     0.0534273    0.000428074   0.00625234       100    
reco:energyrecnc:EnergyReco                           0.00146336    0.00433464     0.0560809    0.00302157    0.00643146       100    
reco:energyrecnumurange:EnergyReco                    0.000144081    0.0197878     0.541478     0.00212544     0.0636459       100    
reco:energyrecnumumcs:EnergyReco                      0.000542415    0.0219584     0.579711     0.00324192     0.067967        100    
reco:opdec:Deconvolution                               0.0480155     0.265881      0.474627      0.267527      0.0917773       100    
reco:ophitspe:OpHitFinderDeco                          0.141632       1.79509       4.56142       1.70703      0.894157        100    
reco:opflash:OpFlashFinder                            0.000158581    0.0013746    0.00570923     0.0013187    0.000786968      100    
reco:rns:RandomNumberSaver                            1.6411e-05    4.27142e-05   0.000533594   3.80055e-05   5.09344e-05      100    
reco:anglereconue:NuAngularReco                       0.000141532   0.00173695     0.0406478    0.000386408   0.00510965       100    
reco:anglereconumu:NuAngularReco                       6.579e-05    0.00148632     0.0385673    0.000191501   0.00490976       100    
reco:anglereconuepfps:NuAngularReco                   0.000103501   0.00161955     0.0384151    0.000330653   0.00488202       100    
reco:anglereconumupfps:NuAngularReco                  9.0981e-05    0.00155564     0.0380959    0.000245072   0.00489448       100    
[art]:TriggerResults:TriggerResultInserter             7.63e-06     1.37617e-05   5.5231e-05     1.272e-05    5.53633e-06      100    
end_path:out1:RootOutput                               2.39e-06     6.46134e-06    2.005e-05     6.215e-06    2.68049e-06      100    
end_path:out1:RootOutput(write)                        0.0112846     0.220752       1.21823      0.200335      0.192907        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21852.5 MB
  Peak resident set size usage (VmHWM): 2429.01 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1024_20231203T071307Z_gen_g4_detsim_hitreco__20240508T072823Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1024_20231203T071307Z_gen_g4_detsim_hitreco__20240508T072823Z_reco2_reco_2024-11-23T_053434Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1024_20231203T071307Z_gen_g4_detsim_hitreco__20240508T072823Z_reco2_reco_data_2024-11-23T_053434Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work