Jobsub ID 108827.72@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108827.72@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:25:34
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 06:10:20
From worker node	Hostname	b9p05p9589.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:27:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_500_20231208T100201Z_gen_g4_detsim_hitreco__20240510T064945Z_reco2.root
Jobscript	Exit code	0
	Real time	42m (2565s)
	CPU time	43m (2589s = 100%)
Outputting started	2024-11-23 06:10:00
Output files
Finished	2024-11-23 06:10:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

00
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:09:42 CET run: 6472938 subRun: 1 event: 50100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:09:42 CET run: 6472938 subRun: 1 event: 50100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:09:42 CET run: 6472938 subRun: 1 event: 50100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:09:42 CET run: 6472938 subRun: 1 event: 50100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:09:42 CET run: 6472938 subRun: 1 event: 50100
1st wire C:0 T:11 P:2 W:715 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:09:43 CET run: 6472938 subRun: 1 event: 50100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 582, 604, 585
minwire 0: 1138
minwire 1: 1403
minwire 2: 955
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0105301, 0.989342, 0.000128146, 
Output 1: 0.995578, 0.00431179, 9.15822e-05, 1.86472e-05, 
Output 2: 0.996554, 0.00344582, 

628
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2349!
23-Nov-2024 07:09:59 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_500_20231208T100201Z_gen_g4_detsim_hitreco__20240510T064945Z_reco2_reco_data_2024-11-23T_052721Z.root"
23-Nov-2024 07:09:59 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/e0/72/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_500_20231208T100201Z_gen_g4_detsim_hitreco__20240510T064945Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.10644       23.6049       405.502       8.57713       50.368         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0169832     0.043916      0.317716      0.0420187     0.0305233       100    
reco:linecluster:LineCluster                           0.149166      0.343328       9.87034      0.172465       1.05006        100    
reco:trajcluster:TrajCluster                           0.0475937     0.575768       24.6363      0.0587231      2.80395        100    
reco:pandora:StandardPandora                           0.826593       1.86301       30.2381       1.2883        3.10064        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000127093   0.00553139     0.0899472    0.00133554     0.012724        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000183024    0.0200586     0.363285     0.00265425     0.0630064       100    
reco:pandoracalo:Calorimetry                          0.000101502    0.0136755     0.365127     0.000981192    0.0467349       100    
reco:pandorapid:Chi2ParticleID                        3.4801e-05    0.000175631   0.000619044   0.000144584    0.0001248       100    
reco:trkshowersplit:TrackShowerHits                   0.000117923    0.0195624     0.611217     0.000573693    0.0820426       100    
reco:pmtrack:PMAlgTrackMaker                          0.00109235      1.98754       55.3189      0.180305       6.61073        100    
reco:pmtrackcalo:Calorimetry                          6.9072e-05    0.00976614     0.363005     0.000795667    0.0401312       100    
reco:pmtrackpid:Chi2ParticleID                        2.6201e-05    0.000279827   0.00410752    0.000145349   0.000477208      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000175414     1.61443       28.574       0.193742       4.08037        100    
reco:pmtrajfitcalo:Calorimetry                        6.2602e-05     0.0145821     0.413216     0.00129366     0.0510589       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4491e-05    0.000195063   0.00106337    0.000155694   0.000169404      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000797958     2.5149        98.4622      0.219554       10.6985        100    
reco:pmtracktccalo:Calorimetry                        5.9481e-05    0.00728597     0.142641     0.000857014    0.0230035       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4861e-05    0.000259455   0.00240643    0.000147068   0.000359142      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000156524     1.45461       26.6774      0.261309       3.88168        100    
reco:pmtrajfittccalo:Calorimetry                      5.9352e-05     0.0111384     0.411025     0.00104475     0.044958        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3581e-05    0.000324586   0.00391354    0.000164903   0.000495149      100    
reco:blurredcluster:BlurredClustering                 0.00522262      1.08956       8.59984      0.455627       1.53301        100    
reco:emtrkmichelid:EmTrackMichelId                     0.163513       8.49048       107.115       2.0563        16.9273        100    
reco:emshower:EMShower                                0.000272616    0.0295397     0.470572     0.00479275     0.066613        100    
reco:cvnmap:CVNMapper                                  2.708e-05     0.0136567     0.0659044     0.0153141     0.0110975       100    
reco:cvneva:CVNEvaluator                              1.7131e-05     0.841578       7.53675      0.967748      0.905115        100    
reco:energyrecnumu:EnergyReco                         0.000381228    0.0231572     0.773816     0.00432309     0.0857338       100    
reco:energyrecnue:EnergyReco                          0.000124003   0.00198877     0.0337043    0.000442335   0.00498625       100    
reco:energyrecnc:EnergyReco                           0.00172071    0.00460951     0.0372938    0.00336328    0.00521146       100    
reco:energyrecnumurange:EnergyReco                    0.000181334    0.0203653      0.75451     0.00283818     0.0838109       100    
reco:energyrecnumumcs:EnergyReco                      0.000726526    0.0221842     0.758262     0.00357345     0.0839198       100    
reco:opdec:Deconvolution                               0.082664      0.278428      0.482944       0.27097      0.0860424       100    
reco:ophitspe:OpHitFinderDeco                          0.349957       2.03241       4.21161       1.88622      0.890941        100    
reco:opflash:OpFlashFinder                            0.000345637   0.00148154     0.0044342    0.00131058    0.000714623      100    
reco:rns:RandomNumberSaver                             2.273e-05    4.32569e-05   0.000337848   4.0761e-05    3.08106e-05      100    
reco:anglereconue:NuAngularReco                       0.000178354   0.00135212     0.0272523    0.000427714   0.00342418       100    
reco:anglereconumu:NuAngularReco                      7.9582e-05    0.00113989     0.0273955    0.00021952    0.00343035       100    
reco:anglereconuepfps:NuAngularReco                   0.000144573   0.00131455     0.0272266    0.000377058   0.00340477       100    
reco:anglereconumupfps:NuAngularReco                  0.000113232   0.00118977     0.0266442    0.000274251    0.0033312       100    
[art]:TriggerResults:TriggerResultInserter             8.92e-06     1.43244e-05   5.1962e-05    1.36755e-05   5.32341e-06      100    
end_path:out1:RootOutput                               3.271e-06    7.75625e-06   2.4851e-05    7.5555e-06    3.05975e-06      100    
end_path:out1:RootOutput(write)                        0.0193375     0.247825      0.840861      0.246209      0.162762        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21956.5 MB
  Peak resident set size usage (VmHWM): 2535.04 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_500_20231208T100201Z_gen_g4_detsim_hitreco__20240510T064945Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_500_20231208T100201Z_gen_g4_detsim_hitreco__20240510T064945Z_reco2_reco_2024-11-23T_052721Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_500_20231208T100201Z_gen_g4_detsim_hitreco__20240510T064945Z_reco2_reco_data_2024-11-23T_052721Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work