Jobsub ID 301750.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301750.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:13:43
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 06:22:34
From worker node	Hostname	b9p02p2167.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:15:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_432_20231205T100259Z_gen_g4_detsim_hitreco__20240510T035434Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4006s)
	CPU time	1h (4077s = 101%)
Outputting started	2024-11-23 06:22:15
Output files
Finished	2024-11-23 06:22:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

eation:pandoraShower@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
1st wire C:0 T:10 P:2 W:516 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:22:11 CET run: 50602933 subRun: 1 event: 43300
1st wire C:0 T:10 P:2 W:516 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:22:12 CET run: 50602933 subRun: 1 event: 43300
Comparing two wires in the same plane: return failure
%MSG
79
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 85!
23-Nov-2024 07:22:14 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_432_20231205T100259Z_gen_g4_detsim_hitreco__20240510T035434Z_reco2_reco_data_2024-11-23T_051534Z.root"
23-Nov-2024 07:22:14 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/bd/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_432_20231205T100259Z_gen_g4_detsim_hitreco__20240510T035434Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.43936        37.94        1080.43       9.57666       125.918        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0169999     0.0442425     0.0936082     0.0450694     0.0138808       100    
reco:linecluster:LineCluster                            0.14858       1.38423       90.7504       0.17662       9.27352        100    
reco:trajcluster:TrajCluster                           0.0476631      2.59164       176.543      0.063534       18.0012        100    
reco:pandora:StandardPandora                           0.996181       3.42768       77.6575       1.33324       10.6292        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000128341   0.00766132     0.221085     0.00174911     0.0253281       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000167581    0.0111101      0.10579     0.00360179     0.0189673       100    
reco:pandoracalo:Calorimetry                          9.2141e-05     0.0401869      1.79292     0.00115032     0.209781        100    
reco:pandorapid:Chi2ParticleID                         3.462e-05    0.000166917   0.00116181    0.000139792   0.000138343      100    
reco:trkshowersplit:TrackShowerHits                   0.000113422    0.0425456      2.1543      0.000798129     0.22891        100    
reco:pmtrack:PMAlgTrackMaker                          0.00108081      4.14585       173.86       0.318689       19.3246        100    
reco:pmtrackcalo:Calorimetry                          6.8691e-05     0.0169671     0.580113     0.00106484     0.0740645       100    
reco:pmtrackpid:Chi2ParticleID                         2.514e-05    0.000340742   0.00434385    0.000157322   0.000654215      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000194572     2.64231       56.0286      0.257788       8.25581        100    
reco:pmtrajfitcalo:Calorimetry                        5.9811e-05     0.0431186      1.77523     0.00194924     0.219279        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.504e-05    0.000197159   0.00107154    0.000152036   0.000162362      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000825538     5.28032       238.499      0.252946       26.2368        100    
reco:pmtracktccalo:Calorimetry                         6.752e-05     0.0102152     0.346185     0.000946925    0.0398017       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5781e-05    0.000301959   0.00452263    0.000148532   0.000545932      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000164681     2.28543       55.2411      0.315655       8.04898        100    
reco:pmtrajfittccalo:Calorimetry                      5.9341e-05     0.0271675     0.875711     0.00133204     0.115645        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.376e-05    0.000456091   0.00640007    0.000159042    0.0009368       100    
reco:blurredcluster:BlurredClustering                  0.0069052      1.09207       9.38046      0.380197       1.90148        100    
reco:emtrkmichelid:EmTrackMichelId                     0.176786       11.278         186.1        2.56202       27.6652        100    
reco:emshower:EMShower                                0.000246113    0.0289263     0.313848     0.00536782     0.0524113       100    
reco:cvnmap:CVNMapper                                  2.422e-05     0.0104087     0.0519926    0.00779175    0.00972887       100    
reco:cvneva:CVNEvaluator                               1.729e-05      0.73653       2.7916       0.897384      0.518175        100    
reco:energyrecnumu:EnergyReco                         0.000327834    0.0386714      1.06413     0.00493375     0.124921        100    
reco:energyrecnue:EnergyReco                          0.000131191   0.00256896     0.0698528    0.00053322     0.0083857       100    
reco:energyrecnc:EnergyReco                           0.00165474    0.00527314     0.0675256    0.00337417    0.00856435       100    
reco:energyrecnumurange:EnergyReco                    0.000160992    0.0359138      1.06003     0.00294087     0.123209        100    
reco:energyrecnumumcs:EnergyReco                      0.000883689    0.0376158      1.05951     0.00395979     0.123355        100    
reco:opdec:Deconvolution                               0.125499       0.29251      0.509135      0.288713      0.0836521       100    
reco:ophitspe:OpHitFinderDeco                          0.457012       2.11367       4.85561       2.12278      0.867655        100    
reco:opflash:OpFlashFinder                            0.000381894   0.00139899    0.00582677    0.00128556    0.000813631      100    
reco:rns:RandomNumberSaver                             1.918e-05    3.85702e-05   0.000418334   3.50705e-05   3.9481e-05       100    
reco:anglereconue:NuAngularReco                       0.000156712   0.00205552     0.0588544    0.000371214   0.00719369       100    
reco:anglereconumu:NuAngularReco                      6.9091e-05    0.00183781     0.0562667    0.000232748   0.00693203       100    
reco:anglereconuepfps:NuAngularReco                   0.000125162   0.00201854     0.0578706    0.000357653   0.00706444       100    
reco:anglereconumupfps:NuAngularReco                  9.3571e-05    0.00192132     0.057537     0.000294323   0.00706328       100    
[art]:TriggerResults:TriggerResultInserter             8.69e-06     1.38097e-05    5.124e-05     1.279e-05    5.36107e-06      100    
end_path:out1:RootOutput                               3.11e-06     6.79726e-06    3.503e-05     6.42e-06     3.86661e-06      100    
end_path:out1:RootOutput(write)                        0.0287439     0.254426        1.491       0.248072      0.228687        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21792.5 MB
  Peak resident set size usage (VmHWM): 2353.87 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_432_20231205T100259Z_gen_g4_detsim_hitreco__20240510T035434Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_432_20231205T100259Z_gen_g4_detsim_hitreco__20240510T035434Z_reco2_reco_2024-11-23T_051534Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_432_20231205T100259Z_gen_g4_detsim_hitreco__20240510T035434Z_reco2_reco_data_2024-11-23T_051534Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work