Jobsub ID 108743.34@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108743.34@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:34:30
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:12:10
From worker node	Hostname	b9p05p5884.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:36:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_450_20231208T003114Z_gen_g4_detsim_hitreco__20240510T031110Z_reco2.root
Jobscript	Exit code	0
	Real time	35m (2117s)
	CPU time	37m (2240s = 105%)
Outputting started	2024-11-23 05:11:53
Output files
Finished	2024-11-23 05:12:10
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ov-2024 06:11:38 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:11:38 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:11:38 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:11:38 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:11:47 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:11:47 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:11:47 CET run: 6470038 subRun: 1 event: 45100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 467, 408, 345
minwire 0: 887
minwire 1: 1185
minwire 2: 1064
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.122547, 0.496976, 0.380477, 
Output 1: 0.0880344, 0.678468, 0.220378, 0.0131191, 
Output 2: 0.207773, 0.792227, 

521
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1760!
23-Nov-2024 06:11:51 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_450_20231208T003114Z_gen_g4_detsim_hitreco__20240510T031110Z_reco2_reco_data_2024-11-23T_043640Z.root"
23-Nov-2024 06:11:51 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cd/24/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_450_20231208T003114Z_gen_g4_detsim_hitreco__20240510T031110Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.535356       19.173        244.376       8.65305       39.0136        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0163419     0.0386829     0.0751954     0.0383449     0.0111226       100    
reco:linecluster:LineCluster                           0.0938324      0.21868       1.72121       0.17325      0.218529        100    
reco:trajcluster:TrajCluster                           0.047326      0.371123       13.0458      0.0605053      1.58744        100    
reco:pandora:StandardPandora                           0.0561656      2.46362       58.8275       1.20232       8.04081        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000117752   0.00421725     0.0651099    0.00114917    0.00950336       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000154424    0.0117641     0.264869     0.00257524     0.0326709       100    
reco:pandoracalo:Calorimetry                          9.1102e-05    0.00952152     0.192166     0.000783202    0.0317195       100    
reco:pandorapid:Chi2ParticleID                        3.3311e-05    0.000151064   0.00158104    0.000114123   0.000171736      100    
reco:trkshowersplit:TrackShowerHits                   6.1501e-05     0.0106547     0.240041     0.000720025    0.0392777       100    
reco:pmtrack:PMAlgTrackMaker                           0.0010182      1.23749       20.644        0.1423        3.45035        100    
reco:pmtrackcalo:Calorimetry                          5.8251e-05    0.00683876     0.134948      0.0007144     0.0227016       100    
reco:pmtrackpid:Chi2ParticleID                        2.3981e-05    0.000205764   0.00170184    0.000134953   0.000272216      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000167764     1.15178       18.5104      0.234254       2.92596        100    
reco:pmtrajfitcalo:Calorimetry                        5.5321e-05     0.0112635     0.273222     0.00115677     0.0383079       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.345e-05    0.000161286   0.000867848   0.000134673   0.000132087      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000787337     1.67994       44.8099      0.216867       5.63092        100    
reco:pmtracktccalo:Calorimetry                        6.1241e-05    0.00688719      0.16067     0.000748926    0.0218986       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3641e-05    0.000222463   0.00191199    0.000135158   0.000316113      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000140903     1.06819       18.0626      0.215173       2.88172        100    
reco:pmtrajfittccalo:Calorimetry                      5.3121e-05    0.00897933     0.237422     0.00100698     0.0323874       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.264e-05    0.000273855   0.00325326    0.000148489   0.000432731      100    
reco:blurredcluster:BlurredClustering                 9.1512e-05     0.823306       7.87356      0.433903       1.17724        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000124623     6.90781       79.9242       2.28339       14.2073        100    
reco:emshower:EMShower                                0.000170203    0.0203243     0.325342     0.00458895     0.0400998       100    
reco:cvnmap:CVNMapper                                  1.894e-05    0.00763663     0.0314751     0.0057547    0.00685146       100    
reco:cvneva:CVNEvaluator                               1.495e-05     0.611527       1.09069      0.754602       0.38321        100    
reco:energyrecnumu:EnergyReco                         0.000295606    0.0524596      1.74126     0.00329753     0.216574        100    
reco:energyrecnue:EnergyReco                          0.000106583   0.00156759     0.0188978    0.000427849   0.00318796       100    
reco:energyrecnc:EnergyReco                           0.00143777    0.00373058     0.0220264    0.00307573     0.0033703       100    
reco:energyrecnumurange:EnergyReco                    0.000150004    0.0492788      1.39854      0.0026278     0.203101        100    
reco:energyrecnumumcs:EnergyReco                      0.000552492    0.0545782      1.97694     0.00337059     0.235479        100    
reco:opdec:Deconvolution                               0.0575956     0.267338      0.551248      0.267323      0.0813504       100    
reco:ophitspe:OpHitFinderDeco                          0.142409       1.85828       3.81961       1.82485      0.777364        100    
reco:opflash:OpFlashFinder                            0.000230215   0.00129408    0.00677625    0.00120706    0.00075204       100    
reco:rns:RandomNumberSaver                            1.8461e-05    3.81171e-05   0.000409258   3.41255e-05   3.88719e-05      100    
reco:anglereconue:NuAngularReco                       0.000140483   0.00105218     0.0160479    0.000364888   0.00237066       100    
reco:anglereconumu:NuAngularReco                      6.9401e-05    0.000856992    0.0156315    0.000203849   0.00229949       100    
reco:anglereconuepfps:NuAngularReco                   0.000106232   0.00100055     0.0158503    0.000339532   0.00233504       100    
reco:anglereconumupfps:NuAngularReco                  8.8672e-05    0.000934426    0.0158058    0.00026211    0.00234387       100    
[art]:TriggerResults:TriggerResultInserter             8.79e-06     1.34537e-05   4.6091e-05     1.245e-05    4.78544e-06      100    
end_path:out1:RootOutput                               2.53e-06     6.65348e-06    2.224e-05    6.6305e-06    2.91267e-06      100    
end_path:out1:RootOutput(write)                        0.0140205      0.20754      0.903389      0.207293      0.138034        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21620.2 MB
  Peak resident set size usage (VmHWM): 2200.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_450_20231208T003114Z_gen_g4_detsim_hitreco__20240510T031110Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_450_20231208T003114Z_gen_g4_detsim_hitreco__20240510T031110Z_reco2_reco_2024-11-23T_043640Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_450_20231208T003114Z_gen_g4_detsim_hitreco__20240510T031110Z_reco2_reco_data_2024-11-23T_043640Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work