Jobsub ID 301709.15@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301709.15@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:37:41
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 05:23:53
From worker node	Hostname	b9p30p0404.cern.ch
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:38:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_299_20231205T083942Z_gen_g4_detsim_hitreco__20240510T033340Z_reco2.root
Jobscript	Exit code	0
	Real time	44m (2683s)
	CPU time	42m (2573s = 95%)
Outputting started	2024-11-23 05:23:34
Output files
Finished	2024-11-23 05:23:53
List job events Wrapper job log
Jobscript log (last 10,000 characters)

in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:23:25 CET run: 50602933 subRun: 1 event: 30000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:23:25 CET run: 50602933 subRun: 1 event: 30000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:23:25 CET run: 50602933 subRun: 1 event: 30000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:23:25 CET run: 50602933 subRun: 1 event: 30000
1st wire C:0 T:11 P:2 W:531 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:23:25 CET run: 50602933 subRun: 1 event: 30000
1st wire C:0 T:11 P:2 W:489 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:23:26 CET run: 50602933 subRun: 1 event: 30000
1st wire C:0 T:11 P:2 W:530 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 69, 129, 61
minwire 0: 536
minwire 1: 1958
minwire 2: 963
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0134046, 0.000669519, 0.985926, 
Output 1: 0.00501428, 0.991201, 0.00376089, 2.38084e-05, 
Output 2: 0.988843, 0.0111573, 

545
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1880!
23-Nov-2024 06:23:33 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_299_20231205T083942Z_gen_g4_detsim_hitreco__20240510T033340Z_reco2_reco_data_2024-11-23T_043856Z.root"
23-Nov-2024 06:23:33 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/4c/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_299_20231205T083942Z_gen_g4_detsim_hitreco__20240510T033340Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.27916       24.8717       383.957       9.73113       45.1452        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0174144     0.039521      0.0759065     0.0386617     0.0107893       100    
reco:linecluster:LineCluster                            0.14912      0.264567       4.53899       0.17716       0.45921        100    
reco:trajcluster:TrajCluster                           0.0478838     0.556834       19.5498      0.0585348      2.08637        100    
reco:pandora:StandardPandora                           0.906189       1.95409       36.356        1.44921       3.52268        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000154933    0.0052999     0.0941326    0.00129486     0.0113787       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000222264    0.0171294     0.385409     0.00300249     0.0513271       100    
reco:pandoracalo:Calorimetry                          0.000105472    0.0130906     0.423391     0.000919195    0.0451751       100    
reco:pandorapid:Chi2ParticleID                         3.981e-05    0.000157938   0.00100728    0.000121652   0.000135787      100    
reco:trkshowersplit:TrackShowerHits                   0.000118652    0.0155183     0.445867     0.000698302    0.0507962       100    
reco:pmtrack:PMAlgTrackMaker                          0.00123438      2.09287       53.9954      0.241367       6.02407        100    
reco:pmtrackcalo:Calorimetry                          5.8481e-05    0.00946027     0.363646     0.000830445    0.0376212       100    
reco:pmtrackpid:Chi2ParticleID                         2.441e-05    0.000277381   0.00323417    0.000122832   0.000421822      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000220664     1.77305       29.392       0.344615       3.81999        100    
reco:pmtrajfitcalo:Calorimetry                        6.8211e-05     0.0133813      0.42195     0.00121807     0.0453447       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7991e-05    0.00017382    0.00131511    0.000133202   0.000166719      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0011789      2.21587       53.192       0.179549       6.30903        100    
reco:pmtracktccalo:Calorimetry                        6.9271e-05    0.00671118     0.185946     0.000798724    0.0205304       100    
reco:pmtracktcpid:Chi2ParticleID                      2.8641e-05    0.000257009   0.00303266    0.000129587   0.000360321      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000201564     1.35049       25.9082      0.224435       3.08332        100    
reco:pmtrajfittccalo:Calorimetry                      6.8711e-05     0.0130633     0.539978     0.000975812    0.0554848       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3461e-05    0.000345763    0.0050866    0.000156798   0.000594383      100    
reco:blurredcluster:BlurredClustering                 0.00801759      1.02345       8.79582      0.480905       1.32283        100    
reco:emtrkmichelid:EmTrackMichelId                     0.276153       10.5098       143.335       2.68667       19.0132        100    
reco:emshower:EMShower                                0.000301035    0.0301633     0.262696     0.00871013     0.054255        100    
reco:cvnmap:CVNMapper                                 2.3811e-05     0.0131553     0.0521879     0.0159974     0.0101733       100    
reco:cvneva:CVNEvaluator                              1.6371e-05     0.673568       4.73931      0.781306      0.585367        100    
reco:energyrecnumu:EnergyReco                         0.000348286    0.0275813     0.399542     0.00379826     0.0634943       100    
reco:energyrecnue:EnergyReco                          0.000122132   0.00167412     0.0301263    0.000465033   0.00383447       100    
reco:energyrecnc:EnergyReco                           0.00185924    0.00402673     0.0349473     0.0030474    0.00424007       100    
reco:energyrecnumurange:EnergyReco                    0.000128572    0.0254803     0.380421     0.00256442     0.0622464       100    
reco:energyrecnumumcs:EnergyReco                      0.00101343     0.0268991     0.384233     0.00329114     0.0625543       100    
reco:opdec:Deconvolution                               0.0811097     0.294023      0.470518       0.30255      0.0869174       100    
reco:ophitspe:OpHitFinderDeco                          0.267996       1.65345       3.24757       1.64307      0.675503        100    
reco:opflash:OpFlashFinder                            0.000328535    0.0014622    0.00401791    0.00142485    0.000649164      100    
reco:rns:RandomNumberSaver                            1.7681e-05    3.86793e-05   0.000241514   3.5726e-05    2.26408e-05      100    
reco:anglereconue:NuAngularReco                       0.000151743   0.00127693     0.0249148    0.000419822   0.00294484       100    
reco:anglereconumu:NuAngularReco                      6.5781e-05    0.00106914     0.0244789    0.000196783   0.00290953       100    
reco:anglereconuepfps:NuAngularReco                   0.000123952   0.00120761     0.0248991    0.000350846    0.0029265       100    
reco:anglereconumupfps:NuAngularReco                  9.3952e-05    0.00112146     0.024679     0.00025839    0.00291821       100    
[art]:TriggerResults:TriggerResultInserter             8.15e-06     1.29118e-05   3.8741e-05    1.20055e-05   4.65286e-06      100    
end_path:out1:RootOutput                                2.8e-06     6.59552e-06   4.7261e-05     6.005e-06    4.78718e-06      100    
end_path:out1:RootOutput(write)                        0.0181152     0.237928      0.924032      0.238925       0.15315        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 13615 MB
  Peak resident set size usage (VmHWM): 2944.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_299_20231205T083942Z_gen_g4_detsim_hitreco__20240510T033340Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_299_20231205T083942Z_gen_g4_detsim_hitreco__20240510T033340Z_reco2_reco_2024-11-23T_043856Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_299_20231205T083942Z_gen_g4_detsim_hitreco__20240510T033340Z_reco2_reco_data_2024-11-23T_043856Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work