Jobsub ID 301666.56@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301666.56@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:09:01
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 04:55:23
From worker node	Hostname	td810.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:11:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_163_20231203T091620Z_gen_g4_detsim_hitreco__20240509T193711Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6224s)
	CPU time	1h (5744s = 92%)
Outputting started	2024-11-23 04:55:05
Output files
Finished	2024-11-23 04:55:23
List job events Wrapper job log
Jobscript log (last 10,000 characters)

un: 50574782 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:54:47 CET run: 50574782 subRun: 1 event: 16400
Need min. 2 hits per view, at least two views.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:54:47 CET run: 50574782 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:54:47 CET run: 50574782 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:54:47 CET run: 50574782 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:48 CET run: 50574782 subRun: 1 event: 16400
1st wire C:0 T:22 P:2 W:808 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:49 CET run: 50574782 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 355, 496, 132
minwire 0: 2024
minwire 1: 91
minwire 2: 1819
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00163504, 6.17441e-05, 0.998303, 
Output 1: 0.630358, 0.295369, 0.0604083, 0.0138651, 
Output 2: 0.986304, 0.0136955, 

522
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1848!
23-Nov-2024 05:55:03 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_163_20231203T091620Z_gen_g4_detsim_hitreco__20240509T193711Z_reco2_reco_data_2024-11-23T_031124Z.root"
23-Nov-2024 05:55:03 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/9f/84/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_163_20231203T091620Z_gen_g4_detsim_hitreco__20240509T193711Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.11054       58.1865       752.631       18.5737       116.176        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0332272     0.0582169     0.0924622     0.0597516     0.0184385       100    
reco:linecluster:LineCluster                           0.192184       1.33956       60.3913       0.34359       6.22104        100    
reco:trajcluster:TrajCluster                           0.0964571      3.69807       135.516      0.138741       16.2299        100    
reco:pandora:StandardPandora                           0.107901       3.79035       45.0931       1.83277       7.10979        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000138382    0.0138232     0.250294     0.00366992     0.0325083       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000191071    0.0290945     0.364524     0.00673775     0.057681        100    
reco:pandoracalo:Calorimetry                          0.000108431    0.046344       1.14207     0.00354518     0.151188        100    
reco:pandorapid:Chi2ParticleID                         4.325e-05    0.000252375    0.0017818    0.000193382    0.0002522       100    
reco:trkshowersplit:TrackShowerHits                   8.0251e-05     0.0622472      1.27392     0.00234431      0.19699        100    
reco:pmtrack:PMAlgTrackMaker                          0.00134893      5.87258       128.791      0.718893       16.8533        100    
reco:pmtrackcalo:Calorimetry                          6.9861e-05     0.0243091     0.470045     0.00284911     0.0700901       100    
reco:pmtrackpid:Chi2ParticleID                        3.1791e-05    0.000521295   0.00432356    0.000222567   0.000775444      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000206933     4.8469        60.145       0.959976       10.7777        100    
reco:pmtrajfitcalo:Calorimetry                         6.784e-05     0.0477197      1.00165     0.00397566     0.151258        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.877e-05    0.000294987   0.00185989    0.000221677   0.000274086      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00116958      6.68204       131.395      0.744071       19.2385        100    
reco:pmtracktccalo:Calorimetry                        7.1481e-05     0.0162378     0.262571     0.00251991     0.0386474       100    
reco:pmtracktcpid:Chi2ParticleID                       2.923e-05    0.000451262   0.00280081    0.000233103   0.000568528      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000167271     3.63515       40.3439      0.819443       7.22263        100    
reco:pmtrajfittccalo:Calorimetry                      6.7291e-05     0.0386857     0.666434     0.00315631     0.114357        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.882e-05    0.000689977   0.00589584    0.000236473   0.00108389       100    
reco:blurredcluster:BlurredClustering                 0.000122381     1.73131       13.1783       1.04277       2.19882        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000139762     21.584        189.386       6.67839       36.7203        100    
reco:emshower:EMShower                                0.000212612    0.0447574     0.308452      0.0157491     0.0695184       100    
reco:cvnmap:CVNMapper                                  2.477e-05     0.0129398     0.0426909     0.0117419    0.00992786       100    
reco:cvneva:CVNEvaluator                              1.8871e-05     0.814648       1.63723      0.922831      0.432587        100    
reco:energyrecnumu:EnergyReco                         0.000563865    0.0764852      1.76843     0.00866261     0.232672        100    
reco:energyrecnue:EnergyReco                          0.000177542   0.00441634     0.0709399    0.00114296    0.00963572       100    
reco:energyrecnc:EnergyReco                           0.00237524    0.00745421     0.0729781    0.00423155     0.0100207       100    
reco:energyrecnumurange:EnergyReco                    0.000347893    0.0735801      1.69242     0.00575589     0.231273        100    
reco:energyrecnumumcs:EnergyReco                      0.00259808      0.07532       1.72839     0.00728742     0.233614        100    
reco:opdec:Deconvolution                               0.199542      0.505668      0.791003       0.51969      0.133867        100    
reco:ophitspe:OpHitFinderDeco                          0.290459       2.63745       5.16809       2.63705       1.01219        100    
reco:opflash:OpFlashFinder                            0.000166001   0.00195683    0.00581175    0.00188292     0.0009211       100    
reco:rns:RandomNumberSaver                             2.019e-05    4.12416e-05   0.000403414   3.59455e-05   3.86622e-05      100    
reco:anglereconue:NuAngularReco                       0.000191602    0.0036181     0.0706205    0.000669016   0.00928909       100    
reco:anglereconumu:NuAngularReco                      7.7431e-05    0.00332368     0.0656428    0.000444814   0.00889548       100    
reco:anglereconuepfps:NuAngularReco                   0.000130861   0.00350433     0.0646567    0.00059683     0.0089189       100    
reco:anglereconumupfps:NuAngularReco                  0.000110951    0.0034285     0.0650988    0.000573905   0.00890811       100    
[art]:TriggerResults:TriggerResultInserter             9.78e-06     1.60461e-05   8.4951e-05    1.4536e-05    9.11979e-06      100    
end_path:out1:RootOutput                               3.32e-06     7.06193e-06    4.013e-05     4.79e-06     5.1862e-06       100    
end_path:out1:RootOutput(write)                        0.0155884     0.396496       1.64071      0.360405      0.290015        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22165.1 MB
  Peak resident set size usage (VmHWM): 2877.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_163_20231203T091620Z_gen_g4_detsim_hitreco__20240509T193711Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_163_20231203T091620Z_gen_g4_detsim_hitreco__20240509T193711Z_reco2_reco_2024-11-23T_031124Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_163_20231203T091620Z_gen_g4_detsim_hitreco__20240509T193711Z_reco2_reco_data_2024-11-23T_031124Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work