Jobsub ID 301742.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301742.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:53:26
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 05:40:59
From worker node	Hostname	tdm003.pic.es
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:54:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_491_20231203T094311Z_gen_g4_detsim_hitreco__20240509T195637Z_reco2.root
Jobscript	Exit code	0
	Real time	46m (2785s)
	CPU time	1h (4620s = 165%)
Outputting started	2024-11-23 05:40:43
Output files
Finished	2024-11-23 05:40:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
1st wire C:0 T:11 P:2 W:1046 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
1st wire C:0 T:11 P:2 W:1108 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:40:17 CET run: 50574782 subRun: 1 event: 49200
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 859, 582, 718
minwire 0: 672
minwire 1: 872
minwire 2: 37
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.680734, 0.0142808, 0.304986, 
Output 1: 0.196516, 0.684503, 0.112077, 0.00690464, 
Output 2: 0.0345628, 0.965437, 

713
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2719!
23-Nov-2024 06:40:41 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_491_20231203T094311Z_gen_g4_detsim_hitreco__20240509T195637Z_reco2_reco_data_2024-11-23T_045421Z.root"
23-Nov-2024 06:40:41 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/36/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_491_20231203T094311Z_gen_g4_detsim_hitreco__20240509T195637Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.59216       23.4948       340.193       11.2894       42.5001        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0461938     0.0901581     0.139433      0.102457      0.0259737       100    
reco:linecluster:LineCluster                            0.27459      0.507098       2.15121      0.460286      0.197853        100    
reco:trajcluster:TrajCluster                           0.135062       0.54863       16.3862      0.152616       1.79455        100    
reco:pandora:StandardPandora                           0.141422       1.81875       17.3843       1.38247       1.89042        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139883   0.00611334     0.162677     0.00153785     0.018044        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000218874    0.0100749     0.104446     0.00285944     0.0189102       100    
reco:pandoracalo:Calorimetry                          0.000104832   0.00971469     0.202957     0.00124266     0.029047        100    
reco:pandorapid:Chi2ParticleID                        4.0181e-05    0.000170914   0.00125717    0.000124772   0.000166479      100    
reco:trkshowersplit:TrackShowerHits                   8.4451e-05     0.0120289     0.357022     0.000985611    0.0428829       100    
reco:pmtrack:PMAlgTrackMaker                          0.00138559      1.69137       45.1896      0.254625       5.3514         100    
reco:pmtrackcalo:Calorimetry                          6.9661e-05    0.00793809     0.233071     0.00105123     0.0293805       100    
reco:pmtrackpid:Chi2ParticleID                         2.818e-05    0.00027701    0.00296632    0.000156147   0.000425864      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000197343     1.2427        19.5133      0.269618       2.66467        100    
reco:pmtrajfitcalo:Calorimetry                        6.6791e-05     0.010627      0.234445     0.00184117     0.0316774       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.802e-05    0.000191606   0.00136782    0.000157017   0.000174253      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00115198      2.13068       59.7575      0.238468       6.99517        100    
reco:pmtracktccalo:Calorimetry                        6.8942e-05    0.00597415     0.142886     0.000970686    0.0206465       100    
reco:pmtracktcpid:Chi2ParticleID                       2.791e-05    0.000248972   0.00200248    0.000162027   0.000315788      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000167153     1.30029       24.3377      0.363391       3.2409         100    
reco:pmtrajfittccalo:Calorimetry                      6.8481e-05     0.0103707     0.306094     0.00140286     0.0365777       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.616e-05     0.0003292    0.00303498    0.000183293   0.000470899      100    
reco:blurredcluster:BlurredClustering                 0.000211883     1.13828       7.70052      0.573994       1.47862        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000104211     9.34953       136.85        3.2504        18.711         100    
reco:emshower:EMShower                                0.000165332    0.0302673     0.310122      0.0106474     0.0513183       100    
reco:cvnmap:CVNMapper                                  2.504e-05     0.0122804     0.0474233    0.00842693     0.010861        100    
reco:cvneva:CVNEvaluator                               1.766e-05     0.901975       1.42591       1.20353      0.550732        100    
reco:energyrecnumu:EnergyReco                         0.000295055    0.029095      0.823313     0.00428849     0.107148        100    
reco:energyrecnue:EnergyReco                          0.000141443    0.001991      0.0564438    0.000543584   0.00624095       100    
reco:energyrecnc:EnergyReco                           0.00226104    0.00441138     0.0617323    0.00292599    0.00660458       100    
reco:energyrecnumurange:EnergyReco                    0.000134542    0.0267144     0.806423       0.00239       0.10486        100    
reco:energyrecnumumcs:EnergyReco                      0.00144092     0.0282325     0.799268      0.0033909     0.104151        100    
reco:opdec:Deconvolution                               0.163918      0.524925      0.803901      0.519453      0.118147        100    
reco:ophitspe:OpHitFinderDeco                          0.498248       1.7775        2.91489       1.74003      0.548348        100    
reco:opflash:OpFlashFinder                            0.000274614   0.00143254    0.00445874    0.00135783    0.000611693      100    
reco:rns:RandomNumberSaver                             1.99e-05     3.71229e-05   0.000344355   3.5065e-05    3.30447e-05      100    
reco:anglereconue:NuAngularReco                       0.000174163   0.00161618     0.0525852    0.000369721   0.00578044       100    
reco:anglereconumu:NuAngularReco                      7.3602e-05    0.00140757     0.0507802    0.000220219   0.00559722       100    
reco:anglereconuepfps:NuAngularReco                   0.000124082   0.00157188     0.0508988    0.00033916    0.00559449       100    
reco:anglereconumupfps:NuAngularReco                  0.000104092   0.00150277     0.0520354    0.000270599   0.00570292       100    
[art]:TriggerResults:TriggerResultInserter             9.74e-06     1.24795e-05   5.0341e-05     1.146e-05    4.24609e-06      100    
end_path:out1:RootOutput                               3.34e-06     6.14969e-06    2.443e-05     6.45e-06     2.49765e-06      100    
end_path:out1:RootOutput(write)                        0.0222824     0.256136       1.03057      0.253347      0.163675        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39418.2 MB
  Peak resident set size usage (VmHWM): 2280.69 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_491_20231203T094311Z_gen_g4_detsim_hitreco__20240509T195637Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_491_20231203T094311Z_gen_g4_detsim_hitreco__20240509T195637Z_reco2_reco_2024-11-23T_045421Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_491_20231203T094311Z_gen_g4_detsim_hitreco__20240509T195637Z_reco2_reco_data_2024-11-23T_045421Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work