Jobsub ID 108780.42@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108780.42@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:05:10
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:41:38
From worker node	Hostname	b9p08p7686.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:06:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_81_20231208T024843Z_gen_g4_detsim_hitreco__20240510T042546Z_reco2.root
Jobscript	Exit code	0
	Real time	35m (2107s)
	CPU time	38m (2286s = 108%)
Outputting started	2024-11-23 05:41:20
Output files
Finished	2024-11-23 05:41:38
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: 8200
1st wire C:0 T:3 P:2 W:623 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:08 CET run: 6471483 subRun: 1 event: 8200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:08 CET run: 6471483 subRun: 1 event: 8200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:08 CET run: 6471483 subRun: 1 event: 8200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:08 CET run: 6471483 subRun: 1 event: 8200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:08 CET run: 6471483 subRun: 1 event: 8200
1st wire C:0 T:3 P:2 W:608 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:08 CET run: 6471483 subRun: 1 event: 8200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 415, 290, 344
minwire 0: 194
minwire 1: 2451
minwire 2: 18
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000583026, 0.999392, 2.4942e-05, 
Output 1: 0.952981, 0.0469325, 8.02771e-05, 6.0141e-06, 
Output 2: 0.999475, 0.000524773, 

425
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1343!
23-Nov-2024 06:41:19 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_81_20231208T024843Z_gen_g4_detsim_hitreco__20240510T042546Z_reco2_reco_data_2024-11-23T_050617Z.root"
23-Nov-2024 06:41:19 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a9/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_81_20231208T024843Z_gen_g4_detsim_hitreco__20240510T042546Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.524041       19.1785       249.37        8.48471       38.6358        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0169605     0.0375393     0.0832481     0.0380547     0.0103479       100    
reco:linecluster:LineCluster                           0.0934086     0.226997       1.99971       0.17462      0.241496        100    
reco:trajcluster:TrajCluster                           0.0470431     0.471937       11.4366      0.0577979      1.65798        100    
reco:pandora:StandardPandora                            0.05415       1.84007       44.9114       1.10963       4.72802        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00011158     0.0037091     0.0691704    0.00119237    0.00877191       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00016342    0.00706528     0.0769984    0.00261202     0.0115386       100    
reco:pandoracalo:Calorimetry                          8.9511e-05     0.0118721     0.258367     0.00086619     0.0398858       100    
reco:pandorapid:Chi2ParticleID                         3.26e-05     0.000136515   0.00103878    0.000114206   0.000130112      100    
reco:trkshowersplit:TrackShowerHits                   9.3681e-05     0.0131818     0.385324     0.000677469    0.0521208       100    
reco:pmtrack:PMAlgTrackMaker                          0.000922795     1.48669       25.4192       0.21642       4.17624        100    
reco:pmtrackcalo:Calorimetry                          5.5431e-05    0.00691307      0.18472     0.000688244    0.0259039       100    
reco:pmtrackpid:Chi2ParticleID                        2.2061e-05    0.000224964   0.00319383    0.000111835   0.000400664      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00014778      1.35876       33.493       0.244435       4.1189         100    
reco:pmtrajfitcalo:Calorimetry                         5.417e-05     0.0121125     0.241018      0.0011773     0.0398459       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.125e-05    0.000144969   0.000679104   0.000128911   0.000107957      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000728964     1.80902       34.7481      0.169248       5.57999        100    
reco:pmtracktccalo:Calorimetry                        5.3921e-05    0.00495788     0.117012     0.000749084    0.0164211       100    
reco:pmtracktcpid:Chi2ParticleID                       2.138e-05    0.000195194   0.00163087    0.000129766   0.000256374      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000121081     1.0244        17.5066       0.24115       2.62005        100    
reco:pmtrajfittccalo:Calorimetry                       4.874e-05     0.0106347     0.340802     0.00082444     0.0425369       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.057e-05    0.000257866   0.00332937    0.000131491    0.0004503       100    
reco:blurredcluster:BlurredClustering                 5.6211e-05     0.795652       7.41811      0.455555       1.07429        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000126031     7.29153       100.609       2.06747       15.3428        100    
reco:emshower:EMShower                                0.000163221    0.0225461     0.261391     0.00708051     0.0390962       100    
reco:cvnmap:CVNMapper                                  1.814e-05     0.0113674     0.0317235     0.0127264    0.00828128       100    
reco:cvneva:CVNEvaluator                               1.47e-05       0.61816       1.11652      0.738396       0.34881        100    
reco:energyrecnumu:EnergyReco                         0.000299652    0.0272303     0.734885     0.00355968     0.0911233       100    
reco:energyrecnue:EnergyReco                          0.000110641   0.00145569     0.0246243    0.000506708   0.00329748       100    
reco:energyrecnc:EnergyReco                           0.00132019    0.00353335     0.0283843    0.00279584    0.00379106       100    
reco:energyrecnumurange:EnergyReco                    0.000147231    0.0236411     0.676429      0.0017234     0.0834731       100    
reco:energyrecnumumcs:EnergyReco                      0.00161988     0.0248725     0.648374     0.00305615     0.0818513       100    
reco:opdec:Deconvolution                               0.0492129      0.25465      0.459761      0.261563      0.080455        100    
reco:ophitspe:OpHitFinderDeco                          0.141791       1.57008       3.15433       1.51847       0.7055         100    
reco:opflash:OpFlashFinder                            0.000140541   0.00118365    0.00347992    0.00119205    0.000524161      100    
reco:rns:RandomNumberSaver                            1.6471e-05    3.66723e-05   0.000311822    3.362e-05    2.88075e-05      100    
reco:anglereconue:NuAngularReco                       0.000127931   0.00102752     0.0217514    0.000345086   0.00273341       100    
reco:anglereconumu:NuAngularReco                       5.849e-05    0.00078995     0.0207633    0.000187136   0.00242663       100    
reco:anglereconuepfps:NuAngularReco                    9.566e-05    0.000919645    0.0213804    0.000311951   0.00246989       100    
reco:anglereconumupfps:NuAngularReco                    8.5e-05     0.000835398    0.020993     0.000243346   0.00243419       100    
[art]:TriggerResults:TriggerResultInserter             8.44e-06      1.264e-05     4.494e-05    1.1375e-05    5.5571e-06       100    
end_path:out1:RootOutput                                2.5e-06     6.03582e-06    2.217e-05     5.77e-06     2.54395e-06      100    
end_path:out1:RootOutput(write)                       0.00928493     0.200448      0.723286      0.195105      0.132658        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21803.8 MB
  Peak resident set size usage (VmHWM): 2370.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_81_20231208T024843Z_gen_g4_detsim_hitreco__20240510T042546Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_81_20231208T024843Z_gen_g4_detsim_hitreco__20240510T042546Z_reco2_reco_2024-11-23T_050617Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_81_20231208T024843Z_gen_g4_detsim_hitreco__20240510T042546Z_reco2_reco_data_2024-11-23T_050617Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work