Jobsub ID 301715.104@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301715.104@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:48:29
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:42:13
From worker node	Hostname	b9p08p3442.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:50:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_22_20231207T135427Z_gen_g4_detsim_hitreco__20240510T061027Z_reco2.root
Jobscript	Exit code	0
	Real time	51m (3107s)
	CPU time	52m (3158s = 101%)
Outputting started	2024-11-23 05:41:53
Output files
Finished	2024-11-23 05:42:13
List job events Wrapper job log
Jobscript log (last 10,000 characters)

6457551 subRun: 1 event: 2300
1st wire C:0 T:11 P:2 W:692 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:34 CET run: 6457551 subRun: 1 event: 2300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:34 CET run: 6457551 subRun: 1 event: 2300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:34 CET run: 6457551 subRun: 1 event: 2300
1st wire C:0 T:11 P:2 W:645 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:34 CET run: 6457551 subRun: 1 event: 2300
Cannot attach back to inner node of other track.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:41:34 CET run: 6457551 subRun: 1 event: 2300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:41:44 CET run: 6457551 subRun: 1 event: 2300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 496, 582, 375
minwire 0: 1490
minwire 1: 1237
minwire 2: 1137
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000258224, 0.999547, 0.000194714, 
Output 1: 0.781615, 0.215404, 0.00290424, 7.7106e-05, 
Output 2: 0.704496, 0.295504, 

692
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2801!
23-Nov-2024 06:41:51 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_22_20231207T135427Z_gen_g4_detsim_hitreco__20240510T061027Z_reco2_reco_data_2024-11-23T_045010Z.root"
23-Nov-2024 06:41:51 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/24/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_22_20231207T135427Z_gen_g4_detsim_hitreco__20240510T061027Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.526751       29.0177       563.852       8.79741       77.7677        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0161469     0.0392147     0.0708136     0.0412659     0.0113113       100    
reco:linecluster:LineCluster                           0.0953932     0.671111       28.0972       0.17901       3.20957        100    
reco:trajcluster:TrajCluster                           0.0478294     0.997053       36.1025      0.0563761      5.05664        100    
reco:pandora:StandardPandora                           0.0529937      2.38924       52.4448       1.31702       6.32804        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00010976    0.00544562      0.10758     0.00139343     0.0128869       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000154201    0.0119744     0.385319     0.00334408     0.040192        100    
reco:pandoracalo:Calorimetry                          0.000102661    0.0238915      1.06864     0.000994576    0.117687        100    
reco:pandorapid:Chi2ParticleID                         3.729e-05    0.000175863   0.00101242     0.0001354    0.000137242      100    
reco:trkshowersplit:TrackShowerHits                    6.672e-05     0.0269719     0.980208     0.000662689     0.12238        100    
reco:pmtrack:PMAlgTrackMaker                          0.000872605     3.2338        109.573       0.13653       14.0995        100    
reco:pmtrackcalo:Calorimetry                           6.648e-05    0.00920273     0.353339     0.000828185    0.0392156       100    
reco:pmtrackpid:Chi2ParticleID                         2.428e-05    0.000287104   0.00389585    0.000137321   0.000502717      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000140881     2.86905       116.183      0.262581       12.7587        100    
reco:pmtrajfitcalo:Calorimetry                         5.14e-05      0.0228293     0.879684     0.00148682     0.101087        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.256e-05    0.000198754   0.000870356   0.000177246   0.000153845      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000752825     2.90887       89.9657      0.200998       10.8433        100    
reco:pmtracktccalo:Calorimetry                         5.507e-05    0.00562746     0.137381     0.00087453     0.016501        100    
reco:pmtracktcpid:Chi2ParticleID                      2.4361e-05    0.000267941   0.00241966    0.000155671   0.000327561      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.00012195      1.5903        36.8575      0.226696       4.52158        100    
reco:pmtrajfittccalo:Calorimetry                       4.995e-05     0.014712      0.400422     0.000923981    0.0544715       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.19e-05     0.000366854   0.00267658    0.000175861   0.000502259      100    
reco:blurredcluster:BlurredClustering                  0.0001008     0.897674       6.74023      0.450259       1.24887        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000132611     9.69305       166.145       2.11567       21.7823        100    
reco:emshower:EMShower                                0.000169891    0.0303797      0.30716     0.00556067     0.0577176       100    
reco:cvnmap:CVNMapper                                  2.035e-05     0.0141811     0.0490155     0.0185568     0.0113996       100    
reco:cvneva:CVNEvaluator                               1.601e-05     0.690898       1.27641      0.879215      0.456948        100    
reco:energyrecnumu:EnergyReco                         0.000331402    0.0400793      1.65489     0.00368841     0.175544        100    
reco:energyrecnue:EnergyReco                          0.00013141    0.00219572     0.0553395    0.000552584   0.00598771       100    
reco:energyrecnc:EnergyReco                            0.0013563    0.00468121     0.059153     0.00339814    0.00627265       100    
reco:energyrecnumurange:EnergyReco                    0.000163691    0.0373421      1.61672     0.00235882     0.172234        100    
reco:energyrecnumumcs:EnergyReco                      0.000646834    0.037731       1.39997     0.00370376     0.152691        100    
reco:opdec:Deconvolution                               0.0549101     0.288648      0.486664      0.297336      0.0876931       100    
reco:ophitspe:OpHitFinderDeco                          0.142197       2.19853       4.4215        2.23905      0.907106        100    
reco:opflash:OpFlashFinder                            0.000178252   0.00167752    0.00598751    0.00160928    0.000890046      100    
reco:rns:RandomNumberSaver                             1.855e-05    4.45059e-05   0.000291912   4.32955e-05   2.68927e-05      100    
reco:anglereconue:NuAngularReco                       0.00015747    0.00174581     0.0485717    0.000518403   0.00519068       100    
reco:anglereconumu:NuAngularReco                      7.6731e-05    0.00143049     0.0464518    0.000255131   0.00495733       100    
reco:anglereconuepfps:NuAngularReco                   0.000115251   0.00160414     0.0469475    0.000432523   0.00497262       100    
reco:anglereconumupfps:NuAngularReco                  0.000118461    0.0013676     0.0328738    0.000358697   0.00373821       100    
[art]:TriggerResults:TriggerResultInserter             8.69e-06     1.43913e-05    3.572e-05    1.4245e-05    3.53181e-06      100    
end_path:out1:RootOutput                               2.65e-06     7.12923e-06    1.987e-05     7.14e-06     2.50725e-06      100    
end_path:out1:RootOutput(write)                       0.00953998     0.251605      0.913149      0.269254      0.162705        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21669.2 MB
  Peak resident set size usage (VmHWM): 2250.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_22_20231207T135427Z_gen_g4_detsim_hitreco__20240510T061027Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_22_20231207T135427Z_gen_g4_detsim_hitreco__20240510T061027Z_reco2_reco_2024-11-23T_045010Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_22_20231207T135427Z_gen_g4_detsim_hitreco__20240510T061027Z_reco2_reco_data_2024-11-23T_045010Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work