Jobsub ID 301707.96@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301707.96@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:33:23
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 05:34:34
From worker node	Hostname	b9p05p7101.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:35:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_86_20231203T163828Z_gen_g4_detsim_hitreco__20240509T200055Z_reco2.root
Jobscript	Exit code	0
	Real time	59m (3546s)
	CPU time	1h (3622s = 102%)
Outputting started	2024-11-23 05:34:17
Output files
Finished	2024-11-23 05:34:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

0
1st wire C:0 T:16 P:2 W:568 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:00 CET run: 6419482 subRun: 1 event: 8700
1st wire C:0 T:16 P:2 W:550 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:00 CET run: 6419482 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:00 CET run: 6419482 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:00 CET run: 6419482 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:00 CET run: 6419482 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:00 CET run: 6419482 subRun: 1 event: 8700
1st wire C:0 T:16 P:2 W:499 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 328, 509, 464
minwire 0: 1975
minwire 1: 882
minwire 2: 1883
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0231164, 0.025602, 0.951282, 
Output 1: 0.236792, 0.75066, 0.0114573, 0.00109054, 
Output 2: 0.155896, 0.844104, 

542
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1866!
23-Nov-2024 06:34:16 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_86_20231203T163828Z_gen_g4_detsim_hitreco__20240509T200055Z_reco2_reco_data_2024-11-23T_043516Z.root"
23-Nov-2024 06:34:16 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/c6/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_86_20231203T163828Z_gen_g4_detsim_hitreco__20240509T200055Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.834382       33.3627       1216.83       9.41501       123.951        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0177924     0.0410637     0.0867821     0.0390672     0.0134485       100    
reco:linecluster:LineCluster                           0.102084      0.497808       23.0105       0.18024       2.2972         100    
reco:trajcluster:TrajCluster                           0.0512461      1.53513       109.628      0.0630962      10.9508        100    
reco:pandora:StandardPandora                           0.0625474      4.28132       268.335       1.24443       26.5887        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000121571   0.00929038     0.445454     0.00142898     0.0449114       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000157991    0.0145865      0.16623     0.00325435     0.0315797       100    
reco:pandoracalo:Calorimetry                          9.6921e-05     0.0312689      1.80754     0.00119581     0.183506        100    
reco:pandorapid:Chi2ParticleID                        3.3411e-05    0.000163582   0.00121145    0.00012136    0.000171271      100    
reco:trkshowersplit:TrackShowerHits                   0.000130011    0.0341337      1.58992     0.000713574     0.17096        100    
reco:pmtrack:PMAlgTrackMaker                          0.000988276     3.31104       161.798       0.27367       16.4787        100    
reco:pmtrackcalo:Calorimetry                           5.755e-05     0.0188104      1.22069     0.00108505     0.122988        100    
reco:pmtrackpid:Chi2ParticleID                         2.309e-05    0.000288432   0.00444218    0.000152336   0.000555907      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000161971     2.71224       93.7509      0.332042       10.4912        100    
reco:pmtrajfitcalo:Calorimetry                        5.3901e-05     0.0345237      2.17003     0.00149852     0.218465        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.456e-05    0.000190812   0.00140663    0.000148401   0.000190457      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000776495     4.04165       196.718      0.302445       20.3463        100    
reco:pmtracktccalo:Calorimetry                         5.603e-05    0.00971535     0.512564      0.0011041     0.0514416       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3641e-05    0.000257845   0.00303493    0.000151156   0.000358698      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000153501     2.14867       86.2708      0.342947       8.81207        100    
reco:pmtrajfittccalo:Calorimetry                       5.469e-05     0.0280966      1.98376     0.00119384     0.197701        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.252e-05    0.00037324    0.00830784    0.000156346   0.000871465      100    
reco:blurredcluster:BlurredClustering                 0.000117961    0.966208       8.99506      0.502647       1.46083        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000153561     10.2339       247.566       2.58724       27.3363        100    
reco:emshower:EMShower                                0.000199332    0.0278324     0.857006     0.00400045     0.090431        100    
reco:cvnmap:CVNMapper                                  2.202e-05     0.0098563     0.0441629     0.0098185    0.00762922       100    
reco:cvneva:CVNEvaluator                               1.682e-05     0.714685       1.24791      0.849623      0.396754        100    
reco:energyrecnumu:EnergyReco                         0.000299882    0.0250448     0.857863     0.00454929     0.0909034       100    
reco:energyrecnue:EnergyReco                          0.000125071   0.00307777     0.131398     0.000552553    0.0133803       100    
reco:energyrecnc:EnergyReco                           0.00160135    0.00536895     0.134668     0.00334439     0.0134397       100    
reco:energyrecnumurange:EnergyReco                    0.000159191     0.02286       0.91814     0.00293138     0.0958319       100    
reco:energyrecnumumcs:EnergyReco                      0.00138507     0.0248974     0.912996     0.00393738     0.0959158       100    
reco:opdec:Deconvolution                               0.0734584     0.281928      0.433456      0.270364      0.0765627       100    
reco:ophitspe:OpHitFinderDeco                          0.400345       2.03519       3.82081       1.93545      0.822728        100    
reco:opflash:OpFlashFinder                            0.000439123   0.00148329    0.00650271    0.00135112    0.000758545      100    
reco:rns:RandomNumberSaver                             1.683e-05    3.99555e-05   0.000337752    3.654e-05    3.18011e-05      100    
reco:anglereconue:NuAngularReco                       0.000154371   0.00282344      0.16174     0.000437178    0.0161672       100    
reco:anglereconumu:NuAngularReco                      7.5551e-05    0.00232293     0.138197     0.000249887    0.0138653       100    
reco:anglereconuepfps:NuAngularReco                   0.000124321   0.00240834     0.132202     0.000383877    0.0132785       100    
reco:anglereconumupfps:NuAngularReco                   0.0001031    0.00226794     0.126992     0.000317397    0.0127641       100    
[art]:TriggerResults:TriggerResultInserter              7.8e-06     1.28124e-05    6.745e-05    1.2335e-05    6.20408e-06      100    
end_path:out1:RootOutput                               2.51e-06     5.2719e-06     2.315e-05     5.44e-06     2.24965e-06      100    
end_path:out1:RootOutput(write)                        0.021008      0.247858       1.41088      0.240767      0.184998        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21954.8 MB
  Peak resident set size usage (VmHWM): 2542.78 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_86_20231203T163828Z_gen_g4_detsim_hitreco__20240509T200055Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_86_20231203T163828Z_gen_g4_detsim_hitreco__20240509T200055Z_reco2_reco_2024-11-23T_043516Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_86_20231203T163828Z_gen_g4_detsim_hitreco__20240509T200055Z_reco2_reco_data_2024-11-23T_043516Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work