Jobsub ID 108742.61@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.61@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:26:10
From worker node	Hostname	b9p05p9589.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:37:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_11_20231204T060351Z_gen_g4_detsim_hitreco__20240509T203537Z_reco2.root
Jobscript	Exit code	0
	Real time	48m (2902s)
	CPU time	49m (2970s = 102%)
Outputting started	2024-11-23 05:25:53
Output files
Finished	2024-11-23 05:26:10
List job events Wrapper job log
Jobscript log (last 10,000 characters)

odule  23-Nov-2024 06:25:44 CET run: 74517585 subRun: 1 event: 1200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:25:44 CET run: 74517585 subRun: 1 event: 1200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:25:44 CET run: 74517585 subRun: 1 event: 1200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:25:44 CET run: 74517585 subRun: 1 event: 1200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:25:44 CET run: 74517585 subRun: 1 event: 1200
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:25:47 CET run: 74517585 subRun: 1 event: 1200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 54, 55, 46
minwire 0: 1805
minwire 1: 924
minwire 2: 1501
Used alternate method to get min and max wires due to vertex determination failure: 329, 528
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 583, 782
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.997152, 0.000112895, 0.00273545, 
Output 1: 0.15619, 0.639143, 0.195264, 0.00940272, 
Output 2: 0.996221, 0.00377855, 

513
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1822!
23-Nov-2024 06:25:52 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_11_20231204T060351Z_gen_g4_detsim_hitreco__20240509T203537Z_reco2_reco_data_2024-11-23T_043736Z.root"
23-Nov-2024 06:25:52 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/3f/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_11_20231204T060351Z_gen_g4_detsim_hitreco__20240509T203537Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.561872       26.9968       695.973       8.25286       76.728         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0166214     0.0431402     0.239204      0.0397323     0.0242343       100    
reco:linecluster:LineCluster                           0.0933672     0.638092       34.5497      0.165485       3.48993        100    
reco:trajcluster:TrajCluster                           0.0469358     0.928625       42.407       0.0534381      4.66014        100    
reco:pandora:StandardPandora                           0.0544874      2.33955       57.9374       1.21258       6.16608        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000124663   0.00681675     0.228048     0.000931966    0.0250088       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000183174   0.00887891     0.145239     0.00218737      0.02087        100    
reco:pandoracalo:Calorimetry                          9.8572e-05     0.0262425      1.15997     0.000607419    0.126641        100    
reco:pandorapid:Chi2ParticleID                         3.512e-05    0.000173104    0.0016612    0.000128413   0.000204787      100    
reco:trkshowersplit:TrackShowerHits                   8.7172e-05     0.0364433      1.65286     0.000539362    0.181431        100    
reco:pmtrack:PMAlgTrackMaker                          0.000963971     2.43847       93.5607      0.129418       10.189         100    
reco:pmtrackcalo:Calorimetry                          6.0051e-05     0.0105388      0.45413     0.000665665    0.0480806       100    
reco:pmtrackpid:Chi2ParticleID                         2.332e-05    0.000281165   0.00380908    0.000137738   0.000492217      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000169944     2.38771       96.6971      0.210218       10.3091        100    
reco:pmtrajfitcalo:Calorimetry                        5.9521e-05     0.0253556      1.1634      0.000843553    0.126505        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7101e-05    0.000218361   0.00232188    0.000136633   0.00032857       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000781028     2.96253       122.989       0.1578        13.0469        100    
reco:pmtracktccalo:Calorimetry                        6.2311e-05    0.00657526     0.257893     0.000746831    0.0268673       100    
reco:pmtracktcpid:Chi2ParticleID                       2.712e-05    0.000252853   0.00299086    0.000144579   0.000356017      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000155493     1.67706       51.8986       0.20535       5.78217        100    
reco:pmtrajfittccalo:Calorimetry                      6.6021e-05     0.0155695     0.541465     0.000768867    0.0638257       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.721e-05    0.000348729   0.00391843    0.000144073   0.000612855      100    
reco:blurredcluster:BlurredClustering                 0.000118213    0.909526       7.72088      0.474265       1.24777        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000127753     9.16657       168.433       1.85794       21.5589        100    
reco:emshower:EMShower                                0.000182814    0.0297054     0.387716     0.00611175     0.057024        100    
reco:cvnmap:CVNMapper                                  2.055e-05    0.00973523     0.0324747     0.010891     0.00782723       100    
reco:cvneva:CVNEvaluator                               1.634e-05     0.641253       1.25374      0.795275      0.434452        100    
reco:energyrecnumu:EnergyReco                         0.000373039    0.0424908      2.48451     0.00350071     0.251597        100    
reco:energyrecnue:EnergyReco                          0.000125583   0.00242293     0.083985     0.00046588    0.00885559       100    
reco:energyrecnc:EnergyReco                           0.00163196    0.00485779     0.0909297    0.00315424    0.00930436       100    
reco:energyrecnumurange:EnergyReco                    0.000211955    0.0388868      2.37538     0.00306245     0.239837        100    
reco:energyrecnumumcs:EnergyReco                      0.00143557     0.0364319      2.02005     0.00362025     0.204318        100    
reco:opdec:Deconvolution                               0.0536159     0.266676      0.463779      0.261663      0.0786298       100    
reco:ophitspe:OpHitFinderDeco                          0.202875       2.04849       4.2479        1.99159      0.848398        100    
reco:opflash:OpFlashFinder                            0.000320067   0.00142053    0.00519513     0.0013461    0.000711378      100    
reco:rns:RandomNumberSaver                             1.686e-05    4.43776e-05   0.000331528   4.2411e-05    3.15597e-05      100    
reco:anglereconue:NuAngularReco                       0.000128573   0.00186943     0.0475226    0.000390789   0.00609305       100    
reco:anglereconumu:NuAngularReco                      6.2022e-05    0.00165185     0.0472187    0.000198079   0.00602414       100    
reco:anglereconuepfps:NuAngularReco                   9.8772e-05    0.00181221     0.0469138    0.000374593   0.00600715       100    
reco:anglereconumupfps:NuAngularReco                  8.4442e-05     0.0017093     0.0463902    0.000279022   0.00596355       100    
[art]:TriggerResults:TriggerResultInserter             8.91e-06     1.5743e-05    4.5041e-05    1.54805e-05   4.89073e-06      100    
end_path:out1:RootOutput                               2.53e-06     7.15108e-06   2.0891e-05     7.635e-06    2.66998e-06      100    
end_path:out1:RootOutput(write)                        0.0157794     0.236475      0.898259      0.232348      0.170517        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21865.8 MB
  Peak resident set size usage (VmHWM): 2412.01 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_11_20231204T060351Z_gen_g4_detsim_hitreco__20240509T203537Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_11_20231204T060351Z_gen_g4_detsim_hitreco__20240509T203537Z_reco2_reco_2024-11-23T_043736Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_11_20231204T060351Z_gen_g4_detsim_hitreco__20240509T203537Z_reco2_reco_data_2024-11-23T_043736Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work