Jobsub ID 301839.81@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301839.81@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:18:31
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-23 07:35:22
From worker node	Hostname	mff2201.farm.particle.cz
	cpuinfo	AMD EPYC 7352 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:19:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1007_20231203T030515Z_gen_g4_detsim_hitreco__20240508T070325Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4537s)
	CPU time	1h (4924s = 108%)
Outputting started	2024-11-23 07:35:08
Output files
Finished	2024-11-23 07:35:22
List job events Wrapper job log
Jobscript log (last 10,000 characters)

%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:01 CET run: 50571580 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:01 CET run: 50571580 subRun: 1 event: 100800
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:01 CET run: 50571580 subRun: 1 event: 100800
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:01 CET run: 50571580 subRun: 1 event: 100800
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:01 CET run: 50571580 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:35:03 CET run: 50571580 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:35:03 CET run: 50571580 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 137, 125, 95
minwire 0: 1284
minwire 1: 795
minwire 2: 1212
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0211835, 0.977422, 0.00139444, 
Output 1: 0.972066, 0.0276278, 0.000260494, 4.53516e-05, 
Output 2: 0.987859, 0.012141, 

516
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1643!
23-Nov-2024 08:35:06 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1007_20231203T030515Z_gen_g4_detsim_hitreco__20240508T070325Z_reco2_reco_data_2024-11-23T_061940Z.root"
23-Nov-2024 08:35:06 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/2b/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1007_20231203T030515Z_gen_g4_detsim_hitreco__20240508T070325Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0633686      43.5591       744.998       10.9523       126.503        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00178401     0.0214055     0.172936      0.0186647     0.0172678       100    
reco:linecluster:LineCluster                          0.00212716     0.621971       40.6016      0.0389874      4.22171        100    
reco:trajcluster:TrajCluster                          0.00119963      2.65508       122.847      0.0135114      14.7184        100    
reco:pandora:StandardPandora                          0.00757534      3.60606       62.7769       1.8763        8.93469        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000126292    0.0114191     0.298249     0.00206833     0.0391511       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000184954    0.0123246     0.219249      0.0026379     0.0304038       100    
reco:pandoracalo:Calorimetry                          0.000107275    0.037578       1.21648     0.00110883     0.177607        100    
reco:pandorapid:Chi2ParticleID                        4.0918e-05    0.00025906    0.00350449    0.000159885   0.000388297      100    
reco:trkshowersplit:TrackShowerHits                   8.6425e-05     0.0329463      1.02953     0.000859617    0.146464        100    
reco:pmtrack:PMAlgTrackMaker                          0.00138238      3.82016       104.903      0.176645       15.8837        100    
reco:pmtrackcalo:Calorimetry                          6.5034e-05     0.0169213      0.48591     0.000958101    0.0694418       100    
reco:pmtrackpid:Chi2ParticleID                        2.6631e-05    0.000416876   0.00381336    0.000179924   0.000688002      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00022463      2.44581       87.7793      0.124502       9.78022        100    
reco:pmtrajfitcalo:Calorimetry                        6.0926e-05     0.0319034      1.01814     0.00139955     0.142993        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4768e-05    0.000272174   0.00187806    0.000184593   0.00027824       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00100705      4.59205       113.019      0.136798       18.1168        100    
reco:pmtracktccalo:Calorimetry                        6.5285e-05     0.0153379     0.493236     0.00101332     0.0634617       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5439e-05    0.000461336   0.00634159    0.000186837   0.000944609      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000144306     2.11295       45.2175      0.258713       7.06386        100    
reco:pmtrajfittccalo:Calorimetry                      5.9775e-05     0.0292828      0.7618       0.0011815      0.12017        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4637e-05    0.000497958   0.00513888    0.000203414   0.000939028      100    
reco:blurredcluster:BlurredClustering                 6.9092e-05      1.12068       10.6001      0.565208       1.65019        100    
reco:emtrkmichelid:EmTrackMichelId                    9.2076e-05      19.3036       361.122       3.67899       52.4559        100    
reco:emshower:EMShower                                0.000133896    0.0328134     0.617517      0.0116667      0.07522        100    
reco:cvnmap:CVNMapper                                 1.7624e-05     0.0164415     0.0885232     0.0154242     0.0146368       100    
reco:cvneva:CVNEvaluator                               1.552e-05      1.09345       2.57068       1.1851       0.723839        100    
reco:energyrecnumu:EnergyReco                           0.00045      0.0732939      5.11214     0.00450745     0.514106        100    
reco:energyrecnue:EnergyReco                          0.000147572    0.0027611     0.0541715    0.000636883   0.00772557       100    
reco:energyrecnc:EnergyReco                           0.00183719    0.00507937     0.0564236    0.00302701    0.00775638       100    
reco:energyrecnumurange:EnergyReco                    0.000174253    0.0394788      2.04261     0.00244598     0.219533        100    
reco:energyrecnumumcs:EnergyReco                      0.00148458     0.0412168      2.0823      0.00368817      0.22297        100    
reco:opdec:Deconvolution                              0.00515694     0.180951      0.438252      0.174249      0.0797114       100    
reco:ophitspe:OpHitFinderDeco                         0.00300695      1.29117       3.42153       1.23984       0.60536        100    
reco:opflash:OpFlashFinder                            9.9069e-05     0.0019561     0.0159141    0.00160229    0.00178991       100    
reco:rns:RandomNumberSaver                            1.7283e-05    7.08747e-05   0.000811661   5.0306e-05    9.80062e-05      100    
reco:anglereconue:NuAngularReco                       8.3289e-05     0.0023902     0.056017     0.000521677    0.0072448       100    
reco:anglereconumu:NuAngularReco                      5.9674e-05    0.00188066     0.0444589    0.000206049   0.00633286       100    
reco:anglereconuepfps:NuAngularReco                   9.2206e-05    0.00213989     0.0438413    0.000386418   0.00645326       100    
reco:anglereconumupfps:NuAngularReco                  9.3158e-05     0.0020186     0.0446733    0.000261725   0.00648897       100    
[art]:TriggerResults:TriggerResultInserter             9.899e-06    1.84311e-05   0.000105802   1.5229e-05    1.30183e-05      100    
end_path:out1:RootOutput                               4.048e-06    9.25547e-06   3.7401e-05     7.84e-06     5.87813e-06      100    
end_path:out1:RootOutput(write)                        0.0103436     0.279209       1.64803      0.243381      0.236771        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17516 MB
  Peak resident set size usage (VmHWM): 2544.56 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1007_20231203T030515Z_gen_g4_detsim_hitreco__20240508T070325Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1007_20231203T030515Z_gen_g4_detsim_hitreco__20240508T070325Z_reco2_reco_2024-11-23T_061940Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1007_20231203T030515Z_gen_g4_detsim_hitreco__20240508T070325Z_reco2_reco_data_2024-11-23T_061940Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work