Jobsub ID 301714.6@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301714.6@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:47:25
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 05:18:23
From worker node	Hostname	b9p08p7421.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:48:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_293_20231203T165017Z_gen_g4_detsim_hitreco__20240509T194838Z_reco2.root
Jobscript	Exit code	0
	Real time	29m (1782s)
	CPU time	31m (1888s = 105%)
Outputting started	2024-11-23 05:18:04
Output files
Finished	2024-11-23 05:18:23
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: 1 event: 29400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:17:43 CET run: 74511854 subRun: 1 event: 29400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 475, 1000, 526
minwire 0: 1384
minwire 1: 918
minwire 2: 1496
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.177483, 0.819584, 0.00293289, 
Output 1: 0.887426, 0.0969393, 0.0102389, 0.00539545, 
Output 2: 0.976483, 0.0235169, 

721
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2983!
23-Nov-2024 06:18:02 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_293_20231203T165017Z_gen_g4_detsim_hitreco__20240509T194838Z_reco2_reco_data_2024-11-23T_044826Z.root"
23-Nov-2024 06:18:02 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/57/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_293_20231203T165017Z_gen_g4_detsim_hitreco__20240509T194838Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.80284       15.926        187.973       9.80944       22.2264        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0169716     0.0379647     0.0979546     0.0385937     0.0126556       100    
reco:linecluster:LineCluster                            0.15107      0.178244      0.970547      0.163156      0.0822721       100    
reco:trajcluster:TrajCluster                           0.0480783     0.215545       10.3456      0.058889       1.02827        100    
reco:pandora:StandardPandora                           0.927737       1.43955       8.55094       1.33833      0.758966        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000142751   0.00333732     0.0449418    0.00138367    0.00583761       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000194131    0.0132732     0.204922     0.00267525     0.0300685       100    
reco:pandoracalo:Calorimetry                          0.000119631   0.00492486     0.107562     0.00101825     0.0127477       100    
reco:pandorapid:Chi2ParticleID                         4.282e-05    0.000169896   0.00112389    0.000139371   0.000133678      100    
reco:trkshowersplit:TrackShowerHits                   0.000112041   0.00561041     0.198396     0.000587409    0.0208127       100    
reco:pmtrack:PMAlgTrackMaker                          0.00111035      1.12232       24.4047      0.135731       3.05498        100    
reco:pmtrackcalo:Calorimetry                           6.531e-05    0.00386942     0.103317     0.00093639     0.0121717       100    
reco:pmtrackpid:Chi2ParticleID                        2.5751e-05    0.000198026   0.00154297    0.000151741   0.000194774      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000178861    0.947945       22.1354      0.239504       2.62683        100    
reco:pmtrajfitcalo:Calorimetry                         6.107e-05    0.00580718     0.122452     0.00145992     0.0151309       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.794e-05    0.000193323   0.000767554   0.000163681   0.000133923      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000945205     1.09056       28.9922      0.216583       3.21511        100    
reco:pmtracktccalo:Calorimetry                        6.2481e-05    0.00366169     0.0757157    0.00099132    0.00959919       100    
reco:pmtracktcpid:Chi2ParticleID                       2.623e-05    0.000200871   0.000924705   0.000155721   0.000168217      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000212991    0.832628       10.6792      0.316775        1.579         100    
reco:pmtrajfittccalo:Calorimetry                       6.963e-05    0.00459682     0.101781     0.00114842     0.0119378       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.916e-05    0.000244479   0.00146486    0.000167111   0.000223388      100    
reco:blurredcluster:BlurredClustering                 0.00741268     0.848472       6.93376       0.40415       1.08739        100    
reco:emtrkmichelid:EmTrackMichelId                     0.208334       5.68327       68.2626       2.6387        9.21529        100    
reco:emshower:EMShower                                0.000237061    0.0222604     0.279091     0.00408944     0.0395109       100    
reco:cvnmap:CVNMapper                                  2.465e-05     0.0122062     0.0347309     0.0137277    0.00804417       100    
reco:cvneva:CVNEvaluator                               1.532e-05     0.758581       1.22618      0.919257      0.418013        100    
reco:energyrecnumu:EnergyReco                         0.000342522    0.0156027     0.404743     0.00398799     0.0474039       100    
reco:energyrecnue:EnergyReco                          0.00011495    0.00118933     0.0130613    0.000457623   0.00183279       100    
reco:energyrecnc:EnergyReco                           0.00162089    0.00387742     0.0153282    0.00341758    0.00209063       100    
reco:energyrecnumurange:EnergyReco                    0.000149491    0.0129321     0.298332     0.00263727     0.0396016       100    
reco:energyrecnumumcs:EnergyReco                      0.000523403    0.0145342     0.298677     0.00415401     0.0383052       100    
reco:opdec:Deconvolution                                0.15327      0.279075      0.564451      0.277682      0.0725743       100    
reco:ophitspe:OpHitFinderDeco                          0.719082       2.11054       3.87702       2.10336      0.715539        100    
reco:opflash:OpFlashFinder                            0.000511733   0.00152939    0.00490436    0.00147171    0.000654793      100    
reco:rns:RandomNumberSaver                             2.291e-05    4.65059e-05   0.000399913   4.12605e-05   4.10389e-05      100    
reco:anglereconue:NuAngularReco                       0.000183931   0.000907885    0.0117383    0.000458797   0.00149121       100    
reco:anglereconumu:NuAngularReco                       6.554e-05    0.000664327    0.010705     0.000229341   0.00140213       100    
reco:anglereconuepfps:NuAngularReco                   0.00012755    0.000843641    0.0109594    0.000401767   0.00142036       100    
reco:anglereconumupfps:NuAngularReco                  8.7891e-05    0.000729411    0.011156     0.000294812   0.00143726       100    
[art]:TriggerResults:TriggerResultInserter             8.581e-06    1.49033e-05    5.393e-05     1.389e-05    5.61864e-06      100    
end_path:out1:RootOutput                               3.72e-06     7.75303e-06    2.44e-05      7.52e-06     2.46428e-06      100    
end_path:out1:RootOutput(write)                        0.036522       0.24571      0.690997      0.271042       0.12529        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21426.2 MB
  Peak resident set size usage (VmHWM): 1999.95 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_293_20231203T165017Z_gen_g4_detsim_hitreco__20240509T194838Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_293_20231203T165017Z_gen_g4_detsim_hitreco__20240509T194838Z_reco2_reco_2024-11-23T_044826Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_293_20231203T165017Z_gen_g4_detsim_hitreco__20240509T194838Z_reco2_reco_data_2024-11-23T_044826Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work