Jobsub ID 108878.35@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108878.35@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:16:18
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:21:12
From worker node	Hostname	wn-choc-031.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:17:31
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_369_20231202T044456Z_gen_g4_detsim_hitreco__20240507T201857Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (207s)
	CPU time	3m (200s = 96%)
Outputting started	2024-11-23 07:20:59
Output files
Finished	2024-11-23 07:21:12
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:57 CET run: 6396715 subRun: 1 event: 36909
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:58 CET run: 6396715 subRun: 1 event: 36909
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:20:58 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8a/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_369_20231202T044456Z_gen_g4_detsim_hitreco__20240507T201857Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0161149      10.5438       40.5343       4.68749       11.8356         9     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000938361    0.0028724     0.0161149    0.00129684    0.00468894        9     
reco:linecluster:LineCluster                           0.0108063     0.0704738      0.41279      0.0192888     0.122585         9     
reco:trajcluster:TrajCluster                          0.00155931     0.313337       2.53849      0.017961      0.788636         9     
reco:pandora:StandardPandora                            1.43356       2.11437       4.14359       1.53777      0.932651         9     
reco:pandoraTrack:LArPandoraTrackCreation             0.000156545   0.00599488     0.0272458    0.00203004    0.00858073        9     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000214798    0.0112823     0.032928      0.0060154     0.0125356        9     
reco:pandoracalo:Calorimetry                          0.000126975    0.0114799     0.0781081    0.00180841     0.023939         9     
reco:pandorapid:Chi2ParticleID                        5.7663e-05    0.000349078   0.00187831    0.000146988   0.000547493       9     
reco:trkshowersplit:TrackShowerHits                   0.000128161    0.0133917      0.10431     0.000788417    0.0322813        9     
reco:pmtrack:PMAlgTrackMaker                          0.00147326      1.05542       4.66949      0.236283       1.57297         9     
reco:pmtrackcalo:Calorimetry                          0.000103867   0.00558571     0.028823     0.00174425    0.00918213        9     
reco:pmtrackpid:Chi2ParticleID                        4.2296e-05    0.000235425   0.000952934   0.000134134   0.00027285        9     
reco:pmtrajfit:PMAlgTrajFitter                        0.000289554     1.37772       8.76978      0.346088       2.7123          9     
reco:pmtrajfitcalo:Calorimetry                        9.5454e-05     0.0120174     0.0840167    0.00191349     0.0258708        9     
reco:pmtrajfitpid:Chi2ParticleID                       4.045e-05    0.000170628   0.000418775   0.000143391   0.00011985        9     
reco:pmtracktc:PMAlgTrackMaker                        0.00134256      1.29871       7.37865      0.278587       2.30828         9     
reco:pmtracktccalo:Calorimetry                        9.8761e-05    0.00250401     0.0108536    0.00157017    0.00332308        9     
reco:pmtracktcpid:Chi2ParticleID                      4.9185e-05    0.000200317   0.000645661   0.000132418   0.000184444       9     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000251927    0.915515       4.67079      0.271661       1.42461         9     
reco:pmtrajfittccalo:Calorimetry                      9.3822e-05    0.00713173     0.0449076    0.00142221     0.0137986        9     
reco:pmtrajfittcpid:Chi2ParticleID                    4.0573e-05    0.000270283   0.00120519    0.000117033   0.000347099       9     
reco:blurredcluster:BlurredClustering                 0.00640941     0.639158       1.44225      0.750156      0.447113         8     
reco:emtrkmichelid:EmTrackMichelId                     0.391563       5.13158       22.4194       2.50281       6.94348         8     
reco:emshower:EMShower                                0.000295556   0.00923472     0.0578467    0.00244777     0.0184758        8     
reco:cvnmap:CVNMapper                                  3.105e-05    0.00853029     0.023315     0.00561291     0.0090425        8     
reco:cvneva:CVNEvaluator                              2.0898e-05     0.602189       1.24033      0.595988      0.602346         8     
reco:energyrecnumu:EnergyReco                         0.000365662   0.00872758     0.024275     0.00560309    0.00843434        8     
reco:energyrecnue:EnergyReco                          0.000195872    0.0011261    0.00333013    0.00072112    0.00110306        8     
reco:energyrecnc:EnergyReco                           0.00253477    0.00323833    0.00569881    0.00280613    0.000991141       8     
reco:energyrecnumurange:EnergyReco                    0.000284738   0.00679303     0.020627     0.00266913    0.00738263        8     
reco:energyrecnumumcs:EnergyReco                      0.00255433    0.00812032     0.0210587    0.00283803    0.00745352        8     
reco:opdec:Deconvolution                               0.028052      0.152791      0.258567      0.154658      0.076128         8     
reco:ophitspe:OpHitFinderDeco                          0.187175       1.10859       1.93129        1.091       0.575057         8     
reco:opflash:OpFlashFinder                            0.000356267   0.00147703    0.00415306    0.00131384     0.001106         8     
reco:rns:RandomNumberSaver                            1.9933e-05    5.83203e-05   0.000296984   2.2451e-05    9.03422e-05       8     
reco:anglereconue:NuAngularReco                       0.00016219    0.000723175   0.00237986    0.000414524   0.000738931       8     
reco:anglereconumu:NuAngularReco                      9.7764e-05    0.000511322   0.00228451    0.000230123   0.000688393       8     
reco:anglereconuepfps:NuAngularReco                   0.000154397   0.000907502   0.00284077    0.000321275   0.00104248        8     
reco:anglereconumupfps:NuAngularReco                  0.000147918   0.000593215   0.00243523    0.00029207    0.000717372       8     
[art]:TriggerResults:TriggerResultInserter            1.2944e-05    1.78671e-05    4.881e-05    1.3417e-05    1.17051e-05       8     
end_path:out1:RootOutput                               2.829e-06     5.499e-06    1.5154e-05     3.759e-06     3.917e-06        8     
end_path:out1:RootOutput(write)                        0.0207988     0.177767      0.374456      0.176557      0.131924         8     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6413.55 MB
  Peak resident set size usage (VmHWM): 1665.7 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:20:58 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6396715 subRun: 1 event: 36909
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-92e1-651b-b07b-1199.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_369_20231202T044456Z_gen_g4_detsim_hitreco__20240507T201857Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_369_20231202T044456Z_gen_g4_detsim_hitreco__20240507T201857Z_reco2_reco_2024-11-23T_071734Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work