Jobsub ID 108883.111@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108883.111@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:26:49
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2024-11-23 07:32:04
From worker node	Hostname	wn-choc-039.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:28:41
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_145_20231028T052642Z_gen_g4_detsim_hitreco__20240503T051038Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (189s)
	CPU time	3m (180s = 95%)
Outputting started	2024-11-23 07:31:50
Output files
Finished	2024-11-23 07:32:04
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ittc@BeginModule  23-Nov-2024 08:31:48 CET run: 1794 subRun: 1 event: 29005
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:48 CET run: 1794 subRun: 1 event: 29005
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:31:49 CET run: 1794 subRun: 1 event: 29005
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:31:49 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/d6/3d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_145_20231028T052642Z_gen_g4_detsim_hitreco__20240503T051038Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00159605      15.4381       45.9272       10.5429       15.7983         5     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000965846   0.00124473    0.00159605    0.00104378    0.000279512       5     
reco:linecluster:LineCluster                           0.0140093     0.0599342     0.130778      0.0288274     0.0490882        5     
reco:trajcluster:TrajCluster                          0.00544892     0.478591       1.69847      0.0291987     0.656439         5     
reco:pandora:StandardPandora                            1.60064       2.3932        3.15518       2.15761      0.612693         5     
reco:pandoraTrack:LArPandoraTrackCreation             0.000660092   0.00779899     0.0243235    0.00590204    0.00854217        5     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00133576     0.0131273     0.0328296    0.00785351     0.0108751        5     
reco:pandoracalo:Calorimetry                          0.000486389    0.0122783     0.0315733    0.00468784     0.0126349        5     
reco:pandorapid:Chi2ParticleID                        0.000106317   0.000446367   0.00153464    0.000141929   0.000548877       5     
reco:trkshowersplit:TrackShowerHits                   0.000281392    0.0129784     0.0371608    0.00278213     0.0149545        5     
reco:pmtrack:PMAlgTrackMaker                           0.235057       2.40841       6.8531       0.619827       2.65054         5     
reco:pmtrackcalo:Calorimetry                          0.000662956    0.0114174     0.0462503    0.00150753     0.0175753        5     
reco:pmtrackpid:Chi2ParticleID                        0.000124575   0.000403586   0.00108397    0.000204305   0.000363082       5     
reco:pmtrajfit:PMAlgTrajFitter                         0.534816       2.52836       6.88175       1.22239       2.41857         5     
reco:pmtrajfitcalo:Calorimetry                        0.00091478     0.012785      0.0356679    0.00230389     0.014205         5     
reco:pmtrajfitpid:Chi2ParticleID                      0.000128393   0.000233254   0.000477747   0.000198134   0.000128313       5     
reco:pmtracktc:PMAlgTrackMaker                         0.116286       2.76721       7.79607      0.703937       3.09267         5     
reco:pmtracktccalo:Calorimetry                        0.000682615   0.00826664     0.0284083    0.00152996     0.010628         5     
reco:pmtracktcpid:Chi2ParticleID                      0.000128844   0.000388337   0.000923538   0.00016977    0.000318297       5     
reco:pmtrajfittc:PMAlgTrajFitter                       0.395481       2.01005       4.7833       0.587933       1.88433         5     
reco:pmtrajfittccalo:Calorimetry                      0.00102191     0.0146269     0.047715     0.00173841     0.0182094        5     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000127919   0.000575824   0.00157213    0.000223917   0.000556992       5     
reco:blurredcluster:BlurredClustering                  0.0319279     0.464268        1.238       0.293574      0.471655         4     
reco:emtrkmichelid:EmTrackMichelId                      1.65903       7.45593       21.3787       3.39299       8.07076         4     
reco:emshower:EMShower                                0.000763949    0.0182006     0.0499449     0.0110469     0.0197939        4     
reco:cvnmap:CVNMapper                                  0.0103461     0.0113993     0.0127655     0.0112428    0.000967314       4     
reco:cvneva:CVNEvaluator                                1.15012       1.16528       1.18135       1.16483      0.0112982        4     
reco:energyrecnumu:EnergyReco                         0.00288458    0.00741579     0.0152173    0.00578062    0.00473853        4     
reco:energyrecnue:EnergyReco                          0.000352689   0.000992305   0.00176134    0.000927594   0.000552456       4     
reco:energyrecnc:EnergyReco                           0.00269282    0.00314417    0.00402033    0.00293177    0.000515625       4     
reco:energyrecnumurange:EnergyReco                    0.00273252    0.00463111    0.00865804    0.00356695    0.00237987        4     
reco:energyrecnumumcs:EnergyReco                      0.00270337    0.00451758    0.00823174     0.0035676    0.00220908        4     
reco:opdec:Deconvolution                               0.174882      0.201797      0.262201      0.185053      0.0354422        4     
reco:ophitspe:OpHitFinderDeco                           1.27047       1.60482       2.3426        1.40311       0.43341         4     
reco:opflash:OpFlashFinder                            0.00124272    0.00207212    0.00399087    0.00152745    0.00112517        4     
reco:rns:RandomNumberSaver                            2.5076e-05    9.20702e-05   0.000282991   3.0107e-05    0.000110249       4     
reco:anglereconue:NuAngularReco                       0.000257748   0.000646111   0.00120707    0.000559815   0.000350619       4     
reco:anglereconumu:NuAngularReco                      0.000179065   0.000364576   0.000542079   0.00036858    0.000130353       4     
reco:anglereconuepfps:NuAngularReco                   0.000268148   0.00103098    0.00220888    0.00082344    0.000729305       4     
reco:anglereconumupfps:NuAngularReco                  0.000241387   0.000425009   0.000593907   0.000432371   0.000130627       4     
[art]:TriggerResults:TriggerResultInserter            1.2423e-05    2.1213e-05    4.5969e-05     1.323e-05    1.42987e-05       4     
end_path:out1:RootOutput                               4.149e-06    7.66325e-06   1.5657e-05    5.4235e-06    4.64863e-06       4     
end_path:out1:RootOutput(write)                        0.262434      0.294645      0.370542      0.272802      0.044622         4     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6399.42 MB
  Peak resident set size usage (VmHWM): 1609.67 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:31:50 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 1794 subRun: 1 event: 29005
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-0979-23be-8b9a-a1a5.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_145_20231028T052642Z_gen_g4_detsim_hitreco__20240503T051038Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_145_20231028T052642Z_gen_g4_detsim_hitreco__20240503T051038Z_reco2_reco_2024-11-23T_072845Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work