On Renyi's entropy estimation with one-dimensional Gaussian kernels

Abstract

Rényi's entropies play a significant role in many signal processing applications. Plug-in kernel density estimation methods have been employed to estimate such entropies with good results. However, they become computationally intractable in higher dimensions, because of the requirement to store intermediate probability density values for a large number of data points. We propose a method to reduce the number of the samples in a plug-in kernel density estimation method for Rényi's entropies of real exponents and to improve the result of the standard plug-in kernel density method. To this end, we derive a univariate estimator, using an Hermite expansion of sums of Gaussian kernels and a hierarchical clustering of the samples. On simulated data from a univariate Gaussian distribution, our method performs better than a k-nearest neigbour algorithm and other kernel density estimation methods.