Estimation of the Entropy Functional from Dependent Samples

Abstract

The differential entropy is importantly used in many disciplines, where the estimation of entropy is often the main research objective or the first step toward it. To estimate entropy, plug-in estimators, such as histogram based entropy estimators or kernel based entropy estimators, are commonly used. Especially, though the histogram itself performs poorly in estimating density, the histogram based entropy estimator is often employed due to its computational benefit. Many efforts have been made to understand the properties of the histogram based entropy estimator theoretically, but most of such efforts are restricted to the case of independently and identically distributed (IID) samples. In this article, we show that two histogram-based entropy estimators by Gyórfi and van der Meulen (1987) are almost surely consistent when samples are from a φ-mixing process. A limited simulation study is implemented to compare those two estimators and to investigate their performance for varying intensity of dependency. In addition, we discuss the extension of -consistency of the estimators in IID setting by Hall (1990) to the case of dependent samples.