Entropy invariance for autoregressive processes constructed by linear filtering

Abstract

In signal processing and in computational techniques for applied mathematics, linear filtering is an important way of turning a time series with independent innovations into a process with highly dependent signals. Stationary autoregressive processes are some of the best-studied linear modes, and their construction involves either full or soft (randomized) linear filters. The entropy rate (per unit time entropy) of a stationary time series is an important quantitative characteristic. When filtering, the entropy for n-dimensional blocks should change as ‘the order’ increases, but the entropy rate of the process could remain invariant. This is the main issue addressed in the paper, where we prove that full or soft linear filtering for Gaussian or uniform innovations produce no change in the entropy rate of the process. That is, linear operators applied to independent innovations do not add or wipe entropy from the innovations family. The plug-in estimators of the entropy rates are also provided, and they can be used to characterize Gaussian or uniform stationary sources. A simulation study is conducted in order to validate the theoretical results and to give a statistical characterization of the plug-in estimator we propose for the entropy rate. Not only it is easy to calculate and has a good precision, but it bridges the lack for nonparametric estimators for the entropy rate of stationary AR processes.