On the entropy of a hidden Markov process

Abstract

In this paper the entropy rate of a binary hidden Markov process (HMP) defined by observing the output of a binary symmetric channel whose input is a first-order binary Markov process is studied. Despite the simplicity of the models involved, the characterization of this entropy is a long standing open problem. By presenting the probability of a sequence under the model as a product of random matrices, and show that the entropy rate sought is a top Lyapunov exponent of the product, which explains the difficulty in its explicit computation. The same product of random matrices to derive an explicit expression for a first order Taylor approximation of the entropy rate with respect to the parameter of the binary symmetric channel is applied. The accuracy of the approximation is validated against empirical simulation results and also extends the results to Renyi's entropy of any order.