On Hidden Markov Processes with Infinite Excess Entropy

Łukasz Dębowski

doi:10.1007/s10959-012-0468-6

Abstract

We investigate stationary hidden Markov processes for which mutual information between the past and the future is infinite. It is assumed that the number of observable states is finite and the number of hidden states is countably infinite. Under this assumption, we show that the block mutual information of a hidden Markov process is upper bounded by a power law determined by the tail index of the hidden state distribution. Moreover, we exhibit three examples of processes. The first example, considered previously, is nonergodic and the mutual information between the blocks is bounded by the logarithm of the block length. The second example is also nonergodic but the mutual information between the blocks obeys a power law. The third example obeys the power law and is ergodic.

Highlights

In recent years, there has been a surge of interdisciplinary interest in excess entropy, which is the Shannon mutual information between the past and the future of a stationary discrete-time process
The initial motivation for this interest was a paper by Hilberg [22], who supposed that certain processes with infinite excess entropy may be useful for modeling texts in natural language
We showed that a power-law growth of mutual information between adjacent blocks of text arises when the text describes certain facts in a logically consistent and highly repetitive way

Summary

Introduction

There has been a surge of interdisciplinary interest in excess entropy, which is the Shannon mutual information between the past and the future of a stationary discrete-time process. We will study several examples of stationary hidden Markov processes over a finite alphabet for which excess entropy is infinite.The first study of such processes was developed by Travers and Crutchfield [26]. To allow for hidden Markov processes with unbounded mutual information, we need to assume that the number of hidden states is at least countably infinite.

Objectives

Results

Conclusion