Predictive models and generative complexity

Abstract

The causal states of computational mechanics define the minimal sufficient memory for a given discrete stationary stochastic process. Their entropy is an important complexity measure called statistical complexity (or true measure complexity). They induce the ɛ-machine, which is a hidden Markov model (HMM) generating the process. But it is not the minimal one, although generative HMMs also have a natural predictive interpretation. This paper gives a mathematical proof of the idea that the ɛ-machine is the minimal HMM with an additional (partial) determinism condition. Minimal internal state entropy of a generative HMM is in analogy to statistical complexity called generative complexity. This paper also shows that generative complexity depends on the process in a nice way. It is, as a function of the process, lower semi-continuous (w.r.t. weak-* topology), concave, and behaves nice under ergodic decomposition of the process.