Non-sufficient Memories That Are Sufficient for Prediction

Abstract

The causal states of computational mechanics define the minimal sufficient (prescient) memory for a given stationary stochastic process. They induce the e-machine which is a hidden Markov model (HMM) generating the process. The e-machine is, however, not the minimal generative HMM and minimal internal state entropy of a generative HMM is a tighter upper bound for excess entropy than provided by statistical complexity. We propose a notion of prediction that does not require sufficiency. The corresponding models can be substantially smaller than the e-machine and are closely related to generative HMMs.