MML Inference of Hierarchical Probabilistic Finite State Machine

Abstract

A Finite State Machine (FSM) is a mathematical model of computation which can effectively model a sequence of words or tokens. A grammar representing a collection of tokens in a finite alphabet might contain regularities that are not fully captured by a deterministic formal grammar. Therefore, the simple FSM model is extended to include some probabilistic structure in the grammar which is now termed as Probabilistic Finite State Machine (PFSM). We extend earlier work on inferring PFSMs using the Bayesian informationtheoretic Minimum Message Length (MML) principle to the case of inferring hierarchical PFSMs (HPFSMs). HPFSMs consist of an outer PFSM whose states can internally contain a PFSM (or, recursively, an HPFSM). The alphabet of each such internally contained PFSM can be smaller than the complete HPFSM. HPFSMs can often represent the behaviour of a PFSM more concisely, and MML's ability to deal with both discrete structures and continuous probabilities renders MML well suited to this more general inference. We empirically compare on pseudo-random data-sets.