Minimum Message Length Principle Research Articles

Minimum Message Length Inference of the Exponential Distribution with Type I Censoring.

Data with censoring is common in many areas of science and the associated statistical models are generally estimated with the method of maximum likelihood combined with a model selection criterion such as Akaike’s information criterion. This manuscript demonstrates how the information theoretic minimum message length principle can be used to estimate statistical models in the presence of type I random and fixed censoring data. The exponential distribution with fixed and random censoring is used as an example to demonstrate the process where we observe that the minimum message length estimate of mean survival time has some advantages over the standard maximum likelihood estimate.

Heterogeneous Graphical Granger Causality by Minimum Message Length.

The heterogeneous graphical Granger model (HGGM) for causal inference among processes with distributions from an exponential family is efficient in scenarios when the number of time observations is much greater than the number of time series, normally by several orders of magnitude. However, in the case of “short” time series, the inference in HGGM often suffers from overestimation. To remedy this, we use the minimum message length principle (MML) to determinate the causal connections in the HGGM. The minimum message length as a Bayesian information-theoretic method for statistical model selection applies Occam’s razor in the following way: even when models are equal in their measure of fit-accuracy to the observed data, the one generating the most concise explanation of data is more likely to be correct. Based on the dispersion coefficient of the target time series and on the initial maximum likelihood estimates of the regression coefficients, we propose a minimum message length criterion to select the subset of causally connected time series with each target time series and derive its form for various exponential distributions. We propose two algorithms—the genetic-type algorithm (HMMLGA) and exHMML to find the subset. We demonstrated the superiority of both algorithms in synthetic experiments with respect to the comparison methods Lingam, HGGM and statistical framework Granger causality (SFGC). In the real data experiments, we used the methods to discriminate between pregnancy and labor phase using electrohysterogram data of Islandic mothers from Physionet databasis. We further analysed the Austrian climatological time measurements and their temporal interactions in rain and sunny days scenarios. In both experiments, the results of HMMLGA had the most realistic interpretation with respect to the comparison methods. We provide our code in Matlab. To our best knowledge, this is the first work using the MML principle for causal inference in HGGM.

Color object segmentation and tracking using flexible statistical model and level-set

This study presents an unsupervised novel algorithm for color image segmentation, object detection and tracking based on unsupervised learning step followed with a post processing step implemented with a variational active contour. Flexible learning method of a finite mixture of bounded generalized Gaussian distributions using the Minimum Message Length (MML) principle is developed to cope with the complexity of color images modeling. We deal here simultaneously with the issues of data-model fitting, determining automatically the optimal number of classes and selecting relevant features. Indeed, a feature selection step based on MML is implemented to eliminate uninformative features and therefore improving the algorithm’s performance. For model’s parameters estimation, the maximum likelihood (ML) was investigated and conducted via expectation maximization (EM) algorithm. The obtained object boundaries in the first step are tracked on each frame of a given sequence using a geometric level-set approach. The implementation has the advantage to help in improving the computational efficiency in high-dimensional spaces. We demonstrate the effectiveness of the developed segmentation method through several experiments. Obtained results reveal that our approach is able to achieve higher precision as compared to several other methods for color image segmentation and object tracking.

Minimum message length inference of the Poisson and geometric models using heavy-tailed prior distributions

Minimum message length is a general Bayesian principle for model selection and parameter estimation that is based on information theory. This paper applies the minimum message length principle to a small-sample model selection problem involving Poisson and geometric data models. Since MML is a Bayesian principle, it requires prior distributions for all model parameters. We introduce three candidate prior distributions for the model parameters with both light- and heavy-tails. The performance of the MML methods is compared with objective Bayesian inference and minimum description length techniques based on the normalized maximum likelihood code. Simulations show that our MML approach with a heavy-tail prior distribution performs well in all tests.

Minimum message length analysis of multiple short time series

This paper applies the Bayesian minimum message length principle to the multiple short time series problem, yielding satisfactory estimates for all model parameters as well as a test for autocorrelation. Connections with the method of conditional likelihood are also discussed.

Knowledge acquisition with forgetting: an incremental and developmental setting

Identifying the balance between remembering and forgetting is the key to abstraction in the human brain and, therefore, the creation of memories and knowledge. We present an incremental, lifelong view of knowledge acquisition which tries to improve task after task by determining what to keep, consolidate and forget, overcoming the stability–plasticity dilemma. Our framework can combine any rule-based inductive engine (which learns new rules) with a deductive engine (which derives a coverage graph for all rules) and integrate them into a lifelong learner. We rate rules by introducing several metrics through the first adaptation, to our knowledge, of the minimum message length (MML) principle to a coverage graph, a hierarchical assessment structure which handles evidence and rules in a unified way. The metrics are used to forget some of the worst rules and also to consolidate those selected rules that are promoted to the knowledge base. This mechanism is also mirrored by a demotion system. We evaluate the framework with a series of tasks in a chess rule learning domain.

Model selection using Minimal Message Length: an example using pollen data

In this paper we examine the use of the minimum message length criterion in the process of evaluating alternative models of data when the samples are serially ordered in space and implicitly in time. Much data from vegetation studies can be arranged in a sequence and in such cases the user may elect to constrain the clustering by zones, in preference to an unconstrained clustering. We use the minimum message length principle to determine if such a choice provides an effective model of the data. Pollen data provide a suitably organised set of samples, but have other properties which make it desirable to examine several different models for the distribution of palynomorphs within the clusters. The results suggest that zonation is not a particularly preferred model since it captures only a small part of the patterns present. It represents a user expectation regarding the nature of variation in the data and results in some patterns being neglected. By using unconstrained clustering within zones, we can recover some of this overlooked pattern. We then examine other evidence for the nature of change in vegetation and finally discuss the usefulness of the minimum message length as a guiding principle in model choice and its relationship to other possible criteria.

Wallace's Approach to Unsupervised Learning: The Snob Program

We describe the Snob program for unsupervised learning as it has evolved from its beginning in the 1960s until its present form. Snob uses the minimum message length principle expounded in Wallace and Freeman (Wallace, C.S. and Freeman, P.R. (1987) Estimation and inference by Compact coding. J. Roy. Statist. Soc. Ser. B, 49, 240– 252.) and we indicate how Snob estimates class parameters using the approach of that paper. We will survey the evolution of Snob from these beginnings to the state that it has reached as described by Wallace and Dowe (Wallace, C.S. and Dowe, D.L. (2000) MMM mixture modelling of multi-state, Poisson, Von Mises Circular and Gaussian distributions. Stat. Comput., 10, 73–83.) We pay particular attention to the revision of Snob in the 1980s where definite assignment of things to classes was abandoned.

High-Dimensional Unsupervised Selection and Estimation of a Finite Generalized Dirichlet Mixture Model Based on Minimum Message Length

We consider the problem of determining the structure of high-dimensional data, without prior knowledge of the number of clusters. Data are represented by a finite mixture model based on the generalized Dirichlet distribution. The generalized Dirichlet distribution has a more general covariance structure than the Dirichlet distribution and offers high flexibility and ease of use for the approximation of both symmetric and asymmetric distributions. This makes the generalized Dirichlet distribution more practical and useful. An important problem in mixture modeling is the determination of the number of clusters. Indeed, a mixture with too many or too few components may not be appropriate to approximate the true model. Here, we consider the application of the minimum message length (MML) principle to determine the number of clusters. The MML is derived so as to choose the number of clusters in the mixture model which best describes the data. A comparison with other selection criteria is performed. The validation involves synthetic data, real data clustering, and two interesting real applications: classification of web pages, and texture database summarization for efficient retrieval.

The Mimetic Technique when the Original Data is Not Available: Model Learning and Revision

The mimetic technique (or Multiple Combined Models, CMM) basically consists of using a generally accurate but incomprehensible model as an oracle to generate and label a random data set. This dataset is used, along with the original training data, to train a second comprehensible model, known as the mimetic model. This technique has been used to provide understandability to black box models without considerably sacrificing their accuracy. In this work we study the mimetic application in a scenario in which the original training data is not available. In this context we first determine the optimal size of the random data set, according to the minimum message length principle (MML). This result can be used in knowledge acquisition for expert systems. Secondly we apply the mimetic technique to model revision and show that in some change situations the mimetic model can be used as a transition model between the original model and the new model.

Unsupervised selection of a finite Dirichlet mixture model: an MML-based approach

This paper proposes an unsupervised algorithm for learning a finite Dirichlet mixture model. An important part of the unsupervised learning problem is determining the number of clusters which best describe the data. We extend the minimum message length (MML) principle to determine the number of clusters in the case of Dirichlet mixtures. Parameter estimation is done by the expectation-maximization algorithm. The resulting method is validated for one-dimensional and multidimensional data. For the one-dimensional data, the experiments concern artificial and real SAP image histograms. The validation for multidimensional data involves synthetic data and two real applications: shadow detection in images and summarization of texture image databases for efficient retrieval. A comparison with results obtained for other selection criteria is provided

Christopher S. WallaceStatistical and Inductive Inference by Minimum Message Length. Springer (2005). ISBN 038723795X. £46.00. 432 pp. Hardbound.

This book is about the Minimum Message Length (MML) Principle, an information-theoretic approach to induction, hypothesis testing, model selection and statistical inference. MML, which can be seen as a mathematically precise version of Occam's Razor, asserts that the ‘best’ explanation of the observed data is the shortest. The book is essentially the manuscript left behind by Professor Chris Wallace when he died on August 7, 2004. Professor Wallace was a remarkably versatile scientist: originally trained as a physicist, he has made significant contributions to various areas of computer science including computer architecture, arithmetic and simulation. Simultaneously, he is the main originator of the MML Principle, which has implications for applied and theoretical statistics as well as for the philosophy of science. MML was developed between 1968 and 2004 in a series of papers by Wallace and several coworkers such as D. Boulton, P. Freeman and D. Dowe. These papers mostly concern practical applications of MML to problems of statistics and machine learning, as well as the mathematical development of the underlying ideas. Since these papers were often quite short and technical, there was much need of an extensive introduction to, and comprehensive overview of, the MML idea. As its first contribution, this book provides such a, most welcome, introduction. Yet its aim and scope are much wider: as its second contribution, the book presents MML as a general theory of inductive inference, and there is extensive discussion of its philosophical foundations and implications. Thus, the book will appeal to a broad audience: the technical part of the book is mainly of interest to researchers in applied machine learning, data mining and statistics who wish to learn about a non-main stream but practically successful, generally applicable statistical method. The philosophical part is interesting for researchers in these fields, as well as philosophers of science, who will enjoy Wallace's original ideas on the foundations of statistical and inductive inference.

Hierarchical clusters of vegetation types

In this paper, we examine possible sources of hierarchical (nested) structure in vegetation data. We then use the Minimum Message length principle to provide a rational means of comparing hierarchical and non-hierarchical clustering. The results indicate that, with the data used, a hierarchical solution was not as efficient as a nonhierarchical one. However, the hierarchical solution seems to provide a more comprehensible solution, separating first isolated types, probably caused from unusual contingent events, then subdividing the more diverse areas before finally subdividing the less diverse. By presenting this in 3 stages, the complexity of the non-hierarchical result is avoided. The result also suggests that a hierarchical analysis may be useful in determining homogeneous areas.

Optimal classification to describe environmental change: pictures from the exposition

In this paper we examine the impact of runnelling on the vegetation of a salt marsh. Runnelling is a form of habitat modification used for mosquito control in Australia. Defining the states of the system through unsupervised clustering of vegetation records using the minimum message length principle, 11 states (or classes) were identified. The runnelled sites have a greater diversity of states present than the unrunnelled ones. The states at each time for each site were then used to develop transition matrices. From these, two different pathways were identified, indicating the patterns of change. The method of showing changes relied on pictures that represent average species size and density. Both the two main pathways of change started with the dominant grass (Sporobolus). One led to a reduction in Sporobolous and ended in bare ground; the other included changes involving variation in the size and density of a mix of Sporobolus and Sarcocornia. The effects can be interpreted in terms of the increased access of seawater to the marsh resulting in an extension of the lower marsh. We note, however, that this methodology does not distinguish between changes of state within a single process and changes associated with a change in the actual processes operating.

Minimum message length clustering: an explication and some applications to vegetation data

In this paper, we examine the application of a particular approach to induction, the minimum message length principle and illustrate some of the problems that can be addressed through its use. The MML principle seeks to identify an optimal model within some specified parameterised class of models and for this paper we have chosen to concentrate on a single model class, that of mixture separation or fuzzy clustering. The first section presents, in outline, an MML methodology for fuzzy clustering. We then present some applications, including the nature of the within-cluster model, examination of the univocality of results for different groups of species and the effectiveness of presence data compared to purely quantitative data. Finally, we examine some possibilities of extending MML methodology to include within-class correlation of species, the existence of dependence between observed samples and the comparison of different classes of models.

Minimum message length clustering, environmental heterogeneity and the variable Poisson model

One possible explanation of variation in vegetation is based on the variable Poisson model. In this model, species occurrence is presumed to follow a Poisson distribution, but the value of the Poisson parameter for any species varies from point to point, as a result of environmental variation. As an extreme, this includes dividing the given habitat into areas favourable to a community and areas which are unfavourable, or at least not occupied. The spatial area can then be viewed as a series of patches within which each species follows a Poisson distribution, although different patchesmay have different values for the Poisson parameter for any particular species. In this paper, I use a method of fuzzy clustering (mixture modelling) based on the minimum message length principle to examine the variation in Poisson parameter of individual species. The method uses the difference between the message length for the null, 1-cluster case and the message length for the optimal cluster solution, appropriately normal...

Complexity Approximation Principle

We propose a new inductive principle, which we call the complexity approximation principle (CAP). This principle is a natural generalization of Rissanen’s minimum description length (MDL) principle and Wallace’s minimum message length (MML) principle and is based on the notion of predictive complexity, a recent generalization of Kolmogorov complexity. Like the MDL principle, CAP can be regarded as an implementation of Occam’s razor.

A beam search algorithm for PFSA inference

In the past, many methods have been proposed for the inference of probabilistic and non-probabilistic finite state automata from positive examples of their behaviour. In this paper, we introduce a search method guided by the information-theoretic Minimum Message Length principle to infer Probabilistic Finite State Automata (PFSA).1 The method is a beam search technique that searches for the best PFSA that accounts for a given dataset. Results of testing this method against some earlier algorithms are presented. A simulated annealing version of the beam search algorithm is also described as ongoing research in the area.