Minimum Description Length Model Selection Research Articles

Description Length Guided Unified Granger Causality Analysis

In this article, we propose a description length guided unified Granger causality analysis (uGCA) framework for sequential medical imaging. While existing efforts of GCA focused on causal relation design and statistical methods for their improvement, our strategy adopts the minimum description length (MDL) principle in the GCA procedure where the MDL principle offers a unified model selection criteria for deciding the optimal model in the sense of description length. Under this framework, we present different description length forms of linear Granger representations under several coding schemes that all achieve the lower bounds on redundancy, thus producing valid MDL model selection criteria. The efforts are validated using a 5-node network synthetic experiment, illustrating its potential advantage over conventional two-stage approach. The subtle distinction between the performance of different uGCA forms is investigated as well. More importantly, the proposed approach gives a more similar network topology than conventional approach in a challenging fMRI dataset, in which neural correlates of mental calculation elicited by visual and auditory stimulation (respectively) in the same task paradigm, allowing one to evaluate the performance of different GCA methods.

On consistency of minimum description length model selection for piecewise autoregressions

The Auto-PARM (Automatic Piecewise AutoRegressive Modeling) procedure, developed by Davis et al. (2006), uses the minimum description length (MDL) principle to estimate the number and locations of structural breaks in a non-stationary time series. Consistency of this model selection procedure has been established when using conditional maximum (Gaussian) likelihood variance estimates. In contrast, the estimate of the number of change-points is inconsistent in general if Yule–Walker variance estimates are used instead. This surprising result is due to an exact cancellation of first-order terms in a Taylor series expansion in the conditional maximum likelihood case, which does not occur in the Yule–Walker case. In order to simplify notation and make the arguments more transparent, we only treat in detail the simple case where the time series follows an AR(p) model with no change-points.

Theory and applications of proper scoring rules

A scoring rule $$S(x; q)$$ provides a way of judging the quality of a quoted probability density $$q$$ for a random variable $$X$$ in the light of its outcome $$x$$ . It is called proper if honesty is your best policy, i.e., when you believe $$X$$ has density $$p$$ , your expected score is optimised by the choice $$q = p$$ . The most celebrated proper scoring rule is the logarithmic score, $$S(x; q) = -\log {q(x)}$$ : this is the only proper scoring rule that is local, in the sense of depending on the density function $$q$$ only through its value at the observed value $$x$$ . It is closely connected with likelihood inference, with communication theory, and with minimum description length model selection. However, every statistical decision problem induces a proper scoring rule, so there is a very wide variety of these. Many of them have additional interesting structure and properties. At a theoretical level, any proper scoring rule can be used as a foundational basis for the theory of subjective probability. At an applied level a proper scoring can be used to compare and improve probability forecasts, and, in a parametric setting, as an alternative tool for inference. In this article we give an overview of some uses of proper scoring rules in statistical inference, including frequentist estimation theory and Bayesian model selection with improper priors.

Consistency of minimum description length model selection for piecewise stationary time series models

This paper establishes the consistency of the minimum description length (MDL) model selection procedure by [10, 11] for a class of non-stationary time series models. We consider a time series model in which the observations are viewed as coming from stationary segments. In other words, the data are assumed to come from a general time series model in which the parameters change at break-points. Each of these segments is modeled by a pre-specified family of parametric stationary time series models. [10, 11] formulated the above problem and used the minimum description length (MDL) principle to estimate the number of break-points, the location of the break-points, the order of the parametric model and the parameter values in each of the segments. The procedure performed well on a variety of examples. In this paper we show consistency of their minimal MDL model selection procedure under general regularity conditions on the likelihood function. Results about the rate of convergence of the break-point-location estimator are also given. Applications are considered for detecting changes in independent random variables, and in ARMA and GARCH processes.

A Probabilistic Interaction Model for Multipitch Tracking With Factorial Hidden Markov Models

We present a simple and efficient feature modeling approach for tracking the pitch of two simultaneously active speakers. We model the spectrogram features of single speakers using Gaussian mixture models in combination with the minimum description length model selection criterion. To obtain a probabilistic representation for the speech mixture spectrogram features of both speakers, we employ the mixture maximization model (MIXMAX) and, as an alternative, a linear interaction model. A factorial hidden Markov model is applied for tracking pitch over time. This statistical model can be used for applications beyond speech, whenever the interaction between individual sources can be represented as MIXMAX or linear model. For tracking, we use the loopy max-sum algorithm, and provide empirical comparisons to exact methods. Furthermore, we discuss a scheduling mechanism of loopy belief propagation for online tracking. We demonstrate experimental results using Mocha-TIMIT as well as data from the speech separation challenge provided by Cooke We show the excellent performance of the proposed method in comparison to a well known multipitch tracking algorithm based on correlogram features. Using speaker-dependent models, the proposed method improves the accuracy of correct speaker assignment, which is important for single-channel speech separation. In particular, we are able to reduce the overall tracking error by 51% relative for the speaker-dependent case. Moreover, we use the estimated pitch trajectories to perform single-channel source separation, and demonstrate the beneficial effect of correct speaker assignment on speech separation performance.

Minimum description length model selection of multinomial processing tree models

Multinomial processing tree (MPT) modeling has been widely and successfully applied as a statistical methodology for measuring hypothesized latent cognitive processes in selected experimental paradigms. In this article, we address the problem of selecting the best MPT model from a set of scientifically plausible MPT models, given observed data. We introduce a minimum description length (MDL) based model-selection approach that overcomes the limitations of existing methods such as the G(2)-based likelihood ratio test, the Akaike information criterion, and the Bayesian information criterion. To help ease the computational burden of implementing MDL, we provide a computer program in MATLAB that performs MDL-based model selection for any MPT model, with or without inequality constraints. Finally, we discuss applications of the MDL approach to well-studied MPT models with real data sets collected in two different experimental paradigms: source monitoring and pair clustering. The aforementioned MATLAB program may be downloaded from http://pbr.psychonomic-journals.org/content/supplemental.

An empirical study of minimum description length model selection with infinite parametric complexity

Parametric complexity is a central concept in Minimum Description Length (MDL) model selection. In practice it often turns out to be infinite, even for quite simple models such as the Poisson and geometric families. In such cases, MDL model selection as based on NML and Bayesian inference based on Jeffreys’ prior cannot be used. Several ways to resolve this problem have been proposed. We conduct experiments to compare and evaluate their behaviour on small sample sizes. We find interestingly poor behaviour for the plug-in predictive code; a restricted NML model performs quite well but it is questionable if the results validate its theoretical motivation. A Bayesian marginal distribution with Jeffreys’ prior can still be used if one sacrifices the first observation to make a proper posterior; this approach turns out to be most dependable.

Optimisation of Gaussian mixture model for satellite image classification

A new methodology for classifying multidimensional satellite remote-sensing data is proposed. This technique is based on the Gaussian mixture modelling of the feature vectors extracted from the satellite image and the Bayesian approach to pattern recognition. The key contribution is the optimisation of the Gaussian mixture model for each class based on the training data. An array of techniques are employed for this purpose, including attribute learning vector quantisation, minimum description length model selection, semi-tied covariance matrices, minimum-classification-error learning, fusion of classifiers and the genetic algorithm. Contextual analysis is also developed for recognition. Experimental results on thematic mapper satellite image data demonstrate that the proposed technique outperforms various existing methods for pattern learning and classification.

Modified AIC and MDL Model Selection Criteria for Short Data Records

The classical model selection rules such as Akaike information criterion (AIC) and minimum description length (MDL) have been derived assuming that the number of samples (measurements) is much larger than the number of estimated model parameters. For short data records, AIC and MDL have the tendency to select overly complex models. This paper proposes modified AIC and MDL rules with improved finite sample behavior. They are useful in those measurement applications where gathering a sample is very time consuming and/or expensive.

Evidence accumulation in decision making: unifying the "take the best" and the "rational" models.

An evidence accumulation model of forced-choice decision making is proposed to unify the fast and frugal take the best (TTB) model and the alternative rational (RAT) model with which it is usually contrasted. The basic idea is to treat the TTB model as a sequential-sampling process that terminates as soon as any evidence in favor of a decision is found and the rational approach as a sequential-sampling process that terminates only when all available information has been assessed. The unified TTB and RAT models were tested in an experiment in which participants learned to make correct judgments for a set of real-world stimuli on the basis of feedback, and were then asked to make additional judgments without feedback for cases in which the TTB and the rational models made different predictions. The results show that, in both experiments, there was strong intraparticipant consistency in the use of either the TTB or the rational model but large interparticipant differences in which model was used. The unified model is shown to be able to capture the differences in decision making across participants in an interpretable way and is preferred by the minimum description length model selection criterion.

Context-tree modeling of observed symbolic dynamics.

Modern techniques invented for data compression provide efficient automated algorithms for the modeling of the observed symbolic dynamics. We demonstrate the relationship between coding and modeling, motivating the well-known minimum description length (MDL) principle, and give concrete demonstrations of the "context-tree weighting" and "context-tree maximizing" algorithms. The predictive modeling technique obviates many of the technical difficulties traditionally associated with the correct MDL analyses. These symbolic models, representing the symbol generating process as a finite-state automaton with probabilistic emission probabilities, provide excellent and reliable entropy estimations. The resimulations of estimated tree models satisfying the MDL model-selection criterion are faithful to the original in a number of measures. The modeling suggests that the automated context-tree model construction could replace fixed-order word lengths in many traditional forms of empirical symbolic analysis of the data. We provide an explicit pseudocode for implementation of the context-tree weighting and maximizing algorithms, as well as for the conversion to an equivalent Markov chain.

Robust full Bayesian learning for radial basis networks.

We propose a hierarchical full Bayesian model for radial basis networks. This model treats the model dimension (number of neurons), model parameters, regularization parameters, and noise parameters as unknown random variables. We develop a reversible-jump Markov chain Monte Carlo (MCMC) method to perform the Bayesian computation. We find that the results obtained using this method are not only better than the ones reported previously, but also appear to be robust with respect to the prior specification. In addition, we propose a novel and computationally efficient reversible-jump MCMC simulated annealing algorithm to optimize neural networks. This algorithm enables us to maximize the joint posterior distribution of the network parameters and the number of basis function. It performs a global search in the joint space of the parameters and number of parameters, thereby surmounting the problem of local minima to a large extent. We show that by calibrating the full hierarchical Bayesian prior, we can obtain the classical Akaike information criterion, Bayesian information criterion, and minimum description length model selection criteria within a penalized likelihood framework. Finally, we present a geometric convergence theorem for the algorithm with homogeneous transition kernel and a convergence theorem for the reversible-jump MCMC simulated annealing method.