Probabilistic Model Selection Research Articles

A Model Selection Algorithm for a Posteriori Probability Estimation With Neural Networks

This paper proposes a novel algorithm to jointly determine the structure and the parameters of a posteriori probability model based on neural networks (NNs). It makes use of well-known ideas of pruning, splitting, and merging neural components and takes advantage of the probabilistic interpretation of these components. The algorithm, so called a posteriori probability model selection (PPMS), is applied to an NN architecture called the generalized softmax perceptron (GSP) whose outputs can be understood as probabilities although results shown can be extended to more general network architectures. Learning rules are derived from the application of the expectation-maximization algorithm to the GSP-PPMS structure. Simulation results show the advantages of the proposed algorithm with respect to other schemes.

Decision making on the sole basis of statistical likelihood

This paper presents a new axiomatic decision theory for choice under uncertainty. Unlike Bayesian decision theory where uncertainty is represented by a probability function, in our theory, uncertainty is given in the form of a likelihood function extracted from statistical evidence. The likelihood principle in statistics stipulates that likelihood functions encode all relevant information obtainable from experimental data. In particular, we do not assume any knowledge of prior probabilities. Consequently, a Bayesian conversion of likelihoods to posterior probabilities is not possible in our setting. We make an assumption that defines the likelihood of a set of hypotheses as the maximum likelihood over the elements of the set. We justify an axiomatic system similar to that used by von Neumann and Morgenstern for choice under risk. Our main result is a representation theorem using the new concept of binary utility. We also discuss how ambiguity attitudes are handled. Applied to the statistical inference problem, our theory suggests a novel solution. The results in this paper could be useful for probabilistic model selection.

Model selection for support vector machine classification

We address the problem of model selection for Support Vector Machine (SVM) classification. For fixed functional form of the kernel, model selection amounts to tuning kernel parameters and the slack penalty coefficient C. We begin by reviewing a recently developed probabilistic framework for SVM classification. An extension to the case of SVMs with quadratic slack penalties is given and a simple approximation for the evidence is derived, which can be used as a criterion for model selection. We also derive the exact gradients of the evidence in terms of posterior averages and describe how they can be estimated numerically using Hybrid Monte-Carlo techniques. Though computationally demanding, the resulting gradient ascent algorithm is a useful baseline tool for probabilistic SVM model selection, since it can locate maxima of the exact (unapproximated) evidence. We then perform extensive experiments on several benchmark data sets. The aim of these experiments is to compare the performance of probabilistic model selection criteria with alternatives based on estimates of the test error, namely the so-called “span estimate” and Wahba's Generalized Approximate Cross-Validation (GACV) error. We find that all the “simple” model criteria (Laplace evidence approximations, and the span and GACV error estimates) exhibit multiple local optima with respect to the hyperparameters. While some of these give performance that is competitive with results from other approaches in the literature, a significant fraction lead to rather higher test errors. The results for the evidence gradient ascent method show that also the exact evidence exhibits local optima, but these give test errors which are much less variable and also consistently lower than for the simpler model selection criteria.

Probability model selection using information-theoretic optimization criterion

Probability models with discrete random varibales are often used for probabilistic inference and decision support. A fundamental issue lies in the choice and the validity of the probability model. An information theoretic-based approach for probability model selection is discussed. It will be shown that the problem of probability model selection is discussed. It will be shown that the problem of probability model selection can be formulated as an optimization problem of probability model selection can be formulated as on optimization problem with linear (in)equality constraints and a non-linear objective function. An algorithm for model discovery/selection based on a primal–dual formulation similar to that of the interior point method is presented. The implementation of the algorithm for solving an algebraic system of linear constraints is based on singular value decomposition and the numerical method proposed by Kuenzi, Tzschach and Zehnder. Preliminary comparative evaluation is also discussed.