Gaussian Process Inference Research Articles

Adaptive Importance Sampling for Bayesian Inference in Gaussian Process models

Gaussian process (GP) models are nowadays considered among the standard tools in modern control system engineering. They are routinely used for model-based control, time- series prediction, modelling and estimation in engineering applications. While the underlying theory is completely in line with the principles of Bayesian inference, in practice this property is lost due to approximation steps in the GP inference. In this paper we propose a novel inference algorithm for GP models, which relies on adaptive importance sampling strategy to numerically evaluate the intractable marginalization over the hyperparameters. This is required in the case of broad-peaked or multi-modal posterior distribution of the hyperparameters where the point approximations turn out to be insufficient. The benefits of the algorithm are that is retains the Bayesian nature of the inference, has sufficient convergence properties, relatively low computational load and does not require heavy prior knowledge due to its adaptive nature. All the key advantages are demonstrated in practice using numerical examples.

Multiple instance learning via Gaussian processes

Multiple instance learning (MIL) is a binary classification problem with loosely supervised data where a class label is assigned only to a bag of instances indicating presence/absence of positive instances. In this paper we introduce a novel MIL algorithm using Gaussian processes (GP). The bag labeling protocol of the MIL can be effectively modeled by the sigmoid likelihood through the max function over GP latent variables. As the non-continuous max function makes exact GP inference and learning infeasible, we propose two approximations: the soft-max approximation and the introduction of witness indicator variables. Compared to the state-of-the-art MIL approaches, especially those based on the Support Vector Machine, our model enjoys two most crucial benefits: (i) the kernel parameters can be learned in a principled manner, thus avoiding grid search and being able to exploit a variety of kernel families with complex forms, and (ii) the efficient gradient search for kernel parameter learning effectively leads to feature selection to extract most relevant features while discarding noise. We demonstrate that our approaches attain superior or comparable performance to existing methods on several real-world MIL datasets including large-scale content-based image retrieval problems.

Tigre: Transcription factor inference through gaussian process reconstruction of expression for bioconductor

tigre is an R/Bioconductor package for inference of transcription factor activity and ranking candidate target genes from gene expression time series. The underlying methodology is based on Gaussian process inference on a differential equation model that allows the use of short, unevenly sampled, time series. The method has been designed with efficient parallel implementation in mind, and the package supports parallel operation even without additional software. The tigre package is included in Bioconductor since release 2.6 for R 2.11. The package and a user's guide are available at http://www.bioconductor.org.

Online Sparse Gaussian Process Regression and Its Applications

We present a new Gaussian process (GP) inference algorithm, called online sparse matrix Gaussian processes (OSMGP), and demonstrate its merits by applying it to the problems of head pose estimation...

Online Sparse Gaussian Process Regression and Its Applications

We present a new Gaussian process (GP) inference algorithm, called online sparse matrix Gaussian processes (OSMGP), and demonstrate its merits by applying it to the problems of head pose estimation and visual tracking. The OSMGP is based upon the observation that for kernels with local support, the Gram matrix is typically sparse. Maintaining and updating the sparse Cholesky factor of the Gram matrix can be done efficiently using Givens rotations. This leads to an exact, online algorithm whose update time scales linearly with the size of the Gram matrix. Further, we provide a method for constant time operation of the OSMGP using matrix downdates. The downdates maintain the Cholesky factor at a constant size by removing certain rows and columns corresponding to discarded training examples. We demonstrate that, using these matrix downdates, online hyperparameter estimation can be included at cost linear in the number of total training examples. We describe a robust appearance-based head pose estimation system based upon the OSMGP. Numerous experiments and comparisons with existing methods using a large dataset system demonstrate the efficiency and accuracy of our system. Further, to showcase the applicability of OSMGP to a wide variety of problems, we also describe a regression-based visual tracking method. Experiments show that our OSMGP algorithm generalizes well using online learning.

Gaussian Processes for Machine Learning (GPML) Toolbox

The GPML toolbox provides a wide range of functionality for Gaussian process (GP) inference and prediction. GPs are specified by mean and covariance functions; we offer a library of simple mean and covariance functions and mechanisms to compose more complex ones. Several likelihood functions are supported including Gaussian and heavy-tailed for regression as well as others suitable for classification. Finally, a range of inference methods is provided, including exact and variational inference, Expectation Propagation, and Laplace's method dealing with non-Gaussian likelihoods and FITC for dealing with large regression tasks.

Kernels, regularization and differential equations

Many common machine learning methods such as support vector machines or Gaussian process inference make use of positive definite kernels, reproducing kernel Hilbert spaces, Gaussian processes, and regularization operators. In this work these objects are presented in a general, unifying framework and interrelations are highlighted. With this in mind we then show how linear stochastic differential equation models can be incorporated naturally into the kernel framework. And vice versa, many kernel machines can be interpreted in terms of differential equations. We focus especially on ordinary differential equations, also known as dynamical systems, and it is shown that standard kernel inference algorithms are equivalent to Kalman filter methods based on such models. In order not to cloud qualitative insights with heavy mathematical machinery, we restrict ourselves to finite domains, implying that differential equations are treated via their corresponding finite difference equations.

Sparse gaussian processes: inference, subspace identification and model selection

Gaussian Process (GP) inference is a probabilistic kernel method where the GP is treated as a latent function. The inference is carried out using the Bayesian online learning and its extension to the more general iterative approach which we call TAP/EP learning (short for TAP (Opperand Winther, 2001) and "expectation-propagation" (EP) (Minka. 2000)).Sparsity is introduced in this context to make the TAP/EP method applicable to large datasets. We address the prohibitive scaling of the number of parameters by defining a subset of the training data that is used as the support the GP. thus the number of required parameters is independent of the training set, similar to the case of "Support-" or "Relevance-Vectors".An advantage of the full probabilistic treatment is that allows the computation of the marginal data likelihood or evidence, leading to hyper-parameter estimation within the GP inference. An EM algorithm to choose the hyper-parameters is proposed. The TAP/EP learning is the E-step and the M-step then updates the hyper-parameters. Due to the sparse E-step the resulting algorithm does not involve manipulation of large matrices. The presented algorithm is applicable to a wide variety of likelihood functions. We present results of applying the algorithm on classification and nonstandard regression problems for artificial and real datasets.

Statistical estimation of nonstationary Gaussian processes with long-range dependence and intermittency

This paper considers statistical inference for nonstationary Gaussian processes with long-range dependence and intermittency. The existence of such a process has been established by Anh et al. (J. Statist. Plann. Inference 80 (1999) 95–110). We systematically consider the case where the spectral density of nonstationary Gaussian processes with stationary increments is of a general and flexible form. The spectral density function of fRBm is thus a special case of this general form. A continuous version of the Gauss–Whittle objective function is proposed. Estimation procedures for the parameters involved in the spectral density function are then investigated. Both the consistency and the asymptotic normality of the estimators of the parameters are established. In addition, a real example is presented to demonstrate the applicability of the estimation procedures.