A joint Gaussian process model of geochemistry, geophysics, and temperature for groundwater TDS in the San Ardo Oil Field, California, USA

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

Physics-aware Gaussian processes in remote sensing

Machine-Learning Assisted X-Ray Spectroscopy for High- Throughput Characterization of Magnetic Materials

This contribution presents a new development in the design of control system based on evolving Gaussian process (GP) models . GP models provide a probabilistic, nonparametric modelling approach for black-box identification of nonlinear dynamic systems. They can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. GP models contain noticeably less coefficients to be optimised than commonly used parametric models. While GP models are Bayesian models, their output is normal distribution, expressed in terms of mean and variance. Latter can be interpreted as a confidence in prediction and used in many fields, especially in control system. Evolving GP model is the concept approach within which various ways of model adaptations can be used. Successful control system needs as much as possible data about process to be controlled . If the prior knowledge about the system to be controlled is scarce or the system varies with time or operating region, this control problem can be solved with an iterative method which adapts model with information obtained with streaming data and concurrently optimises hyperparameter values. This contribution provides: a survey of adaptive control algorithms for dynamic systems described in publications where GP models have been used for control design, a novel and improved closed-loop controller with evolving GP models and an example for the illustration of proposed control algorithm.

This chapter introduces a nonparametric approach for longitudinal data modeling and prediction. This approach is based on the Gaussian process (GP) model. GP is a stochastic process and can be viewed as a distribution over functions with a continuous domain, e.g. time or space. The chapter also introduces the structure of the GP model. It discusses the estimation and prediction methods for the GP model, examines the pairwise GP model. The chapter presents the extension of a single output GP model to the general Multiple Output Gaussian Process (MOGP) model, which plays a critical role in longitudinal data prediction. It discusses the time-to-failure distribution based on the MOGP model. The chapter provides the basic structure of MOGP and explains the MOGP based prediction method. The Gaussian process, also known as Kriging method, is a very flexible yet effective nonparametric model to describe smooth functional data.

Gaussian process (GP) models are nowadays considered among the state-of-the-art tools in modern dynamic system identification. GP models are probabilistic, non-parametric models based on the principles of Bayesian probability. As a kernel methods they do not try to approximate the modelled system by fitting the parameters of the selected basis functions, but rather by searching for relationships among the measured data. While GP models are Bayesian models they are more robust to overfitting. Moreover, their output is normal distribution, expressed in terms of mean and variance. Due to these features they are used in various fields, e.g. model-based control, time-series prediction, modelling and estimation in engineering applications, etc. But, due to the matrix inversion calculation, whose computationally demand increases with the third power of the number of input data, the amount of training data is limited to at most a few thousand cases. Therefore GP models in principle are not applicable for modelling dynamic systems whose states evolve in time, such as chaotic time-series. In this paper we demonstrate an Evolving GP (EGP) models for predicting chaotic time-series. The EGP is iterative method which adapts model with information obtained from streaming data and concurrently optimizes hyperparameter values. To assess the viability of the EGP an empirical tests were carried out together with a comparative study of various evolving fuzzy methods on a benchmark chaotic time-series MacKey-Glass. The results indicate that the EGP can successfully identify MacKey-Glass chaotic time-series and demonstrate superior performance.

We propose a generalized Gaussian process model (GGPM), which is a unifying framework that encompasses many existing Gaussian process (GP) models, such as GP regression, classification, and counting. In the GGPM framework, the observation likelihood of the GP model is itself parameterized using the exponential family distribution. By deriving approximate inference algorithms for the generalized GP model, we are able to easily apply the same algorithm to all other GP models. Novel GP models are created by changing the parameterization of the likelihood function, which greatly simplifies their creation for task-specific output domains. We also derive a closed-form efficient Taylor approximation for inference on the model, and draw interesting connections with other model-specific closed-form approximations. Finally, using the GGPM, we create several new GP models and show their efficacy in building task-specific GP models for computer vision.

In this paper we introduce Gaussian Process (GP) models for music genre classification. Gaussian Processes are widely used for various regression and classification tasks, but there are relatively few studies where GPs are applied in the audio signal processing systems. The GP models are non-parametric discriminative classifiers similar to the well known SVMs in terms of usage. In contrast to SVMs, however, GP models produce truly probabilistic output and allow for kernel function parameters to be learned from the training data. In this work we compare the performance of GP models and SVMs as music genre classifiers using the ISMIR 2004 database. Audio preprocessing is the same for both cases and is based on Constant-Q spectrograms. The experimental results using linear as well as exponential kernel functions and different amounts of training data show that GP models always outperform SVMs with up to 5.6% absolute difference in the classification accuracy.

Based on Gaussian Process (GP), a wavelength selection algorithm named Synergy Interval Gaussian Process (siGP) model is proposed in this paper by using near infrared spectroscopy technology. Full spectrum is divided into a series of unique and equal spacing intervals, before selecting optimal several intervals to establish GP model. Due to the GP model with nonlinear processing ability, the method reduces the disadvantages of nonlinear factor. Taking the near infrared spectrum data of moisture content and pH in solid-state fermentation of monascus as performance verification object of this new algorithm, the prediction correlation coefficient (Rp) of moisture content and pH are 0.956 4 and 0.977 3, respectively. The root mean square errors for prediction set (RMSEP) are 0.012 7 and 0.161 0, respectively. Data points participating in modeling decrease respectively from the original 1 500 to 225 and 375. In the prediction for independent samples, it shows good accuracy. Comparing with traditional synergy interval partial least squares (siPLS) algorithm, the results show that the siGP achieves the best prediction result. The prediction correlation coefficient of moisture content and pH in new algorithm has increased respectively by 3.37% and 3.51% under the model of Gaussian Process, with increases of 29.4% and 34.8% in the root mean square errors for prediction set. This study shows that the combination of siGP and GP model can select wavelength effectively and improves the prediction accuracy of the NIR model. This method is reference for realizing the online detection and optimization control.

Quantitative prediction of subjective pain intensity from whole-brain fMRI data using Gaussian processes

Systems of Gaussian process models for directed chains of solvers

Bayesian optimization with Gaussian Process (GP) models has been proposed for analog synthesis since it is efficient for the optimizations of expensive black-box functions. However, the computational cost for training and prediction of Gaussian process models are O(N3) and O(N2), respectively, where N is the number of data points. The overhead of the Gaussian process modeling would not be negligible as N is relatively large. Recently, a Bayesian optimization approach using neural network has been proposed to address this problem. It reduces the computational cost of training and prediction of Gaussian process models to O(N) and O(1), respectively. However, reducing the infinite-dimensional kernel to finite-dimensional kernel using neural network mapping would weaken the characterization ability of Gaussian process. In this paper, we propose a novel Bayesian optimization approach using Sparse Pseudo-input Gaussian Process (SPGP). The idea is to use M < N so-called inducing points to build a sparse Gaussian process model to approximate the conventional exact Gaussian process model. Without the need to sacrifice the modeling ability of the surrogate model, it also reduces the computational cost of both training and prediction to O(N) and O(1), respectively. Several experiments were provided to demonstrate the efficiency of the proposed approach.

Control system based on evolving Gaussian process (GP) models is an example of self-learning closed-loop control system. It is meant for closed-loop control of dynamic systems where not much prior knowledge exists or where systems dynamics varies with time or operating region. GP models are non-parametric black-box models which represent a new method for system identification. GP models differ from most other frequently used black-box identification approaches as they do not try to approximate the modeled system by fitting the parameters of the selected basis functions, but rather search for the relationships among measured data. While GP models are Bayesian models, their output is normal distribution, expressed in terms of mean and variance. Latter can be interpreted as a confidence in prediction and used in many fields, especially in control system. Successful control system needs as much as possible data about process to be controlled. If the prior knowledge about the system to be controlled is scarce or the system varies with time or operating region, this control problem can be solved with an iterative method which adapts model with information obtained with streaming data and concurrently optimizes hyperparameter values. While that kind of method for GP models does not yet exist, concepts for evolving GP models and control system based on evolving GP models are proposed in this paper. It is flexible approach within which various ways of model adaptations can be used. One of those possibilities is illustrated with a control of a benchmark problem.

Gaussian process (GP) model based optimization is widely applied in simulation and machine learning. In general, it first estimates a GP model based on a few observations from the true response and then uses this model to guide the search, aiming to quickly locate the global optimum. Despite its successful applications, it has several limitations that may hinder its broader use. First, building an accurate GP model can be difficult and computationally expensive, especially when the response function is multimodal or varies significantly over the design space. Second, even with an appropriate model, the search process can be trapped in suboptimal regions before moving to the global optimum because of the excessive effort spent around the current best solution. In this work, we adopt the additive global and local GP (AGLGP) model in the optimization framework. The model is rooted in the inducing points based GP sparse approximations and is combined with independent local models in different regions. With these properties, the AGLGP model is suitable for multimodal responses with relatively large data sizes. Based on this AGLGP model, we propose a combined global and local search for optimization (CGLO) algorithm. It first divides the whole design space into disjoint local regions and identifies a promising region with the global model. Next, a local model in the selected region is fit to guide detailed search within this region. The algorithm then switches back to the global step when a good local solution is found. The global and local natures of CGLO enable it to enjoy the benefits of both global and local search to efficiently locate the global optimum. Summary of Contribution: This work proposes a new Gaussian process based algorithm for stochastic simulation optimization, which is an important area in operations research. This type of algorithm is also regarded as one of the state-of-the-art optimization algorithms for black-box functions in computer science. The aim of this work is to provide a computationally efficient optimization algorithm when the baseline functions are highly nonstationary (the function values change dramatically across the design space). Such nonstationary surfaces are very common in reality, such as the case in the maritime traffic safety problem considered here. In this problem, agent-based simulation is used to simulate the probability of collision of one vessel with the others on a given trajectory, and the decision maker needs to choose the trajectory with the minimum probability of collision quickly. Typically, in a high-congestion region, a small turn of the vessel can result in a very different conflict environment, and thus the response is highly nonstationary. Through our study, we find that the proposed algorithm can provide safer choices within a limited time compared with other methods. We believe the proposed algorithm is very computationally efficient and has large potential in such operational problems.

In the 1960s E. Fama developed the efficient market hypothesis (EMH) which asserts that the financial market is efficient if its prices are formed on the basis of all publicly available information. That means technical analysis cannot be used to predict and beat the market. Since then, it was widely examined and was mostly accepted by mathematicians and financial engineers. However, the predictability of financial-market returns remains an open problem and is discussed in many publications. Usually, it is concluded that a model able to predict financial returns should adapt to market changes quickly and catch local dependencies in price movements. The Bayesian vector autoregression (BVAR) models, support vector machines (SVM) and some other were already applied to financial data quite succesfully. Gaussian process (GP) models are emerging non-parametric Bayesian models and in this paper we test their applicability to financial data. GP model is fitted to daily data from U.S. commodity markets. For a comparison BVAR model and benchmark model that is commonly used in todays financial mathematics are chosen. The results indicate that GP models are applicable to financial data as well as BVAR models.