Heteroscedastic Gaussian Process Research Articles

Replication or Exploration? Sequential Design for Stochastic Simulation Experiments

ABSTRACTWe investigate the merits of replication, and provide methods for optimal design (including replicates), with the goal of obtaining globally accurate emulation of noisy computer simulation experiments. We first show that replication can be beneficial from both design and computational perspectives, in the context of Gaussian process surrogate modeling. We then develop a lookahead-based sequential design scheme that can determine if a new run should be at an existing input location (i.e., replicate) or at a new one (explore). When paired with a newly developed heteroscedastic Gaussian process model, our dynamic design scheme facilitates learning of signal and noise relationships which can vary throughout the input space. We show that it does so efficiently, on both computational and statistical grounds. In addition to illustrative synthetic examples, we demonstrate performance on two challenging real-data simulation experiments, from inventory management and epidemiology. Supplementary materials for the article are available online.

Multi-frequency sparse Bayesian learning for matched field processing in non-stationary noise

Using simulations and data, we localize a quiet source in the presence of an interferer. The SWellEx-96 Event S59 consists of a submerged source towed along an isobath over a 65 min duration with an interferer traversing the source track. This range independent, multi-frequency scenario includes mismatch, non-stationary noise, and operational uncertainty. Mismatch is defined as a misalignment between the actual source field observed at the array and the modeled replica vector. The noise process changes likely with time. This is modelled as a heteroscedastic Gaussian process, meaning that the noise variance is non-stationary across snapshots. Sparse Bayesian learning (SBL) has been applied previously to the matched field processing application [Gemba et al, J. Acoust. Soc. Am., 141:3411-3420, 2017]. Results demonstrate that SBL exhibits desirable robustness properties and improved localization performance when compared to the white noise constraint and Bartlett processors.

Heteroscedastic Gaussian processes for uncertainty modeling in large-scale crowdsourced traffic data

Accurately modeling traffic speeds is a fundamental part of efficient intelligent transportation systems. Nowadays, with the widespread deployment of GPS-enabled devices, it has become possible to crowdsource the collection of speed information to road users (e.g. through mobile applications or dedicated in-vehicle devices). Despite its rather wide spatial coverage, crowdsourced speed data also brings very important challenges, such as the highly variable measurement noise in the data due to a variety of driving behaviors and sample sizes. When not properly accounted for, this noise can severely compromise any application that relies on accurate traffic data. In this article, we propose the use of heteroscedastic Gaussian processes (HGP) to model the time-varying uncertainty in large-scale crowdsourced traffic data. Furthermore, we develop a HGP conditioned on sample size and traffic regime (SSRC-HGP), which makes use of sample size information (probe vehicles per minute) as well as previous observed speeds, in order to more accurately model the uncertainty in observed speeds. Using 6 months of crowdsourced traffic data from Copenhagen, we empirically show that the proposed heteroscedastic models produce significantly better predictive distributions when compared to current state-of-the-art methods for both speed imputation and short-term forecasting tasks.

Humanoids skill learning based on real-time human motion imitation using Kinect

In this paper, a novel framework which enables humanoid robots to learn new skills from demonstration is proposed. The proposed framework makes use of real-time human motion imitation module as a demonstration interface for providing the desired motion to the learning module in an efficient and user-friendly way. This interface overcomes many problems of the currently used interfaces like direct motion recording, kinesthetic teaching, and immersive teleoperation. This method gives the human demonstrator the ability to control almost all body parts of the humanoid robot in real time (including hand shape and orientation which are essential to perform object grasping). The humanoid robot is controlled remotely and without using any sophisticated haptic devices, where it depends only on an inexpensive Kinect sensor and two additional force sensors. To the best of our knowledge, this is the first time for Kinect sensor to be used in estimating hand shape and orientation for object grasping within the field of real-time human motion imitation. Then, the observed motions are projected onto a latent space using Gaussian process latent variable model to extract the relevant features. These relevant features are then used to train regression models through the variational heteroscedastic Gaussian process regression algorithm which is proved to be a very accurate and very fast regression algorithm. Our proposed framework is validated using different activities concerned with both human upper and lower body parts and object grasping also.

Sensor Calibration and Hysteresis Compensation With Heteroscedastic Gaussian Processes

We deal with the problem of estimating the true measured scalar quantity from the output signal of a sensor that is afflicted with hysteresis and noise. We use a probabilistic, nonparametric sensor model based on heteroscedastic Gaussian processes (GPs), which is trained using a data set of sensor output and ground truth pairs. The inference problem is formulated as state estimation in a dynamical system. We exploit the low dimensionality of the latent state space of the sensor to perform exact probabilistic inference of the measured quantity from a time series of the sensor’s output. Compared with the state-of-the-art assumed density filtering algorithm for GPs, which analytically approximates the posterior by a normal distribution during inference, our method reduces the prediction error by 33% on a data set obtained from a novel flexible tactile sensor based on carbon-black filled elastomers. The proposed model can be applied, but is not limited, to any sensor for which the Preisach model of hysteresis holds. The use of probabilistic modeling and inference not only provides a most likely estimate of the measured quantity but also the corresponding confidence interval.

Experimental Sentinel-2 LAI estimation using parametric, non-parametric and physical retrieval methods – A comparison

Given the forthcoming availability of Sentinel-2 (S2) images, this paper provides a systematic comparison of retrieval accuracy and processing speed of a multitude of parametric, non-parametric and physically-based retrieval methods using simulated S2 data. An experimental field dataset (SPARC), collected at the agricultural site of Barrax (Spain), was used to evaluate different retrieval methods on their ability to estimate leaf area index (LAI). With regard to parametric methods, all possible band combinations for several two-band and three-band index formulations and a linear regression fitting function have been evaluated. From a set of over ten thousand indices evaluated, the best performing one was an optimized three-band combination according to (ρ560-ρ1610-ρ2190)/(ρ560+ρ1610+ρ2190) with a 10-fold cross-validation RCV2 of 0.82 (RMSECV: 0.62). This family of methods excel for their fast processing speed, e.g., 0.05s to calibrate and validate the regression function, and 3.8s to map a simulated S2 image. With regard to non-parametric methods, 11 machine learning regression algorithms (MLRAs) have been evaluated. This methodological family has the advantage of making use of the full optical spectrum as well as flexible, nonlinear fitting. Particularly kernel-based MLRAs lead to excellent results, with variational heteroscedastic (VH) Gaussian Processes regression (GPR) as the best performing method, with a RCV2 of 0.90 (RMSECV: 0.44). Additionally, the model is trained and validated relatively fast (1.70s) and the processed image (taking 73.88s) includes associated uncertainty estimates. More challenging is the inversion of a PROSAIL based radiative transfer model (RTM). After the generation of a look-up table (LUT), a multitude of cost functions and regularization options were evaluated. The best performing cost function is Pearson’s χ-square. It led to a R2 of 0.74 (RMSE: 0.80) against the validation dataset. While its validation went fast (0.33s), due to a per-pixel LUT solving using a cost function, image processing took considerably more time (01:01:47). Summarizing, when it comes to accurate and sufficiently fast processing of imagery to generate vegetation attributes, this paper concludes that the family of kernel-based MLRAs (e.g. GPR) is the most promising processing approach.

Probabilistic electricity price forecasting with variational heteroscedastic Gaussian process and active learning

Electricity price forecasting is essential for the market participants in their decision making. Nevertheless, the accuracy of such forecasting cannot be guaranteed due to the high variability of the price data. For this reason, in many cases, rather than merely point forecasting results, market participants are more interested in the probabilistic price forecasting results, i.e., the prediction intervals of the electricity price. Focusing on this issue, this paper proposes a new model for the probabilistic electricity price forecasting. This model is based on the active learning technique and the variational heteroscedastic Gaussian process (VHGP). It provides the heteroscedastic Gaussian prediction intervals, which effectively quantify the heteroscedastic uncertainties associated with the price data. Because the high computational effort of VHGP hinders its application to the large-scale electricity price forecasting tasks, we design an active learning algorithm to select a most informative training subset from the whole available training set. By constructing the forecasting model on this smaller subset, the computational efforts can be significantly reduced. In this way, the practical applicability of the proposed model is enhanced. The forecasting performance and the computational time of the proposed model are evaluated using the real-world electricity price data, which is obtained from the ANEM, PJM, and New England ISO.

Optimal design for correlated processes with input-dependent noise

Optimal design for parameter estimation in Gaussian process regression models with input-dependent noise is examined. The motivation stems from the area of computer experiments, where computationally demanding simulators are approximated using Gaussian process emulators to act as statistical surrogates. In the case of stochastic simulators, which produce a random output for a given set of model inputs, repeated evaluations are useful, supporting the use of replicate observations in the experimental design. The findings are also applicable to the wider context of experimental design for Gaussian process regression and kriging. Designs are proposed with the aim of minimising the variance of the Gaussian process parameter estimates. A heteroscedastic Gaussian process model is presented which allows for an experimental design technique based on an extension of Fisher information to heteroscedastic models. It is empirically shown that the error of the approximation of the parameter variance by the inverse of the Fisher information is reduced as the number of replicated points is increased. Through a series of simulation experiments on both synthetic data and a systems biology stochastic simulator, optimal designs with replicate observations are shown to outperform space-filling designs both with and without replicate observations. Guidance is provided on best practice for optimal experimental design for stochastic response models.

Random errors in carbon and water vapor fluxes assessed with Gaussian Processes

The flow of carbon between terrestrial ecosystems and the atmosphere is mainly driven by nonlinear, complex and time-lagged processes. Understanding the associated ecosystem responses is a key challenge regarding climate change questions such as the future development of the terrestrial carbon sink. However, high temporal resolution measurements of ecosystem variables (with the eddy covariance method) are subject to random error, that needs to be accounted for in model-data fusion, multi-site syntheses and up-scaling efforts.Gaussian Processes (GPs), a nonparametric regression method, have recently been shown to capture relationships in high-dimensional, nonlinear and noisy data. Heteroscedastic Gaussian Processes (HGPs) are a specialized GP method for data with inhomogeneous noise variance, such as eddy covariance measurements.Here, it is demonstrated that the HGP model captures measurement noise variances well, outperforming the model residual method and providing reasonable flux predictions at the same time. Based on meteorological drivers and temporal information, uncertainties of annual sums of carbon flux and water vapor flux at six different tower sites in Europe and North America are estimated. Similar noise patterns with different magnitudes were found across sites. Random uncertainties in annual sums of carbon fluxes were between 9.80 and 31.57gCm−2yr−1 (or 4–9% of the annual flux), and were between 2.54 and 8.13mmyr−1 (or 1–2% of the annual flux) for water vapor fluxes. The empirical HGP model offers a general method to estimate random errors at half-hourly resolution based on entire annual records of measurements. It is introduced as a new tool for random uncertainty assessment widely throughout the FLUXNET network.

Data fusion with Gaussian processes

This paper addresses the problem of fusing multiple sets of heterogeneous sensor data using Gaussian processes (GPs). Experiments on large scale terrain modeling in mining automation are presented. Three techniques in increasing order of model complexity are discussed. The first is based on adding data to an existing GP model. The second approach treats data from different sources as different noisy samples of a common underlying terrain and fusion is performed using heteroscedastic GPs. The final approach, based on dependent GPs, models each data set by a separate GP and learns spatial correlations between data sets through auto and cross covariances. The paper presents a unifying view of approaches to data fusion using GPs, a statistical evaluation that compares these approaches and multiple previously untested variants of them and an insight into the effect of model complexity on data fusion. Experiments suggest that in situations where data being fused is not rich enough to require a complex GP data fusion model or when computational resources are limited, the use of simpler GP data fusion techniques, which are constrained versions of the more generic models, reduces optimization complexity and consequently can enable superior learning of hyperparameters, resulting in a performance gain.

A class of stochastic volatility models for environmental applications

Many environmental data sets have a continuous domain, in time and/or space, and complex features that may be poorly modelled with a stationary (in space and time) Gaussian process (GP). We adapt stochastic volatility modelling to this context, resulting in a stochastic heteroscedastic process (SHP), which is unconditionally stationary and non‐Gaussian. Conditional on a latent GP, the SHP is a heteroscedastic GP with non‐stationary (in space and time) covariance structure. The realizations from SHP are versatile and can represent spatial inhomogeneities. The unconditional correlation functions of SHP form a rich isotropic class that can allow for a smoothed nugget effect. We apply an importance sampling strategy to implement pseudo maximum likelihood parameter estimation for the SHP. To predict the process at unobserved locations, we develop a plug‐in best predictor. We extend the single‐realization SHP model to handle replicates across time of SHP realizations in space. Empirical results with simulated data show that SHP is nearly as efficient as a stationary GP in out‐of‐sample prediction when the true process is a stationary GP, and outperforms a stationary GP substantially when the true process is SHP. The SHP methodology is applied to enhanced vegetation index data and US NO3 deposition data for illustration.

Heteroscedastic kernel ridge regression

In this paper we extend a form of kernel ridge regression (KRR) for data characterised by a heteroscedastic (i.e. input dependent variance) Gaussian noise process, introduced in Foxall et al. (in: Proceedings of the European Symposium on Artificial Neural Networks (ESANN-2002), Bruges, Belgium, April 2002, pp. 19–24). It is shown that the proposed heteroscedastic kernel ridge regression model can give a more accurate estimate of the conditional mean of the target distribution than conventional KRR and also provides an indication of the spread of the target distribution (i.e. predictive error bars). The leave-one-out cross-validation estimate of the conditional mean is used in fitting the model of the conditional variance in order to overcome the inherent bias in maximum likelihood estimates of the variance. The benefits of the proposed model are demonstrated on synthetic and real-world benchmark data sets and for the task of predicting episodes of poor air quality in an urban environment.