Heteroscedastic Gaussian Process Research Articles

Safe Corridor Learning For Model Predictive Path Following Control

Safe collision-free operation of autonomous systems, such as mobile robots in crowded, uncertain, only partially known environments, is challenging. We propose learning a collision-free corridor from demonstration via heteroscedastic Gaussian processes. We incorporate available deterministic obstacle information in the learning procedure to derive safety guarantees for the corridor. The learned passage is utilized in a model predictive path planning controller that steers the system safely through the environment. The achievable results are underlined in simulations considering a mobile robot.

Achieving robustness to aleatoric uncertainty with heteroscedastic Bayesian optimisation

Bayesian optimisation is a sample-efficient search methodology that holds great promise for accelerating drug and materials discovery programs. A frequently-overlooked modelling consideration in Bayesian optimisation strategies however, is the representation of heteroscedastic aleatoric uncertainty. In many practical applications it is desirable to identify inputs with low aleatoric noise, an example of which might be a material composition which displays robust properties in response to a noisy fabrication process. In this paper, we propose a heteroscedastic Bayesian optimisation scheme capable of representing and minimising aleatoric noise across the input space. Our scheme employs a heteroscedastic Gaussian process surrogate model in conjunction with two straightforward adaptations of existing acquisition functions. First, we extend the augmented expected improvement heuristic to the heteroscedastic setting and second, we introduce the aleatoric noise-penalised expected improvement (ANPEI) heuristic. Both methodologies are capable of penalising aleatoric noise in the suggestions. In particular, the ANPEI acquisition yields improved performance relative to homoscedastic Bayesian optimisation and random sampling on toy problems as well as on two real-world scientific datasets. Code is available at: https://github.com/Ryan-Rhys/Heteroscedastic-BO

Modelling and forecasting of SHM strain measurement for a large-scale suspension bridge during typhoon events using variational heteroscedastic Gaussian process

The modelling and forecasting (M&F) of strain measurement (as a kind of local structural responses) during typhoon events provides valuable insight into the structural condition assessment of large suspension bridges. However, the presence of time-dependent noise in reality can pose difficulties for forecasting the field data obtained by structural health monitoring (SHM) systems. Gaussian process regression (GPR), as a nonparametric model, can obtain probabilistic estimation outputs, but its constant noise assumption hampers the reliability of the forecasting model. In this study, Variational Heteroscedastic Gaussian Process (VHGP), a combination of variational approximation and heteroscedastic Gaussian process (HGP), is applied to perform modelling and forecasting for SHM strain field data during typhoon events because of its heteroskedasticity characteristics, higher forecasting accuracy and strong ability to quantify uncertainty. The proposed M&F method is exemplified by using SHM monitoring strain data acquired from the instrumented Tsing Ma Suspension Bridge during typhoon events. The results reveal that VHGP has a better regression accuracy and can obtain varying confidence intervals which reflect noise variations. Meanwhile, VHGP yields more robust forecasting results. The uncertainty analysis shows that VHGP is competent to evaluate the noise level change of strain responses brought by typhoons, providing a basis for conducting structural health condition assessment for large-scale bridges.

A novel hybrid approach based on variational heteroscedastic Gaussian process regression for multi-step ahead wind speed forecasting

Accurate wind speed forecasting is the key to safe and economic operation of electric power and energy systems. As a Bayesian nonparametric method, Gaussian process regression (GPR) has provided competitive forecasting results in recent years. However, conventional GPR model assumes that the noise obeys Gaussian distribution and the variance of the noise in the whole data set is a constant, which is not appropriate for some problems. Motivated by this, this study makes the first attempt to study the ability of the variational heteroscedastic GPR (VHGPR) model in wind speed forecasting. The Marginalized Variational (MV) approximation is employed to approximate the heteroscedastic Gaussian process in the VHGPR model. What’s more, the complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN) is employed to transfer the nonstationary wind speed series into a certain number of subseries with more regularity such that the forecasting performance of VHGPR can be enhanced. The proposed method is compared with the other twelve benchmark models for 1- to 3- step ahead wind speed forecasting. Experiments results on four real-world datasets demonstrate that VHGPR with the CEEMDAN decomposition strategy is able to obtain better forecasting results for wind speed time series.

Modulating scalable Gaussian processes for expressive statistical learning

For a learning task, Gaussian process (GP) is interested in learning the statistical relationship between inputs and outputs, since it offers not only the prediction mean but also the associated variability. The vanilla GP however is hard to learn complicated distribution with the property of, e.g., heteroscedastic noise, multi-modality and non-stationarity, from massive data due to the Gaussian marginal and the cubic complexity. To this end, this article studies new scalable GP paradigms including the non-stationary heteroscedastic GP, the mixture of GPs and the latent GP, which introduce additional latent variables to modulate the outputs or inputs in order to learn richer, non-Gaussian statistical representation. Particularly, we resort to different variational inference strategies to arrive at analytical or tighter evidence lower bounds (ELBOs) of the marginal likelihood for efficient and effective model training. Extensive numerical experiments against state-of-the-art GP and neural network (NN) counterparts on various tasks verify the superiority of these scalable modulated GPs, especially the scalable latent GP, for learning diverse data distributions.

Top-of-Atmosphere Retrieval of Multiple Crop Traits Using Variational Heteroscedastic Gaussian Processes within a Hybrid Workflow.

In support of cropland monitoring, operational Copernicus Sentinel-2 (S2) data became available globally and can be explored for the retrieval of important crop traits. Based on a hybrid workflow, retrieval models for six essential biochemical and biophysical crop traits were developed for both S2 bottom-of-atmosphere (BOA) L2A and S2 top-of-atmosphere (TOA) L1C data. A variational heteroscedastic Gaussian process regression (VHGPR) algorithm was trained with simulations generated by the combined leaf-canopy reflectance model PROSAILat the BOA scale and further combined with the Second Simulation of a Satellite Signal in the Solar Spectrum (6SV) atmosphere model at the TOA scale. Established VHGPR models were then applied to S2 L1C and L2A reflectance data for mapping: leaf chlorophyll content (Cab), leaf water content (Cw), fractional vegetation coverage (FVC), leaf area index (LAI), and upscaled leaf biochemical compounds, i.e., LAI * Cab (laiCab) and LAI * Cw (laiCw). Estimated variables were validated using in situ reference data collected during the Munich-North-Isar field campaigns within growing seasons of maize and winter wheat in the years 2017 and 2018. For leaf biochemicals, retrieval from BOA reflectance slightly outperformed results from TOA reflectance, e.g., obtaining a root mean squared error (RMSE) of 6.5 μg/cm2 (BOA) vs. 8 μg/cm2 (TOA) in the case of Cab. For the majority of canopy-level variables, instead, estimation accuracy was higher when using TOA reflectance data, e.g., with an RMSE of 139 g/m2 (BOA) vs. 113 g/m2 (TOA) for laiCw. Derived maps were further compared against reference products obtained from the ESA Sentinel Application Platform (SNAP) Biophysical Processor. Altogether, the consistency between L1C and L2A retrievals confirmed that crop traits can potentially be estimated directly from TOA reflectance data. Successful mapping of canopy-level crop traits including information about prediction confidence suggests that the models can be transferred over spatial and temporal scales and, therefore, can contribute to decision-making processes for cropland management.

Large-Scale Heteroscedastic Regression via Gaussian Process.

Heteroscedastic regression considering the varying noises among observations has many applications in the fields, such as machine learning and statistics. Here, we focus on the heteroscedastic Gaussian process (HGP) regression that integrates the latent function and the noise function in a unified nonparametric Bayesian framework. Though showing remarkable performance, HGP suffers from the cubic time complexity, which strictly limits its application to big data. To improve the scalability, we first develop a variational sparse inference algorithm, named VSHGP, to handle large-scale data sets. Furthermore, two variants are developed to improve the scalability and capability of VSHGP. The first is stochastic VSHGP (SVSHGP) that derives a factorized evidence lower bound, thus enhancing efficient stochastic variational inference. The second is distributed VSHGP (DVSHGP) that follows the Bayesian committee machine formalism to distribute computations over multiple local VSHGP experts with many inducing points and adopts hybrid parameters for experts to guard against overfitting and capture local variety. The superiority of DVSHGP and SVSHGP compared to the existing scalable HGP/homoscedastic GP is then extensively verified on various data sets.

Danish partnership sets out to build world's first commercial scale green ammonia plant

Electricity price forecasting is essential for the market participants in their decision making. Nevertheless, the accuracy of such forecasting cannot be guaranteed due to the high variability of the price data. For this reason, in many cases, rather than merely point forecasting results, market participants are more interested in the probabilistic price forecasting results, i.e., the prediction intervals of the electricity price. Focusing on this issue, this paper proposes a new model for the probabilistic electricity price forecasting. This model is based on the active learning technique and the variational heteroscedastic Gaussian process (VHGP). It provides the heteroscedastic Gaussian prediction intervals, which effectively quantify the heteroscedastic uncertainties associated with the price data. Because the high computational effort of VHGP hinders its application to the large-scale electricity price forecasting tasks, we design an active learning algorithm to select a most informative training subset from the whole available training set. By constructing the forecasting model on this smaller subset, the computational efforts can be significantly reduced. In this way, the practical applicability of the proposed model is enhanced. The forecasting performance and the computational time of the proposed model are evaluated using the real-world electricity price data, which is obtained from the ANEM, PJM, and New England ISO.

Intelligent Sampling for Vegetation Nitrogen Mapping Based on Hybrid Machine Learning Algorithms

Upcoming satellite imaging spectroscopy missions will deliver spatiotemporal explicit data streams to be exploited for mapping vegetation properties, such as nitrogen (N) content. Within retrieval workflows for real-time mapping over agricultural regions, such crop-specific information products need to be derived precisely and rapidly. To allow fast processing, intelligent sampling schemes for training databases should be incorporated to establish efficient machine learning (ML) models. In this study, we implemented active learning (AL) heuristics using kernel ridge regression (KRR) to minimize and optimize a training database for variational heteroscedastic Gaussian processes regression (VHGPR) to estimate aboveground N content. Several uncertainty and diversity criteria were applied on a lookup table (LUT) composed of aboveground N content and corresponding hyperspectral reflectance simulated by the PROSAIL-PRO model. The best-performing AL criteria were Euclidian distance-based diversity (EBD) resulting in a reduction of the LUT training data set by 81% (50 initial samples plus 141 samples selected from a pool of 1000 samples). This reduced LUT was used for training VHGPR, which is not only a competitive algorithm but also provides uncertainty estimates. Validation against in situ N reference data provided excellent results with a root-mean-square error (RMSE) of 1.84 g/m2 and a coefficient of determination (R2 ) of 0.92. Mapping aboveground N content over an agricultural region yielded reliable estimates and meaningful associated uncertainties. These promising results encourage the transfer of such hybrid workflows into space and time within the frame of future operational N monitoring from satellite imaging spectroscopy data.

Robust weighted Gaussian processes

This paper presents robust weighted variants of batch and online standard Gaussian processes (GPs) to effectively reduce the negative impact of outliers in the corresponding GP models. This is done by introducing robust data weighers that rely on robust and quasi-robust weight functions that come from robust M-estimators. Our robust GPs are compared to various GP models on four datasets. It is shown that our batch and online robust weighted GPs are indeed robust to outliers, significantly outperforming the corresponding standard GPs and the recently proposed heteroscedastic GP method GPz. Our experiments also show that our methods are comparable to and sometimes better than a state-of-the-art robust GP that uses a Student-t likelihood.

A digital workflow for learning the reduced-order structure-property linkages for permeability of porous membranes

Quantitative relationships between the complex porous structure of a membrane (henceforth simply referred to as microstructure) and its effective permeability are critical for improving the performance of membranes used in filtration and separation applications. This paper presents a digital workflow for learning the porous structure-permeability linkages in membranes. The presented workflow establishes the desired linkages by bringing together recent advances in (i) digital generators for three-dimensional representative volume elements (3-D RVEs) reflecting a large and diverse set of porous structures, (ii) numerical approaches for reliable evaluation of permeability of 3D-RVEs, (iii) low dimensional representation of material internal structure using the framework of 2-point spatial correlations and principal component analyses, and (iv) Gaussian process (GP) regression with input-dependent noise (i.e., heteroscedasticity). It is seen that the digital workflow presented in this study can systematically identify the salient features of the 3-D membrane microstructure and train reduced-order heteroscedastic GP models on the data generated using digital microstructure generators and physics-based permeability simulations. It will be shown that the structure-property linkages are able to make high fidelity predictions and assessment of uncertainties for new porous membrane structures at minimal computational cost.

Retrieval of aboveground crop nitrogen content with a hybrid machine learning method

Hyperspectral acquisitions have proven to be the most informative Earth observation data source for the estimation of nitrogen (N) content, which is the main limiting nutrient for plant growth and thus agricultural production. In the past, empirical algorithms have been widely employed to retrieve information on this biochemical plant component from canopy reflectance. However, these approaches do not seek for a cause-effect relationship based on physical laws. Moreover, most studies solely relied on the correlation of chlorophyll content with nitrogen, and thus neglected the fact that most N is bound in proteins. Our study presents a hybrid retrieval method using a physically-based approach combined with machine learning regression to estimate crop N content. Within the workflow, the leaf optical properties model PROSPECT-PRO including the newly calibrated specific absorption coefficients (SAC) of proteins, was coupled with the canopy reflectance model 4SAIL to PROSAIL-PRO. The latter was then employed to generate a training database to be used for advanced probabilistic machine learning methods: a standard homoscedastic Gaussian process (GP) and a heteroscedastic GP regression that accounts for signal-to-noise relations. Both GP models have the property of providing confidence intervals for the estimates, which sets them apart from other machine learners. Moreover, a GP-based sequential backward band removal algorithm was employed to analyze the band-specific information content of PROSAIL-PRO simulated spectra for the estimation of aboveground N. Data from multiple hyperspectral field campaigns, carried out in the framework of the future satellite mission Environmental Mapping and Analysis Program (EnMAP), were exploited for validation. In these campaigns, corn and winter wheat spectra were acquired to simulate spectral EnMAP data. Moreover, destructive N measurements from leaves, stalks and fruits were collected separately to enable plant-organ-specific validation. The results showed that both GP models can provide accurate aboveground N simulations, with slightly better results of the heteroscedastic GP in terms of model testing and against in situ N measurements from leaves plus stalks, with root mean square error (RMSE) of 2.1 g/m². However, the inclusion of fruit N content for validation deteriorated the results, which can be explained by the inability of the radiation to penetrate the thick tissues of stalks, corn cobs and wheat ears. GP-based band analysis identified optimal spectral settings with ten bands mainly situated in the shortwave infrared (SWIR) spectral region. Use of well-known protein absorption bands from the literature showed comparative results. Finally, the heteroscedastic GP model was successfully applied on airborne hyperspectral data for N mapping. We conclude that GP algorithms, and in particular the heteroscedastic GP, should be implemented for global agricultural monitoring of aboveground N from future imaging spectroscopy data.

A Data-Driven Approach to Prediction and Optimal Bucket-Filling Control for Autonomous Excavators

We develop a data-driven, statistical control method for autonomous excavators. Interactions between soil and an excavator bucket are highly complex and nonlinear, making traditional physical modeling difficult to use for real-time control. Here, we propose a data-driven method, exploiting data obtained from laboratory tests. We use the data to construct a nonlinear, non-parametric statistical model for predicting the behavior of soil scooped by an excavator bucket. The prediction model is built for controlling the amount of soil collected with a bucket. An excavator collects soil by dragging the bucket along the soil surface and scooping the soil by rotating the bucket. It is important to switch from the drag phase to the scoop phase with the correct timing to ensure an appropriate amount of soil has accumulated in front of the bucket. We model the process as a heteroscedastic Gaussian process (GP) based on the observation that the variance of the collected soil mass depends on the scooping trajectory, i.e., the input, as well as the shape of the soil surface immediately prior to scooping. We develop an optimal control algorithm for switching from the drag phase to the scoop phase at an appropriate time and for generating a scoop trajectory to capture a desired amount of soil with high confidence. We implement the method on a robotic excavator and collect experimental data. Experiments show promising results in terms of being able to achieve a desired bucket fill factor with low variance.

An improved long short-term memory network for streamflow forecasting in the upper Yangtze River

Characterized by essential complexity, dynamism, and dynamics, streamflow forecasting presents a great challenge to hydrologists. Long short-term memory (LSTM) streamflow forecast model has received a lot of attention in recent years due to its powerful non-linear modeling ability. But probabilistic streamflow forecasting has rarely been addressed by the LSTM approach. In this study, a probabilistic Long Short-Term Memory network coupled with the Gaussian process (GP) is proposed to deal with the probabilistic daily streamflow forecasting. Moreover, considering that changing mean and variance over time exist in the daily streamflow time series, the heteroscedastic Gaussian process regression is adopted to produce a varying prediction interval. The proposed method encapsulates the inductive biases of the LSTM recurrent network and retains the non-parametric, probabilistic property of Gaussian processes. The performance of the proposed model is investigated by predicting the daily streamflow time series collected from the upper Yangtze River and its tributaries. Artificial neuron network, generalized linear model, heteroscedastic GP, and regular LSTM models are also developed for comparison. Results indicated that the performance of the proposed model is satisfying. It improves prediction accuracy as well as provides an adaptive prediction interval, which is of great significance for water resources management and planning.

FOM: Fourth-order moment based causal direction identification on the heteroscedastic data

Identification of the causal direction is a fundamental problem in many scientific research areas. The independence between the noise and the cause variable is a widely used assumption to identify the causal direction. However, such an independence assumption is usually violated due to heteroscedasticity of the real-world data. In this paper, we propose a new criterion for the causal direction identification which is robust to the heteroscedasticity of the data. In detail, the fourth-order moment of noise is proposed to measure the asymmetry between the cause and effect. A heteroscedastic Gaussian process regression-based estimation of the fourth-order moment is proposed accordingly. Under some commonly used assumptions of the causal mechanism, we theoretically show that the noise’s fourth-order moment of the causal direction is smaller than that of the anti-causal direction. Experimental results on both simulated and real-world data illustrate the efficiency of the proposed approach.

Improved Most Likely Heteroscedastic Gaussian Process Regression via Bayesian Residual Moment Estimator

This paper proposes an improved most likely heteroscedastic Gaussian process (MLHGP) algorithm to handle a kind of nonlinear regression problems involving input-dependent noise. The improved MLHGP follows the same learning scheme as the current algorithm by use of two Gaussian processes (GPs), with the first GP for recovering the unknown function and the second GP for modeling the input-dependent noise. Unlike the current MLHGP pursuing an empirical estimate of the noise level which is provably biased in most of local noise cases, the improved algorithm gives rise to an approximately unbiased estimate of the input-dependent noise. The approximately unbiased noise estimate is elicited from Bayesian residuals by the method of moments. As a by-product of this improvement, the expectation maximization (EM)-like procedure in the current MLHGP is avoided such that the improved algorithm requires only standard GP learnings to be performed twice. Four benchmark experiments, consisting of two synthetic cases and two real-world datasets, demonstrate that the improved MLHGP algorithm outperforms the current version not only in accuracy and stability, but also in computational efficiency.

Toward a generalized predictive model of grapevine water status in Douro region from hyperspectral data

The predawn leaf water potential (ѱpd) is an eco-physiological indicator widely used for assessing vines water status and thus supporting irrigation management in several wine regions worldwide. However, the ѱpd is measured in a short time period before sunrise and the collection of a large sample of points is necessary to adequately represent a vineyard, which constitute operational constraints. In the present study, an alternative method based on hyperspectral data derived from a handheld spectroradiometer and machine learning algorithms was tested and validated for assessing grapevine water status. Two test sites in Douro wine region, integrating three grapevine cultivars, were studied for the years of 2014, 2015, and 2017. Four machine learning regression algorithms were tested for predicting the ѱpd as a continuous variable, namely Random Forest (RF), Bagging Trees (BT), Gaussian Process Regression (GPR), and Variational Heteroscedastic Gaussian Process Regression (VH-GPR). Three predicting variables, including two vegetation indices (NRI554,561 and WI900,970) and a time-dynamic variable based on the ѱpd (ѱpd_0), were applied for modelling the response variable (ѱpd). Additionally, the predicted values of ѱpd were aggregated into three classes representing different levels of water deficit (low, moderate, and high) and compared with the corresponding classes of ѱpd observed values. A root mean square error (RMSE) and a mean absolute error (MAE) lower or equal than 0.15 MPa and 0.12 MPa, respectively, were obtained with an external validation data set (n = 71 observations) for the various algorithms. When the modelling results were assessed through classes of values, a high overall accuracy was obtained for all the algorithms (82–83%), with prediction accuracy by class ranging between 79% and 100%. These results show a good performance of the predictive models, which considered a large variability of climatic, environmental, and agronomic conditions, and included various grape cultivars. By predicting both continuous values of ѱpd and classes of ѱpd, the approach presented in this study allowed obtaining 2-levels of accurate information about vines water status, which can be used to feed management decisions of different types of stakeholders.

Probabilistic modelling of wind turbine power curves with application of heteroscedastic Gaussian Process regression

There exists continued interest in building accurate models of wind turbine power curves for better understanding of performance or assessment of the condition of the turbine or both. Better predictions of the power curve allow increased insight into the operation of the turbine, aid operational decision making, and can be a key feature of online monitoring and fault detection strategies. This work proposes the use of a heteroscedastic Gaussian Process model for this task. The model has a number of attractive properties when modelling power curves. These include, removing the need to specify a parametric functional form for the power curve and automatic quantification of the variance in the prediction. The model exists within a Bayesian framework which exhibits built-in protection against over-fitting and robustness to noisy measurements. The model is shown to be effective on data collected from an operational wind turbine, returning accurate mean predictions (<1% normalised mean-squared error) and higher likelihoods than a corresponding homoscedastic model.

Measurement and Forecasting of High-Speed Rail Track Slab Deformation under Uncertain SHM Data Using Variational Heteroscedastic Gaussian Process.

Uncertainty in sensor data complicates the construction of baseline models for the measurement and forecasting (M&F) of high-speed rail (HSR) track slab deformation. Standard Gaussian process (GP) assumes a uniform noise throughout the input space. However, in the application to modelling of HSR structural health monitoring (SHM) data, this assumption can be unrealistic, because of its unique heteroscedastic uncertainty that is induced by dynamic train loading, electromagnetic interference, large temperature variation, and daily maintenance actions of railway track infrastructure. Therefore, this study firstly develops a novel online SHM system enabled by fiber Bragg grating (FBG) technology to eliminate electromagnetic interference on SHM data for continuous and long-term monitoring of track slab deformation, with the capacity of temperature self-compensation. To deal with different sources of uncertainty, the study explores Variational Heteroscedastic Gaussian Process (VHGP) approach while using variational Bayesian and Gaussian approximation for data modelling, estimation of the monitoring data uncertainty, and further data forecasting. The results demonstrate that the VHGP framework yields more robust regression results and the estimated confidence level can better depict the heteroscedastic variances of the noise in HSR data. Higher accuracy for both regression and forecasting is gained through VHGP and the position with maximum noise can be more accurately forecasted with a smooth varying confidence interval. Based on in-situ measurement data, the uncertainty levels for all sensors are estimated together with corresponding deformation profiles for the instrumented segment and three typical types of uncertainty are summarized during the M&F process of HSR track slab deformation.

Accurate Intrinsic and Extrinsic Calibration of RGB-D Cameras With GP-Based Depth Correction

In recent years, more and more robots have been equipped with low-cost RGB-D sensors, such as Microsoft Kinect and Intel Realsense, for safe navigation and active interaction with objects and people. In order to obtain more accurate and reliable fused color and depth information (coloured point clouds), not only the intrinsic and extrinsic parameters of color and depth sensor should be precisely calibrated but also the external corrections of depth measurements are required. In this paper, using motion capture system, we propose a reliable calibration framework that enables the precise estimation of the intrinsic and extrinsic parameters of RGB-D sensors and provide a model-free depth calibration method based on heteroscedastic Gaussian processes. Compared with the existing depth correction techniques, our method can simultaneously estimate the mean and variance of the depth error at different measurement distances, i.e., the probability distribution of the depth error relative to the measured distance, which is essential in the state estimation problems. To verify the effectiveness of our approach, we conduct a thorough qualitative and quantitative analysis of the major steps of our calibration method, and compare our experimental results with other related work. Furthermore, we demonstrate an experiment about the overall improvement of visual SLAM with a Kinect device calibrated by our calibration technique.