Standard Gaussian Process Regression Research Articles

Hyper‐parameter optimized GPR model based on chaos game algorithm for RF power transistors

AbstractIn this paper, a new frequency domain behavior model for radio frequency (RF) power transistors based on a hyper‐parameter optimized Gaussian process regression (GPR) method is presented. The chaos game optimization (CGO) algorithm is used to optimize GPR hyperparameters, resulting in the CGO‐GPR model. The basic theory as well as the details of the modeling process are presented. Validation of the model is conducted using a 10‐watt GaN power transistor. Compared to the standard GPR model, the proposed model achieved a significant improvement. Furthermore, the comparison with the particle swarm optimization (PSO) based GPR model (PSO‐GPR) showed that the proposed model allows achieving superior performance, thereby confirming the effectiveness of the developed modeling technique.

Complementing Drawability Assessment of Deep-Drawn Components With Surrogate-Based Global Sensitivity Analysis

Abstract In the early-stage development of sheet metal parts, key design properties of new structures must be specified. As these decisions are made under significant uncertainty regarding drawing configuration changes, they sometimes result in the development of new parts that, at a later design stage, will not be drawable. As a result, there is a need to increase the certainty of experience-driven drawing configuration decisions. Complementing this process with a global sensitivity analysis (GSA) can provide insight into the impact of various changes in drawing configurations on drawability, unveiling cost-effective strategies to ensure the drawability of new parts. However, when quantitative global sensitivity approaches, such as Sobol's method, are utilized, the computational requirements for obtaining Sobol indices can become prohibitive even for small application problems. To circumvent computational limitations, we evaluate the applicability of different surrogate models engaged in computing global design variable sensitivities for the drawability assessment of a deep-drawn component. Here, we show in an exemplary application problem, that both a standard Gaussian process regression (GPR) model and an ensemble model can provide commendable results at a fraction of the computational cost. We compare our surrogate models to existing approaches in the field. Furthermore, by comparing drawability measures we show that the error introduced by the surrogate models is of the same order of magnitude as that from the choice of drawability measure. In consequence, our surrogate models can improve the cost-effective development of a component in the early design phase.

Retrieving the 21-cm signal from the Epoch of Reionization with learnt Gaussian process kernels

ABSTRACT Direct detection of the Cosmic Dawn and Epoch of Reionization via the redshifted 21-cm line of neutral Hydrogen will have unprecedented implications for studying structure formation in the early Universe. This exciting goal is challenged by the difficulty of extracting the faint 21-cm signal buried beneath bright astrophysical foregrounds and contaminated by numerous systematics. Here, we focus on improving the Gaussian Process Regression (GPR) signal separation method originally developed for LOFAR observations. We address a key limitation of the current approach by incorporating covariance prior models learnt from 21-cm signal simulations using variational autoencoder (VAE) and interpolatory autoencoder (IAE). Extensive tests are conducted to evaluate GPR, VAE–GPR, and IAE–GPR in different scenarios. Our findings reveal that the new method outperforms standard GPR in component separation tasks. Moreover, the improved method demonstrates robustness when applied to signals not represented in the training set. It also presents a certain degree of resilience to data systematics, highlighting its ability to effectively mitigate their impact on the signal recovery process. However, our findings also underscore the importance of accurately characterizing and understanding these systematics to achieve successful detection. Our generative approaches provide good results even with limited training data, offering a valuable advantage when a large training set is not feasible. Comparing the two algorithms, IAE–GPR shows slightly higher fidelity in recovering power spectra compared to VAE–GPR. These advancements highlight the strength of generative approaches and optimize the analysis techniques for future 21-cm signal detection at high redshifts.

Chained Gaussian processes to estimate battery health degradation with uncertainties

Estimating the average health degradation of a new battery design is a crucial objective for manufacturers to estimate its value. Furthermore, to quantify financial risks, associated uncertainties should be modeled precisely. From a data-driven perspective, Gaussian process regression (GPR) is often a method of choice since it simultaneously learns complex models and naturally includes uncertainties. However, GPR methods generally rely on a stationarity assumption which imposes severe constraints on uncertainties. In this paper, we illustrate the limits of standard GPR and show that the Chained Gaussian processes, a more general framework introduced in the Machine learning community, is a useful alternative, allowing more accurate quantification of uncertainties.

Hierarchical Bayesian approach to experimental data fusion: Application to strength prediction of high entropy alloys from hardness measurements

The discovery of materials with improved properties can be accelerated by models with the ability to combine data from multiple experimental information sources. A recurring task in the toolbox of practitioners is to map input physical descriptors to output properties of interest. Typically, both the outputs and many of the inputs are experimentally measured and, thus, noisy. Probabilistic regression methods, e.g., Gaussian process regression, can easily deal with noisy outputs, even if the noise is input-dependent. However, most regression methods cannot process noisy inputs. Ignoring input uncertainty leads to inaccurate predictive uncertainty, a crucial ingredient for the sequential design of experiments. The objective of this paper is to develop a regression methodology that can deal with input uncertainty when one wishes to correlate an inexpensive experimental measurement (e.g., hardness) to an expensive one (e.g., yield strength). Our hierarchical Bayesian approach uses two Gaussian processes. The first one maps noiseless physical descriptors to the inexpensive experimental measurement. The second Gaussian process maps noiseless physical descriptors and the inexpensive experimental measurement to the expensive experimental measurement. The two Gaussian processes form a nested model that is not analytically tractable. To overcome this issue, we propose semi-analytical approximations to both the marginal likelihood and the posterior predictive distribution. The result is a model that is practical to train and use. We demonstrate the merits of the proposed method through a synthetic dataset in which we control all the uncertainties. The statistical tests clearly show that standard Gaussian process regression cannot cope with input uncertainty whereas our proposed method consistently yields better predictive distributions. Finally, we apply the method to the task of predicting the yield strength of high entropy alloys from hardness on an exhaustive dataset compiled from the available literature.

Probabilistic surrogate modeling of offshore wind-turbine loads with chained Gaussian processes

Heteroscedastic Gaussian process regression, based on the concept of chained Gaussian processes, is used to build surrogates to predict site-specific loads on an offshore wind turbine. Stochasticity in the inflow turbulence and irregular waves results in load responses that are best represented as random variables rather than deterministic values. Moreover, the effect of these stochastic sources on the loads depends strongly on the mean environmental conditions - for instance, at low mean wind speeds, inflow turbulence produces much less variability in loads than at high wind speeds. Statistically, this is known as heteroscedasticity. Deterministic and most stochastic surrogates do not account for the heteroscedastic noise, giving an incomplete and potentially misleading picture of the structural response. In this paper, we draw on the recent advancements in statistical inference to train a heteroscedastic surrogate model on a noisy database to predict the conditional pdf of the response. The model is informed via 10-minute load statistics of the IEA-10MW-RWT subject to both aero- and hydrodynamic loads, simulated with OpenFAST. Its performance is assessed against the standard Gaussian process regression. The predicted mean is similar in both models, but the heteroscedastic surrogate approximates the large-scale variance of the responses significantly better.

Physics-informed Gaussian process regression for states estimation and forecasting in power grids

Real-time state estimation and forecasting are critical for the efficient operation of power grids. In this paper, a physics-informed Gaussian process regression (PhI-GPR) method is presented and used for forecasting and estimating the phase angle, angular speed, and wind mechanical power of a three-generator power grid system using sparse measurements. In standard data-driven Gaussian process regression (GPR), parameterized models for the prior statistics are fit by maximizing the marginal likelihood of observed data. In the PhI-GPR method, we propose to compute the prior statistics offline by solving stochastic differential equations (SDEs) governing the power grid dynamics. The short-term forecast of a power grid system dominated by wind generation is complicated by the stochastic nature of the wind and the resulting uncertainty in wind mechanical power. Here, we assume that the power grid dynamics are governed by swing equations, with the wind mechanical power fluctuating randomly in time. We solve these equations for the mean and covariances of the power grid states using the Monte Carlo simulation method.We demonstrate that the proposed PhI-GPR method can accurately forecast and estimate observed and unobserved states. For the considered problem, PhI-GPR has computational advantages over the ensemble Kalman filter (EnKF) method: In PhI-GPR, ensembles are computed offline and independently of the data acquisition process, whereas for EnFK, ensembles are computed online with data acquisition, rendering real-time forecast more challenging. We also demonstrate that the PhI-GPR forecast is more accurate than the EnKF forecast when the random mechanical wind power is non-Markovian. In contrast, the two methods produce similar forecasts for the Markovian mechanical wind power.For observed states, we show that PhI-GPR provides a forecast comparable to the standard data-driven GPR; both forecasts are significantly more accurate than the autoregressive integrated moving average (ARIMA) forecast. We also show that the ARIMA forecast is more sensitive to observation frequency and measurement errors than the PhI-GPR forecast.

A novel method for identifying geomechanical parameters of rock masses based on a PSO and improved GPR hybrid algorithm

In view of the shortcomings of existing artificial neural network (ANN) and support vector regression (SVR) in the application of three-dimensional displacement back analysis, Gaussian process regression (GPR) algorithm is introduced to make up for the shortcomings of existing intelligent inversion methods. In order to improve the generality of the standard GPR algorithm with single kernel function, an improved Gaussian process regression (IGPR) algorithm with combined kernel function is proposed by adding two single kernel functions. In addition, in the training process of IGPR model, the particle swarm optimization (PSO) is combined with the IGPR model (PSO-IGPR) to optimize the parameters of the IGPR model. After the IGPR model can accurately map the relationship between geomechanical parameters and rock mass deformation, the PSO algorithm is directly used to search the best geomechanical parameters to match the deformation calculated by igpr model with the measured deformation of rock mass. The application case of Beikou tunnel shows that the combined kernel function GPR has higher identification accuracy than the single kernel function GPR and SVR model, the IGPR model with automatic correlation determination (ARD) kernel function can obtain higher identification accuracy than the IGPR model with isotropic (ISO) kernel function, and the PSO-IGPR hybrid model based on ARD kernel function has the highest identification accuracy. Therefore, this paper proposes a displacement back analysis method of the PSO-IGPR hybrid algorithm based on ARD kernel function, which can be used to identify the geomechanical parameters of rock mass and solve other engineering problems.

Locally Smoothed Gaussian Process Regression

We develop a novel framework to accelerate Gaussian process regression (GPR). In particular, we consider localization kernels at each data point to down-weigh the contributions from other data points that are far away, and we derive the GPR model stemming from the application of such localization operation. Through a set of experiments, we demonstrate the competitive performance of the proposed approach compared to full GPR, other localized models, and deep Gaussian processes. Crucially, these performances are obtained with considerable speedups compared to standard global GPR due to the sparsification effect of the Gram matrix induced by the localization operation.

ELECTRIC LOAD AND POWER FORECASTING USING ENSEMBLE GAUSSIAN PROCESS REGRESSION

We propose a new forecasting method for predicting load demand and generation scheduling. Accurate week-long forecasting of load demand and optimal power generation is critical for efficient operation of power grid systems. In this work, we use a synthetic data set describing a power grid with 700 buses and 134 generators over a 365-days period with data synthetically generated at an hourly rate. The proposed approach for week-long forecasting is based on the Gaussian process regression (GPR) method, with prior covariance matrices of the quantities of interest (QoI) computed from ensembles formed by up to twenty preceding weeks of QoI observations. Then, we use these covariances within the GPR framework to forecast the QoIs for the following week. We demonstrate that the the proposed ensemble GPR (EGPR) method is capable of accurately forecasting weekly total load demand and power generation profiles. The EGPR method is shown to outperform traditional forecasting methods including the standard GPR and autoregressive integrated moving average (ARIMA) methods.

Learning Dynamic Systems Using Gaussian Process Regression with Analytic Ordinary Differential Equations as Prior Information

Recently the data-driven learning of dynamic systems has become a promising approach because no physical knowledge is needed. Pure machine learning approaches such as Gaussian process regression (GPR) learns a dynamic model from data, with all physical knowledge about the system discarded. This goes from one extreme, namely methods based on optimizing parametric physical models derived from physical laws, to the other. GPR has high flexibility and is able to model any dynamics as long as they are locally smooth, but can not generalize well to unexplored areas with little or no training data. The analytic physical model derived under assumptions is an abstract approximation of the true system, but has global generalization ability. Hence the optimal learning strategy is to combine GPR with the analytic physical model. This paper proposes a method to learn dynamic systems using GPR with analytic ordinary differential equations (ODEs) as prior information. The one-time-step integration of analytic ODEs is used as the mean function of the Gaussian process prior. The total parameters to be trained include physical parameters of analytic ODEs and parameters of GPR. A novel method is proposed to simultaneously learn all parameters, which is realized by the fully Bayesian GPR and more promising to learn an optimal model. The standard Gaussian process regression, the ODE method and the existing method in the literature are chosen as baselines to verify the benefit of the proposed method. The predictive performance is evaluated by both one-time-step prediction and long-term prediction. By simulation of the cart-pole system, it is demonstrated that the proposed method has better predictive performances.

Reducing Ground-based Astrometric Errors with Gaia and Gaussian Processes

Abstract Stochastic field distortions caused by atmospheric turbulence are a fundamental limitation to the astrometric accuracy of ground-based imaging. This distortion field is measurable at the locations of stars with accurate positions provided by the Gaia DR2 catalog; we develop the use of Gaussian process regression (GPR) to interpolate the distortion field to arbitrary locations in each exposure. We introduce an extension to standard GPR techniques that exploits the knowledge that the 2D distortion field is curl-free. Applied to several hundred 90 s exposures from the Dark Energy Survey as a test bed, we find that the GPR correction reduces the variance of the turbulent astrometric distortions ≈12× , on average, with better performance in denser regions of the Gaia catalog. The rms per-coordinate distortion in the riz bands is typically ≈7 mas before any correction and ≈2 mas after application of the GPR model. The GPR astrometric corrections are validated by the observation that their use reduces, from 10 to 5 mas rms, the residuals to an orbit fit to riz-band observations over 5 yr of the r = 18.5 trans-Neptunian object Eris. We also propose a GPR method, not yet implemented, for simultaneously estimating the turbulence fields and the 5D stellar solutions in a stack of overlapping exposures, which should yield further turbulence reductions in future deep surveys.

Physics-derived covariance functions for machine learning in structural dynamics

Abstract This paper attempts to bridge the gap between standard engineering practice and machine learning when modelling stochastic processes. For a number of physical processes of interest, derivation of the (auto)covariance is achievable. This paper suggests their use as priors in a standard Gaussian process regression as a means of enhancing predictive capability in situations where they are reflective of the process of interest. A covariance function of a linear oscillator under random load is derived and used in a regression context to predict the displacements of a vibratory system. A simulation case study is used to demonstrate the enhancement over a standard Gaussian process regression model.

Cross Trajectory Gaussian Process Regression Model for Battery Health Prediction

Accurate battery capacity prediction is important to ensure reliable battery operation and reduce the cost. However, the complex nature of battery degradation and the presence of capacity regeneration phenomenon render the prediction task very challenging. To address this problem, this paper proposes a novel and efficient algorithm to predict the battery capacity trajectory in a multi-cell setting. The proposed method is a new variant of Gaussian process regression (GPR) model, and it utilizes similar trajectories in the historical data to enhance the prediction of desired capacity trajectory. More importantly, the proposed method adds no extra computation cost to the standard GPR. To demonstrate the effectiveness of the proposed method, validation tests on two different battery datasets are implemented in the case studies. The prediction results and the computation cost are carefully benchmarked with cutting-edge GPR approaches for battery capacity prediction.

Optimizing Gaussian Process Regression for Image Time Series Gap-Filling and Crop Monitoring.

Image processing entered the era of artificial intelligence, and machine learning algorithms emerged as attractive alternatives for time series data processing. Satellite image time series processing enables crop phenology monitoring, such as the calculation of start and end of season. Among the promising algorithms, Gaussian process regression (GPR) proved to be a competitive time series gap-filling algorithm with the advantage of, as developed within a Bayesian framework, providing associated uncertainty estimates. Nevertheless, the processing of time series images becomes computationally inefficient in its standard per-pixel usage, mainly for GPR training rather than the fitting step. To mitigate this computational burden, we propose to substitute the per-pixel optimization step with the creation of a cropland-based precalculations for the GPR hyperparameters θ. To demonstrate our approach hardly affects the accuracy in fitting, we used Sentinel-2 LAI time series over an agricultural region in Castile and Leon, North-West Spain. The performance of image reconstructions were compared against the standard per-pixel GPR time series processing. Results showed that accuracies were on the same order (RMSE 0.1767 vs. 0.1564 [m2/m2], 12% RMSE degradation) whereas processing time accelerated about 90 times. We further evaluated the alternative option of using the same hyperparameters for all the pixels within the complete scene. It led to similar overall accuracies over crop areas and computational performance. Crop phenology indicators were also calculated for the three different approaches and compared. Results showed analogous crop temporal patterns, with differences in start and end of growing season of no more than five days. To the benefit of crop monitoring applications, all the gap-filling and phenology indicators retrieval techniques have been implemented into the freely downloadable GUI toolbox DATimeS.

Gaussian process modelling with Gaussian mixture likelihood

Gaussian Process (GP), as a probabilistic non-linear multi-variable regression model, has been widely used in nonparametric Bayesian framework for the data based modelling of complex processes. The noise dynamics in standard GP regression is assumed to follow a Gaussian distribution. In this setting, the point estimation of the model parameters can be obtained analytically using the maximum likelihood (ML) approach in a straight forward fashion. However, in practical scenarios, processes may have been corrupted by the outliers and other disturbances or have multiple modes of operation, resulting a non-Gaussian data likelihood. In this work, to model such scenarios, we propose to employ a mixture of two Gaussian distributions as the noise model to capture both regular noise and irregular noise, thereby enhancing the robustness of the regression model. Further, we present an Expectation Maximization (EM) algorithm-based approach to obtain the optimal parameters set of the proposed GP regression model. The predictive distribution can then be found according to the estimated hyperparameters from the EM algorithm. The efficacy and practicality of the proposed method are illustrated with two sets of synthetic data, a simulated example, as well as an industrial dataset.

An Online Radio Map Update Scheme for WiFi Fingerprint-Based Localization

Fingerprint-based localization relies on an accurate and up-to-date radio map, which is however cumbersome to obtain. In this paper, a novel scheme is proposed to online adapt radio maps to environmental dynamics by using low-cost crowdsourced received signal strength (RSS) measurements. To be specific, a coarse-grained radio map is initially established in the offline phase utilizing the standard Gaussian process regression (GPR) given a limited number of fingerprints (i.e., RSS measurements with location labels), and further can be recursively refined in the online phase given crowdsourced RSS measurements with their noisy location labels obtained through the existing radio map. Differently from existing GPR-based approaches, the proposed scheme adopts extended GPR to alleviate the model inaccuracy induced by such noisy location labels, and then presents a marginalized particle extended Gaussian process (MPEG) to recursively filter the radio map. In addition, pedestrian dead reckoning (PDR) is leveraged to calibrate such noisy location labels. Extensive experiments are carried out in a real scenario with area of nearly 1000 m2 during a five-month period of time, and a thorough comparison with several existing approaches indicates that the proposed scheme gradually improves the localization accuracy on average by as much as 31.2%, while the counterparts result in fluctuant localization performance and improve the localization accuracy on average by 13.3%.

Gaussian Process-Based Hybrid Model for Predicting Oxygen Consumption in the Converter Steelmaking Process

Oxygen is one of the most important energies used in converter steelmaking processes of integrated iron and steel works. Precisely forecasting oxygen consumption before processing can benefit process control and energy optimization. This paper assumes there is a linear relationship between the oxygen consumption and input materials, and random noises are caused by other unmeasurable materials and unobserved reactions. Then, a novel hybrid prediction model integrating multiple linear regression (MLR) and Gaussian process regression (GPR) is introduced. In the hybrid model, the MLR method is developed to figure the global trend of the oxygen consumption, and the GPR method is applied to explore the local fluctuation caused by noise. Additionally, to accelerate the computational speed on the practical data set, a K-means clustering method is devised to respectively train a number of GPR models. The proposed hybrid model is validated with the actual data collected from an integrated iron and steel work in China, and compared with benchmark prediction models including MLR, artificial neural network, support vector machine and standard GPR. The forecasting results indicate that the suggested model is able to not only produce satisfactory point forecasts, but also estimate accurate probabilistic intervals.

Low-Dimensionality of Noise-Free RSS and Its Application in Distributed Massive MIMO

We examine the dimensionality of noise-free uplink received signal strength (RSS) in a distributed massive multiple-input multiple-output system. Upon applying principal component analysis to the noise-free uplink RSS data, we observe that it spans over a low-dimensional principal subspace. We make use of this unique property to propose a reconstruction-based Gaussian process regression (RecGP) method which predicts user locations from their uplink RSS. Considering noise-free RSS for training and noisy RSS for testing purposes, RecGP reconstructs the noisy test RSS from a low-dimensional principal subspace of the noise-free training RSS. This reconstruction allows RecGP to achieve lower prediction error than the standard Gaussian process regression method which directly uses the original test RSS for location prediction.

One-Match-Ahead Forecasting in Two-Team Sports with Stacked Bayesian Regressions

Abstract There is a growing interest in applying machine learning algorithms to real-world examples by explicitly deriving models based on probabilistic reasoning. Sports analytics, being favoured mostly by the statistics community and less discussed in the machine learning community, becomes our focus in this paper. Specifically, we model two-team sports for the sake of one-match-ahead forecasting. We present a pioneering modeling approach based on stacked Bayesian regressions, in a way that winning probability can be calculated analytically. Benefiting from regression flexibility and high standard of performance, Sparse Spectrum Gaussian Process Regression (SSGPR) – an improved algorithm for the standard Gaussian Process Regression (GPR), was used to solve Bayesian regression tasks, resulting in a novel predictive model called TLGProb. For evaluation, TLGProb was applied to a popular sports event – National Basketball Association (NBA). Finally, 85.28% of the matches in NBA 2014/2015 regular season were correctly predicted by TLGProb, surpassing the existing predictive models for NBA.