GP Regression Research Articles

Double-Descent Curves in Neural Networks: A New Perspective Using Gaussian Processes

Double-descent curves in neural networks describe the phenomenon that the generalisation error initially descends with increasing parameters, then grows after reaching an optimal number of parameters which is less than the number of data points, but then descends again in the overparameterized regime. In this paper, we use techniques from random matrix theory to characterize the spectral distribution of the empirical feature covariance matrix as a width-dependent perturbation of the spectrum of the neural network Gaussian process (NNGP) kernel, thus establishing a novel connection between the NNGP literature and the random matrix theory literature in the context of neural networks. Our analytical expressions allow us to explore the generalisation behavior of the corresponding kernel and GP regression. Furthermore, they offer a new interpretation of double-descent in terms of the discrepancy between the width-dependent empirical kernel and the width-independent NNGP kernel.

Cloud-assisted collaborative estimation for next-generation automobile sensing

This research proposes a unique cloud-assisted collaborative estimating system for on-road sensors like potholes, ice, and road profile to develop next-generation automobiles that are safer, more efficient, and more comfortable. Conventional road information finding methods for a single vehicle are prone to model uncertainty and measurement mistakes. The proposed system iteratively improves estimate performance using many heterogeneous vehicles. Each vehicle merges its onboard measurements with pseudomeasurements from previous participating cars and transmits the resulting local onboard estimate back to the cloud for updating. The system uses a Noisy Input Gaussian Process (NIGP) approach to manage uncertain readings. The study includes the road segment length of 40 m, 2 sensors for sprung mass and suspension displacements, time span of 1.5 s, a sampling time of 0.01 s, and 151 estimation points per Kalman Filter (KF). Additionally, the Root Mean Squared Error (RMSE) was calculated for onboard estimation performance, and training was conducted using data from one vehicle and 10 vehicles for the cloud-based NIGP performance evaluation. Using pseudo-measurements from preceding vehicles significantly improves performance. Comparing the standard GP regression, the NIGP regression and a benchmark based on KF averaging estimations. The NIGP provides better variance and reduces uncertainty. Moreover, the pseudo-measurement strategy iteratively improves onboard performance when using NIGP pseudo-measurements. A detailed experimental plan is described to measure the performance of the proposed collaborative estimating method while considering fleet heterogeneity, measurement noise, and model uncertainty.

Moderate deviations inequalities for Gaussian process regression

AbstractGaussian process regression is widely used to model an unknown function on a continuous domain by interpolating a discrete set of observed design points. We develop a theoretical framework for proving new moderate deviations inequalities on different types of error probabilities that arise in GP regression. Two specific examples of broad interest are the probability of falsely ordering pairs of points (incorrectly estimating one point as being better than another) and the tail probability of the estimation error at an arbitrary point. Our inequalities connect these probabilities to the mesh norm, which measures how well the design points fill the space.

RBM-GP with novel kernels coupled deep learning model for autism screening

Autism Spectrum Disorder (ASD) assumes greater significance because of its worldwide prevalence and the need to detect it in its preliminary stage is imperative. No standard medical test exists universally to screen autism. Each and every aspect of our life proves that nature holds the reign. This could be well demonstrated through many prevailing research works which still fails to classify ASD individuals accurately. In the proposed work, a unique framework, RBM-GP (Restricted Boltzmann Machine-Gaussian Process) along with two novel kernels namely RPR and RQ-P are proposed to address the need for automatic classification of ASD individuals and Typical Controls (TC) thereupon with minimum error. Slice Time Correction and Head Motion Correction are carried out as a preparatory step to take care of noisy fMRI (functional Magnetic Resonance Imaging) data. To make the best use of high dimensional fMRI data, the pre-processed fMRI data is normalized for further processing. In the proposed system, deep learning model is executed to extract non-handcrafted features from the pre-processed fMRI data with the notion of attaining less error during classification process. As a result, Bernoulli RBM acts as a feature extractor to derive highly discriminative fMRI features. In order to develop an effective computerized system for the classification of autistic subjects, GP regression, a machine learning classifier is employed in the proposed system to learn deep fMRI features. Kernel plays a vital role in the GP regression. So, RPR and RQ-P kernels are developed and integrated with GP in the proposed work for the accurate classification of autistic subjects. All datasets from the ABIDE I database are utilized for conducting the experiments. The proposed system achieves the minimum Mean Squared Error (MSE) of 20% for USM dataset with RPR kernel and CMU_b dataset with RQ-P kernel. The experimental results and comprehensive result analysis proclaim the superiority of the proposed system than the existing state-of-the-art approaches.

Prediction of Bus Passenger Traffic using Gaussian Process Regression.

The paper summarizes the design and implementation of a passenger traffic prediction model, based on Gaussian Process Regression (GPR). Passenger traffic analysis is the present day requirement for proper bus scheduling and traffic management to improve the efficiency and passenger comfort. Bayesian analysis uses statistical modelling to recursively estimate new data from existing data. GPR is a fully Bayesian process model, which is developed using PyMC3 with Theano as backend. The passenger data is modelled as a Poisson process so that the prior for designing the GP regression model is a Gamma distributed function. It is observed that the proposed GP based regression method outperforms the existing methods like Student-t process model and Kernel Ridge Regression (KRR) process.

Data-driven aerodynamic analysis of structures using Gaussian Processes

An abundant amount of data gathered during wind tunnel testing and health monitoring of structures inspires the use of machine learning methods to replicate the wind forces. This paper presents a data-driven Gaussian Process-Nonlinear Finite Impulse Response (GP-NFIR) model of the nonlinear self-excited forces acting on structures. Constructed in a nondimensional form, the model takes the effective wind angle of attack as lagged exogenous input and outputs a probability distribution of the forces. The nonlinear input/output function is modeled by a GP regression. Consequently, the model is nonparametric, thereby circumventing to set up the function’s structure a priori. The training input is designed as random harmonic motion consisting of vertical and rotational displacements. Once trained, the model can predict the aerodynamic forces for both prescribed input motion and aeroelastic analysis. The concept is first verified for a flat plate’s analytical solution by predicting the self-excited forces and flutter velocity. Finally, the framework is applied to a streamlined and bluff bridge deck based on Computational Fluid Dynamics (CFD) data. The model’s ability to predict nonlinear aerodynamic forces, flutter velocity, and post-flutter behavior are highlighted. Applications of the framework are foreseen in the structural analysis during the design and monitoring of slender line-like structures.

Robust Gaussian process regression with a bias model

This paper presents a new approach to a robust Gaussian process regression, creating a non-parametric Bayesian regression estimate robust to outliers. Most existing approaches replace an outlier-prone Gaussian likelihood with a non-Gaussian likelihood induced from a heavy tail distribution, such as the Laplace distribution and Student-t distribution. However, the use of a non-Gaussian likelihood would incur the need for a computationally expensive Bayesian approximate computation in the posterior inferences. The proposed approach models an outlier as a noisy and biased observation of an unknown regression function, and accordingly, the likelihood contains bias terms to explain the degree of deviations from the regression function. We introduce two bias models that handle the bias terms differently, treating a bias as an unknown and fixed quantity or treating a bias as a random quantity. We entail how the biases can be estimated accurately with other hyperparameters by a regularized maximum likelihood estimation. Conditioned on the bias estimates, the robust GP regression can be reduced to a standard GP regression problem with analytical forms of the predictive mean and variance estimates. Therefore, the proposed approach is simple and very computationally attractive. It also gives a very robust and accurate GP estimate for many tested scenarios. For the numerical evaluation, we perform a comprehensive simulation study to evaluate the proposed approach with the comparison to the existing robust GP approaches under various simulated scenarios of different outlier proportions and different noise levels. The approach is applied to data from two measurement systems, where the predictors are based on robust environmental parameter measurements and the response variables utilize more complex chemical sensing methods that contain a certain percentage of outliers. The utility of the measurement systems and value of the environmental data are improved through the computationally efficient GP regression and bias model.

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification.

Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification. We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM. In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%. To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.

Scalable Fully Bayesian Gaussian Process Modeling and Calibration With Adaptive Sequential Monte Carlo for Industrial Applications

Abstract Gaussian process (GP) regression or kriging has been extensively applied in the engineering literature for the purposes of building a cheap-to-evaluate surrogate, within the contexts of multi-fidelity modeling, model calibration, and design optimization. With the ongoing automation of manufacturing and industrial practices as a part of Industry 4.0, there has been a greater need for advancing GP regression techniques to handle challenges such as high input dimensionality, data paucity or big data problems, these consist primarily of proposing efficient design of experiments, optimal data acquisition strategies, sparsifying covariance kernels, and other mathematical tricks. In this work, our attention is focused on the challenges of efficiently training a GP model, which, to the authors opinion, has attracted very little attention and is to-date poorly addressed. The performance of widely used training approaches such as maximum likelihood estimation and Markov Chain Monte Carlo (MCMC) sampling can deteriorate significantly in high-dimensional and big data problems and can lead to cost deficient implementations of critical importance to many industrial applications. Here, we compare an Adaptive Sequential Monte Carlo (ASMC) sampling algorithm to classic MCMC sampling strategies and we demonstrate the effectiveness of our implementation on several mathematical problems and challenging industry applications of varying complexity. The computational time savings of the ASMC approach manifest in large-scale problems helping us to push the boundaries of applicability and scalability of GPs for model calibration in various domains of the industry, including but not limited to design automation, design engineering, smart manufacturing, predictive maintenance, and supply chain manufacturing.

Scalable Gaussian Process Classification With Additive Noise for Non-Gaussian Likelihoods.

Gaussian process classification (GPC) provides a flexible and powerful statistical framework describing joint distributions over function space. Conventional GPCs, however, suffer from: 1) poor scalability for big data due to the full kernel matrix and 2) intractable inference due to the non-Gaussian likelihoods. Hence, various scalable GPCs have been proposed through: 1) the sparse approximation built upon a small inducing set to reduce the time complexity and 2) the approximate inference to derive analytical evidence lower bound (ELBO). However, these scalable GPCs equipped with analytical ELBO are limited to specific likelihoods or additional assumptions. In this work, we present a unifying framework that accommodates scalable GPCs using various likelihoods. Analogous to GP regression (GPR), we introduce additive noises to augment the probability space for: 1) the GPCs with step, (multinomial) probit, and logit likelihoods via the internal variables and 2) particularly, the GPC using softmax likelihood via the noise variables themselves. This leads to unified scalable GPCs with analytical ELBO by using variational inference. Empirically, our GPCs showcase superiority on extensive binary/multiclass classification tasks with up to two million data points.

Nonparametric Local Pseudopotentials with Machine Learning: A Tin Pseudopotential Built Using Gaussian Process Regression

We present novel nonparametric representation math for local pseudopotentials (PP) based on Gaussian Process Regression (GPR). Local pseudopotentials are needed for materials simulations using Orbital-Free Density Functional Theory (OF-DFT) to reduce computational cost and to allow kinetic energy functional (KEF) application only to the valence density. Moreover, local PPs are important for the development of accurate KEFs for OF-DFT, but they are only available for a limited number of elements. We optimize local PPs of tin (Sn) represented with GPR to reproduce the experimental lattice constants of α- and β-Sn and the energy difference between these two phases as well as their electronic structure and charge density distributions which are obtained with Kohn-Sham Density Functional Theory employing semilocal PPs. The use of a nonparametric GPR-based PP representation avoids difficulties associated with the use of parametrized functions and has the potential to construct an optimal local PP independent of prior assumptions. The GPR-based Sn local PP results in well-reproduced bulk properties of α- and β-tin and electronic valence densities similar to those obtained with semilocal PP.

Scalable Soft Sensor for Nonlinear Industrial Big Data via Bagging Stochastic Variational Gaussian Processes

Traditional Gaussian process regression suffers from the cubic complexity and excessive computation burdens for industrial big data. To get rid of such defect, this work proposes a scalable soft sensor called bagging stochastic variational GP regression (SVGPR). We first formulate the Gaussian process within the sparse and variational approximation framework. Then, the stochastic variational inference (SVI) mechanism is induced, which can significantly break the formidable obstacle to nonlinear big data modeling. In addition, the imposed automatic relevance determination strategy will also leverage model interpretability with relevant feature weighting. Based on that, the bagging mechanism is encompassed by combining a set of distributed predictors to form a powerful ensemble model. As the results, prediction generalization is enhanced, whereas the stability is also well guaranteed. Both SVI and bagging allow for the parallel deployment. Therefore, a distributed diagram is developed for modeling and inference so that the bagging SVGPR can explore big data effectively and efficiently. For case study demonstrations, the proposed method is first evaluated on the numerical example, and then is applied in the real-time oxygen prediction of hydrogen manufacturing unit.

Combining LS-SVM and GP Regression for the Uncertainty Quantification of the EMI of Power Converters Affected by Several Uncertain Parameters

This article deals with the development of a probabilistic surrogate model for the uncertainty quantification of the voltage output spectral envelope of a power converter with several stochastic parameters. The proposed approach relies on the combination of the least-squares support vector machine (LS-SVM) regression with the Gaussian process regression (GPR), but it can suitably be applied to any deterministic regression techniques. As a first step, the LS-SVM regression is used to build an accurate and fast-to-evaluate deterministic model of the system responses starting from a limited set of training samples provided by the full-computational model. Then the GPR is used to provide a probabilistic model of the regression error. The resulting LS-SVM+GPR probabilistic model not only approximates the system responses for any configuration of its input parameters, but also provides an estimation of its prediction uncertainty, such as the confidence intervals (CIs). The above technique has been applied to qualify the uncertainty of the spectral envelope of the output voltage of a buck converter with 17 independent Gaussian parameters. The feasibility and the accuracy of the resulting model have been investigated by comparing its predictions and CI with the ones obtained by five different surrogate models based on state-of-the-art techniques and by the reference Monte Carlo results.

MÉTODOS COMPUTACIONALES PARA LA DELAMINACIÓN DE GRAFITO UTILIZANDO SURFACTANTES ANIONICOS PARA PRODUCIR GRAFENO

Today, using thermal and chemical reduction and solubility, graphene oxide is produced in large scale. Since there are various methods for producing graphene, each of which allocates properties to the produced graphene, the purpose of this research is to investigate graphite delamination using anionic surfactants and produce graphene by means of computational methods. The research method was applied in order to perform molecular dynamics analysis, first, (the minimum) force that each atom imposes to other atoms was calculated. This is the total gradient of the system’s energy according to the coordinates of the related atom. A Bayesian method was used for dynamic modeling, which, on average, uses dynamic parameters instead of their estimates. The Gaussian Process Dynamic Model (GPDM) was completely defined by a set of low-level data representations and was observed by both dynamics and modeling of GP regression (Gaussian process regression). Then, using the Gaussian software, along with empirical results or just using this software, the molecular state and reactions and their mechanisms were simulated. The results indicated that the presence of benzene, ether and carbo xyl groups in the optimal structure facilitates the entry of surfactants into the sheets and that the agent to start the separation of the graphene sheets adhered to each other by comparing the results of this study between the two surfactants, it was found that the gap to change by separation layers between the graphene plates is different for two surfactants. Besides, the difference in the polarity of the surfactants resulted in the final polarization of the surfactant and graphene system. Therefore, the difference in the polarity causes the difference in the solubility.

Robustness Guarantees for Bayesian Inference with Gaussian Processes

Bayesian inference and Gaussian processes are widely used in applications ranging from robotics and control to biological systems. Many of these applications are safety-critical and require a characterization of the uncertainty associated with the learning model and formal guarantees on its predictions. In this paper we define a robustness measure for Bayesian inference against input perturbations, given by the probability that, for a test point and a compact set in the input space containing the test point, the prediction of the learning model will remain δ−close for all the points in the set, for δ > 0. Such measures can be used to provide formal probabilistic guarantees for the absence of adversarial examples. By employing the theory of Gaussian processes, we derive upper bounds on the resulting robustness by utilising the Borell-TIS inequality, and propose algorithms for their computation. We evaluate our techniques on two examples, a GP regression problem and a fully-connected deep neural network, where we rely on weak convergence to GPs to study adversarial examples on the MNIST dataset.

Exit versus Voice: Competition and Dissatisfied Patients’ Responses to General Practitioners

Abstract : Background: Hirschman’s 1970; 1980 analytical conceptualization of the influence of competition on exit vs. voicing, the point of departure for this study, suggests that the presence of competitive providers’ fosters patients’ decisions to switch GPs exit. In Norway, the list patient system introduced in 2001 gives patients the option of changing GPs up to twice within a year. Although patient exiting can signal dissatisfaction with a particular GP, this signal is less informative about which aspect of the service is the issue. A private web-based rating site, Legelisten, was launched in Norway in May of 2012 to give patients a platform for voicing their opinions about the health care they receive, thereby also providing feedback that doctors can use to improve their services. Higher exit opportunities in a competitive market can reduce a patient’s motivation to voice about his or her provider; however, online voicing creates available knowledge about GPs and becomes a public good of high interest, especially in large-market settings, leading to higher participation in voicing in these markets than in smaller ones. Objective : The study aims to test the influence of GP market competition on dissatisfied patients’ choice to exit switch physician or to voice participate in online rating. Method : A panel data set covering the entire population of Norwegian GPs was utilized for every half year from January 2010 to January 2015 for patient exiting and from January 2012 to January 2015 for patient voicing. The analysis was primarily quantitative, involving descriptive statistics and appropriate panel regression methods for both GP and market-level analyses multilevel mixed-effect linear regression for the GP-level exit, multilevel mixed-effects logistic regression for the GP-level voice, linear regression for the market-level exit, and the Tobit regression for market-level voice. Findings : Higher GP market competition both with and without adjusting for municipality size increased the proportion of patients switching exit. However, for voice, higher GP market competition not adjusted for municipality size led to more participation for both dissatisfied and overall voice, while adjusted for municipality size led to less participation. This opposite effect, reflects the municipality size effect and the associated difference in the public good motive for contributing to the online voice. This result is consistent in both the GP-level and market-level analyses, strongly supporting the hypothesis adopted by this study.

Evaluating the quality of steady-state multivariate experimental data relative to various ORC experimental setups

Today, an increasing amount of experimental data is being released in the ORC field. This data is required to assess and compare the performance of different machines, to point out the main sources of losses, or to calibrate and to validate models.Experimental data is subject to different sources of disturbance and errors whose importance should be assessed. The level of noise, the presence of outliers, or a measure of the ”explainability” of the key variables with respect to the externally imposed operating condition are important indicators, but are not straightforward to obtain, especially if the data is sparse and multivariate.Starting from recent experimental campaigns on two different ORC test rigs, this paper proposes a methodology and a suite of tools implementing Gaussian Processes for quality assessment of steady-state experimental data. The aim of the proposed tool is to (1) provide a smooth (de-noised) multivariate operating map of the measured variable with respect to the inputs; (2) determine which inputs are relevant to predict a selected output; (3) provide a sensitivity analysis of the measured variables with respect to the inputs; (4) provide a measure of the accuracy (confidence intervals) for the prediction of the data; (5) detect the observation that are likely to be outliers.In this paper, the ORC test rigs and the obtained experimental data are described. The results are then analysed with the proposed tool, and compared with the results of traditional modelling techniques. It is demonstrated that GP regression provides insightful numerical indicators for these purposes, and that the obtained performance is higher or comparable to alternative modelling techniques. Finally, the datasets and tools developed in this work are provided within the GPexp open-source package.

Plunging hollow jet aerators - oxygen transfer and modelling

Jet aeration is usually associated with creation of large air–water interfacial area and low energy loss, particularly in not-too-big a tank, thus, making it an efficient oxygenation system. If an aeration device is so designed as to produce a plunging hollow jet, the oxygen transfer is reported to have been much enhanced. Experiments have been conducted on three 60° plunging jet aerators having different configurations viz. plain, curved and the plain-modified shaped. Amongst the three configurations tested, the curved-shaped device is observed to be the most efficient one, giving higher oxygen transfer efficiency values particularly for jet velocity exceeds about 4 m/s. Further a non-linear relationship of volumetric oxygen transfer coefficient () is obtained for curved-shaped hollow jet aerator and the modelling performance is compared with GP regression approach. The performance of modelling techniques evaluated on the basis of statistical parameters (R2, RMSE) suggested that GP regression using RBF kernel performed considerably better followed by GP regression using poly kernel and non-linear regression in predicting based on jet input parameters. The modelling approaches were further compared with the relationships proposed by previous researchers.

Auditory prediction errors as individual biomarkers of schizophrenia.

Schizophrenia is a complex psychiatric disorder, typically diagnosed through symptomatic evidence collected through patient interview. We aim to develop an objective biologically-based computational tool which aids diagnosis and relies on accessible imaging technologies such as electroencephalography (EEG). To achieve this, we used machine learning techniques and a combination of paradigms designed to elicit prediction errors or Mismatch Negativity (MMN) responses. MMN, an EEG component elicited by unpredictable changes in sequences of auditory stimuli, has previously been shown to be reduced in people with schizophrenia and this is arguably one of the most reproducible neurophysiological markers of schizophrenia.EEG data were acquired from 21 patients with schizophrenia and 22 healthy controls whilst they listened to three auditory oddball paradigms comprising sequences of tones which deviated in 10% of trials from regularly occurring standard tones. Deviant tones shared the same properties as standard tones, except for one physical aspect: 1) duration - the deviant stimulus was twice the duration of the standard; 2) monaural gap - deviants had a silent interval omitted from the standard, or 3) inter-aural timing difference, which caused the deviant location to be perceived as 90° away from the standards.We used multivariate pattern analysis, a machine learning technique implemented in the Pattern Recognition for Neuroimaging Toolbox (PRoNTo) to classify images generated through statistical parametric mapping (SPM) of spatiotemporal EEG data, i.e. event-related potentials measured on the two-dimensional surface of the scalp over time. Using support vector machine (SVM) and Gaussian processes classifiers (GPC), we were able classify individual patients and controls with balanced accuracies of up to 80.48% (p-values = 0.0326, FDR corrected) and an ROC analysis yielding an AUC of 0.87. Crucially, a GP regression revealed that MMN predicted global assessment of functioning (GAF) scores (correlation = 0.73, R2 = 0.53, p = 0.0006).

Deriving an intelligent model for soil compression index utilizing multi-gene genetic programming

Multi-gene genetic programming (MGGP) is a new nonlinear system modeling approach that integrates the capabilities of standard GP and classical regression. This paper deals with the prediction of compression index of fine-grained soils using this robust technique. The proposed model relates the soil compression index to its liquid limit, plastic limit and void ratio. Several laboratory test results for fine fine-grained were used to develop the models. Various criteria were considered to check the validity of the model. The parametric and sensitivity analyses were performed and discussed. The MGGP method was found to be very effective for predicting the soil compression index. The prediction coefficients of determination were 0.856 and 0.840 for the training and testing data, respectively. A comparative study was further performed to prove the superiority of the MGGP model to the existing soft computing and traditional empirical equations.