Gaussian Process Approximation Research Articles

Distance preserving machine learning for uncertainty aware accelerator capacitance predictions

Abstract Accurate uncertainty estimations are essential for producing reliable machine learning models, especially in safety-critical applications such as accelerator systems. Gaussian process models are generally regarded as the gold standard for this task; however, they can struggle with large, high-dimensional datasets. Combining deep neural networks with Gaussian process approximation techniques has shown promising results, but dimensionality reduction through standard deep neural network layers is not guaranteed to maintain the distance information necessary for Gaussian process models. We build on previous work by comparing the use of the singular value decomposition against a spectral-normalized dense layer as a feature extractor for a deep neural Gaussian process approximation model and apply it to a capacitance prediction problem for the High Voltage Converter Modulators in the Oak Ridge Spallation Neutron Source. Our model shows improved distance preservation and predicts in-distribution capacitance values with less than 1% error.

A Gaussian-process approximation to a spatial SIR process using moment closures and emulators.

The dynamics that govern disease spread are hard to model because infections are functions of both the underlying pathogen as well as human or animal behavior. This challenge is increased when modeling how diseases spread between different spatial locations. Many proposed spatial epidemiological models require trade-offs to fit, either by abstracting away theoretical spread dynamics, fitting a deterministic model, or by requiring large computational resources for many simulations. We propose an approach that approximates the complex spatial spread dynamics with a Gaussian process. We first propose a flexible spatial extension to the well-known SIR stochastic process, and then we derive a moment-closure approximation to this stochastic process. This moment-closure approximation yields ordinary differential equationsfor the evolution of the means and covariances of the susceptibles and infectious through time. Because these ODEs are a bottleneck to fitting our model by MCMC, we approximate them using a low-rank emulator. This approximation serves as the basis for our hierarchical model for noisy, underreported counts of new infections by spatial location and time. We demonstrate using our model to conduct inference on simulated infections from the underlying, true spatial SIR jump process. We then apply our method to model counts of new Zika infections in Brazil from late 2015 through early 2016.

Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes

In recent years, there has been widespread adoption of machine learning-based approaches to automate the solving of partial differential equations (PDEs). Among these approaches, Gaussian processes (GPs) and kernel methods have garnered considerable interest due to their flexibility, robust theoretical guarantees, and close ties to traditional methods. They can transform the solving of general nonlinear PDEs into solving quadratic optimization problems with nonlinear, PDE-induced constraints. However, the complexity bottleneck lies in computing with dense kernel matrices obtained from pointwise evaluations of the covariance kernel, and its partial derivatives, a result of the PDE constraint and for which fast algorithms are scarce. The primary goal of this paper is to provide a near-linear complexity algorithm for working with such kernel matrices. We present a sparse Cholesky factorization algorithm for these matrices based on the near-sparsity of the Cholesky factor under a novel ordering of pointwise and derivative measurements. The near-sparsity is rigorously justified by directly connecting the factor to GP regression and exponential decay of basis functions in numerical homogenization. We then employ the Vecchia approximation of GPs, which is optimal in the Kullback-Leibler divergence, to compute the approximate factor. This enables us to compute ϵ \epsilon -approximate inverse Cholesky factors of the kernel matrices with complexity O ( N log d ⁡ ( N / ϵ ) ) O(N\log ^d(N/\epsilon )) in space and O ( N log 2 d ⁡ ( N / ϵ ) ) O(N\log ^{2d}(N/\epsilon )) in time. We integrate sparse Cholesky factorizations into optimization algorithms to obtain fast solvers of the nonlinear PDE. We numerically illustrate our algorithm’s near-linear space/time complexity for a broad class of nonlinear PDEs such as the nonlinear elliptic, Burgers, and Monge-Ampère equations. In summary, we provide a fast, scalable, and accurate method for solving general PDEs with GPs and kernel methods.

A deep learning paradigm for medical imaging data

We present a novel deep learning framework to address several challenges in medical imaging analysis. Especially for functional magnetic resonance imaging (fMRI) data, we develop a statistical sampling algorithm to augment small data for deep learning modeling. Also, the ComBat harmonization is utilized for multi-source batch effect correction. Finally, we implement the prediction uncertainty in deep learning through a dropout approximation of the deep Gaussian process. The proposed approach offers effective statistical tools to solve persistent problems in deep learning modeling for medical imaging data. To assess the performance of the proposed approach and demonstrate its effectiveness for various types of data, we apply it to autism fMRI and Pneumonia imaging datasets.

An efficient sparse surrogate model for aerodynamic characteristics of a supersonic compressor cascade with uncertain geometric deformations

The increasing demand for high-load compressors necessitates research and design of supersonic compressor blades. Geometric deformations of the blades resulting from manufacturing errors introduce uncertainties in their aerodynamic characteristics. For uncertainty analysis, Polynomial Chaos Expansion (PCE) has gained popularity among engineers in various disciplines. However, as the number of random variables used to describe uncertainties increases, the efficiency of the PCE method diminishes. To overcome this limitation, an Efficient Sparse Surrogate Model (ESSM) was first proposed. The ESSM combines PCE to approximate global characteristics and Gaussian process modeling to capture local variability, resulting in a highly accurate model. Additionally, by utilizing an iteratively diffeomorphic modulation under observable response preserving homotopy, important PCE basis functions are adaptively selected, leading to a sparse surrogate model that significantly reduces the required training samples. Then using the ESSM, efficient Uncertainty Quantification (UQ) and global sensitivity analysis were conducted to evaluate the impact of uncertain geometric deformations on the aerodynamic characteristics of a supersonic compressor cascade. Principal Component Analysis (PCA) was employed in conjunction with an Improved Class Shape Transformation (ICST) to characterize geometric deformations caused by manufacturing errors. The proposed ICST method exhibits enhanced accuracy in fitting near the leading and trailing edges compared to the CST method. PCA enables a 50% reduction in the number of random variables needed to describe geometric deformations. The UQ results establish a quantitative correlation between geometric deformations and supersonic aerodynamics, and the sensitivity analysis identifies some specific forms of geometric deformations that significantly impact the aerodynamics. Finally, the study investigated the flow mechanisms associated with typical geometric deformation forms, further providing recommendations for manufacturing and testing of supersonic blades to prevent performance deterioration.

Data Fusion of Wing Pressure Distributions Using Scalable Gaussian Processes

Aircraft design requires a large volume of aerodynamic data to characterize various flight conditions throughout the aircraft’s flight envelope, and the data are typically obtained through wind tunnel testing and numerical analysis. Data acquisition can be costly and inevitably entails multiple sources of uncertainty. Data fusion techniques aim to bring together the strengths and mitigate the limitations of data from various information sources. A multifidelity data fusion framework employing Gaussian process regression is adopted herein and applied to the surface pressure data of a large aircraft wing model. The modeling approach is non-hierarchical in that there is no established hierarchy of accuracy among the information sources. Scalability issues arising from the large volume of data required for the study of pressure distributions are overcome by using an approximate Gaussian process regression based on stochastic variational inference, enabling the data fusion framework to be applied effectively to industry-relevant analysis and design challenges. The approach is demonstrated for a high-dimensional data set generated through wind tunnel testing in an industrial setting and numerical analysis.

Scalable phylogenetic Gaussian process models improve the detectability of environmental signals on local extinctions for many Red List species

Abstract Conservation biologists have a daunting task of understanding the causes of species decline associated with anthropogenic factors and predicting the extinction risk of a growing number of endangered species. By reducing variances of estimates with information on closely related species, phylogenetic information among species can bridge gaps in information on species with small range sizes when modelling large numbers of endangered species. However, modelling many species with the Gaussian process (GP), which underlies the evolutionary process of phylogenetic random effects, remains a challenge owing to the computational burden in estimating the large variance–covariance matrix. Here, we applied a phylogenetic generalised mixed model with random slopes and random intercepts to 1010 endangered vascular plant taxa in Japan following phylogenetic GPs implemented by nearest neighbour GP (NNGP) approximation. NNGP enables flexibility in changing the proximity on the phylogenetic tree of species from which information is borrowed to reduce the variances of estimates with a realistic computational burden. We evaluated the effectiveness of phylogenetic models by comparing the predictive performance and descriptive power of phylogenetic and non‐phylogenetic models and identified the anthropogenic factors contributing to the decline of each of the studied endangered species. We found that the model with phylogenetic information had better prediction performance than the model without phylogenetic information. The results showed that across all explanatory variables, the phylogenetic model could detect interspecific differences in response to environmental factors in a number of species more clearly. Combined with the phylogenetic signal results, we could also detect a phylogenetic bias in the species that could benefit from the positive effects of protected areas but reduce the local extinctions of 95% of all studied taxa. In conclusion, our model, considering phylogenetic information with NNGP, allows the elucidation of factors causing the decline of many endangered species. In future analyses, the estimation of extinction probability linked to environmental change might be applied to future climate–land use scenarios, advancing the comprehensive assessment of biodiversity degradation and threats to species at multiple scales.

Data-Driven Abstractions via Binary-Tree Gaussian Processes for Formal Verification

To advance formal verification of stochastic systems against temporal logic requirements for handling unknown dynamics, researchers have been designing data-driven approaches inspired by breakthroughs in the underlying machine learning techniques. As one promising research direction, abstraction-based solutions based on Gaussian process (GP) regression have become popular for their ability to learn a representation of the latent system from data with a quantified error. Results obtained based on this model are then translated to the true system via various methods. In a recent publication, GPs using a so-called binary-tree kernel have demonstrated a polynomial speedup w.r.t. the size of the data compared to their vanilla version, outcompeting all existing sparse GP approximations. Incidentally, the resulting binary-tree Gaussian process (BTGP) is characteristic for its piecewise-constant posterior mean and covariance functions, naturally abstracting the input space into discrete partitions. In this paper, we leverage this natural abstraction of the BTGP for formal verification, eliminating the need for cumbersome abstraction and error quantification procedures. We show that the BTGP allows us to construct an interval Markov chain model of the unknown system with a speedup that is polynomial w.r.t. the size of the abstraction compared to alternative approaches. We provide a delocalized error quantification via a unified formula even when the true dynamics do not live in the function space of the BTGP. This allows us to compute upper and lower bounds on the probability of satisfying reachability specifications that are robust to both aleatoric and epistemic uncertainties.

A localized ensemble of approximate Gaussian processes for fast sequential emulation

More attention has been given to the computational cost associated with the fitting of an emulator. Substantially less attention is given to the computational cost of using that emulator for prediction. This is primarily because the cost of fitting an emulator is usually far greater than that of obtaining a single prediction, and predictions can often be obtained in parallel. In many settings, especially those requiring Markov Chain Monte Carlo, predictions may arrive sequentially and parallelization is not possible. In this case, using an emulator procedure which can produce accurate predictions efficiently can lead to substantial time savings in practice. In this paper, we propose a global model approximate Gaussian process framework via extension of a popular local approximate Gaussian process (laGP) framework. Our proposed emulator can be viewed as a treed Gaussian process where the leaf nodes are laGP models, and the tree structure is learned greedily as a function of the prediction stream. The suggested method (called leapGP) has interpretable tuning parameters which control the time‐memory trade‐off. One reasonable choice of settings leads to an emulator with a training cost and makes predictions rapidly with an asymptotic amortized cost of .

Robust Inference for Partially Observed Functional Response Data.

Irregular functional data in which densely sampled curves are observed over different ranges pose a challenge for modeling and inference, and sensitivity to outlier curves is a concern in applications. Motivated by applications in quantitative ultrasound signal analysis, this paper investigates a class of robust M-estimators for partially observed functional data including functional location and quantile estimators. Consistency of the estimators is established under general conditions on the partial observation process. Under smoothness conditions on the class of M-estimators, asymptotic Gaussian process approximations are established and used for large sample inference. The large sample approximations justify a bootstrap approximation for robust inferences about the functional response process. The performance is demonstrated in simulations and in the analysis of irregular functional data from quantitative ultrasound analysis.

Projecting basis functions with tensor networks for Gaussian process regression

This paper presents a method for approximate Gaussian process (GP) regression with tensor networks (TNs). A parametric approximation of a GP uses a linear combination of basis functions, where the accuracy of the approximation depends on the total number of basis functions M. We develop an approach that allows us to use an exponential amount of basis functions without the corresponding exponential computational complexity. The key idea to enable this is using low-rank TNs. We first find a suitable low-dimensional subspace from the data, described by a low-rank TN. In this low-dimensional subspace, we then infer the weights of our model by solving a Bayesian inference problem. Finally, we project the resulting weights back to the original space to make GP predictions. The benefit of our approach comes from the projection to a smaller subspace: It modifies the shape of the basis functions in a way that it sees fit based on the given data, and it allows for efficient computations in the smaller subspace. In an experiment with an 18-dimensional benchmark data set, we show the applicability of our method to an inverse dynamics problem.

AN ADAPTIVE STRATEGY FOR SEQUENTIAL DESIGNS OF MULTILEVEL COMPUTER EXPERIMENTS

Investigating uncertainties in computer simulations can be prohibitive in terms of computational costs, since the simulator needs to be run over a large number of input values. Building an emulator, i.e., a statistical surrogate model of the simulator constructed using a design of experiments made of a comparatively small number of evaluations of the forward solver, greatly alleviates the computational burden to carry out such investigations. Nevertheless, this can still be above the computational budget for many studies. Two major approaches have been used to reduce the budget needed to build the emulator: efficient design of experiments, such as sequential designs, and combining training data of different degrees of sophistication in a so-called multifidelity method, or multilevel in case these fidelities are ordered typically for increasing resolutions. We present here a novel method that combines both approaches, the multilevel adaptive sequential design of computer experiments in the framework of Gaussian process (GP) emulators. We make use of reproducing kernel Hilbert spaces as a tool for our GP approximations of the differences between two consecutive levels. This dual strategy allows us to allocate efficiently limited computational resources over simulations of different levels of fidelity and build the GP emulator. The allocation of computational resources is shown to be the solution of a simple optimization problem in a special case where we theoretically prove the validity of our approach. Our proposed method is compared to other existing models of multifidelity Gaussian process emulation. Gains in orders of magnitudes in accuracy or computing budgets are demonstrated in some numerical examples for some settings.

Learning Non-Parametric Models in Real Time via Online Generalized Product of Experts

In this work, we address the problem of online learning, where models must be continually updated from an incoming stream of data, while retaining past information. We develop an approach that is nonparametric, models uncertainty, and requires minimal hand-tuning. Our proposed algorithm, which we term online generalized product of experts (OGPoE), extends the powerful generalized product of experts (GPoE) framework to the online setting by leveraging methods for sparse, variational Gaussian process approximations, as well as nonparametric clustering. We devise a 1-D example learning problem to illustrate how our method works, and we verify that we achieve competitive results with other popular modeling approaches on a benchmark learning problem. Finally, we demonstrate how our algorithm can produce high accuracy predictions on a physical system, by learning the kinematics for a concentric tube robot, even when the robot is subject to changing, unknown loads.

Scalable spatio‐temporal smoothing via hierarchical sparse Cholesky decomposition

AbstractWe propose an approximation to the forward filter backward sampler (FFBS) algorithm for large‐scale spatio‐temporal smoothing. FFBS is commonly used in Bayesian statistics when working with linear Gaussian state‐space models, but it requires inverting covariance matrices which have the size of the latent state vector. The computational burden associated with this operation effectively prohibits its applications in high‐dimensional settings. We propose a scalable spatio‐temporal FFBS approach based on the hierarchical Vecchia approximation of Gaussian processes, which has been previously successfully used in spatial statistics. On simulated and real data, our approach outperformed a low‐rank FFBS approximation.

Efficient evaluation of stochastic traffic flow models using Gaussian process approximation

This paper studies a Gaussian process approximation for a class of stochastic traffic flow models. It can be used to efficiently and accurately evaluate the joint (in the spatial and temporal sense) distribution of vehicle-density distributions in road traffic networks of arbitrary topology. The Gaussian approximation follows, via a scaling-limit argument, from a Markovian model that is consistent with discrete-space kinematic wave models. We describe in detail how this formal result can be converted into a computational procedure. The performance of our approach is demonstrated through a series of experiments that feature various realistic scenarios. Moreover, we discuss the computational complexity of our approach by assessing how computation times depend on the network size. We also argue that the (debatable) assumption that the vehicles’ headways are exponentially distributed does not negatively impact the accuracy of our approximation.

Emulating loss of coolant simulations in a pressurized heavy water reactor

In an ideal risk-informed approach to safety analysis, all possible operational states of the system and all sources of uncertainty would be factored. Brute force approaches are cost-prohibitive if high-fidelity multiphysics simulation systems are used. To address this barrier, we present a first attempt to develop emulators for coupled neutronics/thermalhydraulics simulations of large break loss-of-coolant accidents in a generic pressurized heavy water reactor. The emulators are based on a linear mean function coupled with a locally approximate Gaussian process model.The emulators are capable of predicting the power pulse amplitude and the maximum bundle enthalpy within mean absolute relative errors of 0.75 and 0.36%, and mean relative margins of error of 1.9 and 1.0%, respectively. In relation to the simulator, the evaluation time is reduced by a factor of nearly 1800. The paper covers the simulation system, training data generation framework, statistical models and uncertainties, and results.

Fast Approximate Multioutput Gaussian Processes

Gaussian processes regression models are an appealing machine learning method as they learn expressive nonlinear models from exemplar data with minimal parameter tuning and estimate both the mean and covariance of unseen points. However, cubic computational complexity growth with the number of samples has been a long standing challenge. Training requires the inversion of $N \times N$N×N kernel at every iteration, whereas regression needs computation of an $m \times N$m×N kernel, where $N$N and $m$m are the number of training and test points, respectively. This work demonstrates how approximating the covariance kernel using eigenvalues and functions leads to an approximate Gaussian process with significant reduction in training and regression complexity. Training now requires computing only an $N \times n$N×n eigenfunction matrix and an $n \times n$n×n inverse, where $n$n is a selected number of eigenvalues. Furthermore, regression now only requires an $m \times n$m×n matrix. Finally, in a special case, the hyperparameter optimization is completely independent from the number of training samples. The proposed method can regress over multiple outputs, learn the correlations between them, and estimate their derivatives to any order. The computational complexity reduction, regression capabilities, multioutput correlation learning, and comparison to the state of the art are demonstrated in simulation examples. Finally we show how the proposed approach can be utilized to model real human data.

How Good Are Low-Rank Approximations in Gaussian Process Regression?

We provide guarantees for approximate Gaussian Process (GP) regression resulting from two common low-rank kernel approximations: based on random Fourier features, and based on truncating the kernel's Mercer expansion. In particular, we bound the Kullback–Leibler divergence between an exact GP and one resulting from one of the afore-described low-rank approximations to its kernel, as well as between their corresponding predictive densities, and we also bound the error between predictive mean vectors and between predictive covariance matrices computed using the exact versus using the approximate GP. We provide experiments on both simulated data and standard benchmarks to evaluate the effectiveness of our theoretical bounds.

High points of a random model of the Riemann-zeta function and Gaussian multiplicative chaos

We study the total mass of high points in a random model for the Riemann-zeta function. We consider the same model as in Harper (2013) and Arguin et al. (2017), and build on the convergence to Gaussian multiplicative chaos proved in Saksman and Webb (2016). We show that the total mass of points which are a linear order below the maximum, divided by their expectation, converges almost surely to the Gaussian multiplicative chaos of the approximating Gaussian process times a random function. We use the second moment method together with a branching approximation to establish this convergence.

Fast increased fidelity samplers for approximate Bayesian Gaussian process regression.

Gaussian processes (GPs) are common components in Bayesian non-parametric models having a rich methodological literature and strong theoretical grounding. The use of exact GPs in Bayesian models is limited to problems containing several thousand observations due to their prohibitive computational demands. We develop a posterior sampling algorithm using -matrix approximations that scales at . We show that this approximation's Kullback-Leibler divergence to the true posterior can be made arbitrarily small. Though multidimensional GPs could be used with our algorithm, d-dimensional surfaces are modeled as tensor products of univariate GPs to minimize the cost of matrix construction and maximize computational efficiency. We illustrate the performance of this fast increased fidelity approximate GP, FIFA-GP, using both simulated and non-synthetic data sets.