Local Gaussian Process Research Articles

Federated Many-Task Bayesian Optimization

Bayesian optimization is a powerful surrogate-assisted algorithm for solving expensive black-box optimization problems. While Bayesian optimization was developed for centralized optimization, the availability of massive distributed data has attracted increased interests in exploring federated Bayesian optimization that can use data on multiple clients without leaking the raw data. However, existing federated Bayesian optimization (FBO) approaches assume that either all clients jointly solve the same optimization task, or only one client solves one target optimization task by transferring knowledge from others in a federated way, making them unsuited for many real-world applications. In this paper, we consider FBO for the scenario where multiple related local black-box tasks associated with different clients are jointly optimized by sharing knowledge between tasks without leaking the data privacy. An efficient federated many-task Bayesian optimization framework is proposed to address not independent and identically distributed (non-IID) data while protecting the data privacy in the federated setting. A novel federated knowledge transfer paradigm is developed for dynamic many-task model aggregation according to a dissimilarity matrix. The dissimilarity is measured based on the rank of the predictions and only the hyperparameters in the local Gaussian process models are shared. In addition, a federated ensemble acquisition function is constructed by integrating the predictions of two surrogates using the global and local hyperparameters, respectively, to effectively search for the optimal solution. Experimental results show that our proposed method has reliable performance on both benchmark problems and a real machine learning problem also in the presence of non-IID data.

Optimizing green splits in high‐dimensional traffic signal control with trust region Bayesian optimization

AbstractCentralized traffic signal control has long been a challenging, high‐dimensional optimization problem. This study establishes a simulation‐based optimization framework and develops a novel optimization algorithm based on trust region Bayesian optimization (TuRBO), which can efficiently obtain an approximate optimal solution to the high‐dimensional traffic signal control problem. Local Gaussian process (GP), trust region, and Thompson sampling are employed in the TuRBO and contribute considerably to performance in terms of computational speed, solution quality, and scalability. Empirical studies are carried out using data from Mudanjiang and Chengdu, China. The performance of TuRBO is compared with that of Bayesian optimization (BO), genetic algorithm and random sampling. The results show that TuRBO converges the fastest because of its ability to balance exploration and exploitation through the trust region and Thompson sampling. Meanwhile, because TuRBO enables more efficient exploitation through the local GP, the solution quality of TuRBO outperforms others significantly. The average waiting time achieved by TuRBO was 2.84% lower than that achieved by BO. Finally, the method has been successfully extended to a large network with 233‐dimensional spaces and 122 signalized intersections, demonstrating that the developed methodology can deal with high‐dimensional traffic signal control effectively for real case applications.

Taking another step: A simple approach to high-dimensional Bayesian optimization

Scaling Bayesian optimization to high-dimensional problems is a meaningful but challenging task. Most of current approaches assume only a few variables are effective to the objective function or the objective function is additively separable, therefore are not suitable when the problem violates these assumptions. In this work, we propose a simple and efficient approach to extend Bayesian optimization to high dimensions. The proposed approach does not make these two assumptions. In each iteration, after locating a candidate point by the global Gaussian process model, we train a local Gaussian process model around the candidate point and locate a new point using the local model. Instead of evaluating the solution located by the global model, we evaluate the solution located by the local model. This simple taking-another-step approach is shown to be able to improve Bayesian optimization's performance significantly on high-dimensional optimization problems. The proposed algorithm also shows competitive performance when compared with four high-dimensional Bayesian optimization algorithms and four surrogate-assisted evolutionary algorithms.

Interpretable machine learning of SPAC system via a mechanism-assisted gaussian process group: Representation of the system mechanism by data

Traditional models of the soil-plant-atmosphere continuum (SPAC) system are physics-based and their practical application has been hindered by issues of high parameterization, prior structural bias, and costly running. In this paper, we developed a machine learning modelling method based on the Gaussian process (GP) to avoid these difficulties of traditional modelling. At the same time, shortcomings of conventional machine learning in terms of interpretability and mechanism representation were addressed. The innovative method consisted of structuring a local GP group framework and improving kernel functions used in each local GP. The resultant model was endowed with advanced capacities, including representing interplays between subprocesses within the complex SPAC system, representing nonstationary subprocess dynamics caused by crop growth stage shifts, as well as automatically interpreting key low-order input interactions and dominant input variables for each subprocess. The performance of our model was examined on synthetic SPAC system datasets that covered three different soil conditions. Results demonstrated that its interpretations regarding subprocess mechanisms were robust across different soil conditions and consistent with domain knowledge. Compared to a conventional global GP model and two deep learning models (DNN and LSTM), our model performed significantly better not only in regular prediction experiments but also in generalization experiments that required models to be transferred across different conditions. Our study suggested that it is promising to enable a machine learning model interpretable by improving its feature representation function. For machine learning modelling of complex systems like the SPAC, it may be more important to enhance the model’s mechanism representation capability than simply using deeper networks.

Deep Reinforcement Multiagent Learning Framework for Information Gathering with Local Gaussian Processes for Water Monitoring

The conservation of hydrological resources involves continuously monitoring their contamination. A multiagent system composed of autonomous surface vehicles is proposed herein to efficiently monitor the water quality. To achieve a safe control of the fleet, the fleet policy should be able to act based on measurements and fleet state. It is proposed to use local Gaussian processes and deep reinforcement learning to jointly obtain effective monitoring policies. Local Gaussian processes, unlike classical global Gaussian processes, can accurately model the information in a dissimilar spatial correlation which captures more accurately the water quality information. A deep convolutional policy is proposed, that bases the decisions on the observation on the mean and variance of this model, by means of an information gain reward. Using a double deep Q‐learning algorithm, agents are trained to minimize the estimation error in a safe manner thanks to a Consensus‐based heuristic. Simulation results indicate an improvement of up to 24% in terms of the mean absolute error with the proposed models. Also, training results with 1–3 agents indicate that our proposed approach returns 20% and 24% smaller average estimation errors for, respectively, monitoring water quality variables and monitoring algae blooms, as compared to state‐of‐the‐art approaches.

Accelerated Bayesian Inference for Molecular Simulations using Local Gaussian Process Surrogate Models.

While Bayesian inference is the gold standard for uncertainty quantification and propagation, its use within physical chemistry encounters formidable computational barriers. These bottlenecks are magnified for modeling data with many independent variables, such as X-ray/neutron scattering patterns and electromagnetic spectra. To address this challenge, we employ local Gaussian process (LGP) surrogate models to accelerate Bayesian optimization over these complex thermophysical properties. The time-complexity of the LGPs scales linearly in the number of independent variables, in stark contrast to the computationally expensive cubic scaling of conventional Gaussian processes. To illustrate the method, we trained a LGP surrogate model on the radial distribution function of liquid neon and observed a 1,760,000-fold speed-up compared to molecular dynamics simulation, beating a conventional GP by three orders-of-magnitude. We conclude that LGPs are robust and efficient surrogate models poised to expand the application of Bayesian inference in molecular simulations to a broad spectrum of experimental data.

D 3 PBO: D ynamic D omain D ecomposition-based P arallel B ayesian O ptimization for Large-scale Analog Circuit Sizing

Bayesian optimization (BO) is an efficient global optimization method for expensive black-box functions, but the expansion for high-dimensional problems and large sample budgets still remains a severe challenge. In order to extend BO for large-scale analog circuit synthesis, a novel computationally efficient parallel BO method, D 3 PBO, is proposed for high-dimensional problems in this work. We introduce the dynamic domain decomposition method based on maximum variance between clusters. The search space is decomposed into subdomains progressively to limit the maximal number of observations in each domain. The promising domain is explored by multi-trust region-based batch BO with the local Gaussian process (GP) model. As the domain decomposition progresses, the basin-shaped domain is identified using a GP-assisted quadratic regression method and exploited by the local search method BOBYQA to achieve a faster convergence rate. The time complexity of D 3 PBO is constant for each iteration. Experiments demonstrate that D 3 PBO obtains better results with significantly less runtime consumption compared to state-of-the-art methods. For the circuit optimization experiments, D 3 PBO achieves up to 10× runtime speedup compared to TuRBO with better solutions.

A Local Gaussian Process Regression Approach to Mass Appraisal of Residential Properties

A Local Gaussian Process Regression Approach to Mass Appraisal of Residential Properties

Active sparse Bayesian committee machine potential for isothermal-isobaric molecular dynamics simulations.

Recent advancements in machine learning potentials (MLPs) have significantly impacted the fields of chemistry, physics, and biology by enabling large-scale first-principles simulations. Among different machine learning approaches, kernel-based MLPs distinguish themselves through their ability to handle small datasets, quantify uncertainties, and minimize over-fitting. Nevertheless, their extensive computational requirements present considerable challenges. To alleviate these, sparsification methods have been developed, aiming to reduce computational scaling without compromising accuracy. In the context of isothermal and isobaric ML molecular dynamics (MD) simulations, achieving precise pressure estimation is crucial for reproducing reliable system behavior under constant pressure. Despite progress, sparse kernel MLPs struggle with precise pressure prediction. Here, we introduce a virial kernel function that significantly enhances the pressure estimation accuracy of MLPs. Additionally, we propose the active sparse Bayesian committee machine (BCM) potential, an on-the-fly MLP architecture that aggregates local sparse Gaussian process regression (SGPR) MLPs. The sparse BCM potential overcomes the steep computational scaling with the kernel size, and a predefined restriction on the size of kernel allows for fast and efficient on-the-fly training. Our advancements facilitate accurate and computationally efficient machine learning-enhanced MD (MLMD) simulations across diverse systems, including ice-liquid coexisting phases, Li10Ge(PS6)2 lithium solid electrolyte, and high-pressure liquid boron nitride.

A Local Gaussian Process Regression Approach to Frequency Response Function Estimation

Frequency response function (FRF) estimation is a classical subject in system Identification. In the past two decades, there have been remarkable advances in developing local methods for this subject, e.g., the local polynomial method, local rational method, and iterative local rational method. The recent concentrations for local methods are two issues: the model order selection and the Identification of lightly damped systems. To address these two issues, we propose a new local method called local Gaussian process regression (LGPR). We show that the frequency response function locally is either analytic or resonant, and this prior knowledge can be embedded into a kernel-based regularized estimate through a dot-product kernel plus a resonance kernel induced by a second-order resonant system. The LGPR provides a new route to tackle the aforementioned issues. In the numerical simulations, the LGPR shows the best FRF estimation accuracy compared with the existing local methods, and moreover, the LGPR is more robust with respect to sample size and noise level.

Nonparametric Modeling and Control of Ship Steering Motion Based on Local Gaussian Process Regression

This paper aims to study the nonparametric modeling and control of ship steering motion. Firstly, the black box response model is derived based on the Nomoto model. Then, the establishment of a nonparametric response model and prediction of ship steering motion are realized by applying the local Gaussian process regression (LGPR) algorithm. To assess the performance of LGPR, two cases are studied, including a Mariner class vessel by using simulation data and a KVLCC2 tanker model by using experimental data. The results reveal that the response model identified by LGPR presents good prediction accuracy and low computational burden. Finally, the identified response model is used as the basis for developing the ship heading controller, and the results demonstrate that the proposed controller is able to achieve good dynamic performance.

Latin hypercubes for constrained design of experiments for data-driven models

Abstract The quality of data used for data-driven modeling affects the model performance significantly. Thus, design of experiments (DoE) is an important part during model development. The design space is constrained in many applications. In this work, the constrained case is investigated. An Latin hypercube based approach is applied and analyzed for strongly constrained design spaces. Contrary to commonly used optimization techniques, an incremental procedure is proposed. In every step, new data are added to the design. Each new point is selected by a distance-based criterion. The performance of the created designs is evaluated by the quality of the trained models. For different constraints, artificial data sets are created with a function generator. The performance of local model networks and Gaussian process regression models trained with those designs is evaluated and compared to models trained on data sets based on Sobol’ sequences.

Local Gaussian Process Extrapolation for BART Models with Applications to Causal Inference

Bayesian additive regression trees (BART) is a semi-parametric regression model offering state-of-the-art performance on out-of-sample prediction. Despite this success, standard implementations of BART typically suffer from inaccurate prediction and overly narrow prediction intervals at points outside the range of the training data. This article proposes a novel extrapolation strategy that grafts Gaussian processes to the leaf nodes in BART for predicting points outside the range of the observed data. The new method is compared to standard BART implementations and recent frequentist resampling-based methods for predictive inference. We apply the new approach to a challenging problem from causal inference, wherein for some regions of predictor space, only treated or untreated units are observed (but not both). In simulation studies, the new approach boasts superior performance compared to popular alternatives, such as Jackknife+. Supplementary materials for this article are available online.

A local Gaussian Processes method for fitting potential surfaces that obviates the need to invert large matrices

In order to compute a vibrational spectrum, one often wishes to start with a set of ab initio Born–Oppenheimer potential values at points, called fitting points, and interpolate or fit to find values of the potential at quadrature or collocation points. It is common to do this once to build a potential energy surface (PES). Once the PES is known, it can be evaluated at any point in configuration space. Gaussian Process (GP) is frequently being used to make a PES. As is the case in other interpolation methods, to use GP one must store and invert a matrix whose size is the number of fitting points. The matrix is sometimes large enough that approximations are introduced to reduce the cost of the calculation. We show that is possible to use many local Gaussian Process fits rather than one global fit. Retaining only local Gaussians and the associated points works well despite the fact that other Gaussians have tails with significant amplitude in the local region. We demonstrate that from the potential values obtained from the local fits it is possible to compute accurate energy levels of formaldehyde. In one calculation, potential values were obtained with N=120,000 fitting points by inverting matrices of size less than m=400. The local idea reduces the cost from N3 to T(m3+N), where T is the number of desired potential points.

An efficient two‐stage algorithm for parameter identification of non‐linear state‐space models‐based on Gaussian process regression

AbstractThis paper aims to improve the efficiency of parameter identification of the nonlinear state‐space model (SSM). The commonly used particle Markov chain Monte Carlo (PMCMC) method is time‐consuming. The surrogate model is a useful acceleration strategy, but it is expensive to establish a global high‐precision surrogate model. This paper proposes an efficient algorithm that gradually estimates the unknown parameters in two stages. In the first stage, a reduced region is established based on the latest method. We train a local Gaussian Process regression (GPR) of the likelihood function in the reduced region based on the optimum Latin hypercube design (OLHD). In the second stage, we identify the unknown parameters more accurately based on the MCMC method. When the proposal sample is in the reduced region, we use GPR to estimate the likelihood; otherwise, BPF is used to estimate the likelihood. The reduced region is usually the high probability density region, which is why the algorithm is efficient. It is proved that the acceptance rate of any two samples based on the proposed algorithm is theoretically convergent to that of the PMCMC algorithm. Two examples demonstrate that the proposed method performs well in accuracy and efficiency.

Partitioned Active Learning for Heterogeneous Systems

Abstract Active learning is a subfield of machine learning that focuses on improving the data collection efficiency in expensive-to-evaluate systems. Active learning-applied surrogate modeling facilitates cost-efficient analysis of demanding engineering systems, while the existence of heterogeneity in underlying systems may adversely affect the performance. In this article, we propose the partitioned active learning that quantifies informativeness of new design points by circumventing heterogeneity in systems. The proposed method partitions the design space based on heterogeneous features and searches for the next design point with two systematic steps. The global searching scheme accelerates exploration by identifying the most uncertain subregion, and the local searching utilizes circumscribed information induced by the local Gaussian process (GP). We also propose Cholesky update-driven numerical remedies for our active learning to address the computational complexity challenge. The proposed method consistently outperforms existing active learning methods in three real-world cases with better prediction and computation time.

A localized ensemble of approximate Gaussian processes for fast sequential emulation

More attention has been given to the computational cost associated with the fitting of an emulator. Substantially less attention is given to the computational cost of using that emulator for prediction. This is primarily because the cost of fitting an emulator is usually far greater than that of obtaining a single prediction, and predictions can often be obtained in parallel. In many settings, especially those requiring Markov Chain Monte Carlo, predictions may arrive sequentially and parallelization is not possible. In this case, using an emulator procedure which can produce accurate predictions efficiently can lead to substantial time savings in practice. In this paper, we propose a global model approximate Gaussian process framework via extension of a popular local approximate Gaussian process (laGP) framework. Our proposed emulator can be viewed as a treed Gaussian process where the leaf nodes are laGP models, and the tree structure is learned greedily as a function of the prediction stream. The suggested method (called leapGP) has interpretable tuning parameters which control the time‐memory trade‐off. One reasonable choice of settings leads to an emulator with a training cost and makes predictions rapidly with an asymptotic amortized cost of .

Local Gaussian process regression with small sample data for temperature and humidity compensation of polyaniline-cerium dioxide NH3 sensor

Gas sensors have made great progress in gas sensing performances (such as response, sensitivity, detection limit and response speed), but they are generally affected by temperature and humidity. Here, we proposed a local Gaussian process regression with small samples for temperature and humidity compensation of gas sensors. Specifically, the above method is used to compensate the temperature and humidity influence of the polyaniline-cerium dioxide (PANI-CeO2) ammonia (NH3) sensor (10–50 °C, 20%−70% relative humidity (RH); It should be noted that the law of humidity influence is messy when the RH is greater than 70%.). The adaptive matching results show that the optimal number of K-neighbor points is 15, which greatly reduces the amount of computation. The temperature and humidity compensation results show that the predicted concentration achieved high accuracy (the mean absolute error is 0.19 ppm, and the mean relative error is 0.65%), and the absolute error showed a good normal distribution (N∼(0.00047, 0.3772)). This work provides an effective compensation strategy with small samples, high precision and low computational cost for the temperature and humidity influences of gas sensor.

Identification modeling of ship maneuvering motion based on local Gaussian process regression

A fast and accurate nonparametric modeling method based on local Gaussian process regression (LGPR) is proposed for the identification modeling and prediction of ship maneuvering motion. The training dataset collected from the free-running model tests of ship maneuvering is automatically divided into a number of clusters according to the similarity criterion by clustering analysis using k-means algorithm. Utilizing the data in each cluster, the corresponding local nonparametric model is identified. The computational cost of training and prediction based on LGPR is reduced compared to that based on the classic Gaussian process regression (CGPR) using the whole training dataset. Taking the KVLCC2 tanker and an unmanned surface vehicle (USV) as study objects, the nonparametric models are identified based on the experimental data of zigzag maneuvers of the KVLCC2 model and random maneuver of the USV. Using the identified models, the zigzag maneuvers of the KVLCC2 model and the random maneuver of the USV, which are not involved in the training data, are predicted. The results show that LGPR has higher computational efficiency than CGPR with acceptable prediction accuracy.

Bayesian Optimization for Expensive Smooth-Varying Functions

Bayesian optimization (BO) is a powerful class of data-driven techniques for the maximization of expensive-to-evaluate objective functions. These techniques construct a Gaussian process (GP) regression for representing the objective function according to the latest available function evaluations and sequentially select samples and evaluate the function by maximizing an acquisition function. The primary assumption in most BO policies is that the objective function has a uniform level of smoothness over the input space, modeled by a kernel function. However, the uniform smoothness assumption is likely to be violated in a wide range of practical problems, primary domains in which the objective function is evaluated differently at various regions of input space (e.g., through different experiments, software, or approximators). This article develops a BO framework capable of optimizing expensive smooth-varying functions. Unlike the existing techniques that rely on a single GP model, the proposed framework constructs a set of local and global GP models to represent the objective function. The predictive mean and variance at any given sample in the input space are computed according to the posterior probabilities of the local and global GP models. Local and global models are adaptively controlled through a single parameter, which can be optimized along with other GP models’ parameters during the optimization process. Using the predicted local and global values, the expected improvement acquisition function is employed as one of the possible acquisition functions for the selection process. The performance of the proposed framework is assessed extensively through two optimization benchmark problems.