Gaussian Process Model Research Articles

PC-Kriging-powered parallelizing Bayesian updating for stochastic vehicle-track dynamical system with contact force measurements and Gaussian process discrepancy model

High-precision vehicle-track coupled dynamical models play a vital role in assessing the running safety of the vehicle, tracking fatigue behavior, and establishing a digital twin for high-speed railways. This study established a high-fidelity vehicle-track coupled dynamical model as a forward solver, which was then updated in a Bayesian inference framework based on the field tests of rail irregularities and wheel/rail normal contact forces. The prediction errors corresponding to different time steps were represented through a Gaussian process (GP) model characterized by a covariance function with the Gaussian kernel to incorporate the correlation between different time steps. To cope with the considerable expense of the likelihood function calculations with large-scale loop operations and the computational burden due to the large batch of repetitive evaluations of the high-fidelity forward model involved in the stochastic sampling from the posterior probability distribution, a metamodel-powered parallelizing stochastic sampling scheme was adopted. The posterior distribution could be formulated by replacing the expensive explicit model evaluation involved in the likelihood function with a Polynomial-Chaos-Kriging (PC-Kriging) predictor providing a surrogate mapping between the wheel/rail normal contact forces and physical parameters. Finally, a parallelizing computing scheme running across multiple CPU cores on high-performance computers was realized to sample from the vectorized formula of the emulator-powered posterior distribution. Field test data from a high-speed railway in China was adopted to demonstrate that the Bayesian inference scheme can quantify the uncertainties of the core physical parameters of the high-fidelity vehicle-track coupled dynamical system by efficiently trading off accuracy and computational efficiency.

Active learning-based metamodeling for hybrid uncertainty quantification of hydro-mechatronic-control systems: A case study of EHA systems

The Electro-Hydrostatic Actuator (EHA) is a typical hydro-mechatronic control system. Due to the limited accuracy of measurement, inadequate knowledge, and vague judgments, hybrid uncertainties, including aleatory or epistemic uncertainty, inevitably exist in the performance assessment of EHA systems. Existing methods ignored the hybrid uncertainties which can hardly obtain a satisfactory result while wasting a lot of time on the experimental design. To overcome this drawback, a metamodeling method for hybrid uncertainty propagation of EHA systems is developed via an active learning Gaussian Process (GP) model. The proposed method is bifurcated into three pillars: (A) Initializing the GP model and generating the optimum candidate sampling set by an Optimized Max-Minimize Distance (OMMD) algorithm, which aims to maximize the minimum distance between the added samples and original samples, (B) maximizing the learning function and generating new samples by a developed farthest or nearest judgment strategy, while updating the original GP model, and (C) judging the convergence by three uncertainty metrics, i.e., area metric, maximum variance metric, and mean value metric. A numerical example is exemplified to evaluate the effectiveness and efficiency of the proposed method. Meanwhile, the EHA system of aircrafts is examined to show the application of the proposed method for high-dimensional problems. The effects of the uncertainties in the Proportional-Integral-Differential (PID) of the EHA system are also examined.

Probabilistic Attention Based on Gaussian Processes for Deep Multiple Instance Learning.

Multiple instance learning (MIL) is a weakly supervised learning paradigm that is becoming increasingly popular because it requires less labeling effort than fully supervised methods. This is especially interesting for areas where the creation of large annotated datasets remains challenging, as in medicine. Although recent deep learning MIL approaches have obtained state-of-the-art results, they are fully deterministic and do not provide uncertainty estimations for the predictions. In this work, we introduce the attention Gaussian process (AGP) model, a novel probabilistic attention mechanism based on Gaussian processes (GPs) for deep MIL. AGP provides accurate bag-level predictions as well as instance-level explainability and can be trained end-to-end. Moreover, its probabilistic nature guarantees robustness to overfit on small datasets and uncertainty estimations for the predictions. The latter is especially important in medical applications, where decisions have a direct impact on the patient's health. The proposed model is validated experimentally as follows. First, its behavior is illustrated in two synthetic MIL experiments based on the well-known MNIST and CIFAR-10 datasets, respectively. Then, it is evaluated in three different real-world cancer detection experiments. AGP outperforms state-of-the-art MIL approaches, including deterministic deep learning ones. It shows a strong performance even on a small dataset with less than 100 labels and generalizes better than competing methods on an external test set. Moreover, we experimentally show that predictive uncertainty correlates with the risk of wrong predictions, and therefore it is a good indicator of reliability in practice. Our code is publicly available.

Federated Many-Task Bayesian Optimization

Bayesian optimization is a powerful surrogate-assisted algorithm for solving expensive black-box optimization problems. While Bayesian optimization was developed for centralized optimization, the availability of massive distributed data has attracted increased interests in exploring federated Bayesian optimization that can use data on multiple clients without leaking the raw data. However, existing federated Bayesian optimization (FBO) approaches assume that either all clients jointly solve the same optimization task, or only one client solves one target optimization task by transferring knowledge from others in a federated way, making them unsuited for many real-world applications. In this paper, we consider FBO for the scenario where multiple related local black-box tasks associated with different clients are jointly optimized by sharing knowledge between tasks without leaking the data privacy. An efficient federated many-task Bayesian optimization framework is proposed to address not independent and identically distributed (non-IID) data while protecting the data privacy in the federated setting. A novel federated knowledge transfer paradigm is developed for dynamic many-task model aggregation according to a dissimilarity matrix. The dissimilarity is measured based on the rank of the predictions and only the hyperparameters in the local Gaussian process models are shared. In addition, a federated ensemble acquisition function is constructed by integrating the predictions of two surrogates using the global and local hyperparameters, respectively, to effectively search for the optimal solution. Experimental results show that our proposed method has reliable performance on both benchmark problems and a real machine learning problem also in the presence of non-IID data.

Towards Improved Turbomachinery Measurements: A Comprehensive Analysis of Gaussian Process Modeling for a Data-Driven Bayesian Hybrid Measurement Technique

Towards Improved Turbomachinery Measurements: A Comprehensive Analysis of Gaussian Process Modeling for a Data-Driven Bayesian Hybrid Measurement Technique

O'Hare Airport roadway traffic prediction via data fusion and Gaussian process regression

This study proposes an approach of leveraging information gathered from multiple traffic data sources at different resolutions to obtain approximate inference on the traffic distribution of Chicago's O'Hare Airport area. Specifically, it proposes the ingestion of traffic datasets at different resolutions to build spatiotemporal models for predicting the distribution of traffic volume on the road network. Due to its good adaptability and flexibility for spatiotemporal data, the Gaussian process (GP) regression was employed to provide short-term forecasts using data collected by loop detectors (sensors) and supplemented by telematics data. The GP regression is used to make predictions of the distribution of the proportion of sensor data traffic volume represented by the telematics data for each location of the sensors. Consequently, the fitted GP model can be used to determine the approximate traffic distribution for a testing location outside of the training points. Policymakers in the transportation sector can find the results of this work helpful for making informed decisions relating to current and future transportation conditions in the area.

Comprehensive evaluation and systematic comparison of Gaussian process (GP) modelling applications in peptide quantitative structure-activity relationship

Peptide quantitative structure-activity relationship (pQSAR) is a specific extension of traditional QSARs from small-molecule drugs to bioactive peptides. Since peptides are linear biopolymers that are essentially different to small-molecule compounds in terms of their structural features such as ordering sequence, large size and intrinsic flexibility, the pQSAR methodology (including structural characterization and regression modelling) should be further exploited relative to traditional QSARs. Gaussian process (GP) serves as a pioneering Bayesian-based machine learning (ML) solution for tackling linear/nonlinear-hybrid regression issues in intricate domains. However, the applications of GP regression in QSAR and, particularly, the pQSAR still remain largely unexplored to date. In this work, we launched a comprehensive pQSAR study with GP regression modelling, aiming to the deep evaluation of GP performance based on different characterizations and also the systematic comparison of GP with other routine MLs. Here, we culled two distinct classes of peptide datasets, which separately comprise 12 panels of sophisticated benchmarks and 46 panels of extended samples, totally containing 8804 peptide samples and systematically resulting in 522 regression models. Our study indicated that the GP can generally provide an effective solution for many pQSAR problems with the potential to promote ML regression modelling in this area, which is comparable with or even better than those widely used methods on both the sophisticated benchmarks and extended samples. In addition, GP also has many advantages as compared to traditional MLs, such as hyperparameter self-consistency, overfitting resistance, interpretable output and estimable uncertainty.

Bayesian optimization-based Model-Agnostic Meta-Learning: Application to predict maximum cyclic moment resistance of steel bolted T-stub connections

Accurately assessing the maximum moment resistance of steel bolted T-stub connections under cyclic loading is crucial for designing earthquake-resistant structures with such connections. Traditional methods based on design standards like ASCE 41-17 often lack precision. Recently, supervised machine learning techniques, particularly Artificial Neural Network (ANN), have been explored. However, conventional ANNs require substantial data for generalization, which is limited for steel bolted T-stub connections. To address these challenges, this study explores the feasibility of using the Model-Agnostic Meta-Learning (MAML) to predict the maximum cyclic moment resistance of steel bolted T-stub connections. MAML adapts task-specific model parameters rapidly and transfers knowledge across tasks to fine-tune global model parameters, potentially enhancing prediction accuracy with limited data. The MAML model is first optimized using Bayesian optimization with a Gaussian Process model to identify ideal hyperparameters. The optimized MAML model is then compared with two ANN models (one with optimized hyperparameters, another matching MAML's neural network architecture) and ASCE 41-17 method. Results demonstrate the optimized MAML model's superior generalization capabilities, offering a promising approach for steel bolted T-stub connections.

Feature Detection of Non-Cooperative and Rotating Space Objects through Bayesian Optimization.

In this paper, we propose a Bayesian Optimization (BO)-based strategy using the Gaussian Process (GP) for feature detection of a known but non-cooperative space object by a chaser with a monocular camera and a single-beam LIDAR in a close-proximity operation. Specifically, the objective of the proposed Space Object Chaser-Resident Assessment Feature Tracking (SOCRAFT) algorithm is to determine the camera directional angles so that the maximum number of features within the camera range is detected while the chaser moves in a predefined orbit around the target. For the chaser-object spatial incentive, rewards are assigned to the chaser states from a combined model with two components: feature detection score and sinusoidal reward. To calculate the sinusoidal reward, estimated feature locations are required, which are predicted by Gaussian Process models. Another Gaussian Process model provides the reward distribution, which is then used by the Bayesian Optimization to determine the camera directional angles. Simulations are conducted in both 2D and 3D domains. The results demonstrate that SOCRAFT can generally detect the maximum number of features within the limited camera range and field of view.

Adaptive machine learning surrogate based multiobjective optimization for scavenging residual saltwater behind subsurface dams

High-fidelity physically based groundwater flow and solute transport models have been limited for seawater intrusion remediation design because of computationally intensive evolutionary algorithms. Data-driven machine learning approaches are promising to substitute expensive-cost groundwater numerical models within optimization due to computing efficiency. However, machine learning surrogates may accumulate error of forecasting and thereby result in infeasible optimal solutions. To achieve desired fidelity level, this study proposes a novel adaptive machine learning surrogate based multiobjective optimization method for coastal aquifer desalination. An adaptive modeling algorithm is newly introduced to iteratively retrain poorly-performed machine learning models and enhance predicting accuracy. The method is demonstrated to seek optimal extraction and injection strategies for scavenging residual saltwater trapped in an upstream aquifer behind a subsurface dam. Two conflicting objectives of minimizing the total extraction-injection rate and maximizing saltwater removal effectiveness are considered. Three machine learning models including artificial neural network (ANN), Gaussian process (GP) and response surface regression model (RSR) are developed to replace a high-fidelity seawater intrusion model for predicting chloride concentration and salinity mass. Non-dominated Sorting Genetic Algorithm II (NSGA-II) is employed to derive Pareto fronts. Pareto optimal solutions obtained from machine learning models are compared against those from the seawater intrusion model. Results indicate that the developed machine learning models do not only have strong predicting capability, but also maintain good quality of Pareto optimal solutions, while achieving substantial computational saving up to 95%. Especially, the adaptively retrained RSR model inhibits error accumulation of forecasting and accelerates correct convergence to the true Pareto front. The proposed method is found superior performance in accurate convergence, widely-spread diversity as well as high computing efficiency than conventional methods.

Bayesian Optimization for Efficient Prediction of Gas Uptake in Nanoporous Materials.

The discovery and optimization of novel nanoporous materials (NPMs) such as Metal-Organic Frameworks (MOFs) and Covalent Organic Frameworks (COFs) are crucial for addressing global challenges like climate change, energy security, and environmental degradation. Traditional experimental approaches for optimizing these materials are time-consuming and resource-intensive. This research paper presents a strategy using Bayesian optimization (BO) to efficiently navigate the complex design spaces of NPMs for gas storage applications. For a MOF dataset drawn from 19 different sources, we present a quantitative evaluation of BO using a curated set of surrogate model and acquisition function couples. In our study, we employed machine learning (ML) techniques to conduct regression analysis on many models. Following this, we identified the three ML models that exhibited the highest accuracy, which were subsequently chosen as surrogates in our investigation, including the conventional Gaussian Process (GP) model. We found that GP with expected improvement (EI) as the acquisition function but without a gamma prior which is standard in Bayesian Optimisation python library (BO Torch) outperforms other surrogate models. Additionally, it should be noted that while the machine learning model that exhibits superior performance in predicting the target variable may be considered the best choice, it may not necessarily serve as the most suitable surrogate model for BO. This observation has significant importance and warrants further investigation. This comprehensive framework accelerates the pace of materials discovery and addresses urgent needs in energy storage and environmental sustainability. It is to be noted that rather than identifying new MOFs, BO primarily enhances computational efficiency by reducing the reliance on more demanding calculations, such as those involved in Grand Canonical Monte Carlo (GCMC) or Density Functional Theory (DFT).

Explainable Gaussian processes: a loss landscape perspective

Prior beliefs about the latent function to shape inductive biases can be incorporated into a Gaussian process (GP) via the kernel. However, beyond kernel choices, the decision-making process of GP models remains poorly understood. In this work, we contribute an analysis of the loss landscape for GP models using methods from chemical physics. We demonstrate ν-continuity for Matérn kernels and outline aspects of catastrophe theory at critical points in the loss landscape. By directly including ν in the hyperparameter optimisation for Matérn kernels, we find that typical values of ν can be far from optimal in terms of performance. We also provide an a priori method for evaluating the effect of GP ensembles and discuss various voting approaches based on physical properties of the loss landscape. The utility of these approaches is demonstrated for various synthetic and real datasets. Our findings provide insight into hyperparameter optimisation for GPs and offer practical guidance for improving their performance and interpretability in a range of applications.

Stochastic machine learning via sigma profiles to build a digital chemical space

This work establishes a different paradigm on digital molecular spaces and their efficient navigation by exploiting sigma profiles. To do so, the remarkable capability of Gaussian processes (GPs), a type of stochastic machine learning model, to correlate and predict physicochemical properties from sigma profiles is demonstrated, outperforming state-of-the-art neural networks previously published. The amount of chemical information encoded in sigma profiles eases the learning burden of machine learning models, permitting the training of GPs on small datasets which, due to their negligible computational cost and ease of implementation, are ideal models to be combined with optimization tools such as gradient search or Bayesian optimization (BO). Gradient search is used to efficiently navigate the sigma profile digital space, quickly converging to local extrema of target physicochemical properties. While this requires the availability of pretrained GP models on existing datasets, such limitations are eliminated with the implementation of BO, which can find global extrema with a limited number of iterations. A remarkable example of this is that of BO toward boiling temperature optimization. Holding no knowledge of chemistry except for the sigma profile and boiling temperature of carbon monoxide (the worst possible initial guess), BO finds the global maximum of the available boiling temperature dataset (over 1,000 molecules encompassing more than 40 families of organic and inorganic compounds) in just 15 iterations (i.e., 15 property measurements), cementing sigma profiles as a powerful digital chemical space for molecular optimization and discovery, particularly when little to no experimental data is initially available.

Spatiotemporal methods for estimating subsurface ocean thermal response to tropical cyclones

Abstract. Tropical cyclones (TCs), driven by heat exchange between the air and sea, pose a substantial risk to many communities around the world. Accurate characterization of the subsurface ocean thermal response to TC passage is crucial for accurate TC intensity forecasts and an understanding of the role that TCs play in the global climate system. However, that characterization is complicated by the high-noise ocean environment, correlations inherent in spatiotemporal data, relative scarcity of in situ observations, and the entanglement of the TC-induced signal with seasonal signals. We present a general methodological framework that addresses these difficulties, integrating existing techniques in seasonal mean field estimation, Gaussian process modeling, and nonparametric regression into an ANOVA decomposition model. Importantly, we improve upon past work by properly handling seasonality, providing rigorous uncertainty quantification, and treating time as a continuous variable, rather than producing estimates that are binned in time. This ANOVA model is estimated using in situ subsurface temperature profiles from the Argo fleet of autonomous floats through a multistep procedure, which (1) characterizes the upper-ocean seasonal shift during the TC season, (2) models the variability in the temperature observations, and (3) fits a thin-plate spline using the variability estimates to account for heteroskedasticity and correlation between the observations. This spline fit reveals the ocean thermal response to the TC passage. Through this framework, we obtain new scientific insights into the interaction between TCs and the ocean on a global scale, including a three-dimensional characterization of the near-surface and subsurface cooling along the TC storm track and the mixing-induced subsurface warming on the track's right side.

Dynamic payment on microtasking platforms using bee colony optimization

Microtasking involves breaking down a job into smaller tasks and assigning them to a group of workers who voluntarily complete the tasks and receive payment. However, the joint estimation of quality and determination of payment remains an underexplored area in microtask crowdsourcing research. This paper addresses the limitation of unestimated payment on microtasking platforms by introducing a dynamic worker quality and payment estimation algorithm known as Dynamic Quality Payment (DQP). Utilizing the Bee Colony Optimization (BCO) algorithm, the proposed approach integrates worker quality estimation and payment determination. DQP incorporates the Gaussian Process Model (GPM) to initially estimate worker quality and then optimize payments based on the quality of work. Empirical testing of the algorithm is conducted using two datasets from Amazon Mechanical Turk. Additionally, the DQP algorithm is compared against novel microtasking and quality estimation algorithms, demonstrating superior performance. The experimental results illustrate that DQP not only reduces the cost of microtasking but also accurately estimates worker quality.

A New Multimodal Map Building Method Using Multiple Object Tracking and Gaussian Process Regression

Recent advancements in simultaneous localization and mapping (SLAM) have significantly improved the handling of dynamic objects. Traditionally, SLAM systems mitigate the impact of dynamic objects by extracting, matching, and tracking features. However, in real-world scenarios, dynamic object information critically influences decision-making processes in autonomous navigation. To address this, we present a novel approach for incorporating dynamic object information into map representations, providing valuable insights for understanding movement context and estimating collision risks. Our method leverages on-site mobile robots and multiple object tracking (MOT) to gather activation levels. We propose a multimodal map framework that integrates occupancy maps obtained through SLAM with Gaussian process (GP) modeling to quantify the activation levels of dynamic objects. The Gaussian process method utilizes a map-based grid cell algorithm that distinguishes regions with varying activation levels while providing confidence measures. To validate the practical effectiveness of our approach, we also propose a method to calculate additional costs from the generated maps for global path planning. This results in path generation through less congested areas, enabling more informative navigation compared to traditional methods. Our approach is validated using a diverse dataset collected from crowded environments such as a library and public square and is demonstrated to be intuitive and to accurately provide activation levels.

Power of prediction: spatiotemporal Gaussian process modeling for predictive control in slope-based wavefront sensing

Power of prediction: spatiotemporal Gaussian process modeling for predictive control in slope-based wavefront sensing

Irregular extended target tracking with unknown measurement noise covariance

To address the challenge of tracking irregular extended target while adapting to unknown measurement noise, this paper presents a novel tracking algorithm for irregular extended target. Considering the flexibility of the Gaussian process (GP) model for representing shape, in the proposed algorithm, we utilize the GP model to delineate the radial function of the tracked extended target’s contour. Then, we incorporate the GP model into the variational Bayesian technique to formulate the posterior distribution for both the unknown measurement noise and the shape state of the tracked target. Subsequently, the square-root cubature Kalman filter is employed to tackle the nonlinearity filtering in the problem of irregular extended target tracking (ETT), enabling simultaneous estimation of intricate shape details and the unknown measurement noise covariance. Through extensive experiments using both simulated and real-time lidar data, the proposed algorithm demonstrates superior performance compared to the existing ETT algorithms for extended target tracking in both accuracy and robustness.

Carbon price forecasting using leaky integrator echo state networks with the framework of decomposition-reconstruction-integration

Carbon trading is one of the strategies to achieve the goal of dual carbon. However, carbon prices exhibit non-stationary and nonlinear characteristics, posing significant challenges for accurate forecasting. Inspired by the ideas of “division and conquest” and “granularity reconstruction”, a decomposition-reconstruction-integration framework is built for carbon price forecasting. The proposed model leverages a comprehensive ensemble empirical mode decomposition with adaptive noise (CEEMDAN) and sample entropy (SE) to extract features from two perspectives of time and complexity, and thus the one-dimensional time series of carbon price is reconstructed into different items. Subsequently, leaky integrator echo state networks (LiESNs) are utilized to forecast each of these items individually. These individual forecasts are then integrated to generate the final point forecast. To evaluate the uncertainty of the point forecast, a Gaussian process model is applied to interval forecast. The proposed model comprehensively considers the time series features of carbon price including the temporal modal features in the decomposition stage, the entropy probability features in the reconstruction stage, and the essential temporal dependencies via dynamic reservoir of interconnected neurons in the integration forecasting stage. Experimental results demonstrate the model's exceptional forecasting performance, surpassing the effectiveness of alternative approaches.

Discovering chiral auxetic structures with near-zero Poisson's ratio using an active learning strategy

In this paper, a set of hexachiral auxetic structural designs with near zero Poisson's ratio (ZPR) characteristics is discovered via the combination of machine learning and experimentally validated finite element simulation. An active learning-enhanced Gaussian process model is utilized to generate multiple designs with near-ZPR properties and discover the boundary of the positive and negative Poisson's ratio. The results show that active learning successfully constructs a probabilistic estimation of the ZPR boundary. A comprehensive analysis of the identified ZPR contour is performed to extract crucial design insights. The findings indicate that the near-ZPR characteristic can be attained through various combinations of geometric parameters. This offers users the flexibility to select the configuration that best aligns with their specific requirements. Additionally, an investigation of the various ZPR configurations that have been discovered is carried out to understand the mechanism that yields near-ZPR property. One discovered near-ZPR design was subsequently fabricated using 3D printing for validation purposes. The experimental outcomes demonstrated a good agreement with the numerical predictions, underscoring the effectiveness of the active learning strategy in uncovering designs that closely approach ZPR conditions.