Sparse Gaussian Process Research Articles

A novel sparse Gaussian process regression with time-aware spatiotemporal kernel for remaining useful life prediction and uncertainty quantification of bearings

A novel sparse Gaussian process regression with time-aware spatiotemporal kernel for remaining useful life prediction and uncertainty quantification of bearings

Optimal Prediction of Individual Vessel Trajectories Based on Sparse Gaussian Processes

Optimal Prediction of Individual Vessel Trajectories Based on Sparse Gaussian Processes

Optimization of Sparse Camera Array Arrangement using Grid-based Method

—This study explores the application of multi-camera array technology in super-resolution (SR) imaging system. We propose an innovative method based on grid representation to optimize camera arrangements using sparse Gaussian processes and variational inference. This approach significantly enhances the image reconstruction quality of sparse camera arrays. Unlike conventional techniques that provide only qualitative conclusions, our grid-based method precisely optimizes camera positions and effectively simulates the impact of assembly errors. The optimized camera arrangement substantially improves the SR reconstruction quality of sparse array imaging systems, reducing contour jaggedness and high-frequency aliasing. Extensive simulations and experimental validations confirm the robustness and practical applicability of our method. The results demonstrate that the optimized camera arrangements can achieve high-resolution imaging with enhanced robustness against assembly errors, indicating their potential for various applications such as biological microscopy, remote sensing, and smartphone photography. This work presents a new and effective strategy for designing sparse camera array imaging systems, offering significant improvements in both image quality and system reliability.

A Kronecker product accelerated efficient sparse Gaussian Process (E-SGP) for flow emulation

In this paper, we introduce an efficient sparse Gaussian process (E-SGP) for the surrogate modelling of fluid mechanics. This novel Bayesian machine learning algorithm enables efficient model training using databases of different structures. It extends the approximated sparse GP algorithm by combining the concept of efficient GP (E-GP) and variational energy free sparse Gaussian processes (VEF-SGP). The developed E-SGP approach leverages the arbitrariness of inducing points and the monotonically increasing nature of the objective function with respect to the number of inducing points in VEF-SGP. By specifying the inducing points on the orthogonal grid/input subspace and using the Kronecker product, E-SGP significantly enhances computational efficiency without imposing constraints on the covariance matrix or increasing the number of parameters that need to be optimised during training.The E-SGP algorithm outperforms E-GP not only in terms of scalability but also in model quality, as evidenced by mean standardised logarithmic loss (MSLL). The computational complexity of E-GP grows cubically with an increasing structured training database, while E-SGP maintains computational efficiency as the model resolution (i.e., the number of inducing points) remains the same. The examples show that E-SGP produces more accurate predictions compared to E-GP when model resolutions are similar. In the application to a partially structured database, E-GP benefits from more training data but comes with higher computational demands, while E-SGP achieves a comparable level of accuracy but is more computationally efficient, making E-SGP a potentially preferable choice for fluid mechanics. Furthermore, E-SGP can produce more reasonable estimates of model uncertainty, whilst E-GP is more likely to produce over-confident predictions. In the application to partially structured databases, the performance of E-SGP is also compared to the structured Gaussian processes latent variable model (SGPLVM). In this case, E-SGP and SGPLVM both exhibit robust performance. However, E-SGP demonstrates a slight advantage over SGPLVM, particularly in terms of MSLL, indicating that E-SGP is better at producing trustworthy and accurate estimates of the uncertainty in the predictions. . While SGPLVM also improves computational efficiency by exploiting the Kronecker product and a small number of inducing points, E-SGP's orthogonal grid specification of inducing points further enhances its scalability and robustness, making it a superior choice for complex fluid mechanics problems with partially structured data.

An efficient method of predicting S-wave velocity using sparse Gaussian process regression for a tight sandstone reservoir

The shear wave (S-wave) velocity plays a crucial role in interpreting the lithology in seismic data, identifying fluids and predicting reservoirs. However, S-wave velocity is often unavailable due to the high cost of measurement and technical constraints. Conventional methods exhibit limitations that potentially impact the accuracy or efficiency on predicting S-wave velocity. Moreover, these methods always ignore the uncertainty quantification associated with the predicted results. This paper proposes a sparse Gaussian process regression (SGPR) method to predict the S-wave velocity in tight sandstone reservoirs. SGPR is a highly efficient regression technique that is based on the Gaussian process regression (GPR) method. In the SGPR method, inducing inputs are introduced to approximate the kernel matrix to decrease the computational complexity. A sparse set of inducing inputs and kernel hyperparameters are optimized through minimizing the Kullback-Leibler (KL) divergence between the exact posterior distribution and the approximate one. In this study, we select several types of logging data, which include porosity, water saturation, shale content, lithology and P-wave velocity, as the inputs for the SGPR method to predict S-wave velocity. To validate its effectiveness, we use the SGPR method to predict S-wave velocity in tight sandstone and compare the results with those from the GPR method, the bidirectional long short-term memory (BiLSTM) method and the Xu-White model. Additionally, we conduct cross-validation to demonstrate the robustness of the SGPR method. Our findings indicate that the SGPR method presents better performance and significant advantages about the accuracy and efficiency. Moreover, the SGPR method offers uncertainty quantification for the predicted S-wave velocity.

A Bayesian optimization approach for data‐driven mixed‐integer nonlinear programming problems

AbstractThe optimization of process systems within the field of Chemical Engineering often confronts uncertainties that can exert significant influences on the performance and dependability of the obtained solutions. This research endeavors to investigate the application of Bayesian optimization in the realm of constrained mixed integer nonlinear problems. A comparative analysis was conducted, exploring different surrogate models, and evaluating diverse kernel functions and acquisition functions. Furthermore, a sampling strategy was devised to assess the enhancement achieved by the acquisition function. The findings of this article reveal the superior performance of sparse Gaussian processes in conjunction with computationally inexpensive acquisition functions, thereby highlighting their suitability for addressing mixed integer nonlinear programming problems characterized by noisy functions and stochastic behavior. Consequently, this article presents a computationally efficient approach to effectively tackle the challenges associated with data‐driven mixed integer nonlinear programming problems within the domain of Process System Engineering.

Complexity of many-body interactions in transition metals via machine-learned force fields from the TM23 data set

This work examines challenges associated with the accuracy of machine-learned force fields (MLFFs) for bulk solid and liquid phases of d-block elements. In exhaustive detail, we contrast the performance of force, energy, and stress predictions across the transition metals for two leading MLFF models: a kernel-based atomic cluster expansion method implemented using sparse Gaussian processes (FLARE), and an equivariant message-passing neural network (NequIP). Early transition metals present higher relative errors and are more difficult to learn relative to late platinum- and coinage-group elements, and this trend persists across model architectures. Trends in complexity of interatomic interactions for different metals are revealed via comparison of the performance of representations with different many-body order and angular resolution. Using arguments based on perturbation theory on the occupied and unoccupied d states near the Fermi level, we determine that the large, sharp d density of states both above and below the Fermi level in early transition metals leads to a more complex, harder-to-learn potential energy surface for these metals. Increasing the fictitious electronic temperature (smearing) modifies the angular sensitivity of forces and makes the early transition metal forces easier to learn. This work illustrates challenges in capturing intricate properties of metallic bonding with current leading MLFFs and provides a reference data set for transition metals, aimed at benchmarking the accuracy and improving the development of emerging machine-learned approximations.

Model Predictive Control with Gaussian-Process-Enhanced Dynamics for Spacecraft Attitude Takeover Maneuvers

This paper aims to deal with the attitude takeover control for combined spacecraft with unknown dynamics and attitude maneuverability of the target. Generally, the dynamics of combined spacecraft executing an on-orbit servicing task is highly nonlinear and costly to identify. To address this issue, a sparse variational Gaussian process (GP)-based model predictive control (MPC) strategy is proposed to achieve efficient attitude takeover maneuvers under multiple constraints and target attitude maneuverability. The GP regression performs a model completion behavior, where the unknown dynamics is compensated by the sparse variational GP with a low computational load. This enhances the control performance of the combined spacecraft during the takeover task execution. Numerical simulations are used to demonstrate the effectiveness of the proposed strategy.

The capacity degradation path prediction for the prismatic lithium-ion batteries based on the multi-features extraction with SGPR

Accurate capacity degradation path estimation of lithium-ion batteries plays a crucial role in ensuring the safety and reliability of electric vehicles. In recent years, the evolution of the status of health (SOH) prediction with machine learning techniques has become a research hotspot because of its powerful computing power and robustness. Thus, this study employes the sparse gaussian process regression (SGPR) data-driven approach with multi-features extracted from the battery cycling processes to project the potential degradation patterns of the lithium-ion batteries. Firstly, a battery life test platform was built. The accelerated life aging tests of batteries at different temperatures (25 °C, or 60 °C) and different discharge rates (1C, or 2C) were conducted to establish the dataset for the multi-features extraction and training of the SGPR algorithm. Secondly, different battery characteristic features were extracted based on the battery cycle charge-discharge curves. Various order-reduction treatments, i.e., the filter-based, embedding-based, and fusion-based selection algorithms, were performed to suppress the over-fitting and improve the estimate's accuracy. Finally, using the extracted features as inputs, SGPR models are constructed to estimate the degradation path of the battery. With the 50 % training data, the SGPR has a higher estimation accuracy than the regular GPR, and the average maximum absolute errors for the batteries are 2.80 %, 1.22 %, 3.57 %, and 1.83 %, respectively.

Modelling and estimating trajectory points from RTK-GNSS based on an integrated modelling approach

The sparse Gaussian process regression (GPR) has been used to model trajectory data from Real time kinematics-global navigation satellite system (RTK-GNSS). However, upon scrutinizing the model residuals; the sparse GPR model poorly fits the data and exhibits presence of correlated noise. This work attempts to address these issues by proposing an integrated modeling approach called GPR-LR-ARIMA where the sparse GPR was integrated with the linear regression with autoregressive integrated moving average errors (LR-ARIMA) to further enhance the description of the trajectory data. In this integrated approach, the predicted trajectory points from the GPR were further described by the LR-ARIMA. Simulation of the GPR-LR-ARIMA on three sets of trajectory data indicated better model fit, revealed in the normally distributed model residuals and symmetrically distributed scatter plots. Correlated noise was also successfully eliminated by the model. The GPR-LR-ARIMA outperformed both the GPR and LRARIMA by its ability to improve mean-absolute-error in 2-dimension positioning by up to 86%. The GPR-LR-ARIMA contributes to enhancement of positioning accuracy of dynamic GNSS measurements in localization and navigation system with good model fit.

A Python Toolbox for Data-Driven Aerodynamic Modeling Using Sparse Gaussian Processes

In aerodynamics, characterizing the aerodynamic behavior of aircraft typically requires a large number of observation data points. Real experiments can generate thousands of data points with suitable accuracy, but they are time-consuming and resource-intensive. Consequently, conducting real experiments at new input configurations might be impractical. To address this challenge, data-driven surrogate models have emerged as a cost-effective and time-efficient alternative. They provide simplified mathematical representations that approximate the output of interest. Models based on Gaussian Processes (GPs) have gained popularity in aerodynamics due to their ability to provide accurate predictions and quantify uncertainty while maintaining tractable execution times. To handle large datasets, sparse approximations of GPs have been further investigated to reduce the computational complexity of exact inference. In this paper, we revisit and adapt two classic sparse methods for GPs to address the specific requirements frequently encountered in aerodynamic applications. We compare different strategies for choosing the inducing inputs, which significantly impact the complexity reduction. We formally integrate our implementations into the open-source Python toolbox SMT, enabling the use of sparse methods across the GP regression pipeline. We demonstrate the performance of our Sparse GP (SGP) developments in a comprehensive 1D analytic example as well as in a real wind tunnel application with thousands of training data points.

Sample-Constrained Black Box Optimization for Audio Personalization

We consider the problem of personalizing audio to maximize user experience. Briefly, we aim to find a filter h*, which applied to any music or speech, will maximize the user’s satisfaction. This is a black-box optimization problem since the user’s satisfaction function is unknown. Substantive work has been done on this topic where the key idea is to play audio samples to the user, each shaped by a different filter hi, and query the user for their satisfaction scores f(hi). A family of “surrogate” functions is then designed to fit these scores and the optimization method gradually refines these functions to arrive at the filter ˆh* that maximizes satisfaction. In certain applications, we observe that a second type of querying is possible where users can tell us the individual elements h*[j] of the optimal filter h*. Consider an analogy from cooking where the goal is to cook a recipe that maximizes user satisfaction. A user can be asked to score various cooked recipes (e.g., tofu fried rice) or to score individual ingredients (say, salt, sugar, rice, chicken, etc.). Given a budget of B queries, where a query can be of either type, our goal is to find the recipe that will maximize this user’s satisfaction. Our proposal builds on Sparse Gaussian Process Regression (GPR) and shows how a hybrid approach can outperform any one type of querying. Our results are validated through simulations and real world experiments, where volunteers gave feedback on music/speech audio and were able to achieve high satisfaction levels. We believe this idea of hybrid querying opens new problems in black-box optimization and solutions can benefit other applications beyond audio personalization.

Secure Distributed Sparse Gaussian Process Models Using Multi-Key Homomorphic Encryption

Distributed sparse Gaussian process (dGP) models provide an ability to achieve accurate predictive performance using data from multiple devices in a time efficient and scalable manner. The distributed computation of model, however, risks exposure of privately owned data to public manipulation. In this paper we propose a secure solution for dGP regression models using multi-key homomorphic encryption. Experimental results show that with a little sacrifice in terms of time complexity, we achieve a secure dGP model without deteriorating the predictive performance compared to traditional non-secure dGP models. We also present a practical implementation of the proposed model using several Nvidia Jetson Nano Developer Kit modules to simulate a real-world scenario. Thus, secure dGP model plugs the data security issues of dGP and provide a secure and trustworthy solution for multiple devices to use privately owned data for model computation in a distributed environment availing speed, scalability and robustness of dGP.

Sparse Variational Student-t Processes

The theory of Bayesian learning incorporates the use of Student-t Processes to model heavy-tailed distributions and datasets with outliers. However, despite Student-t Processes having a similar computational complexity as Gaussian Processes, there has been limited emphasis on the sparse representation of this model. This is mainly due to the increased difficulty in modeling and computation compared to previous sparse Gaussian Processes. Our motivation is to address the need for a sparse representation framework that reduces computational complexity, allowing Student-t Processes to be more flexible for real-world datasets. To achieve this, we leverage the conditional distribution of Student-t Processes to introduce sparse inducing points. Bayesian methods and variational inference are then utilized to derive a well-defined lower bound, facilitating more efficient optimization of our model through stochastic gradient descent. We propose two methods for computing the variational lower bound, one utilizing Monte Carlo sampling and the other employing Jensen's inequality to compute the KL regularization term in the loss function. We propose adopting these approaches as viable alternatives to Gaussian processes when the data might contain outliers or exhibit heavy-tailed behavior, and we provide specific recommendations for their applicability. We evaluate the two proposed approaches on various synthetic and real-world datasets from UCI and Kaggle, demonstrating their effectiveness compared to baseline methods in terms of computational complexity and accuracy, as well as their robustness to outliers.

Predictions of Boron Phase Stability Using an Efficient Bayesian Machine Learning Interatomic Potential.

Thermodynamic phase stability of three elemental boron allotropes, i.e., α-B, β-B, and γ-B, was investigated using a Bayesian interatomic potential trained via a sparse Gaussian process (SGP). SGP potentials trained with data sets from on-the-fly active learning achieve quantum mechanical level accuracy when employed in molecular dynamics (MD) simulations to predict wide-ranging thermodynamic, structural, and vibrational properties. The simulated phase diagram (500-1400 K and 0-16 GPa) agrees with experimental measurements. The SGP-based MD simulations also successfully predicted that the B13 defect is critical in stabilizing β-B below 700 K. At higher temperatures, the entropy becomes the dominant factor, making β-B the more stable phase over α-B. This letter demonstrates that SGP potentials based on a training set consisting of defect-free-only systems could make correct predictions of defect-related phenomena in solid-state crystals, paving the path to investigate crystal phase stability and transitions.

Residual dynamics learning for trajectory tracking for multi-rotor aerial vehicles

This paper presents a technique to model the residual dynamics between a high-level planner and a low-level controller by considering reference trajectory tracking in a cluttered environment as an example scenario. We focus on minimising residual dynamics that arise due to only the kinematical modelling of high-level planning. The kinematical modelling is sufficient for such scenarios due to safety constraints and aggressive manoeuvres that are difficult to perform when the environment is cluttered and dynamic. We used a simplified motion model to represent the motion of the quadrotor when formulating the high-level planner. The Sparse Gaussian Process Regression-based technique is proposed to model the residual dynamics. Recently proposed Data-Driven MPC is targeting aggressive manoeuvres without considering obstacle constraints. The proposed technique is compared with Data-Driven MPC to estimate the residual dynamics error without considering obstacle constraints. The comparison results yield that the proposed technique helps to reduce the nominal model error by a factor of 2 on average. Further, the proposed complete framework was compared with four other trajectory-tracking approaches in terms of tracking the reference trajectory without colliding with obstacles. The proposed approach outperformed the others with less flight time without losing computational efficiency.

A unifying view for the mixture model of sparse Gaussian processes

The mixture of Gaussian processes (MGP) is generally accurate but slow in multimodal probabilistic regression. To reduce the computational burden of MGP, a mixture of sparse Gaussian processes (MSGP) with a fully independent training conditional (FITC) approximation was designed. However, MSGP with FITC is not always accurate. In this paper, we propose MSGP with a variational inference (VI) approximation to overcome this problem and establish a unifying view of MSGPs with FITC and VI based on their marginal likelihoods to explore the relationship between two MSGPs. Experiments on synthetic and real-world datasets show that MSGP with VI is more favorable than MSGP with FITC in certain behaviors, such as lower heteroscedasticity of noise, less clumping behavior of inducing inputs, more effective recovery of the full MGP and higher probabilistic regression accuracy at similar efficiency.

Real-time Learning-based Nonlinear Model Predictive Control of a virtual motorcycle employing grey-box modeling through Gaussian processes

Virtual prototyping tools simulations are nowadays largely adopted methods in the automotive industry. However, to achieve effective simulative tests for two-wheeled vehicles, a virtual rider, i.e. a controller for virtual motorcycles, is typically needed due to the inherent instability of the system. Different control strategies have been employed to deal with this control task, and promising results have been obtained using Nonlinear Model Predictive Control (NMPC). Yet, performance of NMPC highly relies on the plant characterization, that can be highly complex to obtain analytically for 2-wheel vehicles. To improve it, learning dynamics approaches can be used within a Learning-based Nonlinear Model Predictive Control (LbNMPC) framework, exploiting the combination of data-driven techniques and NMPC strategy. In this manuscript, we present a tailored real-time capable LbNMPC for a virtual motorcycle that relies on a continuous grey-box model based on Gaussian Processes (GPs), method that has proven to be effective for 4-wheel vehicle applications. To cope with the real-time requirement, a feature selection procedure and sparse GP approximations have been adopted. A comparison with a physics-based model NMPC implementation taken from the literature has been carried out, analyzing the impact of different sparse GP representations. Remarkable improvements in both model accuracy and tracking performance have been obtained, and the generalization properties of the grey-box model have been assessed.

Fast deep mixtures of Gaussian process experts

AbstractMixtures of experts have become an indispensable tool for flexible modelling in a supervised learning context, allowing not only the mean function but the entire density of the output to change with the inputs. Sparse Gaussian processes (GP) have shown promise as a leading candidate for the experts in such models, and in this article, we propose to design the gating network for selecting the experts from such mixtures of sparse GPs using a deep neural network (DNN). Furthermore, a fast one pass algorithm called Cluster–Classify–Regress (CCR) is leveraged to approximate the maximum a posteriori (MAP) estimator extremely quickly. This powerful combination of model and algorithm together delivers a novel method which is flexible, robust, and extremely efficient. In particular, the method is able to outperform competing methods in terms of accuracy and uncertainty quantification. The cost is competitive on low-dimensional and small data sets, but is significantly lower for higher-dimensional and big data sets. Iteratively maximizing the distribution of experts given allocations and allocations given experts does not provide significant improvement, which indicates that the algorithm achieves a good approximation to the local MAP estimator very fast. This insight can be useful also in the context of other mixture of experts models.

Active sparse Bayesian committee machine potential for isothermal-isobaric molecular dynamics simulations.

Recent advancements in machine learning potentials (MLPs) have significantly impacted the fields of chemistry, physics, and biology by enabling large-scale first-principles simulations. Among different machine learning approaches, kernel-based MLPs distinguish themselves through their ability to handle small datasets, quantify uncertainties, and minimize over-fitting. Nevertheless, their extensive computational requirements present considerable challenges. To alleviate these, sparsification methods have been developed, aiming to reduce computational scaling without compromising accuracy. In the context of isothermal and isobaric ML molecular dynamics (MD) simulations, achieving precise pressure estimation is crucial for reproducing reliable system behavior under constant pressure. Despite progress, sparse kernel MLPs struggle with precise pressure prediction. Here, we introduce a virial kernel function that significantly enhances the pressure estimation accuracy of MLPs. Additionally, we propose the active sparse Bayesian committee machine (BCM) potential, an on-the-fly MLP architecture that aggregates local sparse Gaussian process regression (SGPR) MLPs. The sparse BCM potential overcomes the steep computational scaling with the kernel size, and a predefined restriction on the size of kernel allows for fast and efficient on-the-fly training. Our advancements facilitate accurate and computationally efficient machine learning-enhanced MD (MLMD) simulations across diverse systems, including ice-liquid coexisting phases, Li10Ge(PS6)2 lithium solid electrolyte, and high-pressure liquid boron nitride.