Gaussian Process Research Articles

Parameter-adaptive variational autoencoder for linear/nonlinear blind source separation

AbstractBlind source separation (BSS) serves as an important technique in the field of structural health monitoring (SHM), particularly for solving modal decomposition tasks. This study proposes a novel approach to both linear and nonlinear BSS problems in the Variational Autoencoder (VAE) framework, where the encoding and decoding processes of VAE are interpreted as procedures for inferring sources from observations and remixing these sources, respectively. In this way, the distribution of latent variables inferred by VAE is equivalent to the distribution of sources. We make improvements to the vanilla VAE to augment its effectiveness for BSS. First, we substitute standard normal distributions with trainable Gaussian processes (GP) as priors for latent variables and implement an exponential function as the activation function for adaptive parameters in the GP kernel functions. While the form of the priors is set as GP, the parameters of their kernel functions are not fixed but automatically converge to suitable values during the model training process. Additionally, a hyperparameter is introduced to balance the terms in the loss function. The proposed method is referred to as parameter-adaptive VAE (PAVAE). Then, upon different assumptions of the variances of sources, the proposed PAVAE is branched into two types: homoscedastic PAVAE (Ho-PAVAE) and heteroscedastic PAVAE (He-PAVAE). Through numerical and laboratory experiments, we demonstrate the effectiveness of the proposed method in solving BSS problems and their potential to underpin future research in SHM.

A robust approach to Gaussian process implementation

Abstract. Gaussian process (GP) regression is a flexible modeling technique used to predict outputs and to capture uncertainty in the predictions. However, the GP regression process becomes computationally intensive when the training spatial dataset has a large number of observations. To address this challenge, we introduce a scalable GP algorithm, termed MuyGPs, which incorporates nearest-neighbor and leave-one-out cross-validation during training. This approach enables the evaluation of large spatial datasets with state-of-the-art accuracy and speed in certain spatial problems. Despite these advantages, conventional quadratic loss functions used in the MuyGPs optimization, such as root mean squared error (RMSE), are highly influenced by outliers. We explore the behavior of MuyGPs in cases involving outlying observations and, subsequently, develop a robust approach to handle and mitigate their impact. Specifically, we introduce a novel leave-one-out loss function based on the pseudo-Huber function (LOOPH) that effectively accounts for outliers in large spatial datasets within the MuyGPs framework. Our simulation study shows that the LOOPH loss method maintains accuracy despite outlying observations, establishing MuyGPs as a powerful tool for mitigating unusual observation impacts in the large data regime. In the analysis of US ozone data, MuyGPs provides accurate predictions and uncertainty quantification, demonstrating its utility in managing data anomalies. Through these efforts, we advance the understanding of GP regression in spatial contexts.

Computation and Applications of Gaussian Integrals in Mathematics and Applied Sciences

Gaussian integrals, particularly those involving the function e−x2 , play a central role in various fields, ranging from physics to finance. This paper explores the computation of Gaussian integrals, beginning with the fundamental integral in the interval −∞ to ∞, which yields π . The author extends this analysis to integrals involving powers of x multiplied by e−x2 , showing their relevance in calculating moments in quantum mechanics and probability theory. Additionally, the author discusses the applications of multivariate Gaussian integrals in machine learning and statistical mechanics, where they are key to solving problems in high-dimensional spaces. Practical examples are provided from quantum mechanics (path integrals), statistical mechanics (partition functions), finance (option pricing models), and machine learning (Gaussian processes). Through these examples, the paper highlights the versatility and universality of Gaussian integrals as essential tools in both theoretical and applied contexts. The integral’s widespread applicability reflects its importance in connecting mathematical theory with real-world phenomena. This work highlights the role played by the Gaussian integral.

Estimation of hydrogen solubility in aqueous solutions using machine learning techniques for hydrogen storage in deep saline aquifers

The porous underground structures have recently attracted researchers’ attention for hydrogen gas storage due to their high storage capacity. One of the challenges in storing hydrogen gas in aqueous solutions is estimating its solubility in water. In this study, after collecting experimental data from previous research and eliminating four outliers, nine machine learning methods were developed to estimate the solubility of hydrogen in water. To optimize the parameters used in model construction, a Bayesian optimization algorithm was employed. By examining error functions and plots, the LSBoost method with R² = 0.9997 and RMSE = 4.18E-03 was identified as the most accurate method. Additionally, artificial neural network, CatBoost, Extra trees, Gaussian process regression, bagged trees, regression trees, support vector machines, and linear regression methods had R² values of 0.9925, 0.9907, 0.9906, 0.9867, 0.9866, 0.9808, 0.9464, and 0.7682 and RMSE values of 2.13E-02, 2.43E-02, 2.44E-02, 2.83E-02, 2.85E-02, 3.40E-02, 5.68E-02, and 1.18E-01, respectively. Subsequently, residual error plots were generated, indicating the accurate performance of the LSBoost model across all ranges. The maximum residual error was − 0.0252, and only 4 data points were estimated with an error greater than ± 0.01. A kernel density estimation (KDE) plot for residual errors showed no specific bias in the models except for the linear regression model. To investigate the impact of temperature, pressure, and salinity parameters on the model outputs, the Pearson correlation coefficients for the LSBoost model were calculated, showing that pressure, temperature, and salinity had values of 0.8188, 0.1008, and − 0.5506, respectively, indicating that pressure had the strongest direct relationship, while salinity had an inverse relationship with hydrogen solubility. Considering the results of this research, the LSBoost method, alongside approaches like state equations, can be applied in real-world scenarios for underground hydrogen storage. The findings of this study can help in a better understanding of hydrogen solubility in aqueous solutions, aiding in the optimization of underground hydrogen storage systems.

Spatial pattern and differential expression analysis with spatial transcriptomic data.

The emergence of spatial transcriptomic technologies has opened new avenues for investigating gene activities while preserving the spatial context of tissues. Utilizing data generated by such technologies, the identification of spatially variable (SV) genes is an essential step in exploring tissue landscapes and biological processes. Particularly in typical experimental designs, such as case-control or longitudinal studies, identifying SV genes between groups is crucial for discovering significant biomarkers or developing targeted therapies for diseases. However, current methods available for analyzing spatial transcriptomic data are still in their infancy, and none of the existing methods are capable of identifying SV genes between groups. To overcome this challenge, we developed SPADE for spatial pattern and differential expression analysis to identify SV genes in spatial transcriptomic data. SPADE is based on a machine learning model of Gaussian process regression with a gene-specific Gaussian kernel, enabling the detection of SV genes both within and between groups. Through benchmarking against existing methods in extensive simulations and real data analyses, we demonstrated the preferred performance of SPADE in detecting SV genes within and between groups. The SPADE source code and documentation are publicly available at https://github.com/thecailab/SPADE.

State of health estimation for lithium-ion batteries with Bayesian optimized Gaussian process regression

The estimation of the state of health (SOH) of lithium-ion batteries is of great importance to ensure the safe and stable operation of a lithium-ion battery management system (BMS). Accurate estimation of SOH can effectively prevent unnecessary losses of lithium-ion batteries due to failures. Therefore, a Bayesian-optimized Gaussian process regression (BO-GPR) method is proposed for SOH estimation. Firstly, the ageing characteristics reflecting the SOH of the battery are extracted from the charge/discharge time and incremental capacity (IC) curve, and the correlation between the health characteristics and the SOH is analysed by means of grey correlation (GRA). On the other hand, a Bayesian optimization algorithm is used to automatically find the optimal hyperparameters of the GPR model and trained using the battery dataset to obtain the SOH estimating model. To validate the effectiveness of the method, the datasets provided by NASA and Oxford are used as experimental objects and compared with different data-driven models to verify the accuracy and reliability of the proposed model. The experimental results show that the method proposed in this paper has a high accuracy in the estimation of the SOH, with the maximum average estimation error being within 1%.

A general Bayesian algorithm for the autonomous alignment of beamlines.

Autonomous methods to align beamlines can decrease the amount of time spent on diagnostics, and also uncover better global optima leading to better beam quality. The alignment of these beamlines is a high-dimensional expensive-to-sample optimization problem involving the simultaneous treatment of many optical elements with correlated and nonlinear dynamics. Bayesian optimization is a strategy of efficient global optimization that has proved successful in similar regimes in a wide variety of beamline alignment applications, though it has typically been implemented for particular beamlines and optimization tasks. In this paper, we present a basic formulation of Bayesian inference and Gaussian process models as they relate to multi-objective Bayesian optimization, as well as the practical challenges presented by beamline alignment. We show that the same general implementation of Bayesian optimization with special consideration for beamline alignment can quickly learn the dynamics of particular beamlines in an online fashion through hyperparameter fitting with no prior information. We present the implementation of a concise software framework for beamline alignment and test it on four different optimization problems for experiments on X-ray beamlines at the National Synchrotron Light Source II and the Advanced Light Source, and an electron beam at the Accelerator Test Facility, along with benchmarking on a simulated digital twin. We discuss new applications of the framework, and the potential for a unified approach to beamline alignment at synchrotron facilities.

Modeling nonstationary surface‐level ozone extremes through the lens of US air quality standards: A Bayesian hierarchical approach

AbstractSurface‐level ozone (O) is a harmful air pollutant whose effects may be more deleterious when at its most extreme levels. Current US air quality standards are written in terms of the 3‐year average of the 4th highest annual daily maximum 8‐h O values; therefore, developing approaches based on extreme value theory may be useful. We develop a Bayesian hierarchical approach, where the ‐largest order statistics are parametrized by the generalized extreme value (GEV) distribution, while a Gaussian process is employed to characterize how the GEV parameters depend on the O precursors, namely nitrous oxides (NO) and volatile organic compounds (VOCs). The fitted model is then used to characterize the upper tail of the distribution of O and estimate O noncompliance probabilities. We illustrate the proposed method using data from an air quality station in Providence, Rhode Island (RI). The results suggest that the far upper tail of extreme O values is likely bounded, and the dependence of the upper tail distribution on NO and O is highly nonlinear, consistent with the known relationship, albeit not specifically for extreme values, in the existing scientific literature. A convolution‐based approach is used to estimate noncompliance probabilities for several covariate scenarios. Our results indicate that estimated noncompliance probabilities in recent years are much lower than in the mid‐1990s, primarily due to lower O precursor levels. However, the estimated noncompliance probabilities appear to rise sharply for hypothetical stricter O standards, even for the conditions observed in recent years.

Selection of optimal spectral features for leaf chlorophyll content estimation

Leaf chlorophyll content (LCC) is crucial for monitoring the physiological processes of crops. Many studies have utilized spectral features to develop regression models for accurate LCC estimation, enabling the quantitative assessment and evaluation of crop growth status. The selection of optimal spectral features and regression algorithms significantly affects the precision of LCC estimation. In this study, we compared and analyzed the optimal spectral features for LCC estimation, as well as the consistency of machine learning methods across different crop types, phenology periods, and sensors. First, we extracted various spectral features, including the original spectral features (OS), first-order derivative spectral features (FDS), original continuum-removed spectra (CR) along with their four related derivative spectral features, principal component variables derived from different spectral features, and highly correlated spectral features with LCC. These extracted spectral features were then employed to construct the LCC models using six common regression algorithms on different datasets. Finally, we analyzed the optimal combination of spectral features and regression algorithms for accurate LCC estimation considering various dimensions, such as crop type, phenological period, and sensor used in observation conditions. The results demonstrate that the combinations of the principal component variables of continuum-removed derivative reflectance with the top 10 correlations with LCC (PCA_CRDR_R) combined with Gaussian process regression (GPR) can be considered as the optimal choice for estimating LCC under diverse observation conditions at a canopy scale, and its R2 can reach 0.62 for sugar beet LCC estimation; thus providing valuable theoretical guidance for selecting appropriate spectral features for LCC estimation.

Assessment of the Choked Flow Model of RELAP5 for Application of Inverse Quantification Methods

In the framework of deterministic safety analysis, best estimate plus uncertainty methodologies can provide a more informative detailed analysis of transients than conservative approaches. However, one important step in the application of such methodologies is derivation of uncertainty of the physical models. Probability density functions of the physical model parameters are obtained by applying inverse uncertainty quantification (IUQ) methods to experimental data. The present work deals with evaluation of the experimental database and assessment of the simulation model, which are required steps prior to the application of IUQ methods. The U.S. Nuclear Regulatory Commission system code RELAP5 has been applied for simulation of three experimental databases on choked/critical flow: Sozzi-Sutherland, Super MobyDick, and Marviken Critical Flow Tests. First, the adequacy of the experimental data to the specific target domain is carried out, to then proceed to calibration of the input model parameters by combining the use of Gaussian Process metamodels and optimization schemes. The assessment of the simulation models is performed by evaluating independently accuracy, precision, and consistency through four statistical indicators, including a novel indicator to evaluate the consistency of the results. The final results indicate that the model is capable of reproducing the selected experimental database and overall average accuracy, precision, and consistency below the 10% threshold and can be used for application of IUQ methods.

Flexible Bayesian modeling for longitudinal binary and ordinal responses

Longitudinal studies with binary or ordinal responses are widely encountered in various disciplines, where the primary focus is on the temporal evolution of the probability of each response category. Traditional approaches build from the generalized mixed effects modeling framework. Even amplified with nonparametric priors placed on the fixed or random effects, such models are restrictive due to the implied assumptions on the marginal expectation and covariance structure of the responses. We tackle the problem from a functional data analysis perspective, treating the observations for each subject as realizations from subject-specific stochastic processes at the measured times. We develop the methodology focusing initially on binary responses, for which we assume the stochastic processes have Binomial marginal distributions. Leveraging the logits representation, we model the discrete space processes through continuous space processes. We utilize a hierarchical framework to model the mean and covariance kernel of the continuous space processes nonparametrically and simultaneously through a Gaussian process prior and an Inverse-Wishart process prior, respectively. The prior structure results in flexible inference for the evolution and correlation of binary responses, while allowing for borrowing of strength across all subjects. The modeling approach can be naturally extended to ordinal responses. Here, the continuation-ratio logits factorization of the multinomial distribution is key for efficient modeling and inference, including a practical way of dealing with unbalanced longitudinal data. The methodology is illustrated with synthetic data examples and an analysis of college students’ mental health status data.

Analyzing and forecasting air pollution concentration in the capital and Southern Thailand using a lag-dependent Gaussian process model.

The air pollution problem has now amassed worldwide attention due to its multifaceted harm to human health. Exploring the concentration of air pollution and improving forecast have important consideration worldwide. In this research, we analyze the air pollution concentration of Southern Thailand and compare it with the central region. Also, we proposed a methodology based on the lag-dependent Gaussian process (LDGP), a Bayesian non-parametric machine learning model, with a stable optimization approach, which is a cluster-based multi-starter technique based on the Nelder-Mead optimizer. This model also provides the confidence band for forecasted values. We also used autoregressive deep neural network (AR-DNN), autoregressive random forest (AR-RF), gradient boosting (GB), and K-nearest neighbors (KNN) models. A comparison of the proposed methodology was performed on the daily air pollution data collected from the southern provinces and also from the capital of Thailand from 1 January 2018 to 31 December 2022. We used well-established performance evaluation measures to compare the performance of the models. To evaluate the bias due to overfit, we performed a tenfold cross-validation for all the pollutants in each region and compared the models to choose the best one. Moreover, we explored the concentration of air pollution in these regions. Results of descriptive analysis revealed that Bangkok had a much higher concentration of air pollution as compared to the southern region. However, the southern region had higher exposure to PM air pollutants as per WHO recommendations and also had higher exposure to O3 and CO levels. The proposed LDGP model outperformed the other machine learning models for forecasting all air pollutants. Hence, it is recommended to be used by experts for further research and studies with different kernel functions. This research is also expected to contribute to local government planning and prevention and worldwide use of the same methodology for the sustainability of public health.

Normalizing Basis Functions: Approximate Stationary Models for Large Spatial Data

ABSTRACTIn geostatistics, traditional spatial models often rely on the Gaussian process (GP) to fit stationary covariances to data. It is well known that this approach becomes computationally infeasible when dealing with large data volumes, necessitating the use of approximate methods. A powerful class of methods approximate the GP as a sum of basis functions with random coefficients. Although this technique offers computational efficiency, it does not inherently guarantee a stationary covariance. To mitigate this issue, the basis functions can be ‘normalized’ to maintain a constant marginal variance, avoiding unwanted artefacts and edge effects. This allows for the fitting of nearly stationary models to large, potentially non‐stationary datasets, providing a rigorous base to extend to more complex problems. Unfortunately, the process of normalizing these basis functions is computationally demanding. To address this, we introduce two fast and accurate algorithms to the normalization step, allowing for efficient prediction on fine grids. The practical value of these algorithms is showcased in the context of a spatial analysis on a large dataset, where significant computational speedups are achieved. While implementation and testing are done specifically within the LatticeKrig framework, these algorithms can be adapted to other basis function methods operating on regular grids.

Advancing Vehicle Trajectory Prediction: A Probabilistic Approach Using Combined Sequential Models

Abstract This paper addresses the critical need to quantify vehicle trajectory uncertainty in autonomous driving under environmental variability. We focus on predicting the posterior distribution of vehicle trajectories over a fixed horizon, given an initial state and a sequence of actions. We propose and compare three approaches: a probabilistic seq2seq model based on stochastic variational Gaussian processes, sequential Monte Carlo simulation with a single-step Gaussian process model, and a hybrid model that leverages the strengths of both methods. Each approach incorporates a baseline vehicle kinematics model to enhance stability and convergence. We evaluate these methods using a dataset generated from the CARLA simulator, assessing both point error metrics and probabilistic prediction metrics. This research introduces novel approaches to quantifying vehicle trajectory uncertainty through various uncertainty quantification techniques, with the goal of improving the safety and reliability of autonomous vehicle control systems.

Wind Farm Power Maximisation via Wake Steering: A Gaussian Process‐Based Yaw‐Dependent Parameter Tuning Approach

ABSTRACTMaximising the power production of wind farms is vital to meet the growing demand for wind energy and reduce its cost. Wake effects, resulting from the aerodynamic interactions between turbines in a wind farm, significantly impact farm efficiency, leading to substantial annual power losses. Wake steering, an influential control strategy, involves mitigating wake effects by strategically yaw misaligning upstream turbines to deflect their wakes. Conventional wake steering approaches typically rely on physics‐based analytical wake models with their parameters often calibrated using higher fidelity data. However, these approaches determine a fixed set of parameters prior to conducting wake steering, neglecting each parameter's dependency on yaw misalignment (i.e. the optimisation variables) exhibited throughout the optimisation process, potentially affecting its accuracy. To address this limitation, this paper introduces a novel data‐driven parameter tuning approach that integrates higher fidelity power measurements using Gaussian processes to continuously adapt parameters in lower fidelity wake models based on the current farm's yaw configuration. The effectiveness of the proposed approach is demonstrated on a wind farm and a layout corresponding to the Horns Rev wind farm, where various wind directions are investigated. The results reveal that the approach can enable a lower fidelity model to capture more complex physics, thereby improving its accuracy in wake steering optimisation, while maintaining robustness and computational efficiency. This method holds promise for real‐time control applications and can be extended to other control strategies and closed‐loop frameworks.

Stacking BSRG-PLS: A physical and data-driven real-time stability safety analysis of arch dams during operation

Anti-sliding stability is the foundation for the normal operation of arch dams. In recent years, extreme weather has occurred frequently. It is important to grasp the anti-slide stability of arch dam (ASSAD) under complex load conditions in time. Currently, the ASSAD safety factor is primarily analyzed through the finite element method (FEM), which are time-consuming, labor-intensive, and lack timeliness. To address this, this paper proposes a real-time ASSAD analysis method during operation based on the Stacking BSRG-PLS model, which integrates a Bidirectional long short-term memory network, Residual neural network, Support vector machine, Gaussian process regression model and Partial Least Squares regression method. Firstly, based on the Stacking BSRG-PLS model and measured temperature combing with FEM, the transient temperature field of arch dam is established. Subsequently, a sample dataset containing the load data and the corresponding ASSAD safety factors calculated by FEM is constructed. Whereafter, using the Stacking BSRG-PLS model again, the real-time ASSAD safety factor is obtained based on the sample dataset. Case studies indicate that this method is effective and feasible, and has high analysis precision. It provides an effective way to quickly evaluate the ASSAD safety based on the measured data.

Decentralized Multi-Agent Search for Moving Targets Using Road Network Gaussian Process Regressions

Unmanned aerial vehicles (UAVs) can collaborate as teams to accomplish diverse mission objectives, such as target search and tracking. This paper introduces a method that leverages accumulated target-density information over the course of a UAV mission to adapt path-planning rewards, guiding UAVs toward areas with a higher likelihood of target presence. The target density is modeled using a Gaussian process, which is iteratively updated as the UAVs search the environment. Unlike conventional search algorithms that prioritize unexplored regions, this approach incentivizes revisiting target-rich areas. The target-density information is shared across UAVs using decentralized consensus filters, enabling cooperative path selection that balances the exploration of uncertain regions with the exploitation of known high-density areas. The framework presented in this paper provides an adaptive cooperative search method that can quickly develop an understanding of the region’s target-dense areas, helping UAVs refine their search. Through Monte Carlo simulations, we demonstrate this method in both a 2D grid region and road networks, showing up to a 26% improvement in target density estimates.

Graph neural processes for molecules: an evaluation on docking scores and strategies to improve generalization

Neural processes (NPs) are models for meta-learning which output uncertainty estimates. So far, most studies of NPs have focused on low-dimensional datasets of highly-correlated tasks. While these homogeneous datasets are useful for benchmarking, they may not be representative of realistic transfer learning. In particular, applications in scientific research may prove especially challenging due to the potential novelty of meta-testing tasks. Molecular property prediction is one such research area that is characterized by sparse datasets of many functions on a shared molecular space. In this paper, we study the application of graph NPs to molecular property prediction with DOCKSTRING, a diverse dataset of docking scores. Graph NPs show competitive performance in few-shot learning tasks relative to supervised learning baselines common in chemoinformatics, as well as alternative techniques for transfer learning and meta-learning. In order to increase meta-generalization to divergent test functions, we propose fine-tuning strategies that adapt the parameters of NPs. We find that adaptation can substantially increase NPs' regression performance while maintaining good calibration of uncertainty estimates. Finally, we present a Bayesian optimization experiment which showcases the potential advantages of NPs over Gaussian processes in iterative screening. Overall, our results suggest that NPs on molecular graphs hold great potential for molecular property prediction in the low-data setting.Scientific contributionNeural processes are a family of meta-learning algorithms which deal with data scarcity by transferring information across tasks and making probabilistic predictions. We evaluate their performance on regression and optimization molecular tasks using docking scores, finding them to outperform classical single-task and transfer-learning models. We examine the issue of generalization to divergent test tasks, which is a general concern of meta-learning algorithms in science, and propose strategies to alleviate it.

Analysis of differential scanning calorimetry data for aged plutonium.

Differential scanning calorimetry data for samples of a 52 year old plutonium alloy with 3.3 at. % Ga that were heated beyond the melting point is analyzed using transition state theory to find activation energies for the δ to ɛ and ɛ to liquid phase transitions. A Bayesian statistical method involving a Gaussian process model is used to find mean values and confidence intervals for the activation energies. The activation energy for the δ to ɛ phase transition increases by 3.3 ± 3.8% per decade, relative to the case when all age related plutonium lattice point defects have been removed through annealing. The corresponding increase in activation energy for the ɛ to liquid transition is shown to be 7.1 ± 1.8% per decade. It is postulated that the change in activation energy with age for both phase transitions is caused, in part, by the accumulation of the same type of lattice point defects associated with the observed increase in elastic bulk modulus over time.

A Gaussian process model for stellar activity in 2-D line profile time-series

A Gaussian process model for stellar activity in 2-D line profile time-series