Deep Gaussian Processes Research Articles

Gradient-enhanced deep Gaussian processes for multifidelity modeling

Multifidelity models integrate data from multiple sources to produce a single approximator for the underlying process. Dense low-fidelity samples are used to reduce interpolation error, while sparse high-fidelity samples are used to compensate for bias or noise in the low-fidelity samples. Deep Gaussian processes (GPs) are attractive for multifidelity modeling as they are non-parametric, robust to overfitting, perform well for small datasets, and, critically, can capture nonlinear and input-dependent relationships between data of different fidelities. Many datasets naturally contain gradient data, most commonly when they are generated by computational models that have adjoint solutions or are built in automatic differentiation frameworks.Principally, this work extends deep GPs to incorporate gradient data. We demonstrate this method on an analytical test problem and two realistic aerospace problems: one focusing on a hypersonic waverider with an inviscid gas dynamics truth model and another focusing on the canonical ONERA M6 wing with a viscous Reynolds-averaged Navier-Stokes truth model.In both examples, the gradient-enhanced deep GP outperforms a gradient-enhanced linear GP model and their non-gradient-enhanced counterparts.

Neural Operator Variational Inference Based on Regularized Stein Discrepancy for Deep Gaussian Processes.

Deep Gaussian process (DGP) models offer a powerful nonparametric approach for Bayesian inference, but exact inference is typically intractable, motivating the use of various approximations. However, existing approaches, such as mean-field Gaussian assumptions, limit the expressiveness and efficacy of DGP models, while stochastic approximation can be computationally expensive. To tackle these challenges, we introduce neural operator variational inference (NOVI) for DGPs. NOVI uses a neural generator to obtain a sampler and minimizes the regularized Stein discrepancy (RSD) between the generated distribution and true posterior in L2 space. We solve the minimax problem using Monte Carlo estimation and subsampling stochastic optimization techniques and demonstrate that the bias introduced by our method can be controlled by multiplying the Fisher divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm. Our experiments on datasets ranging from hundreds to millions demonstrate the effectiveness and the faster convergence rate of the proposed method. We achieve a classification accuracy of 93.56 on the CIFAR10 dataset, outperforming state-of-the-art (SOTA) Gaussian process (GP) methods. We are optimistic that NOVI possesses the potential to enhance the performance of deep Bayesian nonparametric models and could have significant implications for various practical applications.

Capacity Estimation for Lithium-Ion Batteries Based on Impedance Spectral Dynamics and Deep Gaussian Process

Capacity Estimation for Lithium-Ion Batteries Based on Impedance Spectral Dynamics and Deep Gaussian Process

Probabilistic Back Analysis Based on Nadam, Bayesian, and Matrix-Variate Deep Gaussian Process for Rock Tunnels

Probabilistic Back Analysis Based on Nadam, Bayesian, and Matrix-Variate Deep Gaussian Process for Rock Tunnels

A deep learning paradigm for medical imaging data

We present a novel deep learning framework to address several challenges in medical imaging analysis. Especially for functional magnetic resonance imaging (fMRI) data, we develop a statistical sampling algorithm to augment small data for deep learning modeling. Also, the ComBat harmonization is utilized for multi-source batch effect correction. Finally, we implement the prediction uncertainty in deep learning through a dropout approximation of the deep Gaussian process. The proposed approach offers effective statistical tools to solve persistent problems in deep learning modeling for medical imaging data. To assess the performance of the proposed approach and demonstrate its effectiveness for various types of data, we apply it to autism fMRI and Pneumonia imaging datasets.

Abstract 2316: A deep Gaussian process (DGP)-based prediction model for immunecheckpoint inhibitor (ICI) response in gastric cancer (GC)

Abstract Background: Immune checkpoint inhibitors (ICIs) targeting PD-1 or PD-L1 have dramatically improved survival of patients with gastric cancer (GC). However, recent clinical trials, including KEYNOTE-059, KEYNOTE-061, and ATTRACTION-2, show that the objective response rate (ORR) for ICIs is still less than 20% in patients with advanced GC. Therefore, predicting who will respond to ICIs in advance is essential to make better treatment plans for GC patients. There are potential biomarkers to predict ICI response, but most of them can only be applied to a limited population of patients. For example, Kim ST., et al., (PMID: 30013197) show that patients with MSI-high or EBV-positive responded well to ICIs, but these subtypes account for less than 30 percent of GC patients (PMID:25079317). In this study, we propose a prediction model of ICI response using the 32 GC-specific genes that we identified from mutation data (PMID: 26635139) and showed their close association with survival and ICI response in GC (PMID: 35140202; PMID: 36508166). In addition, since the number of available GC-ICI samples is small, we design our prediction model in Bayesian and ensemble learning frameworks which are robust to small sample size problems. Methods: We collected mRNA sequencing data from GC-ICI treated patients from our institutions and publications (PMID: 30013197; PMID: 35254400; PMID: 33846173). We used 108 samples previously included in our previous work (PMID: 36508166) as the training data and 27 samples (13 responders and 14 non-responders) from (PMID: 3384617) as the test data. We built and trained a classifier to separate responders from non-responders using the expressions of the 32 genes. We proposed to use a Deep Gaussian process (GP) model, a multi-layered extension of a GP that is itself a Bayesian method. We optimized the parameters of each GP model in variational inference. To implement ensemble learning, we trained multiple DGP models with different initializations of the model parameters and data splitting (training/validation sets). For each training run, the best model was selected based on the prediction performance on the validation set, measured by AUC. The final prediction for a test sample was made by combining the prediction results of all the DGP models by taking the average. Results: We implemented the DGP-based prediction model using PyTorch in Python, and the codes will be publicly available on the GitHub repository (https://github.com/hwanglab/GC_ICI_response_prediction). We achieved a prediction performance of 0.813 AUC on the test data (sensitivity = 0.769 and specificity = 0.714). Conclusion: We propose the prediction model of ICI response, which can handle small sample size problems in the framework of Bayesian and ensemble learning. The experimental results using the GC-ICI data demonstrate that the 32 genes can be used as predictive markers for ICI response in GC. Citation Format: Sunho Park, Minji Kim, Changjin Hong, Tae Hyun Hwang. A deep Gaussian process (DGP)-based prediction model for immunecheckpoint inhibitor (ICI) response in gastric cancer (GC) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2316.

Abstract 7385: Explainable AI model incorporating uncertainty estimation for enhanced MSI-H prediction and immunotherapy response in gastrointestinal cancer

Abstract Background: Microsatellite instability-high (MSI-H) tumors respond well to immune checkpoint blockade and oftentimes not to chemotherapeutics. Despite value of determining MSI status to guide therapy, many patients’ samples are not untested. Deep learning models may be able to predict MSI status by analyzing hematoxylin and eosin (H&E) stained whole-slide images (WSIs). Validation of this tool will provide another tool to increase MSI testing. However, to be deployed into clinical routine care, prediction models need be externally validated in large and diverse patient cohorts. In addition, these models should provide uncertainty of prediction to help clinicians make informed decisions. Results: In this study, we introduce a novel Bayesian prediction model using whole-slide images (WSIs) and deep Gaussian processes (DGPs) in a weakly supervised learning context. Extensively tested on large datasets (n=3,146) from multiple centers, our model not only demonstrated superior performance but also showed that incorporating uncertainty into predictions significantly improves MSI prediction accuracy, achieving AUROC of >0.91 in colon cancer and >0.94 in gastric cancer. Additionally, an in-depth analysis within the gastric cancer cohort indicated a significant correlation between predicted MSI-H regions and immunotherapy response. Patients who responded to immune checkpoint inhibitors had a higher proportion of MSI-H regions (P-value < 0.05). This correlation was also observed in patients categorized as MSS but responsive to ICIs, highlighting the potential of our model to identify MSI-H patterns not detected through standard laboratory tests. This advancement could lead to more precise patient selection for ICI therapy, enhancing personalized treatment strategies in gastrointestinal cancer. Citation Format: Sunho Park, Minj Kim, Ji-Youn Sung, Jeong Hwan Park, Sung Hak Lee, Sam C. Wang, Jaeho Cheong, Tae Hyun Hwang. Explainable AI model incorporating uncertainty estimation for enhanced MSI-H prediction and immunotherapy response in gastrointestinal cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 7385.

Posterior contraction rates for constrained deep Gaussian processes in density estimation and classification

. We provide posterior contraction rates for constrained deep Gaussian processes in non parametric density estimation and classification. The constraints are in the form of bounds on the values and on the derivatives of the Gaussian processes in the layers of the compositional structure. The contraction rates are first given in a general framework, in terms of a new concentration function that we introduce and that takes the constraints into account. Then, the general framework is applied to integrated Brownian motions, Riemann-Liouville processes, and Matérn processes. In each of these examples, we can recover existing rates, both when the compositional structure is dense and sparse.

Deep Gaussian process for enhanced Bayesian optimization and its application in additive manufacturing

ABSTARCT Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

Personalized Prediction of Parkinson's Disease Progression Based on Deep Gaussian Processes.

Parkinson's disease is a chronic progressive neurodegenerative disease with highly heterogeneous symptoms and progression. It is helpful for patient management to establish a personalized model that integrates heterogeneous interpretation methods to predict disease progression. In the study, we propose a novel approach based on a multi-task learning framework to divide Parkinson's disease progression modeling into an unsupervised clustering task and a disease progression prediction task. On the one hand, the method can cluster patients with different progression trajectories and discover new progression patterns of Parkinson's disease. On the other hand, the discovery of new progression patterns helps to predict the future progression of Parkinson's disease markers more accurately through parameter sharing among multiple tasks. We discovered three different Parkinson's disease progression patterns and achieved better prediction performance (MAE=5.015, RMSE=7.284, r2=0.727) than previously proposed methods on Parkinson's Progression Markers Initiative datasets, which is a longitudinal cohort study with newly diagnosed Parkinson's disease.

End-to-End Optimized Image Compression With Deep Gaussian Process Regression

End-to-end optimization via deep neural networks has facilitated lossy image compression. Existing neural network-based entropy models for end-to-end optimized image compression are limited by parameterized Gaussian distributions with deterministic mean and variance and cannot achieve accurate rate estimation for bottleneck representation with varying statistics. In this paper, we propose a novel entropy model based on deep Gaussian process regression (DGPR) to address this problem. Specifically, the proposed entropy model leverages autoregressive DGPR to flexibly predict the channel-wise posterior distributions of high-dimensional bottleneck representation for entropy coding. Consequently, we develop a well-established bit-rate estimation scheme via posterior inference of DGPR using the learned probabilistic distribution. Furthermore, scalable training is achieved via tensor train decomposition and Monte Carlo sampling to enable tractable variational inference of DGPR. To our best knowledge, this paper is the first attempt to develop the learnable probabilistic model for flexible parameter estimation in entropy modeling. Experimental results show that the proposed model outperforms conventional image compression methods ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., JPEG2000 and BPG) as well as recent end-to-end optimized methods on the Kodak and Tecnick datasets in terms of rate-distortion performance.

A Graphical Multi-Fidelity Gaussian Process Model, with Application to Emulation of Heavy-Ion Collisions

With advances in scientific computing and mathematical modeling, complex scientific phenomena such as galaxy formations and rocket propulsion can now be reliably simulated. Such simulations can however be very time-intensive, requiring millions of CPU hours to perform. One solution is multi-fidelity emulation, which uses data of different fidelities to train an efficient predictive model which emulates the expensive simulator. For complex scientific problems and with careful elicitation from scientists, such multi-fidelity data may often be linked by a directed acyclic graph (DAG) representing its scientific model dependencies. We thus propose a new Graphical Multi-fidelity Gaussian Process (GMGP) model, which embeds this DAG structure (capturing scientific dependencies) within a Gaussian process framework. We show that the GMGP has desirable modeling traits via two Markov properties, and admits a scalable algorithm for recursive computation of the posterior mean and variance along at each depth level of the DAG. We also present a novel experimental design methodology over the DAG given an experimental budget, and propose a nonlinear extension of the GMGP via deep Gaussian processes. The advantages of the GMGP are then demonstrated via a suite of numerical experiments and an application to emulation of heavy-ion collisions, which can be used to study the conditions of matter in the Universe shortly after the Big Bang. The proposed model has broader uses in data fusion applications with graphical structure, which we further discuss.

Vulnerability assessment of heat waves within a risk framework using artificial intelligence

Current efforts to adapt to climate change are not sufficient to reduce projected impacts. Vulnerability assessments are essential to allocate resources where they are needed most. However, current assessments that use principal component analysis suffer from multiple shortcomings and are hard to translate into concrete actions. To address these issues, this article proposes a novel data-driven vulnerability assessment within a risk framework. The framework is based on the definitions from the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, but some definitions, such as sensitivity and adaptive capacity, are clarified. Heat waves that occurred between 2001 and 2018 in Quebec (Canada) are used to validate the framework. The studied impact is the daily mortality rates per cooling degree-days (CDD) region. A vulnerability map is produced to identify the distributions of summer mortality rates in aggregate dissemination areas within each CDD region. Socioeconomic and environmental variables are used to calculate impact and vulnerability. We compared abilities of AutoGluon (an AutoML framework), Gaussian process, and deep Gaussian process to model the impact and vulnerability. We offer advice on how to avoid common pitfalls with artificial intelligence and machine-learning algorithms. Gaussian process is a promising approach for supporting the proposed framework. SHAP values provide an explanation for the model results and are consistent with current knowledge of vulnerability. Recommendations are made to implement the proposed framework quantitatively or qualitatively.

Multi-view Deep Gaussian Processes for Supervised Learning.

Multi-view learning is a widely studied topic in machine learning, which considers learning with multiple views of samples to improve the prediction performance. Even though some approaches have sprung up recently, it is still challenging to jointly explore information contained in different views. Multi-view deep Gaussian processes have shown strong advantages in unsupervised representation learning. However, they are limited when dealing with labeled multi-view data for supervised learning, and ignore the application potential of uncertainty estimation. In this paper, we propose a supervised multi-view deep Gaussian process model (named SupMvDGP), which uses the label of the views to further improve the performance, and takes the quantitative uncertainty estimation as a supplement to assist humans to make better use of prediction. According to the diversity of views, the SupMvDGP can establish asymmetric depth structure to better model different views, so as to make full use of the property of each view. We provide a variational inference method to effectively solve the complex model. Finally, we conduct comprehensive comparative experiments on multiple real world datasets to evaluate the performance of SupMvDGP. The experimental results show that the SupMvDGP achieves the state-of-the-art results in multiple tasks, which verifies the effectiveness and superiority of the proposed approach. Meanwhile, we provide a case study to show that the SupMvDGP has the ability to provide uncertainty estimation than alternative deep models, which can alert people to better treat the prediction results in high-risk applications.

Metamodelling of a two-population spiking neural network.

In computational neuroscience, hypotheses are often formulated as bottom-up mechanistic models of the systems in question, consisting of differential equations that can be numerically integrated forward in time. Candidate models can then be validated by comparison against experimental data. The model outputs of neural network models depend on both neuron parameters, connectivity parameters and other model inputs. Successful model fitting requires sufficient exploration of the model parameter space, which can be computationally demanding. Additionally, identifying degeneracy in the parameters, i.e. different combinations of parameter values that produce similar outputs, is of interest, as they define the subset of parameter values consistent with the data. In this computational study, we apply metamodels to a two-population recurrent spiking network of point-neurons, the so-called Brunel network. Metamodels are data-driven approximations to more complex models with more desirable computational properties, which can be run considerably faster than the original model. Specifically, we apply and compare two different metamodelling techniques, masked autoregressive flows (MAF) and deep Gaussian process regression (DGPR), to estimate the power spectra of two different signals; the population spiking activities and the local field potential. We find that the metamodels are able to accurately model the power spectra in the asynchronous irregular regime, and that the DGPR metamodel provides a more accurate representation of the simulator compared to the MAF metamodel. Using the metamodels, we estimate the posterior probability distributions over parameters given observed simulator outputs separately for both LFP and population spiking activities. We find that these distributions correctly identify parameter combinations that give similar model outputs, and that some parameters are significantly more constrained by observing the LFP than by observing the population spiking activities.

Robust deep Gaussian process-based trustworthy fog-haze-caused pollution flashover prediction approach for overhead contact lines

Due to completely open-air operation, fog-haze has become the main cause of contamination on the insulator surface of overhead contact lines (OCLs), further leading to pollution flashover and a series of serious risk consequences. To perceive the fog-haze-caused pollution flashover (FHPF) risk of OCL insulator, a robust deep Gaussian process (DGP)-based uncertainty-aware trustworthy prediction approach is proposed, incorporating epistemic and aleatoric uncertainties. In particular, aiming at the imbalanced dataset with limited fault samples, the prediction of FHPF risk is cast as a classification problem, and solved by DGP using stochastic gradient Hamiltonian Monte Carlo (SGHMC) inference. The key parameters are identified investigating the influences of fog-haze on insulator surface contamination. Furthermore, the SGHMC sampling-based inference is utilized to efficiently capture the intractable posterior distribution, dealing with uncertainty and enhancing the flexibility of the prediction approach. Finally, extensive experiments on high-speed railway line validate the effectiveness and superior of the proposed approach, compared to other advanced predictive classification methods. In addition, it cannot only capture the prediction uncertainty over a limited number of fault samples, but also achieve favorable prediction performance under unseen noisy environments, ultimately ensuring robust and trustworthy FHPF risk predictions for OCLs.

Deep Learning for Time Series Forecasting: Advances and Open Problems

A time series is a sequence of time-ordered data, and it is generally used to describe how a phenomenon evolves over time. Time series forecasting, estimating future values of time series, allows the implementation of decision-making strategies. Deep learning, the currently leading field of machine learning, applied to time series forecasting can cope with complex and high-dimensional time series that cannot be usually handled by other machine learning techniques. The aim of the work is to provide a review of state-of-the-art deep learning architectures for time series forecasting, underline recent advances and open problems, and also pay attention to benchmark data sets. Moreover, the work presents a clear distinction between deep learning architectures that are suitable for short-term and long-term forecasting. With respect to existing literature, the major advantage of the work consists in describing the most recent architectures for time series forecasting, such as Graph Neural Networks, Deep Gaussian Processes, Generative Adversarial Networks, Diffusion Models, and Transformers.

Image Matting With Deep Gaussian Process.

We observe a common characteristic between the classical propagation-based image matting and the Gaussian process (GP)-based regression. The former produces closer alpha matte values for pixels associated with a higher affinity, while the outputs regressed by the latter are more correlated for more similar inputs. Based on this observation, we reformulate image matting as GP and find that this novel matting-GP formulation results in a set of attractive properties. First, it offers an alternative view on and approach to propagation-based image matting. Second, an application of kernel learning in GP brings in a novel deep matting-GP technique, which is pretty powerful for encapsulating the expressive power of deep architecture on the image relative to its matting. Third, an existing scalable GP technique can be incorporated to further reduce the computational complexity to O(n) from O(n3) of many conventional matting propagation techniques. Our deep matting-GP provides an attractive strategy toward addressing the limit of widespread adoption of deep learning techniques to image matting for which a sufficiently large labeled dataset is lacking. A set of experiments on both synthetically composited images and real-world images show the superiority of the deep matting-GP to not only the classical propagation-based matting techniques but also modern deep learning-based approaches.

Gait Prediction and Variable Admittance Control for Lower Limb Exoskeleton With Measurement Delay and Extended-State-Observer.

The measurement delay of the feedback control system is a universal problem in industrial engineering, which will degrade output performance, especially causing undesirable chatter responses. In this study, a deep-Gaussian-process (DGP)-based method for operator's gait prediction is proposed to estimate the real-time motion intention and to compensate for the measurement delay of the inertial measurement unit (IMU). On the basis of these gait prediction uncertainties quantified by the DGP method, a variable admittance controller is designed to reduce real-time human-exoskeleton interaction torque. The reference trajectory is generated by the admittance controller, which is smoothed by the two-order Bessel interpolation. Meanwhile, the admittance parameters are self-regulated based on the defined uncertainty index of gait prediction. The extend-state observer (ESO) with backstepping iteration is adopted to compensate unmeasured system state, model uncertainties, and unmodeled dynamics of lower limb exoskeleton. The effectiveness of the proposed gait prediction and control scheme is verified by both the comparative simulations and experimental results of the human-exoskeleton cooperative motion.

Uncertainty-aware trustworthy weather-driven failure risk predictor for overhead contact lines

Overhead contact lines (OCLs) are electric transmission lines that power railways, which are constantly threatened by external weather and environmental factors due to their outdoor location. Hence, for the long-term functioning of railway lines, a weather-driven risk predictor is an essential tool. Current prediction methods mainly adopt a single-point estimation system with fixed weights of neural networks and therefore cannot propagate the uncertainties within the data and model, resulting in unreliable predictions. To enhance safety-risk prevention capabilities, this paper proposes an uncertainty-aware trustworthy weather-driven failure-risk approach for OCLs, in a probabilistic deep multitask learning framework. Firstly, a deep Gaussian process is employed as the backbone model to deal with imbalanced weather-related failure datasets with limited fault samples. Moreover, a multi-task learning framework is embedded to simultaneously predict the multiple weather-driven failure risks under lightning, wind and haze. Finally, the predictive uncertainty is decomposed into epistemic and aleatory uncertainties, where epistemic and aleatory uncertainties account for the uncertainty within the model and data, respectively. Extensive experiments on actual OCLs are carried out to demonstrate the effectiveness of the proposed approach, which can effectively capture the predictive uncertainty and provide trustworthy predictive decisions of mitigating operational risk for railway operators.