Model Uncertainty Quantification Research Articles

Uncertainty quantification for a deep learning models for image-based crack segmentation

Uncertainty quantification for a deep learning models for image-based crack segmentation

Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications

Abstract Robust quantification of predictive uncertainty is a critical addition needed for machine learning applied to weather and climate problems to improve understanding of what is driving prediction sensitivity. Ensembles of machine learning models provide predictive uncertainty estimates in a conceptually simple way but require multiple models for training and prediction, increasing computational cost and latency. Parametric deep learning can estimate uncertainty with one model by predicting the parameters of a probability distribution but does not account for epistemic uncertainty. Evidential deep learning, a technique that extends parametric deep learning to higher-order distributions, can account for both aleatoric and epistemic uncertainty with one model. This study compares the uncertainty derived from evidential neural networks to that obtained from ensembles. Through applications of classification of winter precipitation type and regression of surface layer fluxes, we show evidential deep learning models attaining predictive accuracy rivaling standard methods, while robustly quantifying both sources of uncertainty. We evaluate the uncertainty in terms of how well the predictions are calibrated and how well the uncertainty correlates with prediction error. Analyses of uncertainty in the context of the inputs reveal sensitivities to underlying meteorological processes, facilitating interpretation of the models. The conceptual simplicity, interpretability, and computational efficiency of evidential neural networks make them highly extensible, offering a promising approach for reliable and practical uncertainty quantification in Earth system science modeling. In order to encourage broader adoption of evidential deep learning, we have developed a new Python package, MILES-GUESS (https://github.com/ai2es/miles-guess), that enables users to train and evaluate both evidential and ensemble deep learning.

Uncertainty Quantification in the Prediction of Remaining Useful Life Considering Multiple Failure Modes

Abstract Despite the substantive literature on remaining useful life (RUL) prediction, less attention is paid to the influence of epistemic uncertainty and aleatory uncertainty in multiple failure behaviors in the accuracy of RUL. The research question in this study was: can uncertainties be quantified in predicting the RUL of systems with multiple failure modes? The first objective was to quantify the uncertainties in the prediction of RUL, considering known multiple failure modes. This objective used vibration data from accelerated degradation experiments of rolling element bearings. The second objective was to calculate the uncertainties in the prediction of RUL, considering the multiple failure modes as unknown. The experimental data used in this objective were from run-to-failure tests of Li-ion batteries. An analysis was performed on how the uncertainties affect the RUL prediction in systems with known multiple failure modes and systems where the multiple failure modes were unknown. A Bayesian neural network (BNN) was used to quantify epistemic and aleatory uncertainty while predicting RUL. The results of the qualitative uncertainties on RUL in systems with multiple failure modes were presented and discussed. Also, the study yielded an RUL uncertainty quantification model for multiple failure modes. The proposed framework's performance in the RUL prediction was demonstrated. Finally, the epistemic and aleatory uncertainties were quantified in the system's RUL. It was shown that systems that fail due to the same failure mode tend to have similar uncertainty values over time. The results in this paper may lead to the design of more reliable systems that exhibit multiple failure modes.

Probabilistic geophysical inversion of complex resistivity measurements using the Hamiltonian Monte Carlo method

Summary In this work, we introduce the probabilistic inversion of tomographic complex resistivity (CR) measurements using the Hamiltonian Monte Carlo (HMC) method. The posterior model distribution on which our approach operates accounts for the underlying complex-valued nature of the CR imaging problem accurately by including the individual errors of the measured impedance magnitude and phase, allowing for the application of independent regularization on the inferred subsurface conductivity magnitude and phase, and incorporating the effects of cross-sensitivities. As the tomographic CR inverse problem is non-linear, of high dimension and features strong correlations between model parameters, efficiently sampling from the posterior model distribution is challenging. To meet this challenge we use HMC, a Markov-chain Monte Carlo method that incorporates gradient information to achieve efficient model updates. To maximize the benefit of a given number of forward calculations, we use the No-U-Turn sampler (NUTS) as a variant of HMC. We demonstrate the probabilistic inversion approach on a synthetic CR tomography measurement. The NUTS succeeds in creating a sample of the posterior model distribution that provides us with the ability to analyze correlations between model parameters and to calculate statistical estimators of interest, such as the mean model and the covariance matrix. Our results provide a strong basis for the characterization of the posterior model distribution and uncertainty quantification in the context of the tomographic CR inverse problem.

Model uncertainty quantification of a degradation model of miter gates using normalizing flow-based likelihood-free inference

Degradation modeling is essential for failure prognostics of miter gates and subsequent risk-informed maintenance or operational decision-making. Typically, degradation models are formulated based on certain assumptions and simplifications and may not fully capture the complexity of underlying degradation mechanisms. Even minor errors in such models may lead to significant discrepancies in failure prognostics due to error accumulation over time. Aiming to improve the accuracy of the degradation model and ensure reliable failure prognostics, this article proposes a degradation model updating framework for miter gates using normalizing flow-based likelihood-free inference, accounting for both model parameter uncertainty and model form uncertainty (i.e., model bias). The proposed work consists of two phases: the offline training phase and the online updating phase. During the offline training phase, the unknown model bias term is modeled as a random noise with uncertain standard deviation. Synthetic data are then generated using a physical model based on prior knowledge of the uncertain model parameters and model bias. The synthetic data are utilized to train an inference model, which has a summary network for data compression and a conditional invertible neural network for parameter estimation. In the online updating phase, the trained inference model uses strain observations to obtain posterior samples. The Gaussian mixture model and dual particle filter techniques are utilized to recursively update posterior samples, thereby improving the estimation accuracy. Subsequently, model bias is inversely estimated using the degradation model, which uses the updated model parameters and the gap length from the inference model. Following that, a regression model is constructed to correct the biases inherent in the simplified degradation model. This process is implemented iteratively over time to perform continuous model updating and failure prognostics. Results of a case study demonstrate the efficacy of the proposed framework in reducing both model parameter uncertainty and recovering model biases and thereby improving the accuracy of failure prognostics of miter gates.

Uncertainty quantification driven machine learning for improving model accuracy in imbalanced regression tasks

Several factors are known to determine the quality of machine learning models, one of which is the dataset quality. One problem related to the quality of a dataset is the imbalance issue. An imbalanced dataset contains significantly more data points for certain values of the output variable which increases the overfitting risk and negatively affects the prediction accuracy. In this article, we propose using epistemic uncertainty quantification (UQ) of machine learning models to identify rare samples in imbalanced regression problems for balancing the dataset. The developed algorithm, uncertainty quantification-driven imbalanced regression (UQDIR), is guided by UQ to restructure the training set with an adequate weight function using existent samples, eliminating the need for new data collection. After identifying rare samples with UQ, the algorithm selects a sample from the training set, assigns a resampling weight using the new weight function, and finally resamples the selected sample according to its assigned weight. We test UQDIR on several benchmark datasets and different machine learning algorithms, then compare its performance with similar imbalanced regression methods. A metamaterial design problem application is also provided for demonstrating the effectiveness of the algorithm in real-world scenarios. We show that improving the quality of UQ metrics results in improved model accuracy.

Confidence-based design optimization using multivariate kernel density estimation under insufficient input data

The uncertainty quantification of the input statistical model in reliability-based design optimization (RBDO) has been widely investigated for accurate reliability analysis, and it could be estimated through its characteristics, cumulative experiences, and available data. However, uncertainty quantification of random variables in existing RBDO studies has exploited parametric distributions quantifying the uncertainty through the Bayes' theorem. In addition, a correlation between random variables is often underestimated due to a lack of knowledge and difficulty to describe the high-dimensional correlation. Hence, it has been a challenge to properly quantify input statistical model and its uncertainty. Therefore, a multivariate kernel density estimation (KDE) is employed to perform data-driven confidence-based design optimization (CBDO) for effective quantification of input model uncertainty. Any assumption on input distribution is not necessary since it is established only with the given input data. Moreover, the input model uncertainty due to insufficient data is quantified using bootstrapping and optimal adaptive bandwidth matrices through the Bayes’ theorem using cross-validation error. Consequently, the proposed CBDO with given input data is capable of finding a conservative optimum of RBDO accounting for both aleatory uncertainty of random variables and epistemic uncertainty induced by a limited number of input data through the multivariate KDE.

Hierarchical deep compartment modeling: A workflow to leverage machine learning and Bayesian inference for hierarchical pharmacometric modeling.

Population pharmacokinetic (PK) modeling serves as the cornerstone for understanding drug behavior within a specific population. It integrates subject covariates to elucidate the variability in PK parameters, thus enhancing predictive accuracy. However, covariate modeling within this framework can be intricate and time-consuming due to the often obscure structural relationship between covariates and PK parameters. Previous attempts, such as deep compartment modeling (DCM), aimed to streamline this process using machine learning techniques. Nonetheless, DCM fell short in assessing residual errors and interindividual variability (IIV), potentially leading to model misspecification and overfitting. Furthermore, DCM lacked the ability to quantify model uncertainty. To address these limitations, we introduce hierarchical deep compartment modeling (HDCM) as an advancement of DCM. HDCM harnesses machine learning to discern the interplay between covariates and PK parameters while simultaneously evaluating diverse levels of random effects and quantifying uncertainty through Bayesian inference. This tutorial provides a comprehensive application of the HDCM workflow using open-source Julia tools.

A multi-scale uncertainty quantification model encompassing minimum-size unit cells for effective properties of plain woven composites

The uncertainty quantification is crucial to the high-precision prediction of composites’ effective properties. However, the unclear input uncertainties of multiscale parameters, complex uncertainty propagations of inter-scale correlations, and unaffordable computational cost of massive simulations are three primary problems at present. In this work, an innovative model with high accuracy and feasible cost for the mechanical property prediction of plain woven composites is developed encompassing minimum-size unit cells and multi-scale uncertainty quantification. For the accuracy holding, an uncertainty analysis process consists of the traceability description, inter-scale propagation and quantification is established. The uncertainties of geometry are described by uniform distributions for fiber, fiber bundle and composite scales, respectively; that of constituent properties is described by normal distributions for fiber and matrix, and its propagations to bundle and composite scales are realized by Nataf transformation methods with the consideration of parameter correlations. For the cost control, minimum-size unit cells are formulated by exhaustive analysis of structural symmetries to reduce the computational cost without accuracy compromising for the single simulation, and 1/8 and 1/16 unit cells compared with traditional full-size ones are obtained. The evolution convergence for statistical uncertainties of effective properties is finally obtained with totally reduced computational cost of 89%.

Uncertainty quantification in dynamic modeling of pole vaulting using a non-intrusive polynomial chaos approach

Various stochastic techniques have been developed to account for uncertainties in analyzing the random nature of input parameters within mechanical systems. In this re-search, we employ the non-intrusive polynomial chaos method (NIPC) to evaluate the dynamic responses of a mass-spring configuration, resembling an inverted pendulum that simulates pole vaulting by incorporating uncertainties related to various parameters. In addition, the effectiveness of the NIPC is described and approved to allow demonstrat-ing how these uncertain parameters affect the performance of the vault in conjunction with the Monte Carlo (MC) method. The results obtained from the NIPC method for uncer-tainty quantification (UQ) are compared with those obtained using the MC reference ap-proach. These results produced by the NIPC method demonstrate remarkable concord-ance with the solutions derived from the reference method. In general, the proposed NIPC method proves to be an effective solution for mitigating problem complexity, reduc-ing computational time, obtaining uncertainty quantification of dynamic responses of the model for pole vaulting.

Improving the performance of Stein variational inference through extreme sparsification of physically-constrained neural network models

Most scientific machine learning (SciML) applications of neural networks involve hundreds to thousands of parameters, and hence, uncertainty quantification for such models is plagued by the curse of dimensionality. Using physical applications, we show that L0 sparsification prior to Stein variational gradient descent (L0+SVGD) is a more robust and efficient means of uncertainty quantification, in terms of computational cost and performance than the direct application of SGVD or projected SGVD methods. Specifically, L0+SVGD demonstrates superior resilience to noise, the ability to perform well in extrapolated regions, and a faster convergence rate to an optimal solution.

Uncertainty quantification in high-dimensional linear models incorporating graphical structures with applications to gene set analysis.

The functions of genes in networks are typically correlated due to their functional connectivity. Variable selection methods have been developed to select important genes associated with a trait while incorporating network graphical information. However, no method has been proposed to quantify the uncertainty of individual genes under such settings. In this paper, we construct confidence intervals (CIs) and provide P-values for parameters of a high-dimensional linear model incorporating graphical structures where the number of variables p diverges with the number of observations. For combining the graphical information, we propose a graph-constrained desparsified LASSO (least absolute shrinkage and selection operator) (GCDL) estimator, which reduces dramatically the influence of high correlation of predictors and enjoys the advantage of faster computation and higher accuracy compared with the desparsified LASSO. Theoretical results show that the GCDL estimator achieves asymptotic normality. The asymptotic property of the uniform convergence is established, with which an explicit expression of the uniform CI can be derived. Extensive numerical results indicate that the GCDL estimator and its (uniform) CI perform well even when predictors are highly correlated. An R package implementing the proposed method is available at https://github.com/XiaoZhangryy/gcdl.

Joint trajectory inference for single-cell genomics using deep learning with a mixture prior

Trajectory inference methods are essential for analyzing the developmental paths of cells in single-cell sequencing datasets. It provides insights into cellular differentiation, transitions, and lineage hierarchies, helping unravel the dynamic processes underlying development and disease progression. However, many existing tools lack a coherent statistical model and reliable uncertainty quantification, limiting their utility and robustness. In this paper, we introduce VITAE (Variational Inference for Trajectory by AutoEncoder), a statistical approach that integrates a latent hierarchical mixture model with variational autoencoders to infer trajectories. The statistical hierarchical model enhances the interpretability of our framework, while the posterior approximations generated by our variational autoencoder ensure computational efficiency and provide uncertainty quantification of cell projections along trajectories. Specifically, VITAE enables simultaneous trajectory inference and data integration, improving the accuracy of learning a joint trajectory structure in the presence of biological and technical heterogeneity across datasets. We show that VITAE outperforms other state-of-the-art trajectory inference methods on both real and synthetic data under various trajectory topologies. Furthermore, we apply VITAE to jointly analyze three distinct single-cell RNA sequencing datasets of the mouse neocortex, unveiling comprehensive developmental lineages of projection neurons. VITAE effectively reduces batch effects within and across datasets and uncovers finer structures that might be overlooked in individual datasets. Additionally, we showcase VITAE's efficacy in integrative analyses of multiomic datasets with continuous cell population structures.

Remaining-Useful-Lifetime Prediction of Proton Exchange Membrane Fuel Cell Considering Model Uncertainty Quantification on the Full-Time Scale

Remaining-Useful-Lifetime Prediction of Proton Exchange Membrane Fuel Cell Considering Model Uncertainty Quantification on the Full-Time Scale

Highly Selective Novel Heme Oxygenase-1 Hits Found by DNA-Encoded Library Machine Learning beyond the DEL Chemical Space.

DNA-encoded library (DEL) technology, especially when combined with machine learning (ML), is a powerful method to discover novel inhibitors. DEL-ML can expand a larger chemical space and boost cost-effectiveness during hit finding. Heme oxygenase-1 (HO-1), a heme-degrading enzyme, is linked to diseases such as cancer and neurodegenerative disorders. The discovery of five series of new scaffold HO-1 hits is reported here, using a DEL-ML workflow, which emphasizes the model's uncertainty quantification and domain of applicability. This model exhibits a strong extrapolation ability, identifying new structures beyond the DEL chemical space. About 37% of predicted molecules showed a binding affinity of K D < 20 μM, with the strongest being 141 nM, amd 14 of those molecules displayed >100-fold selectivity for HO-1 over heme oxygenase-2 (HO-2). These molecules also showed structural novelty compared to existing HO-1 inhibitors. Docking simulations provided insights into possible selectivity rationale.

Bayesian calibration and uncertainty quantification of a rate-dependent cohesive zone model for polymer interfaces

This study presents a rate-dependent cohesive zone model for the fracture of polymeric interfaces and performs a Bayesian calibration, an uncertainty quantification, and a sensitivity analysis for the model. The proposed cohesive zone model accounts for both reversible elastic and irreversible rate-dependent separation sliding deformation at the interface. The viscous dissipation due to the irreversible opening at the interface is modeled using elastic-viscoplastic kinematics that incorporates the effects of strain rate. Inverse calibration of parameters for such complex models through trial and error is challenging due to the large number of parameters of the model. Moreover, the calibrated parameter values are often non-unique and uncertain when the available experimental data is limited. To tackle this challenge, we employ a Bayesian calibration approach to identify parameters from experimental data, the resulting parameters significantly enhance the accuracy of the model. To quantify the uncertainty associated with the inverse parameter estimation, a modular Bayesian approach is employed to calibrate the unknown model parameters, accounting for the parameter uncertainty of the cohesive zone model. The advantages of the Bayesian calibration over a deterministic parameter fit are demonstrated. Further, to quantify the model uncertainties, such as incorrect assumptions or missing physics, a discrepancy function is introduced, which significantly improves the model’s prediction. Finally, the total uncertainty of the model is quantified in a predictive setting. A sensitivity analysis is performed to assess how changes in the input variables of the model affect the peak load, facilitating the identification of a concise set of highly influential parameters. The present approach can be used for calibration and uncertainty quantification for other complex computational mechanics models. It should also facilitate the designing of interface materials under uncertainty.

Hierarchical Bayesian modeling for calibration and uncertainty quantification of constitutive material models: Application to a uniaxial steel material model

Constitutive material models play an integral role in finite element (FE) modeling of structural components and systems and are governed by a set of parameters. The values of the material model parameters are important for the FE model to be able to predict the real structural behavior as closely as possible. Realistic cyclic material constitutive models well calibrated to experimental data are critical when predicting the nonlinear dynamic response of structural systems to extreme loads (e.g., earthquakes, high winds, blast). Hence, this study focuses on the inference and uncertainty quantification of material model parameters from an experimental dataset consisting of data from experiments on multiple specimens of the same kind, which is a topic of great interest to the structural modeling and simulation community. Traditionally, Bayesian inference is performed by pooling the data from all the specimens, or by considering the data from a single specimen, disregarding the inherent variability (aleatory uncertainty) among specimens of the same kind. Consequently, traditional approaches fail to capture the population distribution of material model parameters, which represents the aleatory specimen-to-specimen variability that is essential for conducting reliability analysis and uncertainty propagation studies. Moreover, these approaches cannot accurately simulate the response of new specimens. To overcome these limitations, this study introduces a novel application of hierarchical Bayesian inference for calibrating material model parameters. By simultaneously considering experimental data from multiple specimens of the same kind, hierarchical Bayesian modeling quantifies both epistemic uncertainty and aleatory specimen-to-specimen variability. In this study, the parameters of the well-known Giuffré-Menegotto-Pinto (GMP) uniaxial material model for reinforcing steel are calibrated and validated based on a dataset of experimental cyclic tests conducted on thirty-six steel specimens, or coupons, originating from three different manufacturing mills, complying to two different manufacturing standards, and subjected to different strain histories. These strain histories are similar to those experienced during seismic events by reinforcing steel bars within reinforced concrete structures. The material model parameters are calibrated using both the traditional and the hierarchical Bayesian approach and the results are compared. Based on the experimental dataset employed in this study, a joint probability distribution of the GMP material model parameters that quantifies both epistemic (due to finite sample size) and aleatory (coupon-to-coupon) variability or uncertainty is provided which can be used for uncertainty propagation in reliability and risk analyses of nonlinear dynamic systems.

Dynamical system identification, model selection, and model uncertainty quantification by Bayesian inference.

This study presents a Bayesian maximum a posteriori (MAP) framework for dynamical system identification from time-series data. This is shown to be equivalent to a generalized Tikhonov regularization, providing a rational justification for the choice of the residual and regularization terms, respectively, from the negative logarithms of the likelihood and prior distributions. In addition to the estimation of model coefficients, the Bayesian interpretation gives access to the full apparatus for Bayesian inference, including the ranking of models, the quantification of model uncertainties, and the estimation of unknown (nuisance) hyperparameters. Two Bayesian algorithms, joint MAP and variational Bayesian approximation, are compared to the least absolute shrinkage and selection operator (LASSO), ridge regression, and the sparse identification of nonlinear dynamics (SINDy) algorithms for sparse regression by application to several dynamical systems with added Gaussian or Laplace noise. For multivariate Gaussian likelihood and prior distributions, the Bayesian formulation gives Gaussian posterior and evidence distributions, in which the numerator terms can be expressed in terms of the Mahalanobis distance or "Gaussian norm" ||y-y^||M-12=(y-y^)⊤M-1(y-y^), where y is a vector variable, y^ is its estimator, and M is the covariance matrix. The posterior Gaussian norm is shown to provide a robust metric for quantitative model selection for the different systems and noise models examined.

Structured methods for parameter inference and uncertainty quantification for mechanistic models in the life sciences.

Parameter inference and uncertainty quantification are important steps when relating mathematical models to real-world observations and when estimating uncertainty in model predictions. However, methods for doing this can be computationally expensive, particularly when the number of unknown model parameters is large. The aim of this study is to develop and test an efficient profile likelihood-based method, which takes advantage of the structure of the mathematical model being used. We do this by identifying specific parameters that affect model output in a known way, such as a linear scaling. We illustrate the method by applying it to three toy models from different areas of the life sciences: (i) a predator-prey model from ecology; (ii) a compartment-based epidemic model from health sciences; and (iii) an advection-diffusion reaction model describing the transport of dissolved solutes from environmental science. We show that the new method produces results of comparable accuracy to existing profile likelihood methods but with substantially fewer evaluations of the forward model. We conclude that our method could provide a much more efficient approach to parameter inference for models where a structured approach is feasible. Computer code to apply the new method to user-supplied models and data is provided via a publicly accessible repository.

A systematic review and research recommendations on artificial intelligence for automated cervical cancer detection

AbstractEarly diagnosis of abnormal cervical cells enhances the chance of prompt treatment for cervical cancer (CrC). Artificial intelligence (AI)‐assisted decision support systems for detecting abnormal cervical cells are developed because manual identification needs trained healthcare professionals, and can be difficult, time‐consuming, and error‐prone. The purpose of this study is to present a comprehensive review of AI technologies used for detecting cervical pre‐cancerous lesions and cancer. The review study includes studies where AI was applied to Pap Smear test (cytological test), colposcopy, sociodemographic data and other risk factors, histopathological analyses, magnetic resonance imaging‐, computed tomography‐, and positron emission tomography‐scan‐based imaging modalities. We performed searches on Web of Science, Medline, Scopus, and Inspec. The preferred reporting items for systematic reviews and meta‐analysis guidelines were used to search, screen, and analyze the articles. The primary search resulted in identifying 9745 articles. We followed strict inclusion and exclusion criteria, which include search windows of the last decade, journal articles, and machine/deep learning‐based methods. A total of 58 studies have been included in the review for further analysis after identification, screening, and eligibility evaluation. Our review analysis shows that deep learning models are preferred for imaging techniques, whereas machine learning‐based models are preferred for sociodemographic data. The analysis shows that convolutional neural network‐based features yielded representative characteristics for detecting pre‐cancerous lesions and CrC. The review analysis also highlights the need for generating new and easily accessible diverse datasets to develop versatile models for CrC detection. Our review study shows the need for model explainability and uncertainty quantification to increase the trust of clinicians and stakeholders in the decision‐making of automated CrC detection models. Our review suggests that data privacy concerns and adaptability are crucial for deployment hence, federated learning and meta‐learning should also be explored.This article is categorized under: Fundamental Concepts of Data and Knowledge &gt; Explainable AI Technologies &gt; Machine Learning Technologies &gt; Classification