Uncertainty Quantification Methods Research Articles

Evaluation of uncertainty quantification methods in multi-label classification: A case study with automatic diagnosis of electrocardiogram

Artificial Intelligence (AI) use in automated Electrocardiogram (ECG) classification has continuously attracted the research community’s interest, motivated by their promising results. Despite their great promise, limited attention has been paid to the robustness of their results, which is a key element for their implementation in clinical practice. Uncertainty Quantification (UQ) is a critical for trustworthy and reliable AI, particularly in safety-critical domains such as medicine. Estimating uncertainty in Machine Learning (ML) model predictions has been extensively used for Out-of-Distribution (OOD) detection under single-label tasks. However, the use of UQ methods in multi-label classification remains underexplored.This study goes beyond developing highly accurate models comparing five uncertainty quantification methods using the same Deep Neural Network (DNN) architecture across various validation scenarios, including internal and external validation as well as OOD detection, taking multi-label ECG classification as the example domain. We show the importance of external validation and its impact on classification performance, uncertainty estimates quality, and calibration. Ensemble-based methods yield more robust uncertainty estimations than single network or stochastic methods. Although current methods still have limitations in accurately quantifying uncertainty, particularly in the case of dataset shift, incorporating uncertainty estimates with a classification with a rejection option improves the ability to detect such changes. Moreover, we show that using uncertainty estimates as a criterion for sample selection in active learning setting results in greater improvements in classification performance compared to random sampling.

CFD Uncertainty Quantification using PCE–HDMR: Exemplary Application to a Buoyancy-Driven Mixing Process

For the investigation of uncertainties in high dimensional spaces of computationally expensive engineering applications, reliable Uncertainty Quantification (UQ) methods are needed. These methods should provide accurate and efficient High-Dimensional Model Representations of stochastic results using a reasonable number of calculations. Therefore, the PCE–HDMR approach (Polynomial Chaos Expansion–High-Dimensional Model Representation) is utilized to qualify appropriate UQ methods for large-scale computations in the field of Computational Fluid Dynamics. This technique is a combination of Cut-HDMR, a hierarchical decomposition modeling approach, with PCE. To demonstrate its effectiveness, the PCE–HDMR methodology in conjunction with complementary modeling techniques is applied for the UQ analysis of a buoyancy-driven mixing process between two miscible fluids within the Differentially Heated Cavity of aspect ratio 4. The results include a thorough probabilistic representation of time-dependent response quantities that comprehensively describe the mixing process. The stochastic models are derived from Large Eddy Simulations using PCE–HDMR and the Sparse Grid Method, which serves as a reference for the results from PCE–HDMR. The results show that PCE–HDMR provides accurate statistics of the modeled time-dependent stochastic processes and shows good agreement with the reference results. Thus, PCE–HDMR indicates great potential for UQ of technical-scale computations due to its efficiency and flexibility in the construction of stochastic models.

Automated cross-sectional view selection in CT angiography of aortic dissections with uncertainty awareness and retrospective clinical annotations

Surveillance imaging of patients with chronic aortic diseases, such as aneurysms and dissections, relies on obtaining and comparing cross-sectional diameter measurements along the aorta at predefined aortic landmarks, over time. The orientation of the cross-sectional measuring planes at each landmark is currently defined manually by highly trained operators. Centerline-based approaches are unreliable in patients with chronic aortic dissection, because of the asymmetric flow channels, differences in contrast opacification, and presence of mural thrombus, making centerline computations or measurements difficult to generate and reproduce. In this work, we present three alternative approaches – INS, MCDS, MCDbS – based on convolutional neural networks and uncertainty quantification methods to predict the orientation (ϕ,θ) of such cross-sectional planes. For the monitoring of chronic aortic dissections, we show how a dataset of 162 CTA volumes with overall 3273 imperfect manual annotations routinely collected in a clinic can be efficiently used to accomplish this task, despite the presence of non-negligible interoperator variabilities in terms of mean absolute error (MAE) and 95% limits of agreement (LOA). We show how, despite the large limits of agreement in the training data, the trained model provides faster and more reproducible results than either an expert user or a centerline method. The remaining disagreement lies within the variability produced by three independent expert annotators and matches the current state of the art, providing a similar error, but in a fraction of the time.

Deterministic sampling method using simplex ensemble and scaling method for efficient and robust uncertainty quantification

ABSTRACT Uncertainty quantification (UQ) of the neutron multiplication factor is important to investigate the appropriate safety margin for a target system. Although the random sampling method is a practical and useful UQ method, a large computational cost is required to reduce the statistical error of the estimated uncertainty. Furthermore, if an input variable follows a normal distribution with a large standard deviation, the perturbed input variable by the random sampling method may become a physically inappropriate or negative value. To address these issues for the efficient and robust UQ, a modified deterministic sampling method using the simplex ensemble and the scaling method is proposed. The features of the proposed method are summarized as follows: The sample size is r + 2 , where r corresponds to the effective rank of the covariance matrix between the input variables; depending on a situation of target UQ, the amounts of perturbations for the input parameters can be arbitrarily given by the scaling factor method; the scaling factor can be updated to avoid physically inappropriate in the perturbed input variables. The effectiveness of the proposed method is demonstrated through the UQ of the neutron multiplication factor due to fuel manufacturing uncertainties for a typical PWR pin-cell burnup calculation.

Quantifying Uncertainty in Deep Learning of Radiologic Images.

In recent years, deep learning (DL) has shown impressive performance in radiologic image analysis. However, for a DL model to be useful in a real-world setting, its confidence in a prediction must also be known. Each DL model's output has an estimated probability, and these estimated probabilities are not always reliable. Uncertainty represents the trustworthiness (validity) of estimated probabilities. The higher the uncertainty, the lower the validity. Uncertainty quantification (UQ) methods determine the uncertainty level of each prediction. Predictions made without UQ methods are generally not trustworthy. By implementing UQ in medical DL models, users can be alerted when a model does not have enough information to make a confident decision. Consequently, a medical expert could reevaluate the uncertain cases, which would eventually lead to gaining more trust when using a model. This review focuses on recent trends using UQ methods in DL radiologic image analysis within a conceptual framework. Also discussed in this review are potential applications, challenges, and future directions of UQ in DL radiologic image analysis.

Uncertainty assessment of proarrhythmia predictions derived from multi-level in silico models

In silico methods can be used for an early assessment of arrhythmogenic properties of drug candidates. However, their use for decision-making is conditioned by the possibility to estimate the predictions’ uncertainty. This work describes our efforts to develop uncertainty quantification methods for the predictions produced by multi-level proarrhythmia models. In silico models used in this field usually start with experimental or predicted IC50 values that describe drug-induced ion channel blockade. Using such inputs, an electrophysiological model computes how the ion channel inhibition, exerted by a drug in a certain concentration, translates to an altered shape and duration of the action potential in cardiac cells, which can be represented as arrhythmogenic risk biomarkers such as the APD90. Using this framework, we identify the main sources of aleatory and epistemic uncertainties and propose a method based on probabilistic simulations that replaces single-point estimates predicted using multiple input values, including the IC50s and the electrophysiological parameters, by distributions of values. Two selected variability types associated with these inputs are then propagated through the multi-level model to estimate their impact on the uncertainty levels in the output, expressed by means of intervals. The proposed approach yields single predictions of arrhythmogenic risk biomarkers together with value intervals, providing a more comprehensive and realistic description of drug effects on a human population. The methodology was tested by predicting arrhythmogenic biomarkers on a series of twelve well-characterised marketed drugs, belonging to different arrhythmogenic risk classes.

Uncertainty Assessment in the Prediction of Selected Fission Products in the Case of a Postulated Severe Accident Scenario at VVER-1000

The aim of the paper is to present the results and insights gained from the analysis of the postulated severe accident scenario for the Nuclear Power Plant (NPP) equipped with a VVER-1000, focusing on the uncertainty assessment in the prediction of selected Fission Products (FPs). To this end, the simulation of the Large Break Loss of Coolant Accident (LB LOCA) along with a Station Blackout (SBO) scenario was carried out. The capabilities of the ASTECv2.2.0.1 [1] severe accident (SA) code along with the uncertainty quantification tool – SUNSET [2] in the coupled mode was used. The analysis was carried out for the eight input variables and ten output variables. The uncertainty quantification method incorporated within the SUNSET statistical tool was used in order to conduct the present analysis. Based on the Wilks’ formula, the number of ASTEC code runs was determined. The main statistical data are obtained for the selected fission products as figures of merit. The main steps of the analysis are outlined. Furthermore, a brief overview of the SUNSET/ASTECv2.2.0.1 coupling procedures [3] is also presented and discussed. This work demonstrates the application of the uncertainty quantification methodology in the analyses of the severe accidents in the Nuclear Power Plants and provides the specific recommendations of its use. The main outcomes from the assessment provided will be useful and may be a base for the future analyses related to the evaluation of the safety of the VVER-1000.

A survey of uncertainty in deep neural networks

Over the last decade, neural networks have reached almost every field of science and become a crucial part of various real world applications. Due to the increasing spread, confidence in neural network predictions has become more and more important. However, basic neural networks do not deliver certainty estimates or suffer from over- or under-confidence, i.e. are badly calibrated. To overcome this, many researchers have been working on understanding and quantifying uncertainty in a neural network’s prediction. As a result, different types and sources of uncertainty have been identified and various approaches to measure and quantify uncertainty in neural networks have been proposed. This work gives a comprehensive overview of uncertainty estimation in neural networks, reviews recent advances in the field, highlights current challenges, and identifies potential research opportunities. It is intended to give anyone interested in uncertainty estimation in neural networks a broad overview and introduction, without presupposing prior knowledge in this field. For that, a comprehensive introduction to the most crucial sources of uncertainty is given and their separation into reducible model uncertainty and irreducible data uncertainty is presented. The modeling of these uncertainties based on deterministic neural networks, Bayesian neural networks (BNNs), ensemble of neural networks, and test-time data augmentation approaches is introduced and different branches of these fields as well as the latest developments are discussed. For a practical application, we discuss different measures of uncertainty, approaches for calibrating neural networks, and give an overview of existing baselines and available implementations. Different examples from the wide spectrum of challenges in the fields of medical image analysis, robotics, and earth observation give an idea of the needs and challenges regarding uncertainties in the practical applications of neural networks. Additionally, the practical limitations of uncertainty quantification methods in neural networks for mission- and safety-critical real world applications are discussed and an outlook on the next steps towards a broader usage of such methods is given.

Data space inversion for efficient uncertainty quantification using an integrated surface and sub-surface hydrologic model

Abstract. It is incumbent on decision-support hydrological modelling to make predictions of uncertain quantities in a decision-support context. In implementing decision-support modelling, data assimilation and uncertainty quantification are often the most difficult and time-consuming tasks. This is because the imposition of history-matching constraints on model parameters usually requires a large number of model runs. Data space inversion (DSI) provides a highly model-run-efficient method for predictive uncertainty quantification. It does this by evaluating covariances between model outputs used for history matching (e.g. hydraulic heads) and model predictions based on model runs that sample the prior parameter probability distribution. By directly focusing on the relationship between model outputs under historical conditions and predictions of system behaviour under future conditions, DSI avoids the need to estimate or adjust model parameters. This is advantageous when using integrated surface and sub-surface hydrologic models (ISSHMs) because these models are associated with long run times, numerical instability and ideally complex parameterization schemes that are designed to respect geological realism. This paper demonstrates that DSI provides a robust and efficient means of quantifying the uncertainties of complex model predictions. At the same time, DSI provides a basis for complementary linear analysis that allows the worth of available observations to be explored, as well as of observations which are yet to be acquired. This allows for the design of highly efficient, future data acquisition campaigns. DSI is applied in conjunction with an ISSHM representing a synthetic but realistic river–aquifer system. Predictions of interest are fast travel times and surface water infiltration. Linear and non-linear estimates of predictive uncertainty based on DSI are validated against a more traditional uncertainty quantification which requires the adjustment of a large number of parameters. A DSI-generated surrogate model is then used to investigate the effectiveness and efficiency of existing and possible future monitoring networks. The example demonstrates the benefits of using DSI in conjunction with a complex numerical model to quantify predictive uncertainty and support data worth analysis in complex hydrogeological environments.

Sensitivity-enhanced generalized polynomial chaos for efficient uncertainty quantification

We consider the Least Squares (LSQ) regression method for Uncertainty Quantification (UQ) using generalised polynomial chaos (gPC) and augment the linear system with the gradient of the Quantity of Interest (QoI) with respect to the stochastic variables. The gradient is computed very efficiently for all variables from the adjoint system of equations. To minimise the condition number of the augmented LSQ system, an effective sampling strategy of the stochastic space is required. We compare two strategies. In the first, we apply pivoted QR decomposition to the standard LSQ matrix and evaluate both the QoI and its gradient at the sample points identified. In the second strategy, we apply QR decomposition directly to the augmented matrix. We find that the first strategy is more efficient in terms of accuracy vs number of evaluations. We call the new approach sensitivity-enhanced generalised polynomial chaos, or se-gPC, and apply it to several test cases including an aerodynamic case with 40 stochastic parameters. The method can produce accurate estimations of the statistical moments using a small number of sampling points. The computational cost scales as ∼mp−1, instead of ∼mp of the standard LSQ formulation, where m is the number of stochastic variables and p the chaos order. The solution of the adjoint system of equations is implemented in many computational mechanics packages, thus the infrastructure exists for the application of the method to a wide variety of engineering problems.

Developing an integrated PCE-ANOVA-RF method for uncertainty quantification of hydrological model – The Amu Darya River Basin in Central Asia

Uncertainty quantification of hydrological model parameters plays an important role in the streamflow prediction of river basins. It is still a challenge to evaluate the effect of parameter uncertainty on hydrological models in association with moderate computation and high accuracy. In this study, a PCE-ANOVA-RF (abbreviated as PAR) method is developed to analyze the individual and interactive effects of multiple uncertain parameters in the Soil and Water Assessment Tool (SWAT) model as well as generate probabilistic forecasts of daily streamflow. The proposed PAR method can not only unveil the effect of parameter uncertainty and save computation time without reducing the simulation accuracy, but also expand PCE’s ability to predict future streamflow processes. The Amu Darya River Basin in Central Asia is introduced to verify the feasibility and applicability of PAR. The results disclose that (i) compared with the Sequence Uncertainty Fitting ver-2 (SUFI-2) algorithm, the simulation accuracy of PAR is acceptable (e.g., NSE = 0.85 in calibration period and 0.81 in validation period); (ii) in terms of variance analysis and streamflow simulation, PAR is more efficient and can largely reduce the number of model runs (about 88%); (iii) the parameters related to snow melting process have significant effects on both average flow and peak flow; besides, the interaction among these parameters are more obvious than that of other sensitive parameters.

Stochastic spectral cell method for structural dynamics and wave propagations

AbstractThe spectral element method when enriched with the concept of fictitious domain constitutes the spectral cell method which is appropriately able to model geometrically complicated problems by simplifying the mesh generation. In this paper, we propose a new computational framework called stochastic spectral cell method (SSCM) as a stochastic development of the spectral cell method for uncertainty quantification applied to structural dynamics and wave propagations. The stochastic form of spectral cell method has not been developed yet. Hence, the SSCM is a novel stochastic method comprising the advantages of spectral cell method for computational mechanics considering the geometrical complexity resulting from computer‐aided design. The SSCM is established by incorporating the polynomial chaos and Karhunen‐Loève expansions for the uncertainty quantification, and the interpolation functions of spectral elements for the spatial discretization. In the SSCM, the numerical solution of the Fredholm integral equation of the second kind required in Karhunen‐Loève expansion is obtained using the spectral cell method. The use of Cartesian mesh, diagonal mass matrix, and higher order interpolation functions of spectral elements for stochastic dynamic analysis lead to desirable computational efficiency and accuracy of the SSCM. The capability, effectiveness, and accuracy of the proposed SSCM are demonstrated with several numerical examples of wave propagation as well as dynamic analysis of structures.

Fatigue Life Uncertainty Quantification of Front Suspension Lower Control Arm Design

The purpose of this study is to investigate the uncertainty of the design variables of a front suspension lower control arm under fatigue-loading circumstances to estimate a reliable and robust product. This study offers a method for systematic uncertainty quantification (UQ), and the following steps were taken to achieve this: First, a finite element model was built to predict the fatigue life of the control arm under bump-loading conditions. Second, a sensitivity scheme, based on one of the global analyses, was developed to identify the model’s most and least significant design input variables. Third, physics-based and data-driven uncertainty quantification schemes were employed to quantify the model’s input parameter uncertainties via a Monte Carlo simulation. The simulations were conducted using 10,000 samples of material properties and geometrical uncertainty variables, with the coefficients of variation ranging from 1 to 3%. Finally, the confidence interval results show a deviation of about 21.74% from the mean (the baseline). As a result, by applying systematic UQ, a more reliable and robust automobile suspension control arm can be designed during the early stages of design to produce a more efficient and better approximation of fatigue life under uncertain conditions.

Efficiency Optimization of Wireless Power Transfer System for Electric Vehicle Based on Improved Marine Predators Algorithm

Electric vehicle (EV) is the core part of future automobile technology, and safe and reliable wireless power transfer (WPT) technology is the key link to improve the intelligent driving technology of EV. In this paper, the uncertainty quantification method is proposed to guide the optimization design of WPT structure, so as to improve the efficiency of WPT. First this paper establishes a surrogate model of WPT efficiency based on the adaptive sparse polynomial chaos expansion, and the uncertainty of EVs WPT transmission efficiency is quantified, the computational efficiency is improved by about 8.4 times. Then the surrogate model is combined with the global sensitivity analysis method to quantify the impact of different variables in WPT on efficiency and screen out the variables with greater impact. Finally, this paper uses the improved marine predators algorithm to optimize the selected WPT system structure parameters. Considering the uncertainty, the average efficiency of the optimized WPT system is increased from 73.43% to 94.64%. Compared with other optimization methods, it proves that the method in this paper can optimize WPT more efficiently, and significantly improve the transmission efficiency.

Local sensitivity informed anisotropic sparse grid method for uncertainty quantification of chemical kinetics

The determination of rate coefficient parameters in detailed chemical kinetic mechanisms through experiments often suffers from avoidable aleatory uncertainty, while the use of reduced mechanisms, based on various reduction methods, introduces epistemic uncertainty. Both sources of uncertainty pose significant challenges for modeling and numerical simulation, highlighting the need for calibrating these uncertain parameters to achieve robust chemical mechanisms in the next generation. However, the high-dimensional parameter space of chemical kinetic mechanisms remains a significant obstacle in uncertainty computing. The Anisotropic Sparse Grid (ASG) technique has been successful in dealing with high-dimensional uncertainty problems, but it lacks prior knowledge of anisotropy. To address this issue, we propose the Local Sensitivity-Informed Anisotropic Sparse Grid (LSIASG) method, which utilizes local sensitivity as prior information for the ASG method, thereby accelerating the entire uncertainty quantification process. We test the LSIASG method on a theoretical model, a detailed hydrogen kinetic mechanism, and a methane mechanism and demonstrate that it can capture high-dimensional uncertainty characteristics, including expectation and deviation.

Toward Robust Uncertainty Estimation with Random Activation Functions

Deep neural networks are in the limelight of machine learning with their excellent performance in many data-driven applications. However, they can lead to inaccurate predictions when queried in out-of-distribution data points, which can have detrimental effects especially in sensitive domains, such as healthcare and transportation, where erroneous predictions can be very costly and/or dangerous. Subsequently, quantifying the uncertainty of the output of a neural network is often leveraged to evaluate the confidence of its predictions, and ensemble models have proved to be effective in measuring the uncertainty by utilizing the variance of predictions over a pool of models. In this paper, we propose a novel approach for uncertainty quantification via ensembles, called Random Activation Functions (RAFs) Ensemble, that aims at improving the ensemble diversity toward a more robust estimation, by accommodating each neural network with a different (random) activation function. Extensive empirical study demonstrates that RAFs Ensemble outperforms state-of-the-art ensemble uncertainty quantification methods on both synthetic and real-world datasets in a series of regression tasks.

Robust and Adaptive Deep Learning via Bayesian Principles

Deep learning models have achieved tremendous successes in accurate predictions for computer vision, natural language processing and speech recognition applications. However, to succeed in high-risk and safety-critical domains such as healthcare and finance, these deep learning models need to be made reliable and trustworthy. Specifically, they need to be robust and adaptive to real-world environments which can be drastically different from the training settings. In this talk, I will advocate for Bayesian principles to achieve the goal of building robust and adaptive deep learning models. I will introduce a suite of uncertainty quantification methods for Bayesian deep learning, and demonstrate applications en- abled by accurate uncertainty estimates, e.g., robust predic- tion, continual learning and repairing model failures. I will conclude by discussing the research challenges and potential impact for robust and adaptive deep learning models. This paper is part of the AAAI-23 New Faculty Highlights.

On the optimal control of kinetic epidemic models with uncertain social features

AbstractIt is recognized that social heterogeneities in terms of the contact distribution have a strong influence on the spread of infectious diseases. Nevertheless, few data are available on the group composition of social contacts, and their statistical description does not possess universal patterns and may vary spatially and temporally. It is therefore essential to design robust control strategies, mimicking the effects of non‐pharmaceutical interventions, to limit efficiently the number of infected cases. In this work, starting from a recently introduced kinetic model for epidemiological dynamics that takes into account the impact of social contacts of individuals, we consider an uncertain contact formation dynamics leading to slim‐tailed as well as fat‐tailed distributions of contacts. Hence, we analyse the effects of an optimally robust control strategy of the system of agents. Thanks to classical methods of kinetic theory, we couple uncertainty quantification methods with the introduced mathematical model to assess the effects of social limitations. Finally, using the proposed modeling approach and starting from available data, we show the effectiveness of the proposed selective measures to dampen uncertainties together with the epidemic trends.

A learning-based optimal uncertainty quantification method and its application to ballistic impact problems

This paper concerns the study of optimal (supremum and infimum) uncertainty bounds for systems where the input (or prior) probability measure is only partially/imperfectly known (e.g., with only statistical moments and/or on a coarse topology) rather than fully specified. Such partial knowledge provides constraints on the input probability measures. The theory of Optimal Uncertainty Quantification allows us to convert the task into a constraint optimization problem where one seeks to compute the least upper/greatest lower bound of the system’s output uncertainties by finding the extremal probability measure of the input. Such optimization requires repeated evaluation of the system’s performance indicator (input to performance map) and is high-dimensional and non-convex by nature. Therefore, it is difficult to find the optimal uncertainty bounds in practice. In this paper, we examine the use of machine learning, especially deep neural networks, to address the challenge. We achieve this by introducing a neural network classifier to approximate the performance indicator combined with the stochastic gradient descent method to solve the optimization problem. We demonstrate the learning-based framework on the uncertainty quantification of the impact of magnesium alloys, which are promising light-weight structural and protective materials. Finally, we show that the approach can be used to construct maps for the performance certificate and safety design in engineering practice.

Virtual experiments for the assessment of data analysis and uncertainty quantification methods in scatterometry

Geometrical dimensions in nanostructures can be determined through indirect optical measurements carried out by a scatterometer. This includes solving a non-linear regression task in which a physical model is fitted to the observed optical diffraction pattern. We developed a virtual experiment which produces simulated diffraction patterns in coherent Fourier scatterometry measurements including perturbations due to various error sources. We utilize this virtual experiment to assess the suitability of data analysis and uncertainty quantification methods employed in scatterometry. In addition to investigating relevant physical parameters we explore the impact of deviations between the regression model utilized for the analysis and the scatterometry model used to produce the virtual diffraction pattern. We choose coverage probabilities of interval estimates of the geometrical dimensions as the main metric for the assessment. One of our findings is that the discretization level, expressed as the number of retained Fourier orders, can be relaxed up to order 9 in our case study, which is relevant as calculation times strongly depend on this parameter. Another result is that the least-squares approach considered here for solving the regression task in combination with the propagation of variances yields uncertainties which have somewhat lower coverage probabilities than the envisaged 95%. It turned out that it was critically important to model the oxide layer in order to get proper estimates of the width, or ‘critical dimension’, of the sample, while the uncertainty of the side wall angle had the largest impact on the uncertainty of the measurands. Our findings can help establishing traceability of Fourier scatterometry and underline the usefulness of high-quality virtual experiments in connection with complex measurement principles. At the same time the presented case study can be seen as a generic approach that could be followed for assessing uncertainty quantifications in other applications as well.