Uncertainty Quantification Methods Research Articles

A survey of uncertainty in deep neural networks

Over the last decade, neural networks have reached almost every field of science and become a crucial part of various real world applications. Due to the increasing spread, confidence in neural network predictions has become more and more important. However, basic neural networks do not deliver certainty estimates or suffer from over- or under-confidence, i.e. are badly calibrated. To overcome this, many researchers have been working on understanding and quantifying uncertainty in a neural network’s prediction. As a result, different types and sources of uncertainty have been identified and various approaches to measure and quantify uncertainty in neural networks have been proposed. This work gives a comprehensive overview of uncertainty estimation in neural networks, reviews recent advances in the field, highlights current challenges, and identifies potential research opportunities. It is intended to give anyone interested in uncertainty estimation in neural networks a broad overview and introduction, without presupposing prior knowledge in this field. For that, a comprehensive introduction to the most crucial sources of uncertainty is given and their separation into reducible model uncertainty and irreducible data uncertainty is presented. The modeling of these uncertainties based on deterministic neural networks, Bayesian neural networks (BNNs), ensemble of neural networks, and test-time data augmentation approaches is introduced and different branches of these fields as well as the latest developments are discussed. For a practical application, we discuss different measures of uncertainty, approaches for calibrating neural networks, and give an overview of existing baselines and available implementations. Different examples from the wide spectrum of challenges in the fields of medical image analysis, robotics, and earth observation give an idea of the needs and challenges regarding uncertainties in the practical applications of neural networks. Additionally, the practical limitations of uncertainty quantification methods in neural networks for mission- and safety-critical real world applications are discussed and an outlook on the next steps towards a broader usage of such methods is given.

Data space inversion for efficient uncertainty quantification using an integrated surface and sub-surface hydrologic model

Abstract. It is incumbent on decision-support hydrological modelling to make predictions of uncertain quantities in a decision-support context. In implementing decision-support modelling, data assimilation and uncertainty quantification are often the most difficult and time-consuming tasks. This is because the imposition of history-matching constraints on model parameters usually requires a large number of model runs. Data space inversion (DSI) provides a highly model-run-efficient method for predictive uncertainty quantification. It does this by evaluating covariances between model outputs used for history matching (e.g. hydraulic heads) and model predictions based on model runs that sample the prior parameter probability distribution. By directly focusing on the relationship between model outputs under historical conditions and predictions of system behaviour under future conditions, DSI avoids the need to estimate or adjust model parameters. This is advantageous when using integrated surface and sub-surface hydrologic models (ISSHMs) because these models are associated with long run times, numerical instability and ideally complex parameterization schemes that are designed to respect geological realism. This paper demonstrates that DSI provides a robust and efficient means of quantifying the uncertainties of complex model predictions. At the same time, DSI provides a basis for complementary linear analysis that allows the worth of available observations to be explored, as well as of observations which are yet to be acquired. This allows for the design of highly efficient, future data acquisition campaigns. DSI is applied in conjunction with an ISSHM representing a synthetic but realistic river–aquifer system. Predictions of interest are fast travel times and surface water infiltration. Linear and non-linear estimates of predictive uncertainty based on DSI are validated against a more traditional uncertainty quantification which requires the adjustment of a large number of parameters. A DSI-generated surrogate model is then used to investigate the effectiveness and efficiency of existing and possible future monitoring networks. The example demonstrates the benefits of using DSI in conjunction with a complex numerical model to quantify predictive uncertainty and support data worth analysis in complex hydrogeological environments.

Sensitivity-enhanced generalized polynomial chaos for efficient uncertainty quantification

We consider the Least Squares (LSQ) regression method for Uncertainty Quantification (UQ) using generalised polynomial chaos (gPC) and augment the linear system with the gradient of the Quantity of Interest (QoI) with respect to the stochastic variables. The gradient is computed very efficiently for all variables from the adjoint system of equations. To minimise the condition number of the augmented LSQ system, an effective sampling strategy of the stochastic space is required. We compare two strategies. In the first, we apply pivoted QR decomposition to the standard LSQ matrix and evaluate both the QoI and its gradient at the sample points identified. In the second strategy, we apply QR decomposition directly to the augmented matrix. We find that the first strategy is more efficient in terms of accuracy vs number of evaluations. We call the new approach sensitivity-enhanced generalised polynomial chaos, or se-gPC, and apply it to several test cases including an aerodynamic case with 40 stochastic parameters. The method can produce accurate estimations of the statistical moments using a small number of sampling points. The computational cost scales as ∼mp−1, instead of ∼mp of the standard LSQ formulation, where m is the number of stochastic variables and p the chaos order. The solution of the adjoint system of equations is implemented in many computational mechanics packages, thus the infrastructure exists for the application of the method to a wide variety of engineering problems.

Developing an integrated PCE-ANOVA-RF method for uncertainty quantification of hydrological model – The Amu Darya River Basin in Central Asia

Uncertainty quantification of hydrological model parameters plays an important role in the streamflow prediction of river basins. It is still a challenge to evaluate the effect of parameter uncertainty on hydrological models in association with moderate computation and high accuracy. In this study, a PCE-ANOVA-RF (abbreviated as PAR) method is developed to analyze the individual and interactive effects of multiple uncertain parameters in the Soil and Water Assessment Tool (SWAT) model as well as generate probabilistic forecasts of daily streamflow. The proposed PAR method can not only unveil the effect of parameter uncertainty and save computation time without reducing the simulation accuracy, but also expand PCE’s ability to predict future streamflow processes. The Amu Darya River Basin in Central Asia is introduced to verify the feasibility and applicability of PAR. The results disclose that (i) compared with the Sequence Uncertainty Fitting ver-2 (SUFI-2) algorithm, the simulation accuracy of PAR is acceptable (e.g., NSE = 0.85 in calibration period and 0.81 in validation period); (ii) in terms of variance analysis and streamflow simulation, PAR is more efficient and can largely reduce the number of model runs (about 88%); (iii) the parameters related to snow melting process have significant effects on both average flow and peak flow; besides, the interaction among these parameters are more obvious than that of other sensitive parameters.

Stochastic spectral cell method for structural dynamics and wave propagations

AbstractThe spectral element method when enriched with the concept of fictitious domain constitutes the spectral cell method which is appropriately able to model geometrically complicated problems by simplifying the mesh generation. In this paper, we propose a new computational framework called stochastic spectral cell method (SSCM) as a stochastic development of the spectral cell method for uncertainty quantification applied to structural dynamics and wave propagations. The stochastic form of spectral cell method has not been developed yet. Hence, the SSCM is a novel stochastic method comprising the advantages of spectral cell method for computational mechanics considering the geometrical complexity resulting from computer‐aided design. The SSCM is established by incorporating the polynomial chaos and Karhunen‐Loève expansions for the uncertainty quantification, and the interpolation functions of spectral elements for the spatial discretization. In the SSCM, the numerical solution of the Fredholm integral equation of the second kind required in Karhunen‐Loève expansion is obtained using the spectral cell method. The use of Cartesian mesh, diagonal mass matrix, and higher order interpolation functions of spectral elements for stochastic dynamic analysis lead to desirable computational efficiency and accuracy of the SSCM. The capability, effectiveness, and accuracy of the proposed SSCM are demonstrated with several numerical examples of wave propagation as well as dynamic analysis of structures.

Fatigue Life Uncertainty Quantification of Front Suspension Lower Control Arm Design

The purpose of this study is to investigate the uncertainty of the design variables of a front suspension lower control arm under fatigue-loading circumstances to estimate a reliable and robust product. This study offers a method for systematic uncertainty quantification (UQ), and the following steps were taken to achieve this: First, a finite element model was built to predict the fatigue life of the control arm under bump-loading conditions. Second, a sensitivity scheme, based on one of the global analyses, was developed to identify the model’s most and least significant design input variables. Third, physics-based and data-driven uncertainty quantification schemes were employed to quantify the model’s input parameter uncertainties via a Monte Carlo simulation. The simulations were conducted using 10,000 samples of material properties and geometrical uncertainty variables, with the coefficients of variation ranging from 1 to 3%. Finally, the confidence interval results show a deviation of about 21.74% from the mean (the baseline). As a result, by applying systematic UQ, a more reliable and robust automobile suspension control arm can be designed during the early stages of design to produce a more efficient and better approximation of fatigue life under uncertain conditions.

Efficiency Optimization of Wireless Power Transfer System for Electric Vehicle Based on Improved Marine Predators Algorithm

Electric vehicle (EV) is the core part of future automobile technology, and safe and reliable wireless power transfer (WPT) technology is the key link to improve the intelligent driving technology of EV. In this paper, the uncertainty quantification method is proposed to guide the optimization design of WPT structure, so as to improve the efficiency of WPT. First this paper establishes a surrogate model of WPT efficiency based on the adaptive sparse polynomial chaos expansion, and the uncertainty of EVs WPT transmission efficiency is quantified, the computational efficiency is improved by about 8.4 times. Then the surrogate model is combined with the global sensitivity analysis method to quantify the impact of different variables in WPT on efficiency and screen out the variables with greater impact. Finally, this paper uses the improved marine predators algorithm to optimize the selected WPT system structure parameters. Considering the uncertainty, the average efficiency of the optimized WPT system is increased from 73.43% to 94.64%. Compared with other optimization methods, it proves that the method in this paper can optimize WPT more efficiently, and significantly improve the transmission efficiency.

Local sensitivity informed anisotropic sparse grid method for uncertainty quantification of chemical kinetics

The determination of rate coefficient parameters in detailed chemical kinetic mechanisms through experiments often suffers from avoidable aleatory uncertainty, while the use of reduced mechanisms, based on various reduction methods, introduces epistemic uncertainty. Both sources of uncertainty pose significant challenges for modeling and numerical simulation, highlighting the need for calibrating these uncertain parameters to achieve robust chemical mechanisms in the next generation. However, the high-dimensional parameter space of chemical kinetic mechanisms remains a significant obstacle in uncertainty computing. The Anisotropic Sparse Grid (ASG) technique has been successful in dealing with high-dimensional uncertainty problems, but it lacks prior knowledge of anisotropy. To address this issue, we propose the Local Sensitivity-Informed Anisotropic Sparse Grid (LSIASG) method, which utilizes local sensitivity as prior information for the ASG method, thereby accelerating the entire uncertainty quantification process. We test the LSIASG method on a theoretical model, a detailed hydrogen kinetic mechanism, and a methane mechanism and demonstrate that it can capture high-dimensional uncertainty characteristics, including expectation and deviation.

Toward Robust Uncertainty Estimation with Random Activation Functions

Deep neural networks are in the limelight of machine learning with their excellent performance in many data-driven applications. However, they can lead to inaccurate predictions when queried in out-of-distribution data points, which can have detrimental effects especially in sensitive domains, such as healthcare and transportation, where erroneous predictions can be very costly and/or dangerous. Subsequently, quantifying the uncertainty of the output of a neural network is often leveraged to evaluate the confidence of its predictions, and ensemble models have proved to be effective in measuring the uncertainty by utilizing the variance of predictions over a pool of models. In this paper, we propose a novel approach for uncertainty quantification via ensembles, called Random Activation Functions (RAFs) Ensemble, that aims at improving the ensemble diversity toward a more robust estimation, by accommodating each neural network with a different (random) activation function. Extensive empirical study demonstrates that RAFs Ensemble outperforms state-of-the-art ensemble uncertainty quantification methods on both synthetic and real-world datasets in a series of regression tasks.

Robust and Adaptive Deep Learning via Bayesian Principles

Deep learning models have achieved tremendous successes in accurate predictions for computer vision, natural language processing and speech recognition applications. However, to succeed in high-risk and safety-critical domains such as healthcare and finance, these deep learning models need to be made reliable and trustworthy. Specifically, they need to be robust and adaptive to real-world environments which can be drastically different from the training settings. In this talk, I will advocate for Bayesian principles to achieve the goal of building robust and adaptive deep learning models. I will introduce a suite of uncertainty quantification methods for Bayesian deep learning, and demonstrate applications en- abled by accurate uncertainty estimates, e.g., robust predic- tion, continual learning and repairing model failures. I will conclude by discussing the research challenges and potential impact for robust and adaptive deep learning models. This paper is part of the AAAI-23 New Faculty Highlights.

On the optimal control of kinetic epidemic models with uncertain social features

AbstractIt is recognized that social heterogeneities in terms of the contact distribution have a strong influence on the spread of infectious diseases. Nevertheless, few data are available on the group composition of social contacts, and their statistical description does not possess universal patterns and may vary spatially and temporally. It is therefore essential to design robust control strategies, mimicking the effects of non‐pharmaceutical interventions, to limit efficiently the number of infected cases. In this work, starting from a recently introduced kinetic model for epidemiological dynamics that takes into account the impact of social contacts of individuals, we consider an uncertain contact formation dynamics leading to slim‐tailed as well as fat‐tailed distributions of contacts. Hence, we analyse the effects of an optimally robust control strategy of the system of agents. Thanks to classical methods of kinetic theory, we couple uncertainty quantification methods with the introduced mathematical model to assess the effects of social limitations. Finally, using the proposed modeling approach and starting from available data, we show the effectiveness of the proposed selective measures to dampen uncertainties together with the epidemic trends.

A learning-based optimal uncertainty quantification method and its application to ballistic impact problems

This paper concerns the study of optimal (supremum and infimum) uncertainty bounds for systems where the input (or prior) probability measure is only partially/imperfectly known (e.g., with only statistical moments and/or on a coarse topology) rather than fully specified. Such partial knowledge provides constraints on the input probability measures. The theory of Optimal Uncertainty Quantification allows us to convert the task into a constraint optimization problem where one seeks to compute the least upper/greatest lower bound of the system’s output uncertainties by finding the extremal probability measure of the input. Such optimization requires repeated evaluation of the system’s performance indicator (input to performance map) and is high-dimensional and non-convex by nature. Therefore, it is difficult to find the optimal uncertainty bounds in practice. In this paper, we examine the use of machine learning, especially deep neural networks, to address the challenge. We achieve this by introducing a neural network classifier to approximate the performance indicator combined with the stochastic gradient descent method to solve the optimization problem. We demonstrate the learning-based framework on the uncertainty quantification of the impact of magnesium alloys, which are promising light-weight structural and protective materials. Finally, we show that the approach can be used to construct maps for the performance certificate and safety design in engineering practice.

Virtual experiments for the assessment of data analysis and uncertainty quantification methods in scatterometry

Geometrical dimensions in nanostructures can be determined through indirect optical measurements carried out by a scatterometer. This includes solving a non-linear regression task in which a physical model is fitted to the observed optical diffraction pattern. We developed a virtual experiment which produces simulated diffraction patterns in coherent Fourier scatterometry measurements including perturbations due to various error sources. We utilize this virtual experiment to assess the suitability of data analysis and uncertainty quantification methods employed in scatterometry. In addition to investigating relevant physical parameters we explore the impact of deviations between the regression model utilized for the analysis and the scatterometry model used to produce the virtual diffraction pattern. We choose coverage probabilities of interval estimates of the geometrical dimensions as the main metric for the assessment. One of our findings is that the discretization level, expressed as the number of retained Fourier orders, can be relaxed up to order 9 in our case study, which is relevant as calculation times strongly depend on this parameter. Another result is that the least-squares approach considered here for solving the regression task in combination with the propagation of variances yields uncertainties which have somewhat lower coverage probabilities than the envisaged 95%. It turned out that it was critically important to model the oxide layer in order to get proper estimates of the width, or ‘critical dimension’, of the sample, while the uncertainty of the side wall angle had the largest impact on the uncertainty of the measurands. Our findings can help establishing traceability of Fourier scatterometry and underline the usefulness of high-quality virtual experiments in connection with complex measurement principles. At the same time the presented case study can be seen as a generic approach that could be followed for assessing uncertainty quantifications in other applications as well.

Scalable Physics-Based Maximum Likelihood Estimation Using Hierarchical Matrices

Physics-based covariance models provide a systematic way to construct covariance models that are consistent with the underlying physical laws in Gaussian process analysis. The unknown parameters in the covariance models can be estimated using maximum likelihood estimation, but direct construction of the covariance matrix and classical strategies of computing with it require physical model runs, storage complexity, and computational complexity. To address such challenges, we propose to approximate the discretized covariance function using hierarchical matrices. By utilizing randomized range sketching for individual off-diagonal blocks, the construction process of the hierarchical covariance approximation requires physical model applications and the maximum likelihood computations require effort per iteration. We propose a new approach to compute exactly the trace of products of hierarchical matrices which results in the expected Fisher information matrix being computable in as well. The construction is totally matrix-free and the derivatives of the covariance matrix can then be approximated in the same hierarchical structure by differentiating the whole process. Numerical results are provided to demonstrate the effectiveness, accuracy, and efficiency of the proposed method for parameter estimations and uncertainty quantification.

HYPAD-UQ: A Derivative-Based Uncertainty Quantification Method Using a Hypercomplex Finite Element Method

Abstract A derivative-based uncertainty quantification (UQ) method called HYPAD-UQ that utilizes sensitivities from a computational model was developed to approximate the statistical moments and Sobol' indices of the model output. Hypercomplex automatic differentiation (HYPAD) was used as a means to obtain accurate high-order partial derivatives from computational models such as finite element analyses. These sensitivities are used to construct a surrogate model of the output using a Taylor series expansion and subsequently used to estimate statistical moments (mean, variance, skewness, and kurtosis) and Sobol' indices using algebraic expansions. The uncertainty in a transient linear heat transfer analysis was quantified with HYPAD-UQ using first-order through seventh-order partial derivatives with respect to seven random variables encompassing material properties, geometry, and boundary conditions. Random sampling of the analytical solution and the regression-based stochastic perturbation finite element method were also conducted to compare accuracy and computational cost. The results indicate that HYPAD-UQ has superior accuracy for the same computational effort compared to the regression-based stochastic perturbation finite element method. Sensitivities calculated with HYPAD can allow higher-order Taylor series expansions to be an effective and practical UQ method.

A data-driven robust design optimization method and its application in compressor blade

The probability-based robust optimization methods require a large amount of sample data to build probability distribution models of uncertain parameters. However, it is a common situation that only scarce sampled data are available in practice due to expensive tests. This study proposes a data-driven robust optimization framework by embedding a novel uncertainty quantification (UQ) method, which can quantify the uncertainty based on the statistical moments of scarce input data. The computational robustness and accuracy of the developed UQ methods are validated. Then, the data-driven multi-objective optimization framework is applied to improve the mean performance and aerodynamic robustness of a two-dimensional compressor blade with real stagger angle errors. Uncertainty analysis shows that there is a probability of 47.55% to deviate from the nominal total pressure loss coefficient by more than 1% for the actual performance values at high positive incidence i=7° condition. In the optimization process, the total pressure loss coefficient is selected as the objective function, while the static pressure ratio is used as a constraint. The Gaussian process regression model is trained to improve the robust optimization efficiency. The robust optimization is conducted under the most sensitive conditions. Optimized results indicate that compared with the nominal blade, the mean performance of the selected robust blades is increased by 10.9%, 8.56%, and 0.83%; the performance dispersion is decreased by 19.0%, 24.8%, and 35.3%, respectively. The optimized results can provide useful references for the robust design of compressor blades.

Scalable Probabilistic Optimal Power Flow for High Renewables Using Lite Polynomial Chaos Expansion

This work introduces a novel polynomial chaos expansion (PCE)-based method for uncertainty quantification (UQ) in power systems, addressing probabilistic optimal power flow problem. The proposed approach leverages upon a series of transformations to overcome two main hinders to deploying POPF in practice, i.e., the vast number of random variables (RVs) and their interdependence. The proposed Lite-PCE method requires a small set of optimal power flow samples. The most important transformation is developed based on a Clayton copula to convert substantially large dimensional input variables into a single RV for efficient computing. The transformation allows Lite-PCE to be built on a single RV and offers a significant computational benefit while working with a large number of random renewable injections. The method also handles correlated random injection variables with different types of distributions in a nonparametric fashion. The Lite-PCE method is also used to develop the concept of transmission capacity sensitivity under uncertainty using coefficients of variation. The concept and nonparametric, scalable simulations are demonstrated using various IEEE test cases, up to the 1354-Bus system with 1242 random injection variables. The UQ results illustrate the efficiency and accuracy of the method for systems of considerable size and a high number of renewable injections.

Effects of Inlet Incidence Perturbations on Compressor Cascade Performance using Adaptive Sparse Grid Collocation

The effects of inflow variations due to the working environment and flight attitude changes on turbomachines are considerable in the real world. Nevertheless, uncertainty quantification can be adopted to assess mean performance changes and perform the aerodynamic shape design as well as optimization. Thus, an uncertainty quantification method of adaptive sparse grid collocation (ASGC) was first introduced to address the inflow uncertainties’ effect issue effectively and accurately. Then, ASGC was utilized to evaluate the impacts of inlet incidence perturbations at different perturbation scales and reference inflow Mach numbers on the aerodynamic performance of a controlled diffusion cascade. The results showed that compared with the Monte Carlo simulation and static sparse gird collocation, the statistical accuracy and response accuracy of ASGC were maintained, and meanwhile its model construction efficiency was significantly improved because of the nested adaptive sampling feature. Under the perturbations of inlet incidences with high reference incidences, the mean aerodynamic loss always aggravates. The changes in aerodynamic loss nonlinearly depend on the inlet incidence perturbations, and the nonlinear dependence becomes greater when the perturbation scale. expands. At the same perturbation scale, the nonlinear dependence on the inlet incidence perturbations is further enhanced when the reference inflow Mach number rises. Finally, uncertainty quantification of the flow field revealed that the fluctuation of flow accelerations at the leading edge plays a fundamental role in determining the uncertainty of the aerodynamic loss.

Evaluating point-prediction uncertainties in neural networks for protein-ligand binding prediction

Neural Network (NN) models provide potential to speed up the drug discovery process and reduce its failure rates. The success of NN models requires uncertainty quantification (UQ) as drug discovery explores chemical space beyond the training data distribution. Standard NN models do not provide uncertainty information. Some methods require changing the NN architecture or training procedure, limiting the selection of NN models. Moreover, predictive uncertainty can come from different sources. It is important to have the ability to separately model different types of predictive uncertainty, as the model can take assorted actions depending on the source of uncertainty. In this paper, we examine UQ methods that estimate different sources of predictive uncertainty for NN models aiming at protein-ligand binding prediction. We use our prior knowledge on chemical compounds to design the experiments. By utilizing a visualization method we create non-overlapping and chemically diverse partitions from a collection of chemical compounds. These partitions are used as training and test set splits to explore NN model uncertainty. We demonstrate how the uncertainties estimated by the selected methods describe different sources of uncertainty under different partitions and featurization schemes and the relationship to prediction error.

Uncertainty Quantification Method of Crosstalk Involving Braided-Shielded Cable

In this paper, in view of the uncertainty of geometric parameters of braided-shielded cable and structural parameters of the shield in practical problems, polynomial chaos expansions (PCE) is used to explore the uncertainty quantification of a crosstalk calculation model of braided-shielded cable. First, the model based on multi-conductor transmission line theory is expanded by PCE. Second, the truncation degrees of polynomials and sample size of PCE are determined by the leave-one-out method, and the statistical characteristic parameters of crosstalk are obtained by combining coefficients of polynomials. Compared with the calculation results of the Monte Carlo method, the mean value, standard deviation and probability density function obtained by the two methods are basically consistent, but the PCE method has obvious advantages in calculation efficiency. Finally, the influence of input variable parameters in the crosstalk calculation model of braided-shielded cable are calculated by combining the PCE and the Sobol’s global sensitivity analysis method, which provides theoretical guidance for the electromagnetic compatibility design of electrical and electronic equipment using braided-shielded cable.