Uncertainty Quantification Methods Research Articles

Linearity characterization and uncertainty quantification of spectroradiometers via maximum likelihood and the non-parametric bootstrap

A technique for characterizing and correcting the linearity of radiometric instruments is known by the names the ‘flux-addition method’ and the ‘combinatorial technique’. In this paper, we develop a rigorous uncertainty quantification method for use with this technique and illustrate its use with both synthetic data and experimental data from a ‘beam conjoiner’ instrument. We present a probabilistic model that relates the instrument readout to a set of unknown fluxes via a set of polynomial coefficients. Maximum likelihood estimates (MLEs) of the unknown fluxes and polynomial coefficients are recommended, while a non-parametric bootstrap algorithm enables uncertainty quantification including standard errors and confidence intervals.The synthetic data represent plausible outputs of a radiometric instrument and enable testing and validation of the method. The MLEs for these data are found to be approximately unbiased, and confidence intervals derived from the bootstrap replicates are found to be consistent with their target coverage of 95%. For the polynomial coefficients, the observed coverages range from 91% to 99%. The experimental data set illustrates how a complete calibration with uncertainties can be achieved using the method plus one well-known flux level. The uncertainty contribution attributable to estimation of the instrument’s non-linear response is less than 0.025% over most of its range.

Anisotropy-Based Adaptive Polynomial Chaos Method for Hybrid Uncertainty Quantification and Reliability-Based Design Optimization of Structural-Acoustic System

The evaluation of objective functions and component reliability in the optimisation of structural-acoustic systems with random and interval variables is computationally expensive, especially when strong nonlinearity exhibits between the response and input variables. To reduce the computational cost and improve the computational efficiency, a novel anisotropy-based adaptive polynomial chaos (ABAPC) expansion method was developed in this study. In ABAPC, the anisotropy-based polynomial chaos expansion, namely the retained order of polynomial chaos expansion (PCE) differs from each variable, is used to construct the initial surrogate model instead of first-order polynomial chaos expansion in conventional methods. Then, an anisotropy-based adaptive basis growth strategy was developed to reduce the estimation of the coefficients of the polynomial chaos expansion method and increase its computational efficiency. Finally, to solve problems with probabilistic and interval parameters, an adaptive basis truncation strategy was introduced and implemented. Using the ABAPC method, the computational cost of reliability-based design optimisation for structural-acoustic systems can be efficiently reduced. The effectiveness of the proposed method were demonstrated by solving two numerical examples and optimisation problems of a structural-acoustic system.

Explainable uncertainty quantifications for deep learning-based molecular property prediction

Quantifying uncertainty in machine learning is important in new research areas with scarce high-quality data. In this work, we develop an explainable uncertainty quantification method for deep learning-based molecular property prediction. This method can capture aleatoric and epistemic uncertainties separately and attribute the uncertainties to atoms present in the molecule. The atom-based uncertainty method provides an extra layer of chemical insight to the estimated uncertainties, i.e., one can analyze individual atomic uncertainty values to diagnose the chemical component that introduces uncertainty to the prediction. Our experiments suggest that atomic uncertainty can detect unseen chemical structures and identify chemical species whose data are potentially associated with significant noise. Furthermore, we propose a post-hoc calibration method to refine the uncertainty quantified by ensemble models for better confidence interval estimates. This work improves uncertainty calibration and provides a framework for assessing whether and why a prediction should be considered unreliable.Graphical

An efficient uncertainty quantification and propagation method through skewness and kurtosis fitting region

An efficient uncertainty quantification and propagation method through skewness and kurtosis fitting region

A generalized Bayes framework for probabilistic clustering

Summary Loss-based clustering methods, such as k-means clustering and its variants, are standard tools for finding groups in data. However, the lack of quantification of uncertainty in the estimated clusters is a disadvantage. Model-based clustering based on mixture models provides an alternative approach, but such methods face computational problems and are highly sensitive to the choice of kernel. In this article we propose a generalized Bayes framework that bridges between these paradigms through the use of Gibbs posteriors. In conducting Bayesian updating, the loglikelihood is replaced by a loss function for clustering, leading to a rich family of clustering methods. The Gibbs posterior represents a coherent updating of Bayesian beliefs without needing to specify a likelihood for the data, and can be used for characterizing uncertainty in clustering. We consider losses based on Bregman divergence and pairwise similarities, and develop efficient deterministic algorithms for point estimation along with sampling algorithms for uncertainty quantification. Several existing clustering algorithms, including k-means, can be interpreted as generalized Bayes estimators in our framework, and thus we provide a method of uncertainty quantification for these approaches, allowing, for example, calculation of the probability that a data point is well clustered.

Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed. Further, to help facilitate the deployment of UQ in Scientific Machine Learning research and practice, we present and develop in [1] an open-source Python library (github.com/Crunch-UQ4MI/neuraluq), termed NeuralUQ, that is accompanied by an educational tutorial and additional computational experiments.

Uncertainty quantification in operational modal analysis of time-varying structures based on time-dependent autoregressive moving average model

The identified modal parameters in Operational Modal Analysis (OMA) are always subject to statistical uncertainties from many sources and the uncertainty is also important to assess the quality of the identified results. This study proposes an uncertainty quantification method in OMA for time-varying structures based on the Functional Series Time-dependent AutoRegressive Moving Average (FS-TARMA) model. A multi-stage pseudo-linear optimization scheme is first adopted to obtain the approximate Maximum Likelihood Estimates (MLEs) of FS-TARMA model parameters and the covariance is computed as the inverse Hessian matrix of the Negative Log-Likelihood Function (NLLF) at the MLEs. Finally, the approximate closed-form covariance of modal parameters is derived by the first-order sensitivity method. The proposed method is verified by two numerical examples and an experimental beam. Furthermore, the ability of the uncertainty to extinguish the spurious modes caused by colored noise excitation and the noise modes caused by measurement noise or inappropriate model structure is also demonstrated in these examples.

Uncertainty quantification study of the aerodynamic performance of high-altitude propellers

Performance evaluations for propellers operating at high altitudes are subject to increased uncertainty due to scarce experimental or flight data and difficulties in modeling low Reynolds number flows. For this reason, the Polynomial Chaos Expansion (PCE) method is used in this paper to assess the performance uncertainty of propellers operating at high altitudes. Aleatoric (i.e. linked to the geometry or operating conditions) and epistemic (i.e. linked to the mathematical model describing the flow) uncertainty variables are included in this study to estimate the total uncertainty related to performance predictions made by two physical models, namely 3D RANS with the use of γ−Reθ transition model and Blade Element Momentum Theory (BEMT). In order to validate the proposed method, multipoint uncertainty quantification (UQ) studies are performed for two benchmark propeller geometries under various operating conditions for which experimental data are available. The UQ method is further illustrated on a propeller operating at high altitude. The efficacy of UQ with Computational Fluid Dynamics (CFD) and BEMT is compared and the most influential uncertain variables are found using Sobol's total order indices. As a result of the CFD-based uncertainty quantification studies, two major uncertain variables are identified, providing a direction for more computationally affordable UQ studies.

Sinh-arcsinh-normal distributions to add uncertainty to neural network regression tasks: Applications to tropical cyclone intensity forecasts

Abstract A simple method for adding uncertainty to neural network regression tasks in earth science via estimation of a general probability distribution is described. Specifically, we highlight the sinh-arcsinh-normal distributions as particularly well suited for neural network uncertainty estimation. The methodology supports estimation of heteroscedastic, asymmetric uncertainties by a simple modification of the network output and loss function. Method performance is demonstrated by predicting tropical cyclone intensity forecast uncertainty and by comparing two other common methods for neural network uncertainty quantification (i.e., Bayesian neural networks and Monte Carlo dropout). The simple approach described here is intuitive and applicable when no prior exists and one just wishes to parameterize the output and its uncertainty according to some previously defined family of distributions. The authors believe it will become a powerful, go-to method moving forward.

Automated resonance evaluation; Non-convex decomposition method for resonance regression and uncertainty quantification

This work serves as a proof of concept for an automated tool to assist in the evaluation of experimental neutron cross section data in the resolved resonance range. The resonance characterization problem is posed as a mixed integer nonlinear program (MINLP). Since the number of resonances present is unknown, the model must be able to be determine the number of parameters to properly characterize the cross section curve as well as calculate the appropriate values for those parameters. Due to the size of the problem and the nonconvex nature of the parameterization, the optimization formulation is too difficult to solve as whole. A novel method is developed to decompose the problem into smaller, solvable windows and then stitch them back together via parameter-cardinality and parameter-value agreement routines in order to achieve a global solution. A version of quantile regression is used to provide an uncertainty estimate on the suggested cross section that is appropriate with respect to the experimental data. The results demonstrate the model's ability to find the proper number of resonances, appropriate average values for the parameters, and an uncertainty estimation that is directly reflective of the experimental conditions. The use of synthetic data allows access to the solution, this is leveraged to build-up performance statistics and map the uncertainty driven by the experimental data to an uncertainty on the true cross section.

QUANTIFICATION AND PROPAGATION OF MODEL-FORM UNCERTAINTIES IN RANS TURBULENCE MODELING VIA INTRUSIVE POLYNOMIAL CHAOS

Undeterred by its inherent limitations, Reynolds-averaged Navier-Stokes (RANS) based modeling is still considered the most recognized approach for several computational fluid dynamics (CFD) applications. Recently, in the turbulence modeling community, quantification of model-form uncertainties in RANS has attracted significant interest. We present a stochastic RANS solver with an efficient implementation of the intrusive polynomial chaos (IPC) method in OpenFOAM. The stochastic solver quantifies and propagates the uncertainties associated with the output of the RANS model (eddy viscosity or Reynolds stress tensor). Two distinct high-dimensional variants of the uncertainties are considered, namely, the random eddy viscosity field (REVF) and the random Reynolds stress tensor field (RRSTF). The randomness is introduced in the approximated eddy viscosity field and the Reynolds stress tensor, while asserting the realizability. The stochastic RANS solver has been tested on various benchmark problems for RANS turbulence modeling. In this study, we discuss two important problems where the stochastic RANS solver shows significantly better performance than the traditional uncertainty quantification (UQ) methods. The first problem analyzed is the flow over periodic hills with a REVF, while the second stochastic problem considered is the flow in a square duct with a RRSTF. Along with the comparison for three different RANS turbulence models, a detailed analysis of the stochastic solver based on various influential model parameters is also presented. The IPC based stochastic solver demonstrated the potential to be used in the UQ analysis of further complex CFD applications, especially when a large number of deterministic simulations is not feasible, e.g., wind farm CFD simulations.

Significance of modes in the torus by topological data analysis

AbstractThis paper addresses the problem of identifying modes or density bumps in multivariate angular or circular data, which have diverse applications in fields like medicine, biology and physics. We focus on the use of topological data analysis and persistent homology for this task. Specifically, we extend the methods for uncertainty quantification in the context of a torus sample space, where circular data lie. To achieve this, we employ two types of density estimators, namely, the von Mises kernel density estimator and the von Mises mixture model, to compute persistent homology, and propose a scale‐space view for searching significant bumps in the density. The results of bump hunting are summarised and visualised through a scale‐space diagram. Our approach using the mixture model for persistent homology offers advantages over conventional methods, allowing for dendrogram visualisation of components and identification of mode locations. For testing whether a detected mode is really there, we propose several inference tools based on bootstrap resampling and concentration inequalities, establishing their theoretical applicability. Experimental results on SARS‐CoV‐2 spike glycoprotein torsion angle data demonstrate the effectiveness of our proposed methods in practice.

Data-Driven Method to Quantify Correlated Uncertainties

Polynomial chaos (PC) has been proven to be an efficient method for uncertainty quantification, but its applicability is limited by two strong assumptions: the mutual independence of random variables and the requirement of exact knowledge about the distribution of the random variables. We describe a new data-driven method for dealing with correlated multivariate random variables for uncertainty quantification that requires only observed data of the random variables. It is based on the transformation of correlated random variables into independent random variables. We use singular value decomposition as a transformation strategy that does not require information about the probability distribution. For the transformed random variables, we can construct the PC basis to build a surrogate model. This approach provides an additional benefit of quantifying high-dimensional uncertainties by combining our method with the analysis-of-variance (ANOVA) method. We demonstrate in several numerical examples that our proposed approach leads to accurate solutions with a much smaller number of simulations compared to the Monte Carlo method.

Ensemble Multi-Quantiles: Adaptively Flexible Distribution Prediction for Uncertainty Quantification.

We propose a novel, succinct, and effective approach for distribution prediction to quantify uncertainty in machine learning. It incorporates adaptively flexible distribution prediction of [Formula: see text] in regression tasks. This conditional distribution's quantiles of probability levels spreading the interval (0,1) are boosted by additive models which are designed by us with intuitions and interpretability. We seek an adaptive balance between the structural integrity and the flexibility for [Formula: see text], while Gaussian assumption results in a lack of flexibility for real data and highly flexible approaches (e.g., estimating the quantiles separately without a distribution structure) inevitably have drawbacks and may not lead to good generalization. This ensemble multi-quantiles approach called EMQ proposed by us is totally data-driven, and can gradually depart from Gaussian and discover the optimal conditional distribution in the boosting. On extensive regression tasks from UCI datasets, we show that EMQ achieves state-of-the-art performance comparing to many recent uncertainty quantification methods. Visualization results further illustrate the necessity and the merits of such an ensemble model.

Cycle-Consistency-Based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems

Uncertainty estimation is critical for numerous deep neural network (DNN) applications and has drawn increasing attention from researchers. In this study, we demonstrated an uncertainty quantification approach for DNNs used in inverse problems based on cycle consistency. We built forward–backward cycles using the available physical forward model and a trained DNN solving the inverse problem at hand and accordingly derived uncertainty estimators through regression analysis on the consistency of these forward–backward cycles. We theoretically analyzed the cycle consistency metrics and derived their relationship with the uncertainty, bias, and robustness of neural network inference. To demonstrate the effectiveness of these cycle-consistency-based uncertainty estimators, we classified corrupted and out-of-distribution input image data using widely used image deblurring and super-resolution neural networks as test beds. Our blind tests demonstrated that our method surpassed other models in detecting previously unseen data corruption and distribution shifts. This study provides a simple-to-implement and rapid uncertainty quantification method that can be universally applied to various neural networks used to solve inverse problems.

Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation.

Deep learning based methods for automatic organ segmentation have shown promise in aiding diagnosis and treatment planning. However, quantifying and understanding the uncertainty associated with model predictions is crucial in critical clinical applications. While many techniques have been proposed for epistemic or model-based uncertainty estimation, it is unclear which method is preferred in the medical image analysis setting. This paper presents a comprehensive benchmarking study that evaluates epistemic uncertainty quantification methods in organ segmentation in terms of accuracy, uncertainty calibration, and scalability. We provide a comprehensive discussion of the strengths, weaknesses, and out-of-distribution detection capabilities of each method as well as recommendations for future improvements. These findings contribute to the development of reliable and robust models that yield accurate segmentations while effectively quantifying epistemic uncertainty.

Optimization and Uncertainty Quantification Method for Reservoir Stimulation through Carbonate Acidizing.

Reservoir stimulation is a widely used technique in the oil and gas industry for increasing the productivity of hydrocarbon reservoirs, most notably in carbonate formations. This work aims to develop an optimization workflow under uncertainty for matrix acidizing. A reactive transport model is implemented in a finite-element framework to simulate the initiation and propagation of dissolution channels in porous carbonate rock. The model is verified using an analytical solution. We utilize surrogate modeling based on polynomial chaos expansion (PCE) and Sobol indices to identify the most significant parameters. We investigate the effect of varying 12 identified parameters on the efficiency of the stimulation process using dimensionless groups, including the Damköhler, Peclet, and acid capacity numbers. Furthermore, the surrogate model reproduces the physics-based results accurately, including the dissolution channels, the pore volume to breakthrough, and the effective permeability of the stimulated rock. The developed workflow assesses how uncertainties propagate to the model's response, where the surrogate model is used to calculate the univariate effect. The global sensitivity analysis shows that the acid capacity number is the most significant parameter for the pore volume to breakthrough with the highest Sobol index. The marginal effect calculated for the individual parameter confirms the results from Sobol indices. This work provides a systematic workflow for uncertainty analysis and optimization applied to the processes of rock stimulation. Characterizing the impact of uncertainty provides physical insights and a better understanding of the matrix acidizing process.

Nuclear Data Sensitivity and Uncertainty Study for the Pressurized Water Reactor (PWR) Benchmark Using RMC and SCALE

In order to improve the safety and economy of nuclear reactors, it is necessary to analyze the sensitivity and uncertainty (S/U) of the nuclear data. The capabilities of S/U analysis has been developed in the Reactor Monte Carlo code RMC, using the iterated fission probability (IFP) method and the superhistory method. In this paper, the S/U capabilities of RMC are applied to a typical PWR benchmark B&W’s Core XI, and compared with the multigroup and continuous-energy S/U capabilities in the SCALE code system. The S/U results of the RMC-IFP method and the RMC-superhistory method are compared with TSUNAMI-CE/MG in SCALE. The sensitivity results and the uncertainty results of major nuclides that contribute a lot to the uncertainties in keff are in good agreement in both RMC and SCALE. The RMC-superhistory method has the same precision as the IFP method, but it reduces the memory footprint by more than 95% and only doubles the running time. The superhistory method has obvious advantages when there are many nuclides and reaction types to be analyzed. In addition, the total uncertainties in the keff of the first-order uncertainty quantification method are compared with the stochastic sampling method, and the maximum relative deviation of total uncertainties in the keff is 8.53%. Verification shows that the capabilities of S/U analysis developed in the RMC code has good accuracy.

Implementation of Multidisciplinary and Multifidelity Uncertainty Quantification Methods for Sonic Boom Prediction

To advance a supersonic aircraft design process, an uncertainty quantification study is conducted for sonic boom prediction while considering uncertainties associated with flight and atmospheric conditions. The uncertainty quantification process is implemented within a multidisciplinary analyses framework and assisted with a multifidelity surrogate model based approach. The sonic boom prediction framework requires input from the flowfield pressure distribution solution to generate the near-field pressure signature of the aircraft, which is then propagated throughout the atmosphere to the ground by using aeroacoustic methods. The open-source SU2 suite is employed as a high-fidelity flow solver tool to obtain the aerodynamic solution, while in-house postprocessing scripts are developed to generate the required near-field pressure signature. For low-fidelity flow analysis, A502 PAN AIR, a higher-order panel code which solves flows around slender bodies in low angles of attack for subsonic and supersonic regimes, is used. For nonlinear aeroacoustic propagation, NASA Langley Research Center’s code sBOOM exploits the near-field pressure signature for both high-fidelity and low-fidelity sonic boom calculations. Efficient uncertainty quantification tools are developed in house by implementing multifidelity polynomial chaos expansion and multifidelity Monte Carlo methods. Several flight and atmospheric parameters are selected to include randomness where these uncertainties are propagated into the sonic boom loudness prediction of the JAXA wing-body model, which is a low boom aircraft. Finally, an overall assessment of the multifidelity uncertainty quantification methods is presented in terms of efficiency and numerical accuracy.

A non-intrusive frequency normalisation approach for uncertain response analysis of nonlinear dynamic systems

This paper proposes a non-intrusive technique based on the frequency normalisation for non-intrusive propagations of parameter uncertainties in nonlinear mechanical systems. The multi-solution dilemma found in resonance areas, which prohibits successful applications of uncertainty quantification methods, is alleviated by the additional normalised frequency measure. The spurious peaks around the nonlinear peaks and discontinuous points, known as the Gibbs phenomenon, due to parameter variabilities are resolved as well. The proposed technique coupled with the non-intrusive surrogate modelling is applied to two deliberately constructed nonlinear dynamic systems, i.e., a mass-spring system with interactive cubic nonlinearity and a piece-wise rotor/stator contact problem in rotating machines. The two systems exhibit complex amplitude-frequency response characteristics featured by the softening and hardening effects. Numerous case investigations show the effectiveness of the proposed normalisation method for uncertainty analysis of complex nonlinear vibration systems and its working principle is demonstrated in detail via examples. Accuracy tests against the traditional sampling methods are carried out in both systems. The proposed technique can be easily generalised to other nonlinear systems for random and non-random uncertainty propagations because it works non-intrusively and permits users to choose arbitrary nonlinear tools.