Uncertainty Quantification Methods Research Articles

A bootstrap-based stochastic subspace method for modal parameter identification and uncertainty quantification of high-rise buildings

This paper proposes a bootstrap-based stochastic subspace method for modal parameter identification and uncertainty quantification of high-rise buildings. Firstly, the stochastic subspace method in combination with the bootstrap technique enables the estimation of multiple sets of modal parameters from raw data series. Then, a bootstrap-based stabilization diagram is used to extract the physical modes. Finally, the modal identification and associated uncertainty quantification results are determined via statistical analysis. Through a numerical study of high-rise buildings, the performance of the proposed method is validated, demonstrating that it can provide reliable modal parameter identification and uncertainty quantification as well as has good noise immunity. Furthermore, the developed approach is employed to identify modal parameters of a 600-m-tall skyscraper during a typhoon, proving its applicability to field measurements and structural health monitoring of high-rise buildings. This paper aims to present a novel tool for modal parameter identification and associated uncertainty quantification of high-rise buildings.

A comprehensive study on seismic dynamic responses of stochastic structures using sparse grid-based polynomial chaos expansion

The seismic response is a critical aspect of structural engineering, as it directly influences the design and safety evaluation of such systems. Structural seismic response becomes more complicated when unavoidable uncertainties in the structure are considered. Therefore, solving and analyzing the stochastic dynamic response of structures efficiently and accurately is of great significance. Various uncertainty quantification methods such as the stochastic finite element method, Monte Carlo method, and polynomial chaos expansion method have made significant progress in this field. However, these methods do not provide a comprehensive study of the seismic response of stochastic structures and efficiency problems rapidly emerge as the number of random variables increases. This paper delivers a comprehensive study of the seismic dynamic response of space truss and reticulated shell structure considering uncertainties in material properties, geometry, and loading. To this end, the finite element model of the structure was first modeled and the deterministic solutions were obtained. Then, polynomial surrogates are modeled for the natural frequency, mode participation factor, mode superposition response, and transient response of the structure under stochastic uncertainties. The sparse grid technique is used to select sparse quadrature points, which overcomes the curse of dimensionality caused by full grid integration. The results demonstrate that the sparse grid-based polynomial chaos expansion is more efficient. Uncertainty quantification reveals the uncertainty of modal analysis, response spectrum analysis, and transient response analysis of space truss and reticulated shell under seismic excitation. In summary, this study provides valuable insights and an efficient and accurate uncertainty quantification tool for the seismic dynamic analysis of stochastic structures, ultimately contributing to safer and more robust structural designs.

An uncertainty quantification framework for agent-based modeling and simulation in networked anagram games

ABSTRACT In a networked anagram game, players are provided letters with possible actions of requesting letters from their neighbours, replying to letter requests, or forming words. The objective is to form as many words as possible as a team. The experimental data show that behaviours among players can vary significantly. However, simulations using agent-based models (ABM) in the literature often have not incorporated proper uncertainty quantification methods to characterise diverse behaviours of players. In this work, we propose an uncertainty quantification framework to build, exercise, and evaluate agent behaviour models and simulations for networked group anagram games. Specifically, using the data of game experiments, the proposed framework considers the clustering of game players based on their performance to reflect players’ heterogeneity. Moreover, we also quantify uncertainty within each cluster through statistical modelling and inference. Numerical studies of networked game configurations are conducted to demonstrate the merits of the proposed framework.

Ground Heat Flux Reconstruction Using Bayesian Uncertainty Quantification Machinery and Surrogate Modeling

AbstractGround heat flux (G0) is a key component of the land‐surface energy balance of high‐latitude regions. Despite its crucial role in controlling permafrost degradation due to global warming, G0 is sparsely measured and not well represented in the outputs of global scale model simulation. In this study, an analytical heat transfer model is tested to reconstruct G0 across seasons using soil temperature series from field measurements, Global Climate Model, and climate reanalysis outputs. The probability density functions of ground heat flux and of model parameters are inferred using available G0 data (measured or modeled) for snow‐free period as a reference. When observed G0 is not available, a numerical model is applied using estimates of surface heat flux (dependent on parameters) as the top boundary condition. These estimates (and thus the corresponding parameters) are verified by comparing the distributions of simulated and measured soil temperature at several depths. Aided by state‐of‐the‐art uncertainty quantification methods, the developed G0 reconstruction approach provides novel means for assessing the probabilistic structure of the ground heat flux for regional permafrost change studies.

Extraction of comprehensive feature for active target detection using two hand-crafted acoustic features

We propose a model for creating a new feature suitable for classifying targets and non-targets using a small active sonar data. The proposed model extracts a comprehensive feature from two different hand-crafted features using deep-learning (DL). To extract comprehensive feature, we contrived a complementary learning module (CLM), which consists of two stages. In the first stage, a transformer encoder was adopted to reinforce the input features and then complementarily learned through attention mechanism in the following stage. The output features from the CLM are passed through a shallow CNN to classify targets and non-targets. In addition, an uncertainty quantification method is proposed using two CLMs to quickly add new samples acquired in the real ocean dataset. Using the decisions from two CLMs, we calculate the variance and determine which samples from the entire dataset are worthy of prioritized addition. Two real-ocean datasets were used to examine the model generalization and compare the classification results with conventional DL models. The generalization performance of the proposed model was comparable or superior to other DL models and uncertainty quantificaion method improved the generalization performance. [Work supported by Korea Research Institute for defense Technology planning and advancement(KRIT)—Grant funded by the Korea government (DAPA—Defense Acquisition Program Administration) (No. KRIT-CT-23-026, Integrated Underwater Surveillance Research Center for Adapting Future Technologies, 2024)]

Uncertainty Quantification of Capacitor Switching Transient Location Using Machine Learning

Identification of capacitor switching transient location provides valuable insight into the state of the associated equipment. Machine learning (ML) models, and convolutional neural networks (CNNs) in particular, have demonstrated remarkable performance in signal location. However, ML models are data driven whose predictions are affected by noise in data and may also suffer from large extrapolation errors when applied to new conditions. Uncertainty quantification (UQ) is necessary to ensure model trustworthiness and avoid overconfident predictions in extrapolation. In this work, we propose a novel UQ method, called PI3NN, to quantify prediction uncertainty of ML models and integrate the method with CNNs for transient source location. PI3NN calculates Prediction Intervals by training 3 Neural Networks and uses root-finding methods to determine the interval precisely. Additionally, PI3NN can identify out-of-distribution (OOD) data in a nonstationary condition to avoid overconfident prediction. Results indicate that with PI3NN, transient signals are not only correctly identified, but when said signals are subject to corruptions characteristic of an actual power monitoring system (e.g non-ideal sensors), the model recognizes when it is uncertain about its predictions, effectively letting the user know when to accept or discard the results.

Uncertainty Quantification of Machine Learning Model Performance via Anomaly-Based Dataset Dissimilarity Measures

The use of Machine Learning (ML) models as predictive tools has increased dramatically in recent years. However, data-driven systems (such as ML models) exhibit a degree of uncertainty in their predictions. In other words, they could produce unexpectedly erroneous predictions if the uncertainty stemming from the data, choice of model and model parameters is not taken into account. In this paper, we introduce a novel method for quantifying the uncertainty of the performance levels attained by ML classifiers. In particular, we investigate and characterize the uncertainty of model accuracy when classifying out-of-distribution data that are statistically dissimilar from the data employed during training. A main element of this novel Uncertainty Quantification (UQ) method is a measure of the dissimilarity between two datasets. We introduce an innovative family of data dissimilarity measures based on anomaly detection algorithms, namely the Anomaly-based Dataset Dissimilarity (ADD) measures. These dissimilarity measures process feature representations that are derived from the activation values of neural networks when supplied with dataset items. The proposed UQ method for classification performance employs these dissimilarity measures to estimate the classifier accuracy for unseen, out-of-distribution datasets, and to give an uncertainty band for those estimates. A numerical analysis of the efficacy of the UQ method is conducted using standard Artificial Neural Network (ANN) classifiers and public domain datasets. The results obtained generally demonstrate that the amplitude of the uncertainty band associated with the estimated accuracy values tends to increase as the data dissimilarity measure increases. Overall, this research contributes to the verification and run-time performance prediction of systems composed of ML-based elements.

Stochastic modelling of out-of-autoclave epoxy composite cure cycles under uncertainty

Thermoset polymers and composites are subject to several sources of uncertainty which can produce a range of cure outcomes. Recent research into stochastically modelled thermoset cure has indicated that accounting for raw material and process uncertainty can model this range of expected output parameters. However, the uncertainty quantification methods are highly test-intensive, and the results of the simulations have been validated with limited experimental data. This study proposes a simple approach to cure kinetics uncertainty quantification that can be applied to any cure kinetics model without the need for additional testing. Stochastic cure kinetics and temperature conditions for a popular out-of-autoclave carbon fibre/epoxy prepreg were used to produce output distribution functions for key cure events, and the results were validated using data from ten cure replicates. The quantified variation expected from the cure of this prepreg resulted in processing recommendations to ensure quality metrics are met during processing.

Steady-State Transonic Flowfield Prediction via Deep-Learning Framework

This paper focuses on the development of a deep-learning framework for predicting distributed quantities around aircraft flying in the transonic regime. These quantities play a crucial role in determining aerodynamic loads and conducting aeroelastic analysis. Angle of attack and Mach number are chosen as the two independent parameters for the reduced-order models. A comparative assessment is conducted between the proposed deep-learning framework and the proper orthogonal decomposition approach to identify the strengths and weaknesses of each method. The accuracy of the data-driven machine-learning method in modeling steady-state transonic aerodynamics is assessed against three benchmark cases of three-dimensional test cases: Benchmark Super Critical Wing and ONERA M6 wings, and the wing–body Common Research Model configuration. Despite the challenges of the analyzed scenarios, promising results are obtained for each test case, showing the effectiveness of the model implemented. Furthermore, the paper demonstrates the application of the method for aeroelastic analysis and uncertainty quantification. This quantifies the robustness and versatility of the implemented model.

How to evaluate uncertainty estimates in machine learning for regression?

As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Learnable quantile polynomial chaos expansion: An uncertainty quantification method for interval reliability analysis

Various uncertainties like random input and data noise in aerospace engineering affect flight vehicle safety, making uncertainty quantification important for improving system reliability. Polynomial chaos expansion (PCE) is a versatile method for quantifying stochastic systems’ uncertainty caused by random input, while it cannot quantify data uncertainty caused by noise. To solve this problem, a novel uncertainty quantification method, learnable quantile PCE (LQPCE), is proposed for interval reliability analysis. Unlike the traditional PCE, the multi-dimensional orthonormal basis is constructed for combination input rather than only random variables, and the expansion coefficients are parameterized to learnable weights, ensuring the LQPCE model can capture data noise’s features. The LQPCE Pinball loss function is constructed to learn parameterized expansion coefficients by quantile level iterative sampling, achieving accurate prediction and data uncertainty quantification. The quantified data uncertainty is used for interval reliability analysis, calculating more reliable results about the influence of input uncertainty on stochastic systems’ reliability. Three numerical examples and one engineering application validate the LQPCE method’s effectiveness. The results show that the LQPCE method can quantify data uncertainty accurately and obtain credible interval reliability.

Investigation on uncertainty quantification of transonic airfoil using compressive sensing greedy reconstruction algorithms

Uncertainty quantification constructs the stochastic responses of system output under uncertainties. Traditional uncertainty quantification methods such as Monte Carlo and Full-Rank Polynomial Chaos Expansion need large amounts of samples. Compressive Sensing has the advantages of requiring few observation samples and sparse reconstruction. By combining Compressive Sensing with uncertainty quantification, the samples in demand can be reduced. Nevertheless, the sparse reconstruction algorithms directly affect predicting accuracy, especially in a practical aerodynamic scenario. To compare the effect of two l0-minimization based greedy reconstruction algorithms, Orthogonal Matching Pursuit and Subspace Pursuit, this study adopts them into uncertainty quantification of RAE2822 airfoil considering geometrical uncertainties. The Polynomial Chaos Expansion is used to realize sparse representation and Structured Random Observation Matrix is constructed by randomly selecting from Gaussian quadrature set. The two greedy reconstruction methods are compared to Monte Carlo and Full-Rank Probabilistic Collocations on convergence, required samples and accuracy. The results show that both methods could achieve similar accuracy as Monte Carlo with much fewer samples. Besides, Orthogonal Matching Pursuit is able to precisely reconstruct original signal with proper sparsity estimation, while Subspace Pursuit provides more robust prediction for different cases. This study presents an efficient approach that integrates compressive sensing into uncertainty quantification of airfoils.

Uncertainty quantification for structural response field with ultra-high dimensions

The structural response field is crucial for understanding mechanical behavior, especially under uncertain conditions. However, current uncertainty quantification predominantly address one-dimensional output under multi-dimensional input. For the structural full field response with ultra-high dimension, quantifying corresponding uncertainties becomes a formidable task. This paper introduces an efficient uncertainty quantification method for structural response field. Initially, the manifold learning is introduced to transform uncertainty quantification of ultra-high-dimensional responses into the quantification of low-dimensional manifold projections. Subsequently, polynomial chaos expansions facilitate uncertainty propagation, allowing for the effective evaluation of statistical moments for each projection. Leveraging the established manifold structure enables obtaining analytical solutions for statistical moments of full field responses without additional computational costs. In the absence of prior information, the derivative λ-PDF is utilized to model uncertainty of response at any position, so as to realize the visualization of uncertainty. The proposed uncertainty quantification for structural response field provides a non-intrusive analysis framework, which can conveniently deal with structural multi-field coupling problems. Finally, three engineering examples involving multi field coupling are presented to demonstrate the effectiveness and practicability of proposed uncertainty quantification of structural response field.

Uncertainty Quantification of Vibroacoustics with Deep Neural Networks and Catmull–Clark Subdivision Surfaces

This study proposes an uncertainty quantification method based on deep neural networks and Catmull–Clark subdivision surfaces for vibroacoustic problems. The deep neural networks are utilized as a surrogate model to efficiently generate samples for stochastic analysis. The training data are obtained from numerical simulation by coupling the isogeometric finite element method and the isogeometric boundary element method. In the simulation, the geometric models are constructed with Catmull–Clark subdivision surfaces, and meantime, the physical fields are discretized with the same spline functions as used in geometric modelling. Multiple deep neural networks are trained to predict the sound pressure response for various parameters with different numbers and dimensions in vibroacoustic problems. Numerical examples are provided to demonstrate the effectiveness of the proposed method.

A data-driven polynomial chaos method for uncertainty quantification of a subsonic compressor cascade with stagger angle errors

The probability-based uncertainty quantification (UQ) methods require a large amount of sampled data to construct the probability distribution of uncertain input parameters. However, it is a common situation that only limited and scarce sampled data are available in engineering applications due to expensive tests. In the present paper, the Data-Driven Polynomial Chaos (DDPC) method is adopted, which can propagate input uncertainty in the case of scarce sampled data. The calculation accuracy and convergence of the self-developed DDPC method are validated by a nonlinear test function. Subsequently, the DDPC method is applied to investigate the uncertain impact of stagger angle errors on the aerodynamic performance of a subsonic compressor cascade. A family of manufacturing error data of stagger angles was obtained from the real compressor blades. Based on the limited measurement data, the DDPC method combined with Computational Fluid Dynamics (CFD) simulation is employed to quantify the performance impact of the compressor cascade. The results show that the performance dispersion under off-design conditions is more prominent than that under design conditions. The actual aerodynamic performance deviating from the nominal performance is not a small probability event, and the probability of deviating from the nominal loss coefficient and exit flow angle by more than 1% can reach up to 47.6% and 36.8% under high incidence i = 7°. Detailed analysis shows that stagger angle errors have a significant effect on the flow state near the leading edge, resulting in variations in separation bubble size and boundary layer thickness.

Code and Data Repository for A Shrinkage Approach to Improve Direct Bootstrap Resampling Under Input Uncertainty

The goal of this software is to demonstrate the shrinkage bootstrap methods for input uncertainty quantification proposed in "A Shrinkage Approach to Improve Direct Bootstrap Resampling under Input Uncertainty" by Eunhye Song, Henry Lam, and Russell Barton accepted for publication at INFORMS Journal on Computing.

Uncertainties and confidence in stratospheric aerosol injection modelling: a systematic literature review

Abstract Model projections performed to evaluate the efficacy and impacts of solar geoengineering interventions, such as Stratospheric Aerosol Injection (SAI), include multiple sources of uncertainty, namely scenario, model, and natural variability uncertainty. It is well accepted that a quantitative uncertainty assessment related to SAI modelling is required to provide robust and policy-relevant information on SAI. This study investigates how and to what extent articles using a climate modelling approach on SAI quantify and communicate uncertainty sources. We conducted a systematic literature review of a sample of 60 peer-reviewed articles in order to (i) analyse whether uncertainties were addressed, and if yes, which methods were used to characterize uncertainties, and (ii) study how the articles communicated assumptions and limits that contribute to the estimation of confidence in the used models and the resulting projections. We present statistics on the uncertainty quantification methods used in the articles and we discuss the vocabulary employed for conveying these uncertainties and model confidence. In the studied article sample, the attention paid to uncertainty estimations in the SAI literature increased with time, and overall, uncertainties were treated using a variety of methods. Model confidence was not always explicitly communicated as the models used are already tested in the literature and their strengths and weaknesses are known to the community although this is often implicit. Our results show that it is currently difficult to perform global, quantitative assessments of uncertainty related to SAI research, in line with recent review reports on solar geoengineering.

Uncertainty Quantification of Soil Organic Carbon Estimation from Remote Sensing Data with Conformal Prediction

Soil organic carbon (SOC) contents and stocks provide valuable insights into soil health, nutrient cycling, greenhouse gas emissions, and overall ecosystem productivity. Given this, remote sensing data coupled with advanced machine learning (ML) techniques have eased SOC level estimation while revealing its patterns across different ecosystems. However, despite these advances, the intricacies of training reliable and yet certain SOC models for specific end-users remain a great challenge. To address this, we need robust SOC uncertainty quantification techniques. Here, we introduce a methodology that leverages conformal prediction to address the uncertainty in estimating SOC contents while using remote sensing data. Conformal prediction generates statistically reliable uncertainty intervals for predictions made by ML models. Our analysis, performed on the LUCAS dataset in Europe and incorporating a suite of relevant environmental covariates, underscores the efficacy of integrating conformal prediction with another ML model, specifically random forest. In addition, we conducted a comparative assessment of our results against prevalent uncertainty quantification methods for SOC prediction, employing different evaluation metrics to assess both model uncertainty and accuracy. Our methodology showcases the utility of the generated prediction sets as informative indicators of uncertainty. These sets accurately identify samples that pose prediction challenges, providing valuable insights for end-users seeking reliable predictions in the complexities of SOC estimation.

Predicting primate tongue morphology based on geometrical skull matching. A first step towards an application on fossil hominins.

As part of a long-term research project aiming at generating a biomechanical model of a fossil human tongue from a carefully designed 3D Finite Element mesh of a living human tongue, we present a computer-based method that optimally registers 3D CT images of the head and neck of the living human into similar images of another primate. We quantitatively evaluate the method on a baboon. The method generates a geometric deformation field which is used to build up a 3D Finite Element mesh of the baboon tongue. In order to assess the method's ability to generate a realistic tongue from bony structure information alone, as would be the case for fossil humans, its performance is evaluated and compared under two conditions in which different anatomical information is available: (1) combined information from soft-tissue and bony structures; (2) information from bony structures alone. An Uncertainty Quantification method is used to evaluate the sensitivity of the transformation to two crucial parameters, namely the resolution of the transformation grid and the weight of a smoothness constraint applied to the transformation, and to determine the best possible meshes. In both conditions the baboon tongue morphology is realistically predicted, evidencing that bony structures alone provide enough relevant information to generate soft tissue.

A quantitative comparison study for structural flexibility identification using Accelerometric and computer vision-based vibration data

For civil engineering structures, the dynamics-based structural health monitoring is commonly based on inverse analysis of structural acceleration response data to identify the structural parameters. As the measurement of structural displacement response becomes more feasible and cost-effective thanks to advances in computer vision techniques, a comparison of the displacement and acceleration response data for structural identification has become of considerable interest among researchers. This paper presents a comprehensive analysis and comparison of structural flexibility identification based on displacement versus acceleration response data through uncertainty quantification technique. Firstly, theoretical derivation reveals that the energy spectral density of acceleration response is equal to the fourth power of the frequency of the energy spectral density of displacement response, which affects the distribution of measurement noise across different frequency bands. Secondly, the uncertainty quantification method for modal flexibility from subspace-based system identification is utilized to compare the derivation process of covariance estimation on modal flexibility. Finally, numerical analysis and experimental example are carried out to compare the structural flexibility identification results when using the displacement versus acceleration. It shows that more precise identification results of lower-order modal parameters are obtained when using displacement, which further improves the precision of modal flexibility identification. The displacement is more suitable for identifying the modal flexibility when the measurement noises are at the same level.