Uncertainty Quantification Methods Research Articles

Uncertainty-aware deep learning-based CAD system for breast cancer classification using ultrasound and mammography images

ABSTRACT Breast cancer is one of the most common types of cancer in women. Early and accurate diagnosis of breast cancer can increase the treatment chances and decrease the mortality rate. Thus, the development of accurate and reliable Computer-Aided Diagnosis (CAD) systems using breast cancer images is an immediate priority for early diagnosis. Deep learning methods have had a widespread role in CAD systems. In spite of the advantages, most deep learning-based CADs cannot quantify the uncertainty of their predictions. Uncertainty in the predicted results from a CAD system might endanger people’s life in these crucial steps for breast cancer classification. Therefore, in this study, the Monte Carlo Dropout method is utilised as an uncertainty quantification method. Furthermore, differently from most existing studies, we present an uncertainty-aware deep learning model which can classify accurately from different types of images, namely, mammograms and ultrasound. The results show the proposed model achieved the accuracy of 99.95% in mammogram dataset, namely, DDSM and 98.40% and 91.34% in ultrasound datasets, namely, BUSM and BUSI, respectively. Finally, since we found the proposed model shows high uncertainty in misclassifications, we suggested to use the uncertainty values to find appropriate subsets for further inspection to increase the performance.

A Meta-Model Architecture and Elimination Method for Uncertainty Modeling

Uncertainty exists widely in various fields, especially in industrial manufacturing. From traditional manufacturing to intelligent manufacturing, uncertainty always exists in the manufacturing process. With the integration of rapidly developing intelligent technology, the complexity of manufacturing scenarios is increasing, and the postdecision method cannot fully meet the needs of the high reliability of the process. It is necessary to research the pre-elimination of uncertainty to ensure the reliability of process execution. Here, we analyze the sources and characteristics of uncertainty in manufacturing scenarios and propose a meta-model architecture and uncertainty quantification (UQ) framework for uncertainty modeling. On the one hand, our approach involves the creation of a meta-model structure that incorporates various strategies for uncertainty elimination (UE). On the other hand, we develop a comprehensive UQ framework that utilizes quantified metrics and outcomes to bolster the UE process. Finally, a deterministic model is constructed to guide and drive the process execution, which can achieve the purpose of controlling the uncertainty in advance and ensuring the reliability of the process. In addition, two typical manufacturing process scenarios are modeled, and quantitative experiments are conducted on a simulated production line and open-source data sets, respectively, to illustrate the idea and feasibility of the proposed approach. The proposed UE approach, which innovatively combines the domain modeling from the software engineering field and the probability-based UQ method, can be used as a general tool to guide the reliable execution of the process.

Deep Learning and Likelihood Approaches for Viral Phylogeography Converge on the Same Answers Whether the Inference Model Is Right or Wrong.

Analysis of phylogenetic trees has become an essential tool in epidemiology. Likelihood-based methods fit models to phylogenies to draw inferences about the phylodynamics and history of viral transmission. However, these methods are often computationally expensive, which limits the complexity and realism of phylodynamic models and makes them ill-suited for informing policy decisions in real-time during rapidly developing outbreaks. Likelihood-free methods using deep learning are pushing the boundaries of inference beyond these constraints. In this paper, we extend, compare, and contrast a recently developed deep learning method for likelihood-free inference from trees. We trained multiple deep neural networks using phylogenies from simulated outbreaks that spread among 5 locations and found they achieve close to the same levels of accuracy as Bayesian inference under the true simulation model. We compared robustness to model misspecification of a trained neural network to that of a Bayesian method. We found that both models had comparable performance, converging on similar biases. We also implemented a method of uncertainty quantification called conformalized quantile regression that we demonstrate has similar patterns of sensitivity to model misspecification as Bayesian highest posterior density (HPD) and greatly overlap with HPDs, but have lower precision (more conservative). Finally, we trained and tested a neural network against phylogeographic data from a recent study of the SARS-Cov-2 pandemic in Europe and obtained similar estimates of region-specific epidemiological parameters and the location of the common ancestor in Europe. Along with being as accurate and robust as likelihood-based methods, our trained neural networks are on average over 3 orders of magnitude faster after training. Our results support the notion that neural networks can be trained with simulated data to accurately mimic the good and bad statistical properties of the likelihood functions of generative phylogenetic models.

A novel uncertainty quantification method for determining deformations and reliabilities of stochastic laminated composite plates with geometric nonlinearity

Stochastic finite element method (SFEM) for uncertainty quantification is widely applied for analysis of structures with intrinsic randomness. For determining the geometrically nonlinear deformations of laminated composite plates with random fields, the existing intrusive SFEMs have the limitations of low applicability, insufficient accuracy, or low efficiency. To this end, this paper proposes a novel and efficient non-intrusive SFEM incorporating direct probability integral method to achieve probability density functions (PDFs) of stochastic responses and reliabilities of laminated composite plates with geometric nonlinearity. Firstly, the von Kármán strain-displacement relation based on the third-order shear deformation theory is employed to model the geometric nonlinearity of laminated plates. The random field is discretized via Karhunen–Loève expansion. Secondly, the probability density integral equation (PDIE) is derived from the new perspective of probability conservation. The proposed non-intrusive SFEM decouples the equilibrium equation and PDIE to compute the response PDFs and reliabilities of uncertain laminated composite plates in a unified way. Moreover, the criterion which can judge the applicability of geometrically nonlinear theory is suggested for performing uncertainty quantification. Finally, comparisons of the results in terms of Monte Carlo simulation and the literature demonstrate the high accuracy and efficiency of the proposed method. For stochastic laminated plates, the response statistical moments vary nonlinearly with linear increase of load amplitude due to geometric nonlinearity, the deflection variability increases and structural reliability decreases with the increase in variability and correlation length of random field, and the stacking angle significantly affects the stochastic nonlinear deflections and reliabilities.

Mitigating Diagnostic Errors in Lung Cancer Classification: A Multi-Eyes Principle to Uncertainty Quantification.

In radiology, particularly in lung cancer diagnosis, diagnostic errors and cognitive biases pose substantial challenges. These issues, including perceptual errors, interpretive mistakes, and cognitive biases such as anchoring and premature closure, are often unnoticed by experienced radiologists. To address these challenges, we propose the Multi-Eyes principle approach, which utilises multiple deep learning models to reduce bias and potentially improve diagnostic accuracy. Inspired by the Four-Eyes principle in business and cybersecurity, this methodology employs various 3D and 2D (for validation) deep learning architectures and three uncertainty quantification techniques: Monte Carlo Dropout, Deep Ensemble, and Ensemble Monte Carlo Dropout. Each model functions as an independent reviewer, similar to blind reviews. With entropy selected as the uncertainty measurement, it is averaged, followed by ensemble averaging of predictions. The effectiveness of this approach was demonstrated using the LIDC-IDRI dataset for lung cancer classification. Statistical analysis of the uncertainty's distribution reveals that with more models, uncertainty in incorrect predictions becomes more peaked and left skewed, indicating consensus on uncertainty levels. This results in accuracy and F1 score improvements, even with the best performing model, addressing overconfidence in single-model systems. These findings highlight the potential of the Multi-Eyes principle to significantly improve diagnostic performance in computer-aided diagnostic systems. Future research may explore different uncertainty quantification methods and feedback mechanisms for further advancement.

Estimation and Analysis of Slice Propagation Uncertainty in 3D Anatomy Segmentation.

Supervised methods for 3D anatomy segmentation demonstrate superior performance but are often limited by the availability of annotated data. This limitation has led to a growing interest in self-supervised approaches in tandem with the abundance of available unannotated data. Slice propagation has emerged as a self-supervised approach that leverages slice registration as a self-supervised task to achieve full anatomy segmentation with minimal supervision. This approach significantly reduces the need for domain expertise, time, and the cost associated with building fully annotated datasets required for training segmentation networks. However, this shift toward reduced supervision via deterministic networks raises concerns about the trustworthiness and reliability of predictions, especially when compared with more accurate supervised approaches. To address this concern, we propose integrating calibrated uncertainty quantification (UQ) into slice propagation methods, which would provide insights into the model's predictive reliability and confidence levels. Incorporating uncertainty measures enhances user confidence in self-supervised approaches, thereby improving their practical applicability. We conducted experiments on three datasets for 3D abdominal segmentation using five UQ methods. The results illustrate that incorporating UQ improves not only model trustworthiness but also segmentation accuracy. Furthermore, our analysis reveals various failure modes of slice propagation methods that might not be immediately apparent to end-users. This study opens up new research avenues to improve the accuracy and trustworthiness of slice propagation methods.

Analysis of geometric uncertainties in 3D thermo-fluid problems solved by RBF-FD meshless method

The tolerances of production processes can lead to uncertainties in the behaviour and in the features of the manufactured products. From the point of view of the design of engineering components it is therefore of valuable practical interest to be able to quantify such uncertainties as well as the expected, i.e., averaged, performances. Such uncertainty quantification is carried out in this work by means of the Non-Intrusive Polynomial Chaos (PC) method in order to estimate the propagation of geometrical uncertainties of the boundaries, i.e., when the boundaries are described by stochastic variables. Existing deterministic solvers can be used with the PC method because of its non-intrusive formulation, allowing an accurate and practical prediction of the random response through a simple set of deterministic response simulations. The Radial Basis Function Finite Differences (RBF-FD) method is employed as a black box solver for the computation of the required set of responses defined over deterministic boundaries. The RBF-FD method belongs to the class of meshless methods which do not require a computational mesh/grid, therefore its main capability is to easily deal with practical problems defined over complex-shaped domains. The geometrical flexibility of the RBF-FD method is even more advantageous when coupled to the Non-Intrusive PC method for uncertainty quantification since different deterministic solutions over different geometries are required. The applicability of the proposed approach to practical problems is presented through the prediction of geometric uncertainty effects for a steady-state forced convection problem in a 3D complex-shaped domain.

Uncertainty Quantification and Interpretability for Clinical Trial Approval Prediction.

Background: Clinical trial is a crucial step in the development of a new therapy (e.g., medication) and is remarkably expensive and time-consuming. Forecasting the approval of clinical trials accurately would enable us to circumvent trials destined to fail, thereby allowing us to allocate more resources to therapies with better chances. However, existing approval prediction algorithms did not quantify the uncertainty and provide interpretability, limiting their usage in real-world clinical trial management. Methods: This paper quantifies uncertainty and improves interpretability in clinical trial approval predictions. We devised a selective classification approach and integrated it with the Hierarchical Interaction Network, the state-of-the-art clinical trial prediction model. Selective classification, encompassing a spectrum of methods for uncertainty quantification, empowers the model to withhold decision-making in the face of samples marked by ambiguity or low confidence. This approach not only amplifies the accuracy of predictions for the instances it chooses to classify but also notably enhances the model's interpretability. Results: Comprehensive experiments demonstrate that incorporating uncertainty markedly enhances the model's performance. Specifically, the proposed method achieved 32.37%, 21.43%, and 13.27% relative improvement on area under the precision-recall curve over the base model (Hierarchical Interaction Network) in phase I, II, and III trial approval predictions, respectively. For phase III trials, our method reaches 0.9022 area under the precision-recall curve scores. In addition, we show a case study of interpretability that helps domain experts to understand model's outcome. The code is publicly available at https://github.com/Vincent-1125/Uncertainty-Quantification-on-Clinical-Trial-Outcome-Prediction. Conclusion: Our approach not only measures model uncertainty but also greatly improves interpretability and performance for clinical trial approval prediction.

Leveraging Bayesian deep learning and ensemble methods for uncertainty quantification in image classification: A ranking-based approach

Bayesian deep learning (BDL) has emerged as a powerful technique for quantifying uncertainty in classification tasks, surpassing the effectiveness of traditional models by aligning with the probabilistic nature of real-world data. This alignment allows for informed decision-making by not only identifying the most likely outcome but also quantifying the surrounding uncertainty. Such capabilities hold great significance in fields like medical diagnoses and autonomous driving, where the consequences of misclassification are substantial. To further improve uncertainty quantification, the research community has introduced Bayesian model ensembles, which combines multiple Bayesian models to enhance predictive accuracy and uncertainty quantification. These ensembles have exhibited superior performance compared to individual Bayesian models and even non-Bayesian counterparts. In this study, we propose a novel approach that leverages the power of Bayesian ensembles for enhanced uncertainty quantification. The proposed method exploits the disparity between predicted positive and negative classes and employes it as a ranking metric for model selection. For each instance or sample, the ensemble's output for each class is determined by selecting the top ‘k' models based on this ranking. Experimental results on different medical image classifications demonstrate that the proposed method consistently outperforms or achieves comparable performance to conventional Bayesian ensemble. This investigation highlights the practical application of Bayesian ensemble techniques in refining predictive performance and enhancing uncertainty evaluation in image classification tasks.

Uncertainty-Aware Health Diagnostics via Class-Balanced Evidential Deep Learning.

Uncertainty quantification is critical for ensuring the safety of deep learning-enabled health diagnostics, as it helps the model account for unknown factors and reduces the risk of misdiagnosis. However, existing uncertainty quantification studies often overlook the significant issue of class imbalance, which is common in medical data. In this paper, we propose a class-balanced evidential deep learning framework to achieve fair and reliable uncertainty estimates for health diagnostic models. This framework advances the state-of-the-art uncertainty quantification method of evidential deep learning with two novel mechanisms to address the challenges posed by class imbalance. Specifically, we introduce a pooling loss that enables the model to learn less biased evidence among classes and a learnable prior to regularize the posterior distribution that accounts for the quality of uncertainty estimates. Extensive experiments using benchmark data with varying degrees of imbalance and various naturally imbalanced health data demonstrate the effectiveness and superiority of our method. Our work pushes the envelope of uncertainty quantification from theoretical studies to realistic healthcare application scenarios. By enhancing uncertainty estimation for class-imbalanced data, we contribute to the development of more reliable and practical deep learning-enabled health diagnostic systems.

Chemprop: A Machine Learning Package for Chemical Property Prediction.

Deep learning has become a powerful and frequently employed tool for the prediction of molecular properties, thus creating a need for open-source and versatile software solutions that can be operated by nonexperts. Among the current approaches, directed message-passing neural networks (D-MPNNs) have proven to perform well on a variety of property prediction tasks. The software package Chemprop implements the D-MPNN architecture and offers simple, easy, and fast access to machine-learned molecular properties. Compared to its initial version, we present a multitude of new Chemprop functionalities such as the support of multimolecule properties, reactions, atom/bond-level properties, and spectra. Further, we incorporate various uncertainty quantification and calibration methods along with related metrics as well as pretraining and transfer learning workflows, improved hyperparameter optimization, and other customization options concerning loss functions or atom/bond features. We benchmark D-MPNN models trained using Chemprop with the new reaction, atom-level, and spectra functionality on a variety of property prediction data sets, including MoleculeNet and SAMPL, and observe state-of-the-art performance on the prediction of water-octanol partition coefficients, reaction barrier heights, atomic partial charges, and absorption spectra. Chemprop enables out-of-the-box training of D-MPNN models for a variety of problem settings in fast, user-friendly, and open-source software.

Uncertainties in critical slowing down indicators of observation-based fingerprints of the Atlantic Overturning Circulation

Observations are increasingly used to detect critical slowing down (CSD) to measure stability changes in key Earth system components. However, most datasets have non-stationary missing-data distributions, biases and uncertainties. Here we show that, together with the pre-processing steps used to deal with them, these can bias the CSD analysis. We present an uncertainty quantification method to address such issues. We show how to propagate uncertainties provided with the datasets to the CSD analysis and develop conservative, surrogate-based significance tests on the CSD indicators. We apply our method to three observational sea-surface temperature and salinity datasets and to fingerprints of the Atlantic Meridional Overturning Circulation derived from them. We find that the properties of these datasets and especially the specific gap filling procedures can in some cases indeed cause false indication of CSD. However, CSD indicators in the North Atlantic are still present and significant when accounting for dataset uncertainties and non-stationary observational coverage.

A data-driven B-spline-enhanced Kriging method for uncertainty quantification based on Bayesian compressive sensing

Kriging is a powerful surrogate method for fitting smooth functions, and has been widely used in uncertainty quantification. However, for non-smooth functions, the performance of Kriging may be unsatisfactory. To overcome this drawback, we combine the advantages of spline functions and the Kriging model, and propose a data-driven B-spline-enhanced Kriging method (DDBSEK). Firstly, we establish the data-consistent orthogonal B-spline basis functions with the aid of the sample data of the input variables and replace the trend terms of the traditional Kriging model with these orthogonal B-spline bases. Then, by leveraging the fast marginal likelihood maximization algorithm and the Bayesian compressive sensing technique, an improved analytical Bayesian LASSO (least absolute shrinkage and selection operator) algorithm is developed to retain the active orthogonal B-spline bases. The proposed DDBSEK is validated using two types of numerical examples and one implicit engineering example. Results show that the proposed method is superior to the B-spline function model or the Kriging model alone.

Sources of Uncertainty and Risk Quantification Methods in Supply Chain Management: A Literature Study

Sources of Uncertainty and Risk Quantification Methods in Supply Chain Management: A Literature Study

Evaluating uncertainty-based active learning for accelerating the generalization of molecular property prediction

Deep learning models have proven to be a powerful tool for the prediction of molecular properties for applications including drug design and the development of energy storage materials. However, in order to learn accurate and robust structure–property mappings, these models require large amounts of data which can be a challenge to collect given the time and resource-intensive nature of experimental material characterization efforts. Additionally, such models fail to generalize to new types of molecular structures that were not included in the model training data. The acceleration of material development through uncertainty-guided experimental design has the promise to significantly reduce the data requirements and enable faster generalization to new types of materials. To evaluate the potential of such approaches for electrolyte design applications, we perform comprehensive evaluation of existing uncertainty quantification methods on the prediction of two relevant molecular properties - aqueous solubility and redox potential. We develop novel evaluation methods to probe the utility of the uncertainty estimates for both in-domain and out-of-domain data sets. Finally, we leverage selected uncertainty estimation methods for active learning to evaluate their capacity to support experimental design.

Optimized equivalent linearization for random vibration

A fundamental limitation of various Equivalent Linearization Methods (ELMs) in nonlinear random vibration analysis is that they are approximate by their nature. A quantity of interest estimated from an ELM has no guarantee to be the same as the solution of the original nonlinear system. In this study, we tackle this fundamental limitation. We sequentially address the following two questions: (i) given an equivalent linear system obtained from any ELM, how to construct an estimator such that, as the linear system simulations are guided by a limited number of nonlinear system simulations, the estimator converges on the nonlinear system solution? (ii) how to construct an optimized equivalent linear system such that the estimator approaches the nonlinear system solution as quickly as possible? The first question is theoretically straightforward since classic Monte Carlo techniques, such as the control variates and importance sampling, can improve upon the solution of any surrogate model. We adapt the well-known Monte Carlo theories into the specific context of equivalent linearization. The second question is challenging, especially when rare event probabilities are of interest. We develop specialized methods to construct and optimize linear systems. In the context of uncertainty quantification (UQ), the proposed optimized ELM can be viewed as a physical surrogate model-based UQ method. The embedded physical equations endow the surrogate model with the capability to handle high-dimensional uncertainties in stochastic dynamics analysis.

SS-MASVM: An advanced technique for assessing failure probability of high-dimensional complex systems using the multi-class adaptive support vector machine

The integrity of structures in contemporary engineering practices is heavily influenced by inherent uncertainties, which necessitates the development of reliable structural reliability analysis methods for uncertainty quantification and probabilistic analysis. This paper investigates the integration of surrogate models and subset simulation (SS) to accurately compute the failure probabilities of high-dimensional complex systems. Focus is placed on enhancing the computational efficiency and accuracy through using the multi-class adaptive support vector machine (MASVM). The proposed SS-MASVM method incorporates more efficient learning functions, integrates K-means clustering, and employs SS technology for precise estimation of failure probabilities of high-dimensional complex systems. Numerical and engineering structural examples validate the effectiveness of the proposed method and highlight its potential in practical reliability analysis, and the results disclose that the SS-MASVM is efficient and accurate for assessing the complex system reliability.

Main outcomes of the Phebus FPT1 uncertainty and sensitivity analysis in the EU-MUSA project

The Management and Uncertainties of Severe Accidents (MUSA) project was funded in HORIZON 2020 and is coordinated by CIEMAT (Spain). The project aims at consolidating a harmonized approach for the analysis of uncertainties and sensitivities associated with Severe Accidents (SAs) analysis, focusing on source term figures of merit. The Application of Uncertainty Quantification (UQ) Methods against Integral Experiments (AUQMIE – Work Package 4 (WP4)), led by ENEA (Italy), was devoted to apply and test UQ methodologies adopting the internationally recognized PHEBUS FPT1 test. FPT1 was chosen to test UQ methodologies because, even though it is a simplified SA scenario, it was representative of the in-vessel phase of a severe accident initiated by a break in the cold leg of a PWR primary circuit.WP4 served as a platform to identify and discuss the issues encountered in the application of UQ methodologies to SA analyses (e.g. discuss the UQ methodology, perform the coupling between the SA codes and the UQ tools, define the results post-processing methods, etc.). The purpose of this paper is to describe the MUSA PHEBUS FPT1 uncertainty application exercise with the related specifications and the methodologies used by the partners to perform the UQ exercise. The main outcomes and lessons learned of the analysis are: scripting was in general needed for the SA code and uncertainty tool coupling and to have more flexibility; particular attention should be devoted to the proper choice of the input uncertain parameters; outlier values of figures of merit should be carefully analyzed; the computational time is a key element to perform UQ in SA; the large number of uncertain input parameters may complicate the interpretation of correlation or sensitivity analysis; there is the need for a statistically solid handling of failed calculations.

Hybrid uncertainty quantification method of internal disturbance parameters for reliability assessment of building air-conditioning design load

Hybrid uncertainty quantification method of internal disturbance parameters for reliability assessment of building air-conditioning design load

Preface

The 2023 International Conference on Applied Mathematics, Modelling and Statistics Application (AMMSA2023), was held successfully on August 26-28th, 2023 in Changsha, China. The contributions are the results of conference participant submissions, providing readers with a broader view of mathematical and computational science methods, ideas and tools.There are 152 participants in the AMMSA 2023 conference, in which the program includes 4 keynote reports (35 minutes), 67 oral presentations (12 mins+3 Q&A) in four sessions (advances in mathematics and theory, Statistical Modelling and Data Analysis, computational intelligence and algorithms, and mathematical modelling and in engineering and physical sciences) and 15 posters. The paper entitled “An Improved Multi-dimensional Uncertainty Quantification Method Based on DNN-DRM” won the best presentation after being reviewed by the session chair at the conference. This conference emphasizes the development of theories and applications and promotes interdisciplinary endeavours among mathematicians, statisticians, scientists, engineers and researchers from other disciplines.There are 51 papers selected from 144 submissions and included in this volume. All accepted manuscripts were peer-reviewed by 2 reviewers at least in a double-blind review process. A total of 61 reviewers participated in this conference, who provided their professional and strict comments for authors to ensure the topic and quality of the manuscripts. There are seven topic partitions, including Mathematical Modelling and Theory, Advances in Statistical Modelling and Data Analysis, Computational Intelligence and Algorithms, Mathematical Modelling in Fluid Dynamics and Mechanics, Intelligent Control and Automation, Mathematical Modelling in Optical Engineering, and Mathematical Modelling in Engineering and Physical Sciences. This proceeding constitutes an up-to-date account of principles, techniques, and tools for mathematical simulation & modelling in a wide range of research fields.This volume of proceedings is beneficial for readers from academia and industry to understand and tackle the challenges efficiently and adopt appropriate solutions in the mentioned fields above. Building on the success of this event, we hope AMMSA2023 will be one of the most spectacular events in those areas.Thanks go to all the international committee members, who assisted with the review of the papers submitted and presented at the conference. We would like to give special thanks to Prof. Alexander G. Ramm, Prof. Weide Li, Prof. Yajun Liu and Prof. Lu Leng who gave their keynote talks. Thanks also go to the four session chairs, Prof. Rao Li, Prof. Yichuan Zhao, Prof. Hassan Zedan and Prof. Quanxin Zhu who reviewed all oral presentations as well as posters. We are grateful to everyone who has contributed to the realization of AMMSA 2023.List of Conference Committees, Organization are available in this Pdf.