Uncertainty Estimation Research Articles

PixCUE: Joint Uncertainty Estimation and Image Reconstruction in MRI using Deep Pixel Classification.

Deep learning (DL) models are effective in leveraging latent representations from MR data, emerging as state-of-the-art solutions for accelerated MRI reconstruction. However, challenges arise due to the inherent uncertainties associated with undersampling in k-space, coupled with the over- or under-parameterized and opaque nature of DL models. Addressing uncertainty has thus become a critical issue in DL MRI reconstruction. Monte Carlo (MC) inference techniques are commonly employed to estimate uncertainty, involving multiple reconstructions of the same scan to compute variance as a measure of uncertainty. Nevertheless, these methods entail significant computational expenses, requiring multiple inferences through the DL model. In this context, we propose a novel approach to uncertainty estimation during MRI reconstruction using a pixel classification framework. Our method, PixCUE (Pixel Classification Uncertainty Estimation), generates both the reconstructed image and an uncertainty map in a single forward pass through the DL model. We validate the efficacy of this approach by demonstrating that PixCUE-generated uncertainty maps exhibit a strong correlation with reconstruction errors across various MR imaging sequences and under diverse adversarial conditions. We present an empirical relationship between uncertainty estimations using PixCUE and established reconstruction metrics such as NMSE, PSNR, and SSIM. Furthermore, we establish a correlation between the estimated uncertainties from PixCUE and the conventional MC method. Our findings affirm that PixCUE reliably estimates uncertainty in MRI reconstruction with minimal additional computational cost.

Hierarchical Bayesian modeling for Inverse Uncertainty Quantification of system thermal-hydraulics code using critical flow experimental data

The best estimate plus uncertainty methodology in nuclear system thermal-hydraulic studies necessitates a comprehensive understanding of uncertainties in system code predictions. The forward uncertainty quantification (UQ) process involves the propagation of input uncertainties through the computational models to obtain uncertainties in the outputs. To this end, achieving an accurate estimation of input uncertainties is important, which is the focus of inverse UQ (IUQ). Traditionally, research in Bayesian IUQ within the nuclear engineering domain has largely relied on single-level Bayesian inference. While being effective for relatively small datasets, this approach encounters limitations for cases with large datasets. The use of a single-level model may prove inefficient, as the resultant posterior distributions can significantly differ when distinct subsets of data are employed. To address this issue, we employ an hierarchical Bayesian model for IUQ. This approach involves organizing observations into different groups based on the test conditions, thereby accommodating varying calibration parameters across these distinct groups. In this study, we developed and implemented a hierarchical Bayesian IUQ method to consider the grouping effect of critical flow measurement data from various geometries. Comparing the outcomes of IUQ under different selections of test data using hierarchical Bayesian IUQ against those obtained from single-level Bayesian IUQ, the forward propagation of hierarchical Bayesian IUQ results demonstrates a notably improved agreement with the experimental data.

Discrete‐Time Integral Fast Terminal Sliding Mode Predictive Control for Dynamic Positioning Ships With Lumped Uncertainties and Input Constraints

ABSTRACTThis paper proposes a robust discrete‐time integral fast terminal sliding mode predictive control (DIFTSMPC) scheme for DP ship trajectory tracking under the consideration of unmodeled dynamics, environmental disturbance, and input constraints. An improved single closed‐loop control strategy is designed by adopting the earth‐fixed reference velocity as the virtual velocity. The improved nonlinear PID discrete‐time integral fast terminal sliding mode is designed to achieve faster convergence. The unknown nonlinear dynamics and the environmental disturbance are combined as a lumped uncertainty term and estimated using the one‐step delay observation method. Then the corrected sliding model state is optimized to track the reference sliding reaching law in the prediction horizon. The lumped uncertainty estimation is integrated into the sliding mode prediction to guarantee the robustness of the control scheme. The stability of the proposed scheme has been verified. Comparison results demonstrate the effectiveness and superiority in chattering suppressing and constraint handling of the proposed scheme.

Guaranteed Coverage Prediction Intervals With Gaussian Process Regression.

Gaussian Process Regression (GPR) is a popular regression method, which unlike most Machine Learning techniques, provides estimates of uncertainty for its predictions. These uncertainty estimates however, are based on the assumption that the model is well-specified, an assumption that is violated in most practical applications, since the required knowledge is rarely available. As a result, the produced uncertainty estimates can become very misleading; for example the prediction intervals (PIs) produced for the 95% confidence level may cover much less than 95% of the true labels. To address this issue, this paper introduces an extension of GPR based on a Machine Learning framework called, Conformal Prediction (CP). This extension guarantees the production of PIs with the required coverage even when the model is completely misspecified. The proposed approach combines the advantages of GPR with the valid coverage guarantee of CP, while the performed experimental results demonstrate its superiority over existing methods.

Machine learning materials properties with accurate predictions, uncertainty estimates, domain guidance, and persistent online accessibility

Abstract One compelling vision of the future of materials discovery and design involves the use of machine learning (ML) models to predict materials properties and then rapidly find materials tailored for specific applications. However, realizing this vision requires both providing detailed uncertainty quantification (model prediction errors and domain of applicability) and making models readily usable. At present, it is common practice in the community to assess ML model performance only in terms of prediction accuracy (e.g. mean absolute error), while neglecting detailed uncertainty quantification and robust model accessibility and usability. Here, we demonstrate a practical method for realizing both uncertainty and accessibility features with a large set of models. We develop random forest ML models for 33 materials properties spanning an array of data sources (computational and experimental) and property types (electrical, mechanical, thermodynamic, etc). All models have calibrated ensemble error bars to quantify prediction uncertainty and domain of applicability guidance enabled by kernel-density-estimate-based feature distance measures. All data and models are publicly hosted on the Garden-AI infrastructure, which provides an easy-to-use, persistent interface for model dissemination that permits models to be invoked with only a few lines of Python code. We demonstrate the power of this approach by using our models to conduct a fully ML-based materials discovery exercise to search for new stable, highly active perovskite oxide catalyst materials.

Exploring transformer reliability in clinically significant prostate cancer segmentation: A comprehensive in-depth investigation.

Despite the growing prominence of transformers in medical image segmentation, their application to clinically significant prostate cancer (csPCa) has been overlooked. Minimal attention has been paid to domain shift analysis and uncertainty assessment, critical for safely implementing computer-aided diagnosis (CAD) systems. Domain shift in medical imagery refers to differences between the data used to train a model and the data evaluated later, arising from variations in imaging equipment, protocols, patient populations, and acquisition noise. While recent models enhance in-domain performance, areas such as robustness and uncertainty estimation in out-of-domain distributions have received limited investigation, creating indecisiveness about model reliability. In contrast, our study addresses csPCa at voxel, lesion, and image levels, investigating models from traditional U-Net to cutting-edge transformers. We focus on four key points: robustness, calibration, out-of-distribution (OOD), and misclassification detection (MD). Findings show that transformer-based models exhibit enhanced robustness at image and lesion levels, both in and out of domain. However, this improvement is not fully translated to the voxel level, where Convolutional Neural Networks (CNNs) outperform in most robustness metrics. Regarding uncertainty, hybrid transformers and transformer encoders performed better, but this trend depends on misclassification or out-of-distribution tasks.

Inductive State-Relabeling Adversarial Active Learning With Heuristic Clique Rescaling.

Active learning (AL) is to design label-efficient algorithms by labeling the most representative samples. It reduces annotation cost and attracts increasing attention from the community. However, previous AL methods suffer from the inadequacy of annotations and unreliable uncertainty estimation. Moreover, we find that they ignore the intra-diversity of selected samples, which leads to sampling redundancy. In view of these challenges, we propose an inductive state-relabeling adversarial AL model (ISRA) that consists of a unified representation generator, an inductive state-relabeling discriminator, and a heuristic clique rescaling module. The generator introduces contrastive learning to leverage unlabeled samples for self-supervised training, where the mutual information is utilized to improve the representation quality for AL selection. Then, we design an inductive uncertainty indicator to learn the state score from labeled data and relabel unlabeled data with different importance for better discrimination of instructive samples. To solve the problem of sampling redundancy, the heuristic clique rescaling module measures the intra-diversity of candidate samples and recurrently rescales them to select the most informative samples. The experiments conducted on eight datasets and two imbalanced scenarios show that our model outperforms the previous state-of-the-art AL methods. As an extension on the cross-modal AL task, we apply ISRA to the image captioning and it also achieves superior performance.

Advanced AI, Machine Learning and Deep Learning Techniques for Climate Change Studies: A Review

Climate models have become an important tool in understanding as well as projecting future climate conditions. They thus become very critical to decision-making processes, especially in mitigation and adaptation strategies. This paper provides a general overview of methodologies that are applied in climate modeling, its validation, and uncertainty mapping, notably their strengths, limitations, and challenges. The paper takes the reader through all the intricacies associated with the modeling of climate systems: global circulation models, regional climate models, and statistical downscaling. Much emphasis is given to the issue of model choice depending on the research objectives and spatial scales. Thorough model validation is considered, emphasizing that observational data forms the core and good quality and availability are issues. Climate modeling should incorporate uncertainty quantification as a core activity. This paper debates estimation and representation techniques of model uncertainties applied, such as probabilistic approaches, sensitivity analysis, and ensemble modeling. Introduce an idea called here uncertainty mapping, as a means of communicating the climate-related risks. This paper sets this tone by asserting that uncertainties in climate modeling have to be communicated as clearly as possible to decision-makers. No matter how intrinsically complex and limited they are, climate models represent, by themselves, very valuable means for the task of estimating the probable futures of climates. This quest ranges from enhancing model accuracy to ever-lowering estimates of uncertainties, all impelled by new advances made in computational powers and deeper insights into the climate systems.

A probabilistic integration of LSTM and Gaussian process regression for uncertainty-aware reservoir water level predictions

ABSTRACT Reservoir-level forecasting, while being crucial for optimal operation, is challenged by complex physical processes and changing climate conditions. Machine learning approaches offer deterministic predictions but often neglect system physics and uncertainty. This article presents a probabilistic data-driven approach combining long short-term memory (LSTM) and Gaussian process regression (GPR) to provide both point forecasts and uncertainty estimates. The hybrid model leverages LSTM’s fitting capabilities with GPR’s robust Bayesian frameworks for uncertainty estimation in non-linear problems, offering accurate predictions without extensive high-fidelity modelling, and avoiding frequent training and parameter optimization. Evaluation with real reservoir data from India shows the model’s superiority over the vanilla LSTM for both univariate and multivariate scenarios. The proposed model achieved a Nash-Sutcliffe efficiency of 0.97 to 0.98, a mean biased error of −0.5634 to −1.0314 for 10-day forecasts, and a continuous ranked probability score of 5.80 and 1.87 for the Bhakra and Pong reservoirs, respectively.

Adaptive neural boundary control for multi-agent manipulators system with uncertainties through cooperative disturbance observers network

This paper addresses vibration control problem of multi-agent flexible manipulators systems in the presence of simultaneous uncertainty and unknown external disturbance. Particularly, the goal is to suppress vibration of both flexible link and joint angular. In this paper, the dynamic model of the considered flexible manipulator is described by the fourth order partial differential equation. Without control, the system is unstable and vibrate constantly due to initial states, the external unknown disturbances and system uncertainties. To compensate the uncertainty in each agent, the neural networks are employed and novel adaptation laws are developed to update weighting parameters in the neural networks. While for the compensation of the external disturbance a cooperative network of disturbance observers is proposed to enhance the observation reliability. With the resulting estimations of uncertainties and the unknown disturbance, adaptive distributed boundary controllers are derived to suppress vibration in-domain and keep joint angular position to zero. The closed-loop system is proven to be uniform ultimately bounded through Lyapunov stability theory. Numerical simulations result shows that compared with the proportional–derivative control, the proposed method almost reduces all overshoot and steady-state error.

Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances

Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances

DTAUE: A multi-input multi-branch decoupled semi-supervised 3D network based on threshold adaptive uncertainty estimation for echocardiography segmentation

Echocardiography is a non-invasive examination method based on sound waves to generate images of the heart. Segmentation of the left ventricle (LV) and right ventricle (RV) is crucial for evaluating cardiac function. However, due to the limitations in ultrasound imaging angles and the high prevalence of noise in ultrasound data, automatic and accurate segmentation of the left and right ventricles is a challenging task. In this paper, we present a decoupled semi-supervised 3D echocardiography segmentation network based on threshold adaptive uncertainty estimation. To minimize the impact of missing targets caused by blind zones on segmentation accuracy, pixel-wise segmentation maps and target skeleton maps are jointly predicted by decoupled semi-supervised 3D network (DTAUE) so that our network is more effective to perceive the morphological features of segmented targets. And quantifying the uncertainty inherent to a model’s prediction is a promising endeavor to address the lack of reliability and overconfidence in the model. Previous uncertainty estimation methods set a uniform threshold for all training samples. However, this leads to the loss of edge detail information. Different from the existing methods, in our work, we propose a method of adaptive threshold uncertainty estimation to set different confidence threshold for different targets. Experiments on our self-collected 3D echocardiographic dataset and two publicly available datasets show that our method outperforms state-of-the-art semi-supervised learning methods. DTAUE would potentially serve as a new tool for diagnosing and monitoring various heart conditions.

Calibrated Adaptive Teacher for Domain-Adaptive Intelligent Fault Diagnosis

Intelligent fault diagnosis (IFD) based on deep learning can achieve high accuracy from raw condition monitoring signals. However, models usually perform well on the training distribution only, and experience severe performance drops when applied to a different distribution. This is also observed in fault diagnosis, where assets are often operated in working conditions different from the ones in which the labeled data have been collected. The scenario where labeled data are available in a source domain and only unlabeled data are available in a target domain has been addressed recently by unsupervised domain adaptation (UDA) approaches for IFD. Recent methods have relied on self-training with confident pseudo-labels for the unlabeled target samples. However, the confidence-based selection of pseudo-labels is hindered by poorly calibrated uncertainty estimates in the target domain, primarily due to over-confident predictions, which limits the quality of pseudo-labels and leads to error accumulation. In this paper, we propose a novel method called Calibrated Adaptive Teacher (CAT), where we propose to calibrate the predictions of the teacher network on target samples throughout the self-training process, leveraging post hoc calibration techniques. We evaluate CAT on domain-adaptive IFD and perform extensive experiments on the Paderborn University (PU) benchmark for fault diagnosis of rolling bearings under varying operating conditions, using both time- and frequency-domain inputs. We compare four different calibration techniques within our framework, where temperature scaling is both the most effective and lightweight one. The resulting method—CAT+TempScaling—achieves state-of-the-art performance on most transfer tasks, with on average 7.5% higher accuracy and 4 times lower calibration error compared to domain-adversarial neural networks (DANNs) across the twelve PU transfer tasks.

On the importance of discharge observation uncertainty when interpreting hydrological model performance

Abstract. For users of hydrological models, the suitability of models can depend on how well their simulated outputs align with observed discharge. This study emphasizes the crucial role of factoring in discharge observation uncertainty when assessing the performance of hydrological models. We introduce an ad hoc approach, implemented through the eWaterCycle platform, to evaluate the significance of differences in model performance while considering the uncertainty associated with discharge observations. The analysis of the results encompasses 299 catchments from the Catchment Attributes and MEteorology for Large-sample Studies Great Britain (CAMELS-GB) large-sample catchment dataset, addressing three practical use cases for model users. These use cases involve assessing the impact of additional calibration on model performance using discharge observations, conducting conventional model comparisons, and examining how the variations in discharge simulations resulting from model structural differences compare with the uncertainties inherent in discharge observations. Based on the 5th to 95th percentile range of observed flow, our results highlight the substantial influence of discharge observation uncertainty on interpreting model performance differences. Specifically, when comparing model performance before and after additional calibration, we find that, in 98 out of 299 instances, the simulation differences fall within the bounds of discharge observation uncertainty. This underscores the inadequacy of neglecting discharge observation uncertainty during calibration and subsequent evaluation processes. Furthermore, in the model comparison use case, we identify numerous instances where observation uncertainty masks discernible differences in model performance, underscoring the necessity of accounting for this uncertainty in model selection procedures. While our assessment of model structural uncertainty generally indicates that structural differences often exceed observation uncertainty estimates, a few exceptions exist. The comparison of individual conceptual hydrological models suggests no clear trends between model complexity and subsequent model simulations falling within the uncertainty bounds of discharge observations. Based on these findings, we advocate integrating discharge observation uncertainty into the calibration process and the reporting of hydrological model performance, as has been done in this study. This integration ensures more accurate, robust, and insightful assessments of model performance, thereby improving the reliability and applicability of hydrological modelling outcomes for model users.

Efficient Trans-Dimensional Bayesian Inversion of C-Response Data from Geomagnetic Observatory and Satellite Magnetic Data

To investigate deep Earth information, researchers often utilize geomagnetic observatories and satellite data to obtain the conversion function of geomagnetic sounding, C-response data, and employ traditional inversion techniques to reconstruct subsurface structures. However, the traditional gradient-based inversion produces geophysical models with artificial structure constraint enforced subjectively to guarantee a unique solution. This method typically requires the model parameterization knowledge a priori (e.g., based on personal preference) without uncertainty estimation. In this paper, we apply an efficient trans-dimensional (trans-D) Bayesian algorithm to invert C-response data from observatory and satellite geomagnetic data for the electrical conductivity structure of the Earth’s mantle, with the model parameterization treated as unknown and determined by the data. In trans-D Bayesian inversion, the posterior probability density (PPD) represents a complete inversion solution, based on which useful inversion inferences about the model can be made with the requirement of high-dimensional integration of PPD. This is realized by an efficient reversible-jump Markov-chain Monte Carlo (rjMcMC) sampling algorithm based on the birth/death scheme. Within the trans-D Bayesian algorithm, the model parameter is perturbated in the principal-component parameter space to minimize the effect of inter-parameter correlations and improve the sampling efficiency. A parallel tempering scheme is applied to guarantee the complete sampling of the multiple model space. Firstly, the trans-D Bayesian inversion is applied to invert C-response data from two synthetic models to examine the resolution of the model structure constrained by the data. Then, C-response data from geomagnetic satellites and observatories are inverted to recover the global averaged mantle conductivity structure and the local mantle structure with quantitative uncertainty estimation, which is consistent with the data.

Calculated ionization energies, orbital eigenvalues (HOMO), and related QSAR descriptors of organic molecules: a set of 61 experimental values enables elimination of systematic errors and provides realistic error estimates.

Ionization energies (IEs) of organic compounds come in different forms-adiabatic, vertical, as electrode potentials, or as orbital eigenvalues via Koopmans' theorem. They have been linked to the reactivity towards electrophiles and have been used to quantitatively describe electron transfer processes. The de novo prediction of IEs is only meaningful when an estimate of the prediction's uncertainty is included. Pulsed-field ionization (PFI) experiments have quantified adiabatic IEs with unprecedented precision. In this work, a new set of PFI-derived IEs is compiled from the literature as a benchmark for prediction methods. This set includes many common functional groups, a size range from diatomics to two aromatic rings, and IEs between 7 and 14 eV. The first-principles CCSD(T)/CBS protocol presently used reproduces these values within 0.05 eV. For adiabatic IEs and vertical IEs/orbital eigenvalues predicted using approximate density functional theory (DFT), linear regression models are proposed, so that IEs calculated using different methods can be directly compared on a physical scale. This elimination of systematic errors improves the error statistics and allows the performance of predicted IEs to be evaluated if used in quantitative structure-property or -activity relationships, as the latter implicitly correct a descriptor's bias. Owing to the structural scope of the test set, the minimum and maximum deviations from experiment should correspond to those expected for common organic molecules. Deviations from reference values found for orbital eigenvalues but also for IEs calculated explicitly with HF or semi-empirical MO methods were as large as 0.5 eV to 2.0 eV. Such large errors could also propagate into quantitative structure-property models, as shown in illustrative examples of oxidation rate constants in solution.

NetGSR: Towards Efficient and Reliable Network Monitoring with Generative Super Resolution

Network monitoring systems are a key building block in today's networks. They all follow a common framework where measurement data from network elements is aggregated at a central collector for network-wide visibility. When designing network monitoring systems, two key properties have to be taken into account: (1) efficiency, to minimize the communication overhead from network elements to the collector; (2) high-fidelity, to faithfully represent the network status. However, in presence of network dynamics, tracking the right operating point to ensure both high fidelity and efficiency is hard and we observe that prior monitoring approaches trade off one for the other. In this paper, we show that it is possible to satisfy both these properties with NetGSR, a new deep learning based solution we introduce that reconstructs the fine-grained behavior of network status at the collector while requiring low resolution measurement data from network elements. This is achieved through a combination of a new custom-tailored conditional deep generative model (DistilGAN), and a new feedback mechanism (Xaminer) based on model uncertainty estimation and denoising that allows the collector to adjust the sampling rate for measurement data from network elements, at run-time. We extensively evaluate NetGSR using three different network scenarios with corresponding real-world network monitoring datasets as well as two downstream use cases. We show that NetGSR can faithfully reconstruct fine-grained network status with 25x greater measurement efficiency than prior approaches while requiring only few ms of inference time at the collector.

Estimation of experimental uncertainties. A simplified proposal for indirect measurements in the laboratory

Estimation of experimental uncertainties. A simplified proposal for indirect measurements in the laboratory

Uncertainty quantification of collective nuclear observables from the chiral potential parametrization

We perform an uncertainty estimate of quadrupole moments and B(E2) transition rates that inform nuclear collectivity. In particular, we study the low-lying states of 6Li and 12C using the ab initio symmetry-adapted no-core–shell model. For a narrow standard deviation of approximately 1% on the low-energy constants which parametrize high-precision chiral potentials, we find output standard deviations in the collective observables ranging from approximately 3%–6%. The results mark the first step towards a rigorous uncertainty quantification of collectivity in nuclei that aims to account for all sources of uncertainty in ab initio descriptions of challenging collective and clustering observables.

Deep learning applied to the segmentation of rodent brain MRI data outperforms noisy ground truth on full-fledged brain atlases

Translational magnetic resonance imaging of the rodent brain provides invaluable information for preclinical drug development. However, the automated segmentation of such images for quantitative analyses is limited compared to human brain imaging mainly due to the inferior anatomical contrast and the resulting less advanced registration and atlasing tools. Here, we investigated the potential of deep learning models for the segmentation of magnetic resonance images of rat brains into an entire set of multiple regions of interest (rather than individual loci), focusing on the development of a robust method that accommodates changes in the input based on differences in animal strain (genotype) and size. Manually generated labels are expensive, so we tested the ability of neural networks to learn brain structures from noisy but inexpensive registration-based labels, allowing very large datasets to be leveraged for training. We compared three distinct model architectures (U-Net, Attention-U-Net and DeepLab) by training them on a dataset of >10,000 magnetic resonance images of rat brains and found that each model was able to segment the entire brain into predefined sets of 29 and 58 regions, respectively, with the Attention U-Net achieving the best performance. The models canceled out unstructured label noise in the imperfect training data to provide smoother and more symmetric segmentations than registration-based labeling, and were more robust when presented with input variations, thus outperforming the noisy ground truth. Our pipeline also includes uncertainty estimation and an explainability mechanism, hence providing features essential for anomaly detection and quality assurance. In summary, our study shows that deep learning models do achieve accurate brain segmentation in high-throughput quantitative preclinical imaging without the need for expensive expert-generated labels.