Uncertainty Estimation Research Articles

The effect of correlated color temperature (CCT) variation on spectral mismatch factor (SMF) for incandescent luminous flux lamps

In this research, a setup including NIS Spectroradiometer Ocean Optics HR 2000 with an uncertainty of 4.7%, a photometric bench, and a group of NIS total luminous flux standard lamps calibrated at the National Physical Laboratory (NPL) in England with an uncertainty of 0.8%, was used to measure the correlated color temperature (CCT) and spectral power distribution (SPD) of six incandescent lamps. The correlated color temperature (CCT) curves and their equations are crucial for determining the uncertainty and variations in the spectral mismatch correction factor (SMF) for each lamp. Uncertainty calculations and analyses were conducted, revealing that the correlated color temperature (CCT) of the standard lamps ranged from 2400 K and 2750 K, and the correlated color temperature (CCT) of the tested lamps ranged from 2043 K to 2095 K influencing the uncertainty. Additionally, data analysis was performed, and an uncertainty model was developed in conjunction with the measurements. The uncertainty estimation for spectral mismatch correction factors varied from 0.0011 to 0.0084 depending on the correlated color temperature.

A Gaussian Process-Enhanced Non-Linear Function and Bayesian Convolution–Bayesian Long Term Short Memory Based Ultra-Wideband Range Error Mitigation Method for Line of Sight and Non-Line of Sight Scenarios

Relative positioning accuracy between two devices is dependent on the precise range measurements. Ultra-wideband (UWB) technology is one of the popular and widely used technologies to achieve centimeter-level accuracy in range measurement. Nevertheless, harsh indoor environments, multipath issues, reflections, and bias due to antenna delay degrade the range measurement performance in line-of-sight (LOS) and non-line-of-sight (NLOS) scenarios. This article proposes an efficient and robust method to mitigate range measurement error in LOS and NLOS conditions by combining the latest artificial intelligence technology. A GP-enhanced non-linear function is proposed to mitigate the range bias in LOS scenarios. Moreover, NLOS identification based on the sliding window and Bayesian Conv-BLSTM method is utilized to mitigate range error due to the non-line-of-sight conditions. A novel spatial–temporal attention module is proposed to improve the performance of the proposed model. The epistemic and aleatoric uncertainty estimation method is also introduced to determine the robustness of the proposed model for environment variance. Furthermore, moving average and min-max removing methods are utilized to minimize the standard deviation in the range measurements in both scenarios. Extensive experimentation with different settings and configurations has proven the effectiveness of our methodology and demonstrated the feasibility of our robust UWB range error mitigation for LOS and NLOS scenarios.

Uncertainty-Aware High-Fidelity Anatomical MRI Synthesis using Deep Convolutional Network with Monte Carlo Dropout

Multi-modality high-resolution MRI is beneficial for studying the brain structure and function in research and clinical settings. However, its acquisition is time-consuming, which reduces its feasibility for wider adoption especially for certain populations who cannot tolerate long scans. In this study, we propose a convolution neural network to obtain high-resolution T1-weighted MRI from lower-resolution T2-weighted input that can be acquired within a shorter scan time. By leveraging Monte Carlo dropout, our model not only produces high-fidelity anatomical T1-weighted image with higher accuracy compared to baseline model, but also generates uncertainty estimation similar to the actual error map. Our method is validated on the Human Connectome Project, and the experiments indicate our method has the potential to improve the robustness and reliability of deep learning image synthesis and accurately accelerate multi-modality MRI to benefit research and clinical practice.

PixCUE: Joint Uncertainty Estimation and Image Reconstruction in MRI using Deep Pixel Classification.

Deep learning (DL) models are effective in leveraging latent representations from MR data, emerging as state-of-the-art solutions for accelerated MRI reconstruction. However, challenges arise due to the inherent uncertainties associated with undersampling in k-space, coupled with the over- or under-parameterized and opaque nature of DL models. Addressing uncertainty has thus become a critical issue in DL MRI reconstruction. Monte Carlo (MC) inference techniques are commonly employed to estimate uncertainty, involving multiple reconstructions of the same scan to compute variance as a measure of uncertainty. Nevertheless, these methods entail significant computational expenses, requiring multiple inferences through the DL model. In this context, we propose a novel approach to uncertainty estimation during MRI reconstruction using a pixel classification framework. Our method, PixCUE (Pixel Classification Uncertainty Estimation), generates both the reconstructed image and an uncertainty map in a single forward pass through the DL model. We validate the efficacy of this approach by demonstrating that PixCUE-generated uncertainty maps exhibit a strong correlation with reconstruction errors across various MR imaging sequences and under diverse adversarial conditions. We present an empirical relationship between uncertainty estimations using PixCUE and established reconstruction metrics such as NMSE, PSNR, and SSIM. Furthermore, we establish a correlation between the estimated uncertainties from PixCUE and the conventional MC method. Our findings affirm that PixCUE reliably estimates uncertainty in MRI reconstruction with minimal additional computational cost.

Estimation of uncertainties in the density driven flow in fractured porous media using MLMC

AbstractWe use the Multi Level Monte Carlo method to estimate uncertainties in a Henry-like salt water intrusion problem with a fracture. The flow is induced by the variation of the density of the fluid phase, which depends on the mass fraction of salt. While the fracture’s location is fixed, its aperture is uncertain. In our setting, porosity and permeability vary spatially and recharge is time-dependent. So we introduce three random variables, one controlling both the porosity and permeability fields, one for the fracture width and one for the intensity of recharge. For each realization of these uncertain parameters, the evolution of mass fraction and pressure fields is modeled using a system of non-linear, time-dependent PDEs with a solution discontinuity at the fracture. These uncertainties propagate, affecting the distribution of salt concentration, a key factor in water resource quality. We show that the MLMC method can be successfully applied to this problem. It significantly reduces the computational cost compared to classical Monte Carlo methods by effectively balancing discretisation and statistical errors, and by evaluating multiple scenarios over different spatial and temporal mesh levels. The deterministic PDE solver, using the ug4 library, runs in parallel to compute all stochastic scenarios.

Data-driven modelling of hydraulic-head time series: results and lessons learned from the 2022 Groundwater Time Series Modelling Challenge

Abstract. This paper presents the results of the 2022 Groundwater Time Series Modelling Challenge, where 15 teams from different institutes applied various data-driven models to simulate hydraulic-head time series at four monitoring wells. Three of the wells were located in Europe and one was located in the USA in different hydrogeological settings in temperate, continental, or subarctic climates. Participants were provided with approximately 15 years of measured heads at (almost) regular time intervals and daily measurements of weather data starting some 10 years prior to the first head measurements and extending around 5 years after the last head measurement. The participants were asked to simulate the measured heads (the calibration period), to provide a prediction for around 5 years after the last measurement (the validation period for which weather data were provided but not head measurements), and to include an uncertainty estimate. Three different groups of models were identified among the submissions: lumped-parameter models (three teams), machine learning models (four teams), and deep learning models (eight teams). Lumped-parameter models apply relatively simple response functions with few parameters, while the artificial intelligence models used models of varying complexity, generally with more parameters and more input, including input engineered from the provided data (e.g. multi-day averages). The models were evaluated on their performance in simulating the heads in the calibration period and in predicting the heads in the validation period. Different metrics were used to assess performance, including metrics for average relative fit, average absolute fit, fit of extreme (high or low) heads, and the coverage of the uncertainty interval. For all wells, reasonable performance was obtained by at least one team from each of the three groups. However, the performance was not consistent across submissions within each group, which implies that the application of each method to individual sites requires significant effort and experience. In particular, estimates of the uncertainty interval varied widely between teams, although some teams submitted confidence intervals rather than prediction intervals. There was not one team, let alone one method, that performed best for all wells and all performance metrics. Four of the main takeaways from the model comparison are as follows: (1) lumped-parameter models generally performed as well as artificial intelligence models, which means they capture the fundamental behaviour of the system with only a few parameters. (2) Artificial intelligence models were able to simulate extremes beyond the observed conditions, which is contrary to some persistent beliefs about these methods. (3) No overfitting was observed in any of the models, including in the models with many parameters, as performance in the validation period was generally only a bit lower than in the calibration period, which is evidence of appropriate application of the different models. (4) The presented simulations are the combined results of the applied method and the choices made by the modeller(s), which was especially visible in the performance range of the deep learning methods; underperformance does not necessarily reflect deficiencies of any of the models. In conclusion, the challenge was a successful initiative to compare different models and learn from each other. Future challenges are needed to investigate, for example, the performance of models in more variable climatic settings to simulate head series with significant gaps or to estimate the effect of drought periods.

Hierarchical Bayesian modeling for Inverse Uncertainty Quantification of system thermal-hydraulics code using critical flow experimental data

The best estimate plus uncertainty methodology in nuclear system thermal-hydraulic studies necessitates a comprehensive understanding of uncertainties in system code predictions. The forward uncertainty quantification (UQ) process involves the propagation of input uncertainties through the computational models to obtain uncertainties in the outputs. To this end, achieving an accurate estimation of input uncertainties is important, which is the focus of inverse UQ (IUQ). Traditionally, research in Bayesian IUQ within the nuclear engineering domain has largely relied on single-level Bayesian inference. While being effective for relatively small datasets, this approach encounters limitations for cases with large datasets. The use of a single-level model may prove inefficient, as the resultant posterior distributions can significantly differ when distinct subsets of data are employed. To address this issue, we employ an hierarchical Bayesian model for IUQ. This approach involves organizing observations into different groups based on the test conditions, thereby accommodating varying calibration parameters across these distinct groups. In this study, we developed and implemented a hierarchical Bayesian IUQ method to consider the grouping effect of critical flow measurement data from various geometries. Comparing the outcomes of IUQ under different selections of test data using hierarchical Bayesian IUQ against those obtained from single-level Bayesian IUQ, the forward propagation of hierarchical Bayesian IUQ results demonstrates a notably improved agreement with the experimental data.

Discrete‐Time Integral Fast Terminal Sliding Mode Predictive Control for Dynamic Positioning Ships With Lumped Uncertainties and Input Constraints

ABSTRACTThis paper proposes a robust discrete‐time integral fast terminal sliding mode predictive control (DIFTSMPC) scheme for DP ship trajectory tracking under the consideration of unmodeled dynamics, environmental disturbance, and input constraints. An improved single closed‐loop control strategy is designed by adopting the earth‐fixed reference velocity as the virtual velocity. The improved nonlinear PID discrete‐time integral fast terminal sliding mode is designed to achieve faster convergence. The unknown nonlinear dynamics and the environmental disturbance are combined as a lumped uncertainty term and estimated using the one‐step delay observation method. Then the corrected sliding model state is optimized to track the reference sliding reaching law in the prediction horizon. The lumped uncertainty estimation is integrated into the sliding mode prediction to guarantee the robustness of the control scheme. The stability of the proposed scheme has been verified. Comparison results demonstrate the effectiveness and superiority in chattering suppressing and constraint handling of the proposed scheme.

Guaranteed Coverage Prediction Intervals With Gaussian Process Regression.

Gaussian Process Regression (GPR) is a popular regression method, which unlike most Machine Learning techniques, provides estimates of uncertainty for its predictions. These uncertainty estimates however, are based on the assumption that the model is well-specified, an assumption that is violated in most practical applications, since the required knowledge is rarely available. As a result, the produced uncertainty estimates can become very misleading; for example the prediction intervals (PIs) produced for the 95% confidence level may cover much less than 95% of the true labels. To address this issue, this paper introduces an extension of GPR based on a Machine Learning framework called, Conformal Prediction (CP). This extension guarantees the production of PIs with the required coverage even when the model is completely misspecified. The proposed approach combines the advantages of GPR with the valid coverage guarantee of CP, while the performed experimental results demonstrate its superiority over existing methods.

Machine learning materials properties with accurate predictions, uncertainty estimates, domain guidance, and persistent online accessibility

Abstract One compelling vision of the future of materials discovery and design involves the use of machine learning (ML) models to predict materials properties and then rapidly find materials tailored for specific applications. However, realizing this vision requires both providing detailed uncertainty quantification (model prediction errors and domain of applicability) and making models readily usable. At present, it is common practice in the community to assess ML model performance only in terms of prediction accuracy (e.g. mean absolute error), while neglecting detailed uncertainty quantification and robust model accessibility and usability. Here, we demonstrate a practical method for realizing both uncertainty and accessibility features with a large set of models. We develop random forest ML models for 33 materials properties spanning an array of data sources (computational and experimental) and property types (electrical, mechanical, thermodynamic, etc). All models have calibrated ensemble error bars to quantify prediction uncertainty and domain of applicability guidance enabled by kernel-density-estimate-based feature distance measures. All data and models are publicly hosted on the Garden-AI infrastructure, which provides an easy-to-use, persistent interface for model dissemination that permits models to be invoked with only a few lines of Python code. We demonstrate the power of this approach by using our models to conduct a fully ML-based materials discovery exercise to search for new stable, highly active perovskite oxide catalyst materials.

Exploring transformer reliability in clinically significant prostate cancer segmentation: A comprehensive in-depth investigation.

Despite the growing prominence of transformers in medical image segmentation, their application to clinically significant prostate cancer (csPCa) has been overlooked. Minimal attention has been paid to domain shift analysis and uncertainty assessment, critical for safely implementing computer-aided diagnosis (CAD) systems. Domain shift in medical imagery refers to differences between the data used to train a model and the data evaluated later, arising from variations in imaging equipment, protocols, patient populations, and acquisition noise. While recent models enhance in-domain performance, areas such as robustness and uncertainty estimation in out-of-domain distributions have received limited investigation, creating indecisiveness about model reliability. In contrast, our study addresses csPCa at voxel, lesion, and image levels, investigating models from traditional U-Net to cutting-edge transformers. We focus on four key points: robustness, calibration, out-of-distribution (OOD), and misclassification detection (MD). Findings show that transformer-based models exhibit enhanced robustness at image and lesion levels, both in and out of domain. However, this improvement is not fully translated to the voxel level, where Convolutional Neural Networks (CNNs) outperform in most robustness metrics. Regarding uncertainty, hybrid transformers and transformer encoders performed better, but this trend depends on misclassification or out-of-distribution tasks.

Modelling the effect of base component properties and processing conditions on mixture products using probabilistic, knowledge-guided neural networks

Development of materials by mixing different base components is a widespread methodology to create materials with improved properties compared to those of its base components. However, efficient determination of the properties of mixture-based materials during design remains challenging without prior knowledge of the underlying physical phenomena. In this work a new data-based methodology is proposed involving the use of probabilistic, knowledge-guided artificial neural networks to jointly model the properties of the base components, the proportions in which they are mixed and the processing conditions used during manufacture to predict properties of final products. The method proposed does not involve any assumptions in terms of ideal mixing rules of the base components, and allows for estimation of aleatoric uncertainty in the prediction. Additionally, an extension is presented that incorporates expert knowledge into the model by the implementation of monotonicity constraints between certain inputs and outputs. The methodology is illustrated with a case study involving the formulation of drug products using direct compression. The model is used to predict pharmaceutical tablets’ quality attributes (mass variation, tensile strength, disintegration time, friability and ejection force), showing that the method is able to predict properties of the final product overcoming gaps currently present in previous modelling approaches.

Inductive State-Relabeling Adversarial Active Learning With Heuristic Clique Rescaling.

Active learning (AL) is to design label-efficient algorithms by labeling the most representative samples. It reduces annotation cost and attracts increasing attention from the community. However, previous AL methods suffer from the inadequacy of annotations and unreliable uncertainty estimation. Moreover, we find that they ignore the intra-diversity of selected samples, which leads to sampling redundancy. In view of these challenges, we propose an inductive state-relabeling adversarial AL model (ISRA) that consists of a unified representation generator, an inductive state-relabeling discriminator, and a heuristic clique rescaling module. The generator introduces contrastive learning to leverage unlabeled samples for self-supervised training, where the mutual information is utilized to improve the representation quality for AL selection. Then, we design an inductive uncertainty indicator to learn the state score from labeled data and relabel unlabeled data with different importance for better discrimination of instructive samples. To solve the problem of sampling redundancy, the heuristic clique rescaling module measures the intra-diversity of candidate samples and recurrently rescales them to select the most informative samples. The experiments conducted on eight datasets and two imbalanced scenarios show that our model outperforms the previous state-of-the-art AL methods. As an extension on the cross-modal AL task, we apply ISRA to the image captioning and it also achieves superior performance.

A Global Thermospheric Density Prediction Framework Based on a Deep Evidential Method

AbstractThermospheric density influences the atmospheric drag and is crucial for space missions. This paper introduces a global thermospheric density prediction framework based on a deep evidential method. The proposed framework predicts thermospheric density at the required time and geographic position with given geomagnetic and solar indices. It is called global to differentiate it from existing research that only predicts density along a satellite orbit. Through the deep evidential method, we assimilate data from various sources including solar and geomagnetic conditions, accelerometer‐derived density data, and empirical models including the Jacchia‐Bowman model (JB‐2008) and the Naval Research Laboratory Mass Spectrometer and Incoherent Scatter Radar Extended (NRLMSISE‐00) model. The framework is investigated on five test cases along various satellites from 2003 to 2015 involving geomagnetic storms with Disturbance Storm Time (Dst) values smaller than −50 . Results show that the proposed framework can generate density with higher accuracy than the two empirical models. It can also obtain reliable uncertainty estimations. Global density estimations at altitudes from 200 to 550 km are also presented and compared with empirical models on both quiet and storm conditions.

Advanced AI, Machine Learning and Deep Learning Techniques for Climate Change Studies: A Review

Climate models have become an important tool in understanding as well as projecting future climate conditions. They thus become very critical to decision-making processes, especially in mitigation and adaptation strategies. This paper provides a general overview of methodologies that are applied in climate modeling, its validation, and uncertainty mapping, notably their strengths, limitations, and challenges. The paper takes the reader through all the intricacies associated with the modeling of climate systems: global circulation models, regional climate models, and statistical downscaling. Much emphasis is given to the issue of model choice depending on the research objectives and spatial scales. Thorough model validation is considered, emphasizing that observational data forms the core and good quality and availability are issues. Climate modeling should incorporate uncertainty quantification as a core activity. This paper debates estimation and representation techniques of model uncertainties applied, such as probabilistic approaches, sensitivity analysis, and ensemble modeling. Introduce an idea called here uncertainty mapping, as a means of communicating the climate-related risks. This paper sets this tone by asserting that uncertainties in climate modeling have to be communicated as clearly as possible to decision-makers. No matter how intrinsically complex and limited they are, climate models represent, by themselves, very valuable means for the task of estimating the probable futures of climates. This quest ranges from enhancing model accuracy to ever-lowering estimates of uncertainties, all impelled by new advances made in computational powers and deeper insights into the climate systems.

A probabilistic integration of LSTM and Gaussian process regression for uncertainty-aware reservoir water level predictions

ABSTRACT Reservoir-level forecasting, while being crucial for optimal operation, is challenged by complex physical processes and changing climate conditions. Machine learning approaches offer deterministic predictions but often neglect system physics and uncertainty. This article presents a probabilistic data-driven approach combining long short-term memory (LSTM) and Gaussian process regression (GPR) to provide both point forecasts and uncertainty estimates. The hybrid model leverages LSTM’s fitting capabilities with GPR’s robust Bayesian frameworks for uncertainty estimation in non-linear problems, offering accurate predictions without extensive high-fidelity modelling, and avoiding frequent training and parameter optimization. Evaluation with real reservoir data from India shows the model’s superiority over the vanilla LSTM for both univariate and multivariate scenarios. The proposed model achieved a Nash-Sutcliffe efficiency of 0.97 to 0.98, a mean biased error of −0.5634 to −1.0314 for 10-day forecasts, and a continuous ranked probability score of 5.80 and 1.87 for the Bhakra and Pong reservoirs, respectively.

Adaptive neural boundary control for multi-agent manipulators system with uncertainties through cooperative disturbance observers network

This paper addresses vibration control problem of multi-agent flexible manipulators systems in the presence of simultaneous uncertainty and unknown external disturbance. Particularly, the goal is to suppress vibration of both flexible link and joint angular. In this paper, the dynamic model of the considered flexible manipulator is described by the fourth order partial differential equation. Without control, the system is unstable and vibrate constantly due to initial states, the external unknown disturbances and system uncertainties. To compensate the uncertainty in each agent, the neural networks are employed and novel adaptation laws are developed to update weighting parameters in the neural networks. While for the compensation of the external disturbance a cooperative network of disturbance observers is proposed to enhance the observation reliability. With the resulting estimations of uncertainties and the unknown disturbance, adaptive distributed boundary controllers are derived to suppress vibration in-domain and keep joint angular position to zero. The closed-loop system is proven to be uniform ultimately bounded through Lyapunov stability theory. Numerical simulations result shows that compared with the proportional–derivative control, the proposed method almost reduces all overshoot and steady-state error.

Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances

Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances

DTAUE: A multi-input multi-branch decoupled semi-supervised 3D network based on threshold adaptive uncertainty estimation for echocardiography segmentation

Echocardiography is a non-invasive examination method based on sound waves to generate images of the heart. Segmentation of the left ventricle (LV) and right ventricle (RV) is crucial for evaluating cardiac function. However, due to the limitations in ultrasound imaging angles and the high prevalence of noise in ultrasound data, automatic and accurate segmentation of the left and right ventricles is a challenging task. In this paper, we present a decoupled semi-supervised 3D echocardiography segmentation network based on threshold adaptive uncertainty estimation. To minimize the impact of missing targets caused by blind zones on segmentation accuracy, pixel-wise segmentation maps and target skeleton maps are jointly predicted by decoupled semi-supervised 3D network (DTAUE) so that our network is more effective to perceive the morphological features of segmented targets. And quantifying the uncertainty inherent to a model’s prediction is a promising endeavor to address the lack of reliability and overconfidence in the model. Previous uncertainty estimation methods set a uniform threshold for all training samples. However, this leads to the loss of edge detail information. Different from the existing methods, in our work, we propose a method of adaptive threshold uncertainty estimation to set different confidence threshold for different targets. Experiments on our self-collected 3D echocardiographic dataset and two publicly available datasets show that our method outperforms state-of-the-art semi-supervised learning methods. DTAUE would potentially serve as a new tool for diagnosing and monitoring various heart conditions.

Calibrated Adaptive Teacher for Domain-Adaptive Intelligent Fault Diagnosis.

Intelligent fault diagnosis (IFD) based on deep learning can achieve high accuracy from raw condition monitoring signals. However, models usually perform well on the training distribution only, and experience severe performance drops when applied to a different distribution. This is also observed in fault diagnosis, where assets are often operated in working conditions different from the ones in which the labeled data have been collected. The scenario where labeled data are available in a source domain and only unlabeled data are available in a target domain has been addressed recently by unsupervised domain adaptation (UDA) approaches for IFD. Recent methods have relied on self-training with confident pseudo-labels for the unlabeled target samples. However, the confidence-based selection of pseudo-labels is hindered by poorly calibrated uncertainty estimates in the target domain, primarily due to over-confident predictions, which limits the quality of pseudo-labels and leads to error accumulation. In this paper, we propose a novel method called Calibrated Adaptive Teacher (CAT), where we propose to calibrate the predictions of the teacher network on target samples throughout the self-training process, leveraging post hoc calibration techniques. We evaluate CAT on domain-adaptive IFD and perform extensive experiments on the Paderborn University (PU) benchmark for fault diagnosis of rolling bearings under varying operating conditions, using both time- and frequency-domain inputs. We compare four different calibration techniques within our framework, where temperature scaling is both the most effective and lightweight one. The resulting method-CAT+TempScaling-achieves state-of-the-art performance on most transfer tasks, with on average 7.5% higher accuracy and 4 times lower calibration error compared to domain-adversarial neural networks (DANNs) across the twelve PU transfer tasks.