Reliable Uncertainty Estimates Research Articles

Renormalization-group improved resummation of super-leading logarithms

A new strategy is presented for systematically treating super-leading logarithmic contributions including higher-order Glauber exchanges for non-global LHC observables in renormalization-group (RG) improved perturbation theory. This represents an important improvement over previous approaches, as it allows for the consistent inclusion of the scale dependence of the strong coupling, thereby providing more reliable estimates of the scale uncertainties in theoretical predictions. The key idea is to rearrange the relevant RG evolution operator in such a way that all double-logarithmic corrections are exponentiated from the outset. This forms the starting point for the first resummation of super-leading logarithms at leading order in RG-improved perturbation theory for arbitrary 2 → M scattering processes. Moreover, the asymptotic scaling of subleading logarithmic corrections from higher-order Glauber exchanges is determined, demonstrating their parametric suppression.

Sensitivity of GNSS to vertical land motion over Europe: effects of geophysical loadings and common-mode errors

We perform a statistical sensitivity analysis on a parametric fit to vertical daily displacement time series of 244 European Permanent GNSS stations, with a focus on linear vertical land motion (VLM), i.e., station velocity. We compare two independent corrections to the raw (uncorrected) observed displacements. The first correction is physical and accounts for non-tidal atmospheric, non-tidal oceanic and hydrological loading displacements, while the second approach is an empirical correction for the common-mode errors. For the uncorrected case, we show that combining power-law and white noise stochastic models with autoregressive models yields adequate noise approximations. With this as a realistic baseline, we report improvement rates of about 14% to 24% in station velocity sensitivity, after corrections are applied. We analyze the choice of the stochastic models in detail and outline potential discrepancies between the GNSS-observed displacements and those predicted by the loading models. Furthermore, we apply restricted maximum likelihood estimation (RMLE), to remove low-frequency noise biases, which yields more reliable velocity uncertainty estimates. RMLE reveals that for a number of stations noise is best modeled by a combination of random walk, flicker noise, and white noise. The sensitivity analysis yields minimum detectable VLM parameters (linear velocities, seasonal periodic motions, and offsets), which are of interest for geophysical applications of GNSS, such as tectonic or hydrological studies.

Bayesian Uncertainty Estimation by Hamiltonian Monte Carlo: Applications to Cardiac MRI Segmentation

Deep learning (DL)-based methods have achieved state-of-the-art performance for many medical image segmentation tasks. Nevertheless, recent studies show that deep neural net- works (DNNs) can be miscalibrated and overconfident, leading to ”silent failures” that are risky for clinical applications. Bayesian DL provides an intuitive approach to DL failure de- tection, based on posterior probability estimation. However, the posterior is intractable for large medical image segmentation DNNs. To tackle this challenge, we propose a Bayesian learning framework using Hamiltonian Monte Carlo (HMC), tempered by cold posterior (CP) to accommodate medical data augmentation, named HMC-CP. For HMC compu- tation, we further propose a cyclical annealing strategy, capturing both local and global geometries of the posterior distribution, enabling highly efficient Bayesian DNN training with the same computational budget as training a single DNN. The resulting Bayesian DNN outputs an ensemble segmentation along with the segmentation uncertainty. We evaluate the proposed HMC-CP extensively on cardiac magnetic resonance image (MRI) segmentation, using in-domain steady-state free precession (SSFP) cine images as well as out-of-domain datasets of quantitative T1 and T2 mapping. Our results show that the proposed method improves both segmentation accuracy and uncertainty estimation for in- and out-of-domain data, compared with well-established baseline methods such as Monte Carlo Dropout and Deep Ensembles. Additionally, we establish a conceptual link between HMC and the commonly known stochastic gradient descent (SGD) and provide general insight into the uncertainty of DL. This uncertainty is implicitly encoded in the training dynamics but often overlooked. With reliable uncertainty estimation, our method provides a promising direction toward trustworthy DL in clinical applications. We release our code in <a href='https://gitlab.tudelft.nl/yidongzhao/hmc_uncertainty'>https://gitlab.tudelft.nl/yidongzhao/hmc_uncertainty</a>

Deep Kernel learning for reaction outcome prediction and optimization

Recent years have seen a rapid growth in the application of various machine learning methods for reaction outcome prediction. Deep learning models have gained popularity due to their ability to learn representations directly from the molecular structure. Gaussian processes (GPs), on the other hand, provide reliable uncertainty estimates but are unable to learn representations from the data. We combine the feature learning ability of neural networks (NNs) with uncertainty quantification of GPs in a deep kernel learning (DKL) framework to predict the reaction outcome. The DKL model is observed to obtain very good predictive performance across different input representations. It significantly outperforms standard GPs and provides comparable performance to graph neural networks, but with uncertainty estimation. Additionally, the uncertainty estimates on predictions provided by the DKL model facilitated its incorporation as a surrogate model for Bayesian optimization (BO). The proposed method, therefore, has a great potential towards accelerating reaction discovery by integrating accurate predictive models that provide reliable uncertainty estimates with BO.

Reduced‐Order Probabilistic Emulation of Physics‐Based Ring Current Models: Application to RAM‐SCB Particle Flux

AbstractIn this work, we address the computational challenge of large‐scale physics‐based simulation models for the ring current. Reduced computational cost allows for significantly faster than real‐time forecasting, enhancing our ability to predict and respond to dynamic changes in the ring current, valuable for space weather monitoring and mitigation efforts. Additionally, it can also be used for a comprehensive investigation of the system. Thus, we aim to create an emulator for the Ring current‐Atmosphere interactions Model with Self‐Consistent magnetic field (RAM‐SCB) particle flux that not only improves efficiency but also facilitates forecasting with reliable estimates of prediction uncertainties. The probabilistic emulator is built upon the methodology developed by Licata and Mehta (2023), https://doi.org/10.1029/2022sw003345. A novel discrete sampling is used to identify 30 simulation periods over 20 years of solar and geomagnetic activity. Focusing on a subset of particle flux, we use Principal Component Analysis for dimensionality reduction and Long Short‐Term Memory (LSTM) neural networks to perform dynamic modeling. Hyperparameter space was explored extensively resulting in about 5% median symmetric accuracy across all data sets for one‐step dynamic prediction. Using a hierarchical ensemble of LSTMs, we have developed a reduced‐order probabilistic emulator (ROPE) tailored for time‐series forecasting of particle flux in the ring current. This ROPE offers accurate predictions of omnidirectional flux at a single energy with no pitch angle information, providing robust predictions on the test set with an error score below 11% and calibration scores under 8% with bias under 2% providing a significant speed up as compared to the full RAM‐SCB run.

An Optimized Uncertainty-Aware Training Framework for Neural Networks.

Uncertainty quantification (UQ) for predictions generated by neural networks (NNs) is of vital importance in safety-critical applications. An ideal model is supposed to generate low uncertainty for correct predictions and high uncertainty for incorrect predictions. The main focus of state-of-the-art training algorithms is to optimize the NN parameters to improve the accuracy-related metrics. Training based on uncertainty metrics has been fully ignored or overlooked in the literature. This article introduces a novel uncertainty-aware training algorithm for classification tasks. A novel predictive uncertainty estimate-based objective function is defined and optimized using the stochastic gradient descent method. This new multiobjective loss function covers both accuracy and uncertainty accuracy (UA) simultaneously during training. The performance of the proposed training framework is compared from different aspects with other UQ techniques for different benchmarks. The obtained results demonstrate the effectiveness of the proposed framework for developing the NN models capable of generating reliable uncertainty estimates.

DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentation

The intersection of deep learning and photogrammetry unveils a critical need for balancing the power of deep neural networks with interpretability and trustworthiness, especially for safety-critical application like autonomous driving, medical imaging, or machine vision tasks with high demands on reliability. Quantifying the predictive uncertainty is a promising endeavour to open up the use of deep neural networks for such applications. Unfortunately, most current available methods are computationally expensive. In this work, we present a novel approach for efficient and reliable uncertainty estimation for semantic segmentation, which we call Deep Uncertainty Distillation using Ensembles for Segmentation (DUDES). DUDES applies student-teacher distillation with a Deep Ensemble to accurately approximate predictive uncertainties with a single forward pass while maintaining simplicity and adaptability. Experimentally, DUDES accurately captures predictive uncertainties without sacrificing performance on the segmentation task and indicates impressive capabilities of highlighting wrongly classified pixels and out-of-domain samples through high uncertainties on the Cityscapes and Pascal VOC 2012 dataset. With DUDES, we manage to simultaneously simplify and outperform previous work on Deep-Ensemble-based Uncertainty Distillation.

Integrating Dropout and Kullback-Leibler Regularization in Bayesian Neural Networks for improved uncertainty estimation in Regression

The objective of the study is to enhance uncertainty prediction in regression problems by introducing a revolutionary Bayesian Neural Network (BNN) model. Experimental results reveal significant improvements in uncertainty prediction and point forecasts with the integrated BNN model compared to the plain BNN. Performance metrics, including mean squared error (MSE), mean absolute error (MAE), and R-squared (R²), demonstrate superior results for the proposed BNN. The experimental results show that for plain BNN, MSE is 87.3, MAE is 6.62 and R2 is −0.0492, whereas for the proposed BNN model MSE was found to be 44.64, MAE is 4.4 and R2 is 0.46. This research brings a fresh approach to Bayesian Neural Networks by incorporating both dropout and KL regularization techniques, resulting in a powerful tool for handling regression tasks with certainty. By combining these techniques, the study enhances model stability, avoids overfitting, and achieves more reliable uncertainty estimation. This study adds to our knowledge of uncertainty-aware machine learning models and offers a valuable solution for accurately assessing uncertainty in various applications.•The innovative BNN model merges the power of Bayesian principles with the effectiveness of dropout and KL regularization.•To test and refine our model, the study utilizes the Boston Housing dataset for both training and evaluation purposes.

Computation of prediction intervals for forest aboveground biomass predictions using generalized linear models in a large-extent boreal forest region

Abstract Remotely sensed data have an important application for estimation of forest variables, e.g. height, volume, and aboveground biomass (AGB). The increased use of remotely sensed data implemented along with model-based inference has shown improved efficiency in prediction and mapping of such forest variables. In this study, plot-level airborne laser scanning data and Swedish National Forest Inventory field reference data were used to predict AGB using generalized linear models (GLMs) assuming Gamma and Tweedie distributions for the field observed AGB. The GLMs were selected considering the convenience of not correcting transformation bias as it is required in other regression models with transformed response variable. To overcome the challenge in providing reliable uncertainty estimates for the estimated forest variable map products at individual pixel-scale, we focused on computing 95% prediction intervals (PIs) for Gamma and Tweedie GLMs with a square root link function. The relative uncertainties were computed as the ratio between the half-width of the PIs and the predicted AGBs. The AGB-airborne laser scanning models were developed with root mean square error values of 22.6 Mgha−1 (26%) and 21.7 Mgha−1 (25%), respectively, for the Gamma and Tweedie GLMs. Two methods were applied to compute PIs for the Gamma GLM, one using the R package ‘ciTools’ and another derived through asymptotic theory. It was found that the 95% PIs computed using ‘ciTools’ had the most accurate coverage probability in comparison to the other method. An extended version of these PIs was also utilized for the Tweedie GLMs. The range of PIs associated with the prediction of AGB were narrower for lower predicted AGB values compared with the length of higher predicted AGB values. Comparing the two fitted models, the Gamma GLM showed lower relative uncertainties for the lower range of predicted AGBs, whereas the Tweedie GLM showed lower relative uncertainties for the higher range of predicted AGBs. Overall, the Tweedie GLM provided a better model fit for AGB predictions.

Bayesian Proper Orthogonal Decomposition for Learnable Reduced-Order Models With Uncertainty Quantification

Designing and/or controlling complex systems in science and engineering relies on appropriate mathematical modeling of systems dynamics. Classical differential equation based solutions in applied and computational mathematics are often computationally demanding. Recently, the connection between reduced-order models of high-dimensional differential equation systems and surrogate machine learning models has been explored. However, the focus of both existing reduced-order and machine learning models for complex systems has been how to best approximate the high fidelity model of choice. Due to high complexity and often limited training data to derive reduced-order or machine learning surrogate models, it is critical for derived reduced-order models to have reliable uncertainty quantification at the same time. In this paper, we propose such a novel framework of Bayesian reduced-order models naturally equipped with uncertainty quantification as it learns the distributions of the parameters of the reduced-order models instead of their point estimates. In particular, we develop <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">learnable Bayesian proper orthogonal decomposition</i> (BayPOD) that learns the distributions of both the POD projection bases and the mapping from the system input parameters to the projected scores/coefficients so that the learned BayPOD can help predict high-dimensional systems dynamics/fields as quantities of interest in different setups with reliable uncertainty estimates. The developed learnable BayPOD inherits the capability of embedding physics constraints when learning the POD-based surrogate reduced-order models, a desirable feature when studying complex systems in science and engineering applications where the available training data are limited. Furthermore, the proposed BayPOD method is an end-to-end solution, which unlike other surrogate-based methods, does not require separate POD and machine learning steps. The results from a real-world case study of the pressure field around an airfoil have shown the potential of learnable BayPOD as a new family of reduced-order models with reliable uncertainty estimates.

Reconstructing axion-like particles from beam dumps with simulation-based inference

Axion-like particles (ALPs) that decay into photon pairs pose a challenge for experiments that rely on the construction of a decay vertex in order to search for long-lived particles. This is particularly true for beam-dump experiments, where the distance between the unknown decay position and the calorimeter can be very large. In this work we use machine learning to explore the possibility to reconstruct the ALP properties, in particular its mass and lifetime, from such inaccurate observations. We use a simulation-based inference approach based on conditional invertible neural networks to reconstruct the posterior probability of the ALP parameters for a given set of events. We find that for realistic angular and energy resolution, such a neural network significantly outperforms parameter reconstruction from conventional high-level variables while at the same time providing reliable uncertainty estimates. Moreover, the neural network can quickly be re-trained for different detector properties, making it an ideal framework for optimizing experimental design.

Uncertainty-aware Health Diagnostics via Class-balanced Evidential Deep Learning.

Uncertainty quantification is critical for ensuring the safety of deep learning-enabled health diagnostics, as it helps the model account for unknown factors and reduces the risk of misdiagnosis. However, existing uncertainty quantification studies often overlook the significant issue of class imbalance, which is common in medical data. In this paper, we propose a class-balanced evidential deep learning framework to achieve fair and reliable uncertainty estimates for health diagnostic models. This framework advances the state-of-the-art uncertainty quantification method of evidential deep learning with two novel mechanisms to address the challenges posed by class imbalance. Specifically, we introduce a pooling loss that enables the model to learn less biased evidence among classes and a learnable prior to regularize the posterior distribution that accounts for the quality of uncertainty estimates. Extensive experiments using benchmark data with varying degrees of imbalance and various naturally imbalanced health data demonstrate the effectiveness and superiority of our method. Our work pushes the envelope of uncertainty quantification from theoretical studies to realistic healthcare application scenarios. By enhancing uncertainty estimation for class-imbalanced data, we contribute to the development of more reliable and practical deep learning-enabled health diagnostic systems1.

A Solution Verification Study for URANS Simulations of Flow Over a 5:1 Rectangular Cylinder Using Grid Convergence Index and Least Squares Procedures

Abstract A verification study was conducted for URANS (Unsteady Reynolds-Averaged Navier–Stokes) simulations of flow around a 5:1 rectangular cylinder at a Reynolds number of 56,700 (based on the cylinder depth) using the k–ω SST (shear stress transport) turbulence model and the γ−Reθ transition model for three types of grids (a fully structured grid and two hybrid grids generated using Delaunay and advancing front techniques). The grid convergence index (GCI) and least squares (LS) procedures were employed to estimate discretization error and associated uncertainties. The result indicates that the LS procedure provides the most reliable estimates of discretization error uncertainties for solution variables in the structured grid from the k–ω SST model. From the six solution variables of interest, the highest relative uncertainty was typically observed in the root-mean-square (rms) of lift coefficient, followed by time-averaged reattachment length and peak rms of pressure coefficient on the top and bottom surfaces of the cylinder. The solution variable with the lowest uncertainty was Strouhal number, followed by time-averaged drag coefficient. It is also noted that the GCI and LS procedures produce noticeably different uncertainty estimates, primarily due to inconsistences in their estimated observed orders of accuracy and safety factors. To successfully apply the procedures to practical problems, further research is required to reliably estimate uncertainties in solutions with “noisy” grid convergence behaviors and observed orders of accuracy.

Data-driven based fracture prediction of notched components.

A data-driven approach is developed to predict the fracture load of a notched component. To do so, more than 1500 fracture tests (507 unique experimental data points) on mixed-mode I/II loading of notched brittle samples were collected from the literature. After pre-processing the raw data, six features of maximum tangential stress [Formula: see text], maximum tangential stress angle [Formula: see text], ultimate tensile strength [Formula: see text], fracture toughness [Formula: see text], notch opening angle [Formula: see text] and notch tip radius [Formula: see text] were selected by using the neighbourhood component analysis (NCA) technique. To predict the fracture load of various types of notched samples, several machine learning (ML) models were trained using the methods of Gaussian process regression (GPR), decision tree ensemble and artificial neural network (ANN). Then, the Bayesian optimization algorithm was applied to find the optimum hyperparameters for each model. Lastly, the performance of the models in predicting fracture load was evaluated against 124 unseen data points. The results revealed the high potential of data-driven methods for assessing the fracture load of notched brittle components with acceptable precisions of 92%, 89% and 88% accuracy, respectively, for GPR, decision tree ensemble and ANN models. The superior performance of the GPR method can be attributed to its ability to capture complex nonlinear relationships in the data while providing reliable uncertainty estimates. Furthermore, thanks to its interpolation capabilities, GPR is able to seamlessly fill the gaps between data points, resulting in more comprehensive and precise predictions across the entire range of input data. Additionally, the presented models were capable of predicting the fracture load of VO-shaped notched samples with acceptable accuracy, though this type of notch was not used in the model training process. This article is part of the theme issue 'Physics-informed machine learning and its structural integrity applications (Part 2)'.

Compound Continental Risk of Multiple Extreme Floods in the United States

AbstractUnderstanding spatially correlated floods and modeling joint hazard associated with threshold exceedances across multiple locations is crucial for accurate estimation of continental‐scale portfolio risk. This work uses a non‐parametric copula‐based spatial simulator to analyze peak floods across the United States to derive the first‐of‐its‐kind continental portfolio risk estimates at the 10‐ and 100‐year return levels. We find significant interdependence in floods across the nation, revealing the recurring pattern of extreme events affecting the Northeast, Central, West, and Northwest United States in the same year. The stochastic simulator effectively manages high‐dimensional data and offers reliable uncertainty estimates for both spatially dependent floods and the aggregated flood losses at the continental level. El Niño‐Southern Oscillation and Atlantic Multidecadal Oscillation are identified as statistically significant tele‐connectors of aggregate loss. This research aims to advance the understanding of compound continental flood hazard and the potential large‐scale climate teleconnections that lead to such compound floods.

Observationally Constrained Projections of Twenty-First-Century Regional Warming in the Extratropical Northern Hemisphere

Abstract Physically based observational constraint methods can effectively reduce uncertainty in global warming projections but have not been widely applied at regional scales. We first develop and apply multivariate linear regression models for constraining projections of surface air temperature averaged over subcontinental regions in the extratropical Northern Hemisphere, based on a set of potential constraints including climatological metrics derived from tropical and subtropical low-level cloud and global average past warming trend, as well as a set of regional climate metrics previously used in the literature. We evaluate the performance of the multivariate linear regression models based on cross-validated tests using output from phases 5 and 6 of the Coupled Model Intercomparison Projects (CMIP). We find that linear regression models using global-scale low-cloud metrics alone perform more robustly than linear regression models using the past global mean warming trend or regional climate metrics as constraints. These results, while favoring global constraints over the set of regional constraints considered, do not preclude the existence of even better regional constraints for particular regions. Through model-based cross-validation, the projections constrained using low-level cloud metrics exhibit more accurate best estimate projections, narrower uncertainty ranges, and more reliable uncertainty estimates in most Northern Hemisphere regions when compared with unconstrained projections. Application of the approach to climate projections based on both Shared Socioeconomic Pathway (SSP) 1-2.6 and SSP5-8.5 using observed low-cloud metrics results in considerably narrower 5%–95% uncertainty ranges of twenty-first-century warming over subcontinental Northern Hemisphere land regions compared to unconstrained projections.

Improving the statistical reliability of river model predictions via simple state adjustments

A major challenge in hydrologic modelling is producing reliable uncertainty estimates outside of calibration periods. One obvious strategy is to improve model structures utilising advancements in process knowledge and observations. Another statistically relevant approach is to develop improved error models so that the uncertainty can be more reliably estimated. The recently introduced river bed/bank storage (RBS) model is an improved representation of transmission losses/gains within basin-scale river system models. The novelty of the current research is the way the RBS model is combined with State and Parameter Uncertainty Estimation (SPUE) to produce more statistically reliable predictions. Using the RBS model with SPUE resulted in better matched predictive distributions in 13 out of 16 test cases, and higher proportions of observed values within the predictive ranges for all cases. The paper demonstrates that improving model structures combined with better characterisation of state error can alleviate issues of overfitting in predictions.

Uncertainty of feed forward neural networks recognizing quantum contextuality

The usual figure of merit characterizing the performance of neural networks applied to problems in the quantum domain is their accuracy, being the probability of a correct answer on a previously unseen input. Here we append this parameter with the uncertainty of the prediction, characterizing the degree of confidence in the answer. A powerful technique for estimating uncertainty is provided by Bayesian neural networks (BNNs). We first give simple illustrative examples of advantages brought forward by BNNs, out of which we wish to highlight their ability of reliable uncertainty estimation even after training with biased datasets. Then we apply BNNs to the problem of recognition of quantum contextuality, which shows that the uncertainty itself is an independent parameter identifying the chance of misclassification of contextuality.

Explainable machine learning model for multi-step forecasting of reservoir inflow with uncertainty quantification

We propose an explainable machine learning (ML) model with uncertainty quantification (UQ) to improve multi-step reservoir inflow forecasting. Traditional ML methods have challenges in forecasting inflows multiple days ahead, and lack explainability and UQ. To address these limitations, we introduce an encoder–decoder long short-term memory (ED-LSTM) network for multi-step forecasting, employ the SHapley Additive exPlanation (SHAP) technique for understanding the influence of hydrometeorological factors on inflow prediction, and develop a novel UQ method for prediction trustworthiness. We apply these methods to forecast 7-day inflow in snow-dominant and rain-driven reservoirs. The results demonstrate the effectiveness of the ED-LSTM model, with high forecasting accuracy for short lead times. Our UQ method provides reliable uncertainty estimates, covering 90% of data with a 90% confidence level. The SHAP analysis reveals the importance of historical inflow and precipitation as influential factors. These findings and methods may support reservoir operators in optimizing water resources management decisions.

Constrained optimization of offshore wind turbine positions under uncertain wind conditions with correlated data

Offshore wind power production is subject to substantial uncertainty due to variability in wind conditions. Planning of new wind parks should take these uncertainties into account by means of stochastic modeling and uncertainty quantification. Wind speed and wind direction exhibit dependence that needs to be properly modeled to yield reliable uncertainty estimates. In particular, the dependence is strong when data are averaged over a short time-horizon, e.g., on a monthly rather than an annual basis. In this work we introduce a stochastic model for wind speed and wind direction using Rosenblatt transformation and an empirical model. This allows efficient numerical quadrature that replaces thousands of Monte Carlos samples by a few tens of model evaluations at quadrature points in parametric space. A robust design formulation for maximizing power production honoring dependent data is proposed and demonstrated on data from the North Sea site Sørlige Nordsjø II.