Model Uncertainty Quantification Research Articles

Uncertainty quantification and sensitivity analysis of neuron models with ion concentration dynamics.

This paper provides a comprehensive and computationally efficient case study for uncertainty quantification (UQ) and global sensitivity analysis (GSA) in a neuron model incorporating ion concentration dynamics. We address how challenges with UQ and GSA in this context can be approached and solved, including challenges related to computational cost, parameters affecting the system's resting state, and the presence of both fast and slow dynamics. Specifically, we analyze the electrodiffusive neuron-extracellular-glia (edNEG) model, which captures electrical potentials, ion concentrations (Na+, K+, Ca2+, and Cl-), and volume changes across six compartments. Our methodology includes a UQ procedure assessing the model's reliability and susceptibility to input uncertainty and a variance-based GSA identifying the most influential input parameters. To mitigate computational costs, we employ surrogate modeling techniques, optimized using efficient numerical integration methods. We propose a strategy for isolating parameters affecting the resting state and analyze the edNEG model dynamics under both physiological and pathological conditions. The influence of uncertain parameters on model outputs, particularly during spiking dynamics, is systematically explored. Rapid dynamics of membrane potentials necessitate a focus on informative spiking features, while slower variations in ion concentrations allow a meaningful study at each time point. Our study offers valuable guidelines for future UQ and GSA investigations on neuron models with ion concentration dynamics, contributing to the broader application of such models in computational neuroscience.

Quantification of modeling uncertainty in the Rayleigh damping model

AbstractUnderstanding and accurately characterizing energy dissipation mechanisms in civil structures during earthquakes is an important element of seismic assessment and design. The most commonly used model is attributed to Rayleigh. This paper proposes a systematic approach to quantify the uncertainty associated with Rayleigh's damping model. Bayesian calibration with embedded model error is employed to treat the coefficients of the Rayleigh model as random variables using modal damping ratios. Through a numerical example, we illustrate how this approach works and how the calibrated model can address modeling uncertainty associated with the Rayleigh damping model.

Uncertainty qualification in seismic analysis of concrete dams based on model order reduction accelerated stochastic SBFEM

A pioneering framework for uncertainty quantification in seismic analysis of concrete dams rooted in the stochastic scaled boundary finite element method (SSBFEM) is proposed. For seismic analysis of dams, octree decomposition is used for mesh generation. SSBFEM is developed to predict the structural responses with randomly distributed material properties. The script in this paper can integrate octree mesh and SSBFEM into ABAQUS user element (UEL). Implementing octree SSBFEM on commercial software provides accurate and convenient full order models for uncertainty quantification. Monte Carlo simulation (MCs) is deployed to calculate the statistical characteristics of structural responses under random variables. Additionally, singular value decomposition (SVD) and radial basis function (RBF) are leveraged to refine traditional MCs. The solution space of MCs is decomposed into lower-order subspaces. Any structural responses can be rapidly evaluated from linear combinations of subspaces. Particularly, the method has been successfully applied to seismic analysis of concrete dams with different random material properties. Finally, several illustrative examples are provided to validate the accuracy and efficacy of the algorithm.

Towards trustworthy cybersecurity operations using Bayesian Deep Learning to improve uncertainty quantification of anomaly detection

Uncertainty quantification of cybersecurity anomaly detection results provides critical guidance for decision makers on whether or not to accept the results. Improving the trustworthiness of anomaly predictions can reduce the amount of alert false positives that security teams have to process. In this work we investigate the use of Bayesian Autoencoder (BAE) models for uncertainty quantification in anomaly detection. A novel heteroscedastic aleatoric uncertainty modeling method is explored that jointly considers aleatoric and epistemic uncertainty. Heteroscedastic aleatoric uncertainty is modelled on the latent layer of the BAE and further explored through considering the variational lower bound. An uncertainty quantification framework for cybersecurity is designed and verified on UNSW-NB15 and CIC-IDS-2017 data sets. This research enhances the modelling of uncertainty in the BAE model and expands its application in cybersecurity.

Uncertainty quantification of mass models using ensemble Bayesian model averaging

Uncertainty quantification of mass models using ensemble Bayesian model averaging

REHEATFUNQ (REgional HEAT-Flow Uncertainty and aNomaly Quantification) 2.0.1: a model for regional aggregate heat flow distributions and anomaly quantification

Abstract. Surface heat flow is a geophysical variable that is affected by a complex combination of various heat generation and transport processes. The processes act on different lengths scales, from tens of meters to hundreds of kilometers. In general, it is not possible to resolve all processes due to a lack of data or modeling resources, and hence the heat flow data within a region is subject to residual fluctuations. We introduce the REgional HEAT-Flow Uncertainty and aNomaly Quantification (REHEATFUNQ) model, version 2.0.1. At its core, REHEATFUNQ uses a stochastic model for heat flow within a region, considering the aggregate heat flow to be generated by a gamma-distributed random variable. Based on this assumption, REHEATFUNQ uses Bayesian inference to (i) quantify the regional aggregate heat flow distribution (RAHFD) and (ii) estimate the strength of a given heat flow anomaly, for instance as generated by a tectonically active fault. The inference uses a prior distribution conjugate to the gamma distribution for the RAHFDs, and we compute parameters for a uninformed prior distribution from the global heat flow database by Lucazeau (2019). Through the Bayesian inference, our model is the first of its kind to consistently account for the variability in regional heat flow in the inference of spatial signals in heat flow data. Interpretation of these spatial signals and in particular their interpretation in terms of fault characteristics (particularly fault strength) form a long-standing debate within the geophysical community. We describe the components of REHEATFUNQ and perform a series of goodness-of-fit tests and synthetic resilience analyses of the model. While our analysis reveals to some degree a misfit of our idealized empirical model with real-world heat flow, it simultaneously confirms the robustness of REHEATFUNQ to these model simplifications. We conclude with an application of REHEATFUNQ to the San Andreas fault in California. Our analysis finds heat flow data in the Mojave section to be sufficient for an analysis and concludes that stochastic variability can allow for a surprisingly large fault-generated heat flow anomaly to be compatible with the data. This indicates that heat flow alone may not be a suitable quantity to address fault strength of the San Andreas fault.

Variational inference: uncertainty quantification in additive models

AbstractMarkov chain Monte Carlo (MCMC)-based simulation approaches are by far the most common method in Bayesian inference to access the posterior distribution. Recently, motivated by successes in machine learning, variational inference (VI) has gained in interest in statistics since it promises a computationally efficient alternative to MCMC enabling approximate access to the posterior. Classical approaches such as mean-field VI (MFVI), however, are based on the strong mean-field assumption for the approximate posterior where parameters or parameter blocks are assumed to be mutually independent. As a consequence, parameter uncertainties are often underestimated and alternatives such as semi-implicit VI (SIVI) have been suggested to avoid the mean-field assumption and to improve uncertainty estimates. SIVI uses a hierarchical construction of the variational parameters to restore parameter dependencies and relies on a highly flexible implicit mixing distribution whose probability density function is not analytic but samples can be taken via a stochastic procedure. With this paper, we investigate how different forms of VI perform in semiparametric additive regression models as one of the most important fields of application of Bayesian inference in statistics. A particular focus is on the ability of the rivalling approaches to quantify uncertainty, especially with correlated covariates that are likely to aggravate the difficulties of simplifying VI assumptions. Moreover, we propose a method, where we combine both advantages of MFVI and SIVI and compare its performance. The different VI approaches are studied in comparison with MCMC in simulations and an application to tree height models of douglas fir based on a large-scale forestry data set.

Quantification of Uncertainty Cost Functions for Controllable Solar Power Modeling

Navigating the secheduling of generation resources of energy in power systems marked by a significant presence of renewable generation involves intricate optimization challenges. The conventional tools for resolving such challenges include programming techniques and heuristic approaches, both contingent upon a precisely articulated target function for optimization. Traditional optimization tools rely on precisely defined target functions, but the evolving landscape of power systems introduces complexity, especially with unpredictable behaviors of renewable sources. The research specifically quantifies penalty costs associated with photovoltaic (PV) generators, employing probabilistic methods for a robust mathematical analysis. The developed analytical model enhances adaptability in economic dispatch problems, considering uncertainty in decision-making. Validation using Monte Carlo simulation emphasizes uncertainty in PV generation and highlights the advantages of the proposed analytic model. The quadratic form of the model aligns coherently with simulation outcomes, contributing significantly to understanding uncertainty quantification in solar power modeling. The research aims to refine controllable solar power models, establish robust uncertainty cost functions, and improve the accuracy of economic dispatch strategies. Ultimately, this work promotes the seamless integration of solar energy into diverse and dynamic energy grids.

A probabilistic spatio-temporal neural network to forecast COVID-19 counts

AbstractGeo-referenced and temporal data are becoming more and more ubiquitous in a wide range of fields such as medicine and economics. Particularly in the realm of medical research, spatio-temporal data play a pivotal role in tracking and understanding the spread and dynamics of diseases, enabling researchers to predict outbreaks, identify hot spots, and formulate effective intervention strategies. To forecast these types of data we propose a Probabilistic Spatio-Temporal Neural Network that (1) estimates, with computational efficiency, models with spatial and temporal components; and (2) combines the flexibility of a Neural Network—which is free from distributional assumptions—with the uncertainty quantification of probabilistic models. Our architecture is compared with the established INLA method, as well as with other baseline models, on COVID-19 data from Italian regions. Our empirical analysis demonstrates the superior predictive effectiveness of our method across multiple temporal ranges and offers insights for shaping targeted health interventions and strategies.

Uncertainty-aware image classification on 3D CT lung

Early detection is crucial for lung cancer to prolong the patient’s survival. Existing model architectures used in such systems have shown promising results. However, they lack reliability and robustness in their predictions and the models are typically evaluated on a single dataset, making them overconfident when a new class is present. With the existence of uncertainty, uncertain images can be referred to medical experts for a second opinion. Thus, we propose an uncertainty-aware framework that includes three phases: data preprocessing and model selection and evaluation, uncertainty quantification (UQ), and uncertainty measurement and data referral for the classification of benign and malignant nodules using 3D CT images. To quantify the uncertainty, we employed three approaches; Monte Carlo Dropout (MCD), Deep Ensemble (DE), and Ensemble Monte Carlo Dropout (EMCD). We evaluated eight different deep learning models consisting of ResNet, DenseNet, and the Inception network family, all of which achieved average F1 scores above 0.832, and the highest average value of 0.845 was obtained using InceptionResNetV2. Furthermore, incorporating the UQ demonstrated significant improvement in the overall model performance. Upon evaluation of the uncertainty estimate, MCD outperforms the other UQ models except for the metric, URecall, where DE and EMCD excel, implying that they are better at identifying incorrect predictions with higher uncertainty levels, which is vital in the medical field. Finally, we show that using a threshold for data referral can greatly improve the performance further, increasing the accuracy up to 0.959.

Future flood envelope curves for the estimation of design flood magnitudes for highway bridges at river crossings

Creager flood envelope curves, which serve as the upper bound/limit of observed extreme flows, are commonly used by practitioners to estimate design flood magnitudes, which in the case of most river-crossing highway bridges is 75-year flood magnitude in Canada. This study proposes a novel framework for climate change adaption of Creager curves for estimating future design floods. These curves, for the current period, are assessed considering 417 observation stations, located in seven major Canadian river basins (i.e., Fraser, Nelson, Mackenzie, Yukon, Churchill, St Lawrence and St John). The Creager coefficient C, which defines flood envelope curves, varies between 1 and 45 across the studied river basins. To adapt Creager curves for future changes in streamflow, a correction factor, RC, which is the ratio of future to current period C values, is proposed. These factors are obtained for observation sites, using streamflow data from an ensemble of Regional Climate Model (RCM) simulations for current and future periods, through two Regional Frequency Analysis approaches. The first approach, considering only the RCM cells where the stations are located, suggests RC in the 0.3–1.6 range, with southeasterly basins showing values < 1. The second approach, considering all RCM cells for a given region, yields a wider range for RC and adds useful information in that RC values can also be established at ungauged locations. From a practical viewpoint, the proposed framework for estimating future design floods is robust and transferrable to other basins, but can benefit using streamflow projections from other models for better uncertainty quantification.

Quantifying model uncertainty for semantic segmentation of Fluorine-19 MRI using stochastic gradient MCMC

Fluorine-19 (19F) MRI is an emerging theranostic tool for studying diseases and treatments simultaneously, particularly in challenging neuroinflammatory conditions. However, the low signal-to-noise ratio (SNR) of 19F MRI necessitates computational methods to reliably detect 19F signal regions and segment these from the background. In this study, we demonstrate that Bayesian fully convolutional neural networks provide a means to increase sensitivity in 19F MRI and simultaneously provide estimates of data uncertainty. While our model effectively denoises the data, uncertain areas remain, particularly in boundary regions of the foreground. The uncertainty estimates are beneficial in preventing overconfident downstream analysis on noisy data and providing crucial information for rectifying prediction errors. Our results demonstrate that our model significantly outperforms other commonly used methods for 19F MRI signal detection in terms of sensitivity, while also providing valuable uncertainty estimates.

Artificial intelligence for atrial fibrillation detection, prediction, and treatment: A systematic review of the last decade (2013–2023)

AbstractAtrial fibrillation (AF) affects more than 30 million individuals worldwide, making it the most prevalent cardiac arrhythmia on a global scale. This systematic review summarizes recent advancements in applying artificial intelligence (AI) techniques for AF detection, prediction, and guiding treatment selection and risk stratification. In adherence with the PRISMA guidelines (Preferred Reporting Items for Systematic Reviews and Meta‐Analyses), a total of 171 pertinent studies conducted between 2013 and 2023 were examined. Studies applying machine learning (ML) and deep learning (DL) to electrocardiogram (ECG), photoplethysmography (PPG), wearable data, and other sources were evaluated. For AF detection, most works employed DL (48 studies) and ML (28 studies) on ECG data. DL methods directly analyzed ECG waveforms and outperformed approaches relying on hand‐crafted features. For prediction and risk stratification, 22 studies used ML while 7 leveraged DL on ECG. An emerging trend showed the growing potential of PPG for AF screening. Overall, AI demonstrated promising capabilities across various AF‐related tasks. However, real‐world implementation faces challenges including a lack of interpretability, the need for multimodal data integration, prospective performance validation, and regulatory compliance. Future research directions involve quantifying model uncertainty, enhancing transparency, and conducting population‐based clinical trials to facilitate translation into practice.This article is categorized under: Application Areas &gt; Health Care Application Areas &gt; Science and Technology Technologies &gt; Artificial Intelligence

Prediction of the SYM‐H Index Using a Bayesian Deep Learning Method With Uncertainty Quantification

AbstractWe propose a novel deep learning framework, named SYMHnet, which employs a graph neural network and a bidirectional long short‐term memory network to cooperatively learn patterns from solar wind and interplanetary magnetic field parameters for short‐term forecasts of the SYM‐H index based on 1‐ and 5‐min resolution data. SYMHnet takes, as input, the time series of the parameters' values provided by NASA's Space Science Data Coordinated Archive and predicts, as output, the SYM‐H index value at time point t + w hours for a given time point t where w is 1 or 2. By incorporating Bayesian inference into the learning framework, SYMHnet can quantify both aleatoric (data) uncertainty and epistemic (model) uncertainty when predicting future SYM‐H indices. Experimental results show that SYMHnet works well at quiet time and storm time, for both 1‐ and 5‐min resolution data. The results also show that SYMHnet generally performs better than related machine learning methods. For example, SYMHnet achieves a forecast skill score (FSS) of 0.343 compared to the FSS of 0.074 of a recent gradient boosting machine (GBM) method when predicting SYM‐H indices (1 hr in advance) in a large storm (SYM‐H = −393 nT) using 5‐min resolution data. When predicting the SYM‐H indices (2 hr in advance) in the large storm, SYMHnet achieves an FSS of 0.553 compared to the FSS of 0.087 of the GBM method. In addition, SYMHnet can provide results for both data and model uncertainty quantification, whereas the related methods cannot.

Efficient thermodynamic model optimization and uncertainty quantification via integration of combinatorial materials chip and Bayesian approach

Combinatorial materials chips (CMC) are composition spread thin-films deposited on a substrate that can rapidly determine the compositional-temperature information of ternary phase diagrams by high-throughput characterization. In this study, phase boundary information determined by CMC from the recent studies was used as input to optimize the thermodynamic model parameters by ESPEI based on Bayesian approach. The uncertainties in the model were also quantified. Fe–Cr–Ni and Fe–Co–Ni systems were chosen as examples to demonstrate the effectiveness of process. The newly optimized phase diagrams are in good agreement with the experimental results. It is also verified that the model can be effectively optimized by the dense CMC phase boundary data only without including the scattered experimental data. Moreover, the uncertainty in the model is reduced with the addition of CMC data. Finally, probability distributions of various phase combinations in the phase diagram were calculated, providing confidence intervals for material design.

A practice-oriented framework for stationary and nonstationary flood frequency analysis

In flood frequency analysis (FFA), choices of distribution and methods can hinder the reproducibility of results. Besides, changes in climate, land use/cover, and water management can induce nonstationarity. Frameworks to select between stationary FFA (S-FFA) and nonstationary FFA (NS-FFA) are lacking, and NS-FFA tools are limited. Therefore, this paper introduces a systematic and software-supported framework enabling repeatable workflows for both S-FFA and NS-FFA. The framework has three modules to a) process flood series for exploratory data analysis (EDA) and NS-FFA model determination (if needed), b) select the S-FFA or NS-FFA approach underpinned by the EDA, and c) perform FFA including model determination, parameter estimation, uncertainty quantification, and model performance assessment. The framework incorporates various distributions, methods, and metrics, and recent advancements in NS-FFA for model determination and uncertainty quantification and allows for the modeller's intervention while ensuring reproducibility. The software is freely available to the public.

Local Identifiability Analysis, Parameter Subset Selection and Verification for a Minimal Brain PBPK Model.

Physiologically-based pharmacokinetic (PBPK) modeling is important for studying drug delivery in the central nervous system, including determining antibody exposure, predicting chemical concentrations at target locations, and ensuring accurate dosages. The complexity of PBPK models, involving many variables and parameters, requires a consideration of parameter identifiability; i.e., which parameters can be uniquely determined from data for a specified set of concentrations. We introduce the use of a local sensitivity-based parameter subset selection algorithm in the context of a minimal PBPK (mPBPK) model of the brain for antibody therapeutics. This algorithm is augmented by verification techniques, based on response distributions and energy statistics, to provide a systematic and robust technique to determine identifiable parameter subsets in a PBPK model across a specified time domain of interest. The accuracy of our approach is evaluated for three key concentrations in the mPBPK model for plasma, brain interstitial fluid and brain cerebrospinal fluid. The determination of accurate identifiable parameter subsets is important for model reduction and uncertainty quantification for PBPK models.

A framework for parameter estimation, sensitivity analysis, and uncertainty analysis for holistic hydrologic modeling using SWAT+

Abstract. Parameter sensitivity analysis plays a critical role in efficiently determining main parameters, enhancing the effectiveness of the estimation of parameters and uncertainty quantification in hydrologic modeling. In this paper, we demonstrate an uncertainty and sensitivity analysis technique for the holistic Soil and Water Assessment Tool (SWAT+) model coupled with new gwflow module, spatially distributed, physically based groundwater flow modeling. The main calculated groundwater inflows and outflows include boundary exchange, pumping, saturation excess flow, groundwater–surface water exchange, recharge, groundwater–lake exchange and tile drainage outflow. We present the method for four watersheds located in different areas of the United States for 16 years (2000–2015), emphasizing regions of extensive tile drainage (Winnebago River, Minnesota, Iowa), intensive surface–groundwater interactions (Nanticoke River, Delaware, Maryland), groundwater pumping for irrigation (Cache River, Missouri, Arkansas) and mountain snowmelt (Arkansas Headwaters, Colorado). The main parameters of the coupled SWAT+gwflow model are estimated utilizing the parameter estimation software PEST. The monthly streamflow of holistic SWAT+gwflow is evaluated based on the Nash–Sutcliffe efficiency index (NSE), percentage bias (PBIAS), determination coefficient (R2) and Kling–Gupta efficiency coefficient (KGE), whereas groundwater head is evaluated using mean absolute error (MAE). The Morris method is employed to identify the key parameters influencing hydrological fluxes. Furthermore, the iterative ensemble smoother (iES) is utilized as a technique for uncertainty quantification (UQ) and parameter estimation (PE) and to decrease the computational cost owing to the large number of parameters. Depending on the watershed, key identified selected parameters include aquifer specific yield, aquifer hydraulic conductivity, recharge delay, streambed thickness, streambed hydraulic conductivity, area of groundwater inflow to tile, depth of tiles below ground surface, hydraulic conductivity of the drain perimeter, river depth (for groundwater flow processes), runoff curve number (for surface runoff processes), plant uptake compensation factor, soil evaporation compensation factor (for potential and actual evapotranspiration processes), soil available water capacity and percolation coefficient (for soil water processes). The presence of gwflow parameters permits the recognition of all key parameters in the surface and/or subsurface flow processes, with results substantially differing if the base SWAT+ models are utilized.

Remaining-useful-lifetime prediction of proton exchange membrane fuel cell considering model uncertainty quantification on the full-time scale

Remaining-useful-lifetime prediction of proton exchange membrane fuel cell considering model uncertainty quantification on the full-time scale

Implementing measurement error models with mechanistic mathematical models in a likelihood-based framework for estimation, identifiability analysis and prediction in the life sciences.

Throughout the life sciences, we routinely seek to interpret measurements and observations using parametrized mechanistic mathematical models. A fundamental and often overlooked choice in this approach involves relating the solution of a mathematical model with noisy and incomplete measurement data. This is often achieved by assuming that the data are noisy measurements of the solution of a deterministic mathematical model, and that measurement errors are additive and normally distributed. While this assumption of additive Gaussian noise is extremely common and simple to implement and interpret, it is often unjustified and can lead to poor parameter estimates and non-physical predictions. One way to overcome this challenge is to implement a different measurement error model. In this review, we demonstrate how to implement a range of measurement error models in a likelihood-based framework for estimation, identifiability analysis and prediction, called profile-wise analysis. This frequentist approach to uncertainty quantification for mechanistic models leverages the profile likelihood for targeting parameters and understanding their influence on predictions. Case studies, motivated by simple caricature models routinely used in systems biology and mathematical biology literature, illustrate how the same ideas apply to different types of mathematical models. Open-source Julia code to reproduce results is available on GitHub.