Quantification Of Uncertainty In Predictions Research Articles

Gaussian process regression for three-dimensional soil mapping over multiple spatial supports

This study investigates the complexity of spatial soil modelling, particularly focusing on the challenge of variable vertical support in traditional soil data collection. Traditional soil sampling, described in terms of horizons, often fails to accurately pinpoint the specific depths for specific soil properties. This gap is significant, as depth-specific data is crucial for a thorough understanding of soil formation processes and for assessing potential environmental impacts. In digital soil mapping (DSM), the prevalent reliance on standardised depth intervals and mass-preserving spline functions for data resampling results in a modelling approach that tends to disregard depth-related details, thereby introducing potential uncertainties.To address these limitations, this work explores how effectively Gaussian process regression (GPR) can model soil in 3D. This technique comprises two key components: a mean function and a semivariogram-like kernel. Unlike conventional methods that make a single prediction, GPR provides detailed probability distributions. This capability allows for the quantification of prediction uncertainty at various points, offering insights for decision-making and risk assessment purposes. Moreover, GPR has the capability to make volume or block estimates and assess associated uncertainties. Enabling volume-based predictions enriches the range of strategies available for land management.In this research, we employ GPR as a novel 3D soil modelling technique and compare its performance with traditional spline-based methods. The comparison is conducted through a case study on a farm in northern New South Wales, Australia, focusing on the 3D mapping of soil pH and electrical conductivity (EC). Our results demonstrate the ability of GPR to estimate soil properties across various volumes, utilising data from multiple vertical supports, thereby offering a more versatile approach for soil modelling in diverse spatial contexts.

A Bayesian inference approach to extract circuit model parameters and analyze photovoltaic degradation from power production data

Power production data can be a valuable resource to analyze photovoltaic (PV) performance without the need for field surveys. Recent work has demonstrated the exciting possibility of leveraging this data to extract circuit model parameters and current-voltage properties of a PV system, but further development is needed to bolster these initial findings. Here, instead of using a classical optimization approach, we switch to the Bayesian framework to solve this complex multi-solution problem. This allows us to construct probability distributions over the model parameters, get a comprehensive picture of the solution space, and quantify prediction uncertainty. As a result, we can define confidence intervals for the system’s electrical properties and consistently track their daily evolution. Our results are validated with laboratory measurements for five silicon and thin-film modules, and our scalable approach works with on-site as well as online weather data, which opens new prospects for remote PV monitoring, modeling, and degradation analysis for real-life applications.

Connecting spaceborne lidar with NFI networks: A method for improved estimation of forest structure and biomass

Spaceborne lidar provides a unique opportunity to supplement the field plot measurements of national forest inventories (NFIs) by providing dense measurements of vertical canopy structure. For full waveform instruments such as the Global Ecosystem Dynamics Investigation (GEDI), measurements take the form of reflected energy as a function of height within an observed footprint. Many forest attributes cannot be directly computed from the waveforms, and thus statistical models relating the measurements to target attributes must be trained using field plot data. Because the discrete footprint samples produced by spaceborne lidar instruments have little chance of collocation with pre-existing field plots, model calibration remains a primary challenge. We leveraged the density and spatial correlation of GEDI observations to make predictions of the cumulative waveform over 326,787 NFI plots distributed across the contiguous United States. The predictions yield probability distributions, giving not only a predicted waveform but quantification of prediction uncertainty. The product of this work is a data set comprised of the NFI plots paired with the predictive waveform distributions. The predicted waveforms and their uncertainties can be used to train models with “error-in-variables” techniques that account for and filter the uncertainty in the predicted waveforms. This data set and the corresponding techniques allow statistically rigorous model training without requirements of collocation, yielding downstream forest attribute predictions that are consistent with NFI measurements and estimates. We demonstrate the data set by training a model for forest above ground biomass density (AGBD) and then use the model to produce a spatially complete 1 km map of AGBD for the contiguous United States. We further compare our AGBD predictions to US NFI estimates at a 64,000 hectare scale, showing the increase in precision to be proportional to the collection of almost 500,000 additional NFI plots across forested regions of CONUS.

A multi-model evaluation of probabilistic streamflow predictions via residual error modelling

Probabilistic streamflow predictions are valuable tools for predictive uncertainty estimation, hydrologic risk management, and support for decision-making in water resources. Usually, predictive uncertainty quantification is developed and assessed using only a single hydrological model, making it difficult to generalize to other model configurations. To tackle this issue, we assess changes in the model performance ranking of diverse streamflow models by applying a residual error model post-processing approach to multiple basins and multiple models. This assessment employed 141 basins from the Great Lakes watershed covering the USA and Canada, and 13 diverse streamflow models, which are evaluated using deterministic and probabilistic performance metrics. As the first study to implement probabilistic methods to diverse streamflow models applied to a multitude of basins, the analysis here examines the dependence of probabilistic streamflow estimation quality on model quality. Our findings show that streamflow model choice influences the robustness of probabilistic predictions. It was found that moving from deterministic to probabilistic predictions using a post-processing approach does not change the streamflow model performance ranking for the best and worst deterministic models, but models of intermediate rank in deterministic evaluation do not have consistent ranking when evaluated in probabilistic mode. Post-processing residual errors of long short-term memory (LSTM) network models are consistently the best-performing model in terms of deterministic and probabilistic metrics. This study highlights the significance of combining deterministic streamflow model predictions with residual error models for improving the quality and increasing the value of hydrological predictions, quantifying uncertainty, and facilitating decision-making in operational water management. It also clarifies the degree to which probabilistic predictions depend upon good model performance and can compensate for poor model performance.

Piecewise Linear Transformation – Propagating Aleatoric Uncertainty in Neural Networks

Real-world data typically exhibit aleatoric uncertainty which has to be considered during data-driven decision-making to assess the confidence of the decision provided by machine learning models. To propagate aleatoric uncertainty represented by probability distributions (PDs) through neural networks (NNs), both sampling-based and function approximation-based methods have been proposed. However, these methods suffer from significant approximation errors and are not able to accurately represent predictive uncertainty in the NN output. In this paper, we present a novel method, Piecewise Linear Transformation (PLT), for propagating PDs through NNs with piecewise linear activation functions (e.g., ReLU NNs). PLT does not require sampling or specific assumptions about the PDs. Instead, it harnesses the piecewise linear structure of such NNs to determine the propagated PD in the output space. In this way, PLT supports the accurate quantification of predictive uncertainty based on the criterion exactness of the propagated PD. We assess this exactness in theory by showing error bounds for our propagated PD. Further, our experimental evaluation validates that PLT outperforms competing methods on publicly available real-world classification and regression datasets regarding exactness. Thus, the PDs propagated by PLT allow to assess the uncertainty of the provided decisions, offering valuable support.

Correcting missingness in passively-generated mobile data with Multi-Task Gaussian Processes

The prevalence of mobile devices and the ubiquity of network connectivity have generated a massive amount of temporally- and spatially-stamped data. A key characteristic of this mobile data is the prevalence of sparsity—for every recorded user, there is a significant amount of missing data. Sparsity leads to bias in the inferred mobility patterns and thus, correcting bias by imputing the missing data potentially creates opportunities for directly using the corrected mobile trajectory data for large-scale simulations in various applications. We propose a multi-task Gaussian process regression model to correct missingness in mobile data. Gaussian processes (GPs) allow for flexible modeling of diverse (i.e., non-linear, locally periodic) data patterns and quantify prediction uncertainty in an interpretable manner. We develop a methodological framework for applying GPs to mobile data. In doing so, we consider the correlations between users’ coordinates (latitudes and longitudes) through multi-task learning and adjust for individual-level differences in data characteristics through parameter initialization and optimization. We introduce and demonstrate the use of smooth (i.e., rational quadratic) and periodic kernels in modeling human mobility data. Relatedly, we analyze our model’s parameters and imputation accuracy with respect to different types of trips (e.g. slower vs faster trips). We also demonstrate our model’s performance in two experiments with real app-based data, in which it outperforms alternative imputation methods. Our implementation is open-source at https://github.com/ekinugurel/GPSImpute.

Uncertainty Quantification of Capacitor Switching Transient Location Using Machine Learning

Identification of capacitor switching transient location provides valuable insight into the state of the associated equipment. Machine learning (ML) models, and convolutional neural networks (CNNs) in particular, have demonstrated remarkable performance in signal location. However, ML models are data driven whose predictions are affected by noise in data and may also suffer from large extrapolation errors when applied to new conditions. Uncertainty quantification (UQ) is necessary to ensure model trustworthiness and avoid overconfident predictions in extrapolation. In this work, we propose a novel UQ method, called PI3NN, to quantify prediction uncertainty of ML models and integrate the method with CNNs for transient source location. PI3NN calculates Prediction Intervals by training 3 Neural Networks and uses root-finding methods to determine the interval precisely. Additionally, PI3NN can identify out-of-distribution (OOD) data in a nonstationary condition to avoid overconfident prediction. Results indicate that with PI3NN, transient signals are not only correctly identified, but when said signals are subject to corruptions characteristic of an actual power monitoring system (e.g non-ideal sensors), the model recognizes when it is uncertain about its predictions, effectively letting the user know when to accept or discard the results.

Machinery health prognostic with uncertainty for mineral processing using TSC-TimeGAN

Deploying health state (HS) assessment and remaining useful life (RUL) prognostic on the industrial equipment can effectively avoid shutdown and reduce productivity losses. This paper proposes a temporal self-clustering (TSC) and time-series generative adversarial network (TimeGAN) based uncertainty-aware health prognostic algorithm for the hydrocyclone, a critical machine in the mineral processing. To label the HS of degradation data, an autoencoder based TSC algorithm is proposed. It integrates the extraction and clustering of health indicators into a unified framework, so that the correlation information between feature extraction and clustering can be captured for cluster analysis. A Benders decomposition based optimization approach is designed to synthetically optimize network weights and clustering centers, thereby extracting and labeling data features more reasonably. For quantifying prediction uncertainty, an uncertainty-aware prognostic algorithm is proposed to predict the RUL distribution, and the uncertainty can be quantified as the standard deviation (Std) of the distribution. Furthermore, a mechanism and TimeGAN fusion temporal sequence generation algorithm is proposed to augment the degradation process sequences of the equipment. Verification experiments using the actual data show that the proposed algorithm can effectively evaluate the HS of the equipment and provide a sensible and convincing RUL prediction.

Short-term prediction of celestial pole offsets with interpretable machine learning

The difference between observed and modelled precession/nutation reveals unmodelled signals commonly referred to as Celestial Pole Offsets (CPO), denoted by dX and dY. CPO are currently observed only by Very Long Baseline Interferometry (VLBI), but there is nearly 4 weeks of latency by which the data centers provide the most accurate, final CPO series. This latency problem necessitates predicting CPO for high-accuracy, real-time applications that require information regarding Earth rotation, such as spacecraft navigation. Even though the International Earth Rotation and Reference Systems Service (IERS) provides so-called rapid CPO, they are usually less accurate and therefore, may not satisfy the requirements of the mentioned applications. To enhance the quality of CPO predictions, we present a new methodology based on Neural Additive Models (NAMs), a class of interpretable machine learning algorithms. We formulate the problem based on long short-term memory neural networks and derive simple analytical relations for the quantification of prediction uncertainty and feature importance, thereby enhancing the intelligibility of predictions made by machine learning. We then focus on the short-term prediction of CPO with a forecasting horizon of 30 days. We develop an operational framework that consistently provides CPO predictions. Using the CPO series of Jet Propulsion Laboratory as the input to the algorithm, we show that NAMs predictions improve the IERS rapid products on average by 57% for dX and 25% for dY under fully operational conditions. Our predictions are both accurate and overcome the latency issue of final CPO series and thus, can be used in real-time applications.Graphical

An integrated Bayesian framework for multi-omics prediction and classification.

With the growing commonality of multi-omics datasets, there is now increasing evidence that integrated omics profiles lead to more efficient discovery of clinically actionable biomarkers that enable better disease outcome prediction and patient stratification. Several methods exist to perform host phenotype prediction from cross-sectional, single-omics data modalities but decentralized frameworks that jointly analyze multiple time-dependent omics data to highlight the integrative and dynamic impact of repeatedly measured biomarkers are currently limited. In this article, we propose a novel Bayesian ensemble method to consolidate prediction by combining information across several longitudinal and cross-sectional omics data layers. Unlike existing frequentist paradigms, our approach enables uncertainty quantification in prediction as well as interval estimation for a variety of quantities of interest based on posterior summaries. We apply our method to four published multi-omics datasets and demonstrate that it recapitulates known biology in addition to providing novel insights while also outperforming existing methods in estimation, prediction, and uncertainty quantification. Our open-source software is publicly available at https://github.com/himelmallick/IntegratedLearner.

Empirical Quantification of Predictive Uncertainty Due to Model Discrepancy by Training with an Ensemble of Experimental Designs: An Application to Ion Channel Kinetics

When using mathematical models to make quantitative predictions for clinical or industrial use, it is important that predictions come with a reliable estimate of their accuracy (uncertainty quantification). Because models of complex biological systems are always large simplifications, model discrepancy arises—models fail to perfectly recapitulate the true data generating process. This presents a particular challenge for making accurate predictions, and especially for accurately quantifying uncertainty in these predictions. Experimentalists and modellers must choose which experimental procedures (protocols) are used to produce data used to train models. We propose to characterise uncertainty owing to model discrepancy with an ensemble of parameter sets, each of which results from training to data from a different protocol. The variability in predictions from this ensemble provides an empirical estimate of predictive uncertainty owing to model discrepancy, even for unseen protocols. We use the example of electrophysiology experiments that investigate the properties of hERG potassium channels. Here, ‘information-rich’ protocols allow mathematical models to be trained using numerous short experiments performed on the same cell. In this case, we simulate data with one model and fit it with a different (discrepant) one. For any individual experimental protocol, parameter estimates vary little under repeated samples from the assumed additive independent Gaussian noise model. Yet parameter sets arising from the same model applied to different experiments conflict—highlighting model discrepancy. Our methods will help select more suitable ion channel models for future studies, and will be widely applicable to a range of biological modelling problems.

BayFlux: A Bayesian method to quantify metabolic Fluxes and their uncertainty at the genome scale.

Metabolic fluxes, the number of metabolites traversing each biochemical reaction in a cell per unit time, are crucial for assessing and understanding cell function. 13C Metabolic Flux Analysis (13C MFA) is considered to be the gold standard for measuring metabolic fluxes. 13C MFA typically works by leveraging extracellular exchange fluxes as well as data from 13C labeling experiments to calculate the flux profile which best fit the data for a small, central carbon, metabolic model. However, the nonlinear nature of the 13C MFA fitting procedure means that several flux profiles fit the experimental data within the experimental error, and traditional optimization methods offer only a partial or skewed picture, especially in "non-gaussian" situations where multiple very distinct flux regions fit the data equally well. Here, we present a method for flux space sampling through Bayesian inference (BayFlux), that identifies the full distribution of fluxes compatible with experimental data for a comprehensive genome-scale model. This Bayesian approach allows us to accurately quantify uncertainty in calculated fluxes. We also find that, surprisingly, the genome-scale model of metabolism produces narrower flux distributions (reduced uncertainty) than the small core metabolic models traditionally used in 13C MFA. The different results for some reactions when using genome-scale models vs core metabolic models advise caution in assuming strong inferences from 13C MFA since the results may depend significantly on the completeness of the model used. Based on BayFlux, we developed and evaluated novel methods (P-13C MOMA and P-13C ROOM) to predict the biological results of a gene knockout, that improve on the traditional MOMA and ROOM methods by quantifying prediction uncertainty.

Perspectives of 3D Probabilistic Subsoil Modeling for BIM

Building information modeling (BIM) in the planning and construction of infrastructure projects, such as roads, tunnels, and excavations, requires the generation of comprehensive 3D subsoil models that encompass relevant geological and geotechnical information. Presently, this process relies on the deterministic interpolation of discrete data points obtained from exploratory boreholes and soundings, resulting in a single deterministic prediction. Commonly employed interpolation methods for this purpose include radial basis function and kriging. This contribution introduces probabilistic methods for quantifying prediction uncertainty. The proposed modeling approach is illustrated using simple examples, demonstrating how to use sequential Gaussian and Indicator Simulation techniques to model sedimentary processes such as erosion and lenticular bedding. Subsequently, a site in Munich serves as a case study. The widely used industry foundation classes (IFC) schema allows the integration of the model into the BIM environment. A mapping procedure allows transferring voxel models to the IFC schema. This article discusses the significance of incorporating uncertainty quantification into subsoil modeling and shows its integration into the BIM framework. The proposed approach and its efficient integration with evolving BIM standards and methodologies provides valuable insights for the planning and construction of infrastructure projects.

A Calibration Technique for Measuring Nickel Plating Thickness

A calibration technique was developed to predict nickel plating thickness of a manufactured part. The technique was used to replace a destructive scanning electron microscope (SEM) measurement, with a nondestructive X-ray fluorescence (XRF) measurement. The XRF measurement was used to screen suspect parts from a rejected lot, and to replace lot-by-lot sampling with screening for ongoing product acceptance. The key to this approach was calibration and quantification of prediction uncertainty. Guardbanding, based on prediction uncertainty, was used to minimize the probability of accepting defective product. A significant number of suspect parts from a rejected lot were screened and deemed acceptable for use in next assembly with minimal risk.

Quantifying Prediction Uncertainty in Regression Using Random Fuzzy Sets: The ENNreg Model

We introduce a neural network model for regression in which prediction uncertainty is quantified by Gaussian random fuzzy numbers (GRFNs), a newly introduced family of random fuzzy subsets of the real line that generalizes both Gaussian random variables and Gaussian possibility distributions. The output GRFN is constructed by combining GRFNs induced by prototypes using a combination operator that generalizes Dempster's rule of Evidence Theory. The three output units indicate the most plausible value of the response variable, variability around this value, and epistemic uncertainty. The network is trained by minimizing a loss function that generalizes the negative log-likelihood. Comparative experiments show that this method is competitive, both in terms of prediction accuracy and calibration error, with state-of-the-art techniques such as random forests or deep learning with Monte Carlo dropout. In addition, the model outputs a predictive belief function that can be shown to be calibrated, in the sense that it allows us to compute conservative prediction intervals with specified belief degree.

Scalable GP with hyperparameters sharing based on transfer learning for solving expensive optimization problems

Surrogates are essential in surrogate-assisted evolutionary algorithms (SAEAs) for solving expensive optimization problems. Gaussian processes (GPs) are often used as surrogates for their accuracy in prediction and ability to quantify prediction uncertainty. However, calculating the inverse and determinant of the covariance matrix in GPs is computationally expensive because of its cubic complexity. To tackle the issue, this paper proposes a scalable GP with hyperparameters sharing based on transfer learning. A linear predictor adaptively transfers the hyperparameters knowledge from the source full GPs (FGPs) to the target GP. The transfer is performed probabilistically based on the similarity between the distributions of the training data of FGPs and target GP. In this way, the number of building FGPs is significantly reduced. As a result, the optimization cost of the algorithms is also reduced. The scalable GP can be used in SAEAs to solve expensive optimization problems. The effectiveness of the proposed method is confirmed through testing on expensive benchmark problems and a real-world antenna design problem.

Approximate blocked Gibbs sampling for Bayesian neural networks

In this work, minibatch MCMC sampling for feedforward neural networks is made more feasible. To this end, it is proposed to sample subgroups of parameters via a blocked Gibbs sampling scheme. By partitioning the parameter space, sampling is possible irrespective of layer width. It is also possible to alleviate vanishing acceptance rates for increasing depth by reducing the proposal variance in deeper layers. Increasing the length of a non-convergent chain increases the predictive accuracy in classification tasks, so avoiding vanishing acceptance rates and consequently enabling longer chain runs have practical benefits. Moreover, non-convergent chain realizations aid in the quantification of predictive uncertainty. An open problem is how to perform minibatch MCMC sampling for feedforward neural networks in the presence of augmented data.

MagicScaler: Uncertainty-Aware, Predictive Autoscaling

Predictive autoscaling is a key enabler for optimizing cloud resource allocation in Alibaba Cloud's computing platforms, which dynamically adjust the Elastic Compute Service (ECS) instances based on predicted user demands to ensure Quality of Service (QoS). However, user demands in the cloud are often highly complex, with high uncertainty and scale-sensitive temporal dependencies, thus posing great challenges for accurate prediction of future demands. These in turn make autoscaling challenging---autoscaling needs to properly account for demand uncertainty while maintaining a reasonable trade-off between two contradictory factors, i.e., low instance running costs vs. low QoS violation risks. To address the above challenges, we propose a novel predictive autoscaling framework MagicScaler , consisting of a Multi-scale attentive Gaussian process based predictor and an uncertainty-aware scaler. First, the predictor carefully bridges the best of two successful prediction methodologies---multi-scale attention mechanisms, which are good at capturing complex, multi-scale features, and stochastic process regression, which can quantify prediction uncertainty, thus achieving accurate demand prediction with quantified uncertainty. Second, the scaler takes the quantified future demand uncertainty into a judiciously designed loss function with stochastic constraints, enabling flexible trade-off between running costs and QoS violation risks. Extensive experiments on three clusters of Alibaba Cloud in different Chinese cities demonstrate the effectiveness and efficiency of MagicScaler , which outperforms other commonly adopted scalers, thus justifying our design choices.

Data space inversion for efficient uncertainty quantification using an integrated surface and sub-surface hydrologic model

Abstract. It is incumbent on decision-support hydrological modelling to make predictions of uncertain quantities in a decision-support context. In implementing decision-support modelling, data assimilation and uncertainty quantification are often the most difficult and time-consuming tasks. This is because the imposition of history-matching constraints on model parameters usually requires a large number of model runs. Data space inversion (DSI) provides a highly model-run-efficient method for predictive uncertainty quantification. It does this by evaluating covariances between model outputs used for history matching (e.g. hydraulic heads) and model predictions based on model runs that sample the prior parameter probability distribution. By directly focusing on the relationship between model outputs under historical conditions and predictions of system behaviour under future conditions, DSI avoids the need to estimate or adjust model parameters. This is advantageous when using integrated surface and sub-surface hydrologic models (ISSHMs) because these models are associated with long run times, numerical instability and ideally complex parameterization schemes that are designed to respect geological realism. This paper demonstrates that DSI provides a robust and efficient means of quantifying the uncertainties of complex model predictions. At the same time, DSI provides a basis for complementary linear analysis that allows the worth of available observations to be explored, as well as of observations which are yet to be acquired. This allows for the design of highly efficient, future data acquisition campaigns. DSI is applied in conjunction with an ISSHM representing a synthetic but realistic river–aquifer system. Predictions of interest are fast travel times and surface water infiltration. Linear and non-linear estimates of predictive uncertainty based on DSI are validated against a more traditional uncertainty quantification which requires the adjustment of a large number of parameters. A DSI-generated surrogate model is then used to investigate the effectiveness and efficiency of existing and possible future monitoring networks. The example demonstrates the benefits of using DSI in conjunction with a complex numerical model to quantify predictive uncertainty and support data worth analysis in complex hydrogeological environments.

A two-stage approach based on Bayesian deep learning for predicting remaining useful life of rolling element bearings

Remaining useful life (RUL) prediction of rolling element bearings is critical to maintaining rotating machinery and lowering industrial costs. There are many RUL prediction techniques, but most of them ignore two factors that may have a significant impact on prediction accuracy. One is the detection of the first predicting time (FPT) while the other is the predictive uncertainty. This paper proposes a two-stage approach to incorporating both factors into the prediction process based on Bayesian deep learning (BDL). In stage one, the state change of the bearing is identified and the FPT is determined according to a proposed detection technique. In stage two, RUL prediction is performed according to a new BDL model, and the results provide RUL point estimates and quantification of predictive uncertainty. The proposed two-stage approach has been validated on two publicly available bearing datasets, and the experimental results have demonstrated the effectiveness of the proposed approach in detecting FPT and its superiority over competitive BDL models.