Measure Of Prediction Uncertainty Research Articles

Accelerating Optimal Synthesis of Atomically Thin MoS2: A Constrained Bayesian Optimization Guided Brachistochrone Approach

AbstractA machine learning (ML) guided approach is presented for the accelerated optimization of chemical vapor deposition (CVD) synthesis of 2D materials toward the highest quality, starting from low‐quality or unsuccessful synthesis conditions. Using 26 sets of these synthesis conditions as the initial training dataset, our method systematically guides experimental synthesis towards optoelectronic‐grade monolayer MoS2 flakes. A‐exciton linewidth (σA) as narrow as 38 meV could be achieved in 2D MoS2 flakes after only an additional 35 trials (reflecting 15% of the full factorial design dataset for training purposes). In practical terms, this reflects a decrease of the possible experimental time to optimize the parameters from up to one year to about two months. This remarkable efficiency was achieved by formulating a constrained sequencing optimization problem solved via a combination of constraint learning and Bayesian Optimization with the narrowness of σA as the single target metric. By employing graph‐based semi‐supervised learning with data acquired through a multi‐criteria sampling method, the constraint model effectively delineates and refines the feasible design space for monolayer flake production. Additionally, the Gaussian Process regression effectively captures the relationships between synthesis parameters and outcomes, offering high predictive capability along with a measure of prediction uncertainty. This method is scalable to a higher number of synthesis parameters and target metrics and is transferrable to other materials and types of reactors. This study envisions that this method will be fundamental for CVD and similar techniques in the future.

A Bayesian Deep Learning Approach to Near‐Term Climate Prediction

AbstractSince model bias and associated initialization shock are serious shortcomings that reduce prediction skills in state‐of‐the‐art decadal climate prediction efforts, we pursue a complementary machine‐learning‐based approach to climate prediction. The example problem setting we consider consists of predicting natural variability of the North Atlantic sea surface temperature on the interannual timescale in the pre‐industrial control simulation of the Community Earth System Model. While previous works have considered the use of recurrent networks such as convolutional LSTMs and reservoir computing networks in this and other similar problem settings, we currently focus on the use of feedforward convolutional networks. In particular, we find that a feedforward convolutional network with a Densenet architecture is able to outperform a convolutional LSTM in terms of predictive skill. Next, we go on to consider a probabilistic formulation of the same network based on Stein variational gradient descent and find that in addition to providing useful measures of predictive uncertainty, the probabilistic (Bayesian) version improves on its deterministic counterpart in terms of predictive skill. Finally, we characterize the reliability of the ensemble of machine learning models obtained in the probabilistic setting by using analysis tools developed in the context of ensemble numerical weather prediction.

Employing Bayesian Quadrature to Improve Fitting of Surrogate Models to Wind Turbine Loads

Stochastic, high-fidelity simulations are increasingly used to estimate wind turbine and plant loads and performance. While these simulations are more accurate, they are prohibitively computationally expensive. Surrogate models can be used to replace direct simulation for lower computational cost; for stochastic surfaces, a Gaussian Process (GP) is appropriate for random variables that follow Gaussian distributions, such as turbine loads and performance.In this study, we employ an advanced surrogate modeling technique to estimate the loads, reliability, and energy production of wind turbines in a wind plant, based on mid-fidelity aero-elastic simulations. Specifically, we use Bayesian Quadrature (BQ) to produce a training data set that, for a given size, minimizes the variance over the entire domain of a GP surrogate model to estimate rotor torque load on a waked turbine. When applied to a GP that was trained with aero-elastic simulation data, the BQ-enabled algorithm reduced variance—a measure of predictive uncertainty—over the entire GP to < 0.3 with 30 samples. When compared to a uniform sampling approach with the same solution space, the BQ method reduces computational time by 99.88%, or over 590 simulation evaluations.

Mapping of soil organic carbon using machine learning models: Combination of optical and radar remote sensing data

AbstractSoil organic C (SOC) plays an important role in soil quality. Thus, it is of great significance to know the spatial distribution characteristics of SOC. Environmental variables, such as natural predictors, remote sensing (RS) variables, and digital soil mapping approaches have been widely used in SOC prediction. However, it is still challenging to determine which methods and variables are effective to map SOC in farmland. In this study, we compare the performance of three machine learning models, including random forest, Cubist, and the support vector machine model for the prediction of SOC contents using RS variables, topographic variables, climate variables, soil property variables, and spatial position variables as environmental covariates. The recursive feature elimination (RFE) technique was used to select optimal variables. Of the 130 sampling sites of the cultivated layer that were collected in Shanggao county, 70% of the data were used for training and 30% for validation. Prediction accuracy and uncertainty was quantified using the bootstrap approach with 100 iterations. The performance of the model was verified by the mean absolute error, RMSE, R2, and Lin's concordance correlation coefficient (CCC); the SD was used as a measure of prediction uncertainty. Results showed that the Cubist model combined with the RFE method (Cubist_RFE) had a mean R2, RMSE, and SD of .84, 2.39 g kg–1, and 0.73 g kg–1, respectively, which showed the highest prediction accuracy and the lowest uncertainty. The soil property variables (soil alkaline hydrolysis N and pH) and RS variables (soil adjusted vegetation index, enhanced vegetation index, and band7 of Sentinel‐2) were the controlling factors of SOC prediction. Thus, the results indicate the potential of the Cubist model in SOC prediction, and the combination of the optical and radar RS data is a viable way for predicting SOC.

Automatic segmentation with detection of local segmentation failures in cardiac MRI

Segmentation of cardiac anatomical structures in cardiac magnetic resonance images (CMRI) is a prerequisite for automatic diagnosis and prognosis of cardiovascular diseases. To increase robustness and performance of segmentation methods this study combines automatic segmentation and assessment of segmentation uncertainty in CMRI to detect image regions containing local segmentation failures. Three existing state-of-the-art convolutional neural networks (CNN) were trained to automatically segment cardiac anatomical structures and obtain two measures of predictive uncertainty: entropy and a measure derived by MC-dropout. Thereafter, using the uncertainties another CNN was trained to detect local segmentation failures that potentially need correction by an expert. Finally, manual correction of the detected regions was simulated in the complete set of scans of 100 patients and manually performed in a random subset of scans of 50 patients. Using publicly available CMR scans from the MICCAI 2017 ACDC challenge, the impact of CNN architecture and loss function for segmentation, and the uncertainty measure was investigated. Performance was evaluated using the Dice coefficient, 3D Hausdorff distance and clinical metrics between manual and (corrected) automatic segmentation. The experiments reveal that combining automatic segmentation with manual correction of detected segmentation failures results in improved segmentation and to 10-fold reduction of expert time compared to manual expert segmentation.

BayesCap: A Bayesian Approach to Brain Tumor Classification Using Capsule Networks

Convolutional neural networks (CNNs), which have been the state-of-the-art in many image-related applications, are prone to losing important spatial information between image instances. Capsule networks (CapsNets), on the other hand, are capable of leveraging such information through their routing by agreement process, making them powerful architectures for small datasets, such as medical imaging ones. Within the domain of medical imaging problems, brain tumor classification is of paramount importance, due to the deadly nature of this cancer and the consequences of the tumor misclassification. In our recent works, we showed potentials of developing CapsNet architecture for the task of brain tumor type classification. Similar to other deep learning models, however, CapsNets do not capture prediction uncertainty (coming from the uncertainty in the model weights, which is significantly important in keeping the human experts in the loop, by returning the uncertain samples. In this paper, we propose a Bayesian CapsNet framework, referred to as the BayesCap, that can provide not only the mean predictions, but also entropy as a measure of prediction uncertainty. Results show that filtering out the uncertain predictions can improve the accuracy, confirming that returning the uncertain predictions is an appropriate strategy for improving interpretability of the network.

Adaptive Gaussian process emulators for efficient reliability analysis

• Computational resources are used efficiently through combination of subset simulations and Gaussian process emulation. • Multimodal failure domains are efficiently handled with the use of clustering techniques. • The selection of new points, duration of learning and emulator quality are controlled adaptively. • The algorithm is readily extensible for use in standard and reliability-based design optimisation . This paper presents an approximation method for performing efficient reliability analysis with complex computer models. The computational cost of industrial-scale models can cause problems when performing sampling-based reliability analysis. This is due to the fact that the failure modes of the system typically occupy a small region of the performance space and thus require relatively large sample sizes to accurately estimate their characteristics. The sequential sampling method proposed in this article, combines Gaussian process-based optimisation and subset simulation . Gaussian process emulators construct a statistical approximation to the output of the original code, which is both affordable to use and has its own measure of predictive uncertainty. Subset simulation is used as an integral part of the algorithm to efficiently populate those regions of the surrogate which are likely to lead to the performance function exceeding a predefined critical threshold. The emulator itself is used to inform decisions about efficiently using the original code to augment its predictions. The iterative nature of the method ensures that an arbitrarily accurate approximation of the failure region is developed at a reasonable computational cost. The presented method is applied to an industrial model of a biodiesel filter.

Predictive Inference for Big, Spatial, Non‐Gaussian Data: MODIS Cloud Data and its Change‐of‐Support

SummaryRemote sensing of the earth with satellites yields datasets that can be massive in size, nonstationary in space, and non‐Gaussian in distribution. To overcome computational challenges, we use the reduced‐rank spatial random effects (SRE) model in a statistical analysis of cloud‐mask data from NASA's Moderate Resolution Imaging Spectroradiometer (MODIS) instrument on board NASA's Terra satellite. Parameterisations of cloud processes are the biggest source of uncertainty and sensitivity in different climate models’ future projections of Earth's climate. An accurate quantification of the spatial distribution of clouds, as well as a rigorously estimated pixel‐scale clear‐sky‐probability process, is needed to establish reliable estimates of cloud‐distributional changes and trends caused by climate change. Here we give a hierarchical spatial‐statistical modelling approach for a very large spatial dataset of 2.75 million pixels, corresponding to a granule of MODIS cloud‐mask data, and we use spatial change‐of‐Support relationships to estimate cloud fraction at coarser resolutions. Our model is non‐Gaussian; it postulates a hidden process for the clear‐sky probability that makes use of the SRE model, EM‐estimation, and optimal (empirical Bayes) spatial prediction of the clear‐sky‐probability process. Measures of prediction uncertainty are also given.

The Role of Regression Performance on Multimodel Analysis

In this work, we provide suggestions for designing experiments where calibration of many models is required and guidance for identifying problematic calibrations. Calibration of many conceptual models which have different representations of the physical processes in the system, as is done in cross-validation studies or multi-model analysis, often uses computationally frugal inversion techniques to achieve tractable execution times. However, because these frugal methods are usually local methods, and the inverse problem is almost always nonlinear, there is no guarantee that the optimal solution will be found. Furthermore, evaluation of each inverse model's performance to identify poor calibrations can be tedious. Results of this study show that if poorly calibrated models are included in the analysis, simulated predictions and measures of prediction uncertainty can be affected in unexpected ways. Guidelines are provided to help identify problematic regressions and correct them.

Sequential kriging optimization using multiple-fidelity evaluations

When cost per evaluation on a system of interest is high, surrogate systems can provide cheaper but lower-fidelity information. In the proposed extension of the sequential kriging optimization method, surrogate systems are exploited to reduce the total evaluation cost. The method utilizes data on all systems to build a kriging metamodel that provides a global prediction of the objective function and a measure of prediction uncertainty. The location and fidelity level of the next evaluation are selected by maximizing an augmented expected improvement function, which is connected with the evaluation costs. The proposed method was applied to test functions from the literature and a metal-forming process design problem via finite element simulations. The method manifests sensible search patterns, robust performance, and appreciable reduction in total evaluation cost as compared to the original method.

Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models

This paper proposes a new method that extends the efficient global optimization to address stochastic black-box systems. The method is based on a kriging meta-model that provides a global prediction of the objective values and a measure of prediction uncertainty at every point. The criterion for the infill sample selection is an augmented expected improvement function with desirable properties for stochastic responses. The method is empirically compared with the revised simplex search, the simultaneous perturbation stochastic approximation, and the DIRECT methods using six test problems from the literature. An application case study on an inventory system is also documented. The results suggest that the proposed method has excellent consistency and efficiency in finding global optimal solutions, and is particularly useful for expensive systems.

Bayesian applications of belief networks and multilayer perceptrons for ovarian tumor classification with rejection

Incorporating prior knowledge into black-box classifiers is still much of an open problem. We propose a hybrid Bayesian methodology that consists in encoding prior knowledge in the form of a (Bayesian) belief network and then using this knowledge to estimate an informative prior for a black-box model (e.g. a multilayer perceptron). Two technical approaches are proposed for the transformation of the belief network into an informative prior. The first one consists in generating samples according to the most probable parameterization of the Bayesian belief network and using them as virtual data together with the real data in the Bayesian learning of a multilayer perceptron. The second approach consists in transforming probability distributions over belief network parameters into distributions over multilayer perceptron parameters. The essential attribute of the hybrid methodology is that it combines prior knowledge and statistical data efficiently when prior knowledge is available and the sample is of small or medium size. Additionally, we describe how the Bayesian approach can provide uncertainty information about the predictions (e.g. for classification with rejection). We demonstrate these techniques on the medical task of predicting the malignancy of ovarian masses and summarize the practical advantages of the Bayesian approach. We compare the learning curves for the hybrid methodology with those of several belief networks and multilayer perceptrons. Furthermore, we report the performance of Bayesian belief networks when they are allowed to exclude hard cases based on various measures of prediction uncertainty.

Nonlocal and localized analyses of conditional mean steady state flow in bounded, randomly nonuniform domains: 1. Theory and computational approach

We consider the effect of measuring randomly varying hydraulic conductivitiesK(x) on one's ability to predict numerically, without resorting to either Monte Carlo simulation or upscaling, steady state flow in bounded domains driven by random source and boundary terms. Our aim is to allow optimum unbiased prediction of hydraulic heads h(x) and fluxes q(x) by means of their ensemble moments, 〈h(x)〉c and 〈q(x)〉c, respectively, conditioned on measurements of K(x). These predictors have been shown by Neuman and Orr [1993a] to satisfy exactly an integrodifferential conditional mean flow equation in which 〈q(x)〉c is nonlocal and non‐Darcian. Here we develop complementary integrodifferential equations for second conditional moments of head and flux which serve as measures of predictive uncertainty; obtain recursive closure approximations for both the first and second conditional moment equations through expansion in powers of a small parameter σY which represents the standard estimation error of ln K(x); and show how to solve these equations to first order in σY2 by finite elements on a rectangular grid in two dimensions. In the special case where one treats K(x) as if it was locally homogeneous and mean flow as if it was locally uniform, one obtains a localized Darcian approximation 〈q(x)〉c ≈ −Kc(x)∇〈h(x)〉c in which Kc(x) is a space‐dependent conditional hydraulic conductivity tensor. This leads to the traditional deterministic, Darcian steady state flow equation which, however, acquires a nontraditional meaning in that its parameters and state variables are data dependent and therefore inherently nonunique. It further explains why parameter estimates obtained by traditional inverse methods tend to vary as one modifies the database. Localized equations yield no information about predictive uncertainty. Our stochastic derivation of these otherwise standard deterministic flow equations makes clear that uncertainty measures associated with estimates of head and flux, obtained by traditional inverse methods, are generally smaller (often considerably so) than measures of corresponding predictive uncertainty, which can be assessed only by means of stochastic models such as ours. We present a detailed comparison between finite element solutions of nonlocal and localized moment equations and Monte Carlo simulations under superimposed mean‐uniform and convergent flow regimes in two dimensions. Paper 1 presents the theory and computational approach, and paper 2 [Guadagnini and Neuman, this issue] describes unconditional and conditional computational results.