Estimates Of Prediction Uncertainty Research Articles

Information Aware max-norm Dirichlet networks for predictive uncertainty estimation

Precise estimation of uncertainty in predictions for AI systems is a critical factor in ensuring trust and safety. Deep neural networks trained with a conventional method are prone to over-confident predictions. In contrast to Bayesian neural networks that learn approximate distributions on weights to infer prediction confidence, we propose a novel method, Information Aware Dirichlet networks, that learn an explicit Dirichlet prior distribution on predictive distributions by minimizing a bound on the expected max norm of the prediction error and penalizing information associated with incorrect outcomes. Properties of the new cost function are derived to indicate how improved uncertainty estimation is achieved. Experiments using real datasets show that our technique outperforms, by a large margin, state-of-the-art neural networks for estimating within-distribution and out-of-distribution uncertainty, and detecting adversarial examples.

An efficient model for predicting the train-induced ground-borne vibration and uncertainty quantification based on Bayesian neural network

The uncertainty in the prediction of train-induced ground-borne vibration is mainly attributed to the randomness of excitation, the variability of soil, the uncertainty of models, etc. Quantification of the uncertainty in prediction is an intractable problem using traditional models. Herein, an efficient model based on the Bayesian neural network is presented to predict the train-induced ground-borne vibration and quantify the uncertainty. In this model, vibration prediction is performed using a probabilistic framework. The aleatoric uncertainty is quantified by assuming a Gaussian noise over the observation data of vibration level, and the epistemic uncertainty is quantified by delivering the posterior of the fitting parameters in the model using Bayesian inference. In addition to the mean value of prediction, the model can provide a probability distribution to describe the uncertainty in prediction. A case study is presented in which both the weighted vibration level and the frequency-dependent vibration level are predicted. The proposed model performed well about the prediction accuracy and uncertainty estimation, as indicated by a comparison of the results with previously published measurements.

Investment Sizing with Deep Learning Prediction Uncertainties for High-Frequency Eurodollar Futures Trading

In this article, the authors show that prediction uncertainty estimates gleaned from deep learning models can be useful inputs for influencing the relative allocation of risk capital across trades. In this way, consideration of uncertainty is important because it permits the scaling of investment size across trade opportunities in a principled and data-driven way. The authors showcase this insight with a prediction model and, based on a Sharpe ratio metric, find clear outperformance relative to trading strategies that either do not take uncertainty into account or use an alternative market-based statistic as a proxy for uncertainty. Of added novelty is their modeling of high-frequency data at the top level of the Eurodollar futures limit order book for each trading day of 2018, whereby they predict interest rate curve changes on small time horizons. The authors are motivated to study the market for these popularly traded interest rate derivatives because it is deep and liquid and contributes to the efficient functioning of global finance—though there is relatively little by way of its modeling contained in the academic literature. Hence, they verify the utility of prediction models and uncertainty estimates for trading applications in this complex and multidimensional asset price space. <b>TOPICS:</b>Big data/machine learning, derivatives, simulations, statistical methods <b>Key Findings</b> ▪ The authors model high-frequency Eurodollar Futures limit order book data using state-of-the-art deep learning to predict interest rate curve changes on small time horizons. ▪ They further augment their models to yield estimates of prediction uncertainty. ▪ In certain settings, the uncertainty estimates can be used in conjunction with return predictions for scaling bankroll allocation between trades. This can lead to clear trading outperformance relative to the case that uncertainty is not taken into account.

Artificial neural networks for monitoring network optimisation—a practical example using a national insect survey

Monitoring networks are improved by additional sensors. Optimal configurations of sensors give better representations of the process of interest, maximising its exploration while minimising the need for costly infrastructure. By modelling the monitored process, we can identify gaps in its representation, i.e. uncertain predictions, where additional sensors should be located. Here, with data collected from the Rothamsted Insect Survey network, we train an artificial neural network to predict the seasonal aphid arrival from environmental variables. We focus on estimating prediction uncertainty across the UK to guide the addition of a sensor to the network. We first illustrate how to estimate uncertainty in neural networks, hence making them relevant for model-based monitoring network optimisation. Then we highlight critical areas of agricultural importance where additional traps would improve decision support and crop protection in the UK. Possible applications include most ecological monitoring and surveillance activities, but also the weather or pollution monitoring.

High-Throughput Phenotyping of Soybean Maturity Using Time Series UAV Imagery and Convolutional Neural Networks

Soybean maturity is a trait of critical importance for the development of new soybean cultivars, nevertheless, its characterization based on visual ratings has many challenges. Unmanned aerial vehicles (UAVs) imagery-based high-throughput phenotyping methodologies have been proposed as an alternative to the traditional visual ratings of pod senescence. However, the lack of scalable and accurate methods to extract the desired information from the images remains a significant bottleneck in breeding programs. The objective of this study was to develop an image-based high-throughput phenotyping system for evaluating soybean maturity in breeding programs. Images were acquired twice a week, starting when the earlier lines began maturation until the latest ones were mature. Two complementary convolutional neural networks (CNN) were developed to predict the maturity date. The first using a single date and the second using the five best image dates identified by the first model. The proposed CNN architecture was validated using more than 15,000 ground truth observations from five trials, including data from three growing seasons and two countries. The trained model showed good generalization capability with a root mean squared error lower than two days in four out of five trials. Four methods of estimating prediction uncertainty showed potential at identifying different sources of errors in the maturity date predictions. The architecture developed solves limitations of previous research and can be used at scale in commercial breeding programs.

A Survey of Computational Intelligence Techniques for Wind Power Uncertainty Quantification in Smart Grids.

The high penetration level of renewable energy is thought to be one of the basic characteristics of future smart grids. Wind power, as one of the most increasing renewable energy, has brought a large number of uncertainties into the power systems. These uncertainties would require system operators to change their traditional ways of decision-making. This article provides a comprehensive survey of computational intelligence techniques for wind power uncertainty quantification in smart grids. First, prediction intervals (PIs) are introduced as a means to quantify the uncertainties in wind power forecasts. Various PI evaluation indices, including the latest trends in comprehensive evaluation techniques, are compared. Furthermore, computational intelligence-based PI construction methods are summarized and classified into traditional methods (parametric) and direct PI construction methods (nonparametric). In the second part of this article, methods of incorporating wind power forecast uncertainties into power system decision-making processes are investigated. Three techniques, namely, stochastic models, fuzzy logic models, and robust optimization, and different power system applications using these techniques are reviewed. Finally, future research directions, such as spatiotemporal and hierarchical forecasting, deep learning-based methods, and integration of predictive uncertainty estimates into the decision-making process, are discussed. This survey can benefit the readers by providing a complete technical summary of wind power uncertainty quantification and decision-making in smart grids.

Ensemble-based deep learning for estimating PM2.5 over California with multisource big data including wildfire smoke

IntroductionEstimating PM2.5 concentrations and their prediction uncertainties at a high spatiotemporal resolution is important for air pollution health effect studies. This is particularly challenging for California, which has high variability in natural (e.g, wildfires, dust) and anthropogenic emissions, meteorology, topography (e.g. desert surfaces, mountains, snow cover) and land use. MethodsUsing ensemble-based deep learning with big data fused from multiple sources we developed a PM2.5 prediction model with uncertainty estimates at a high spatial (1 km × 1 km) and temporal (weekly) resolution for a 10-year time span (2008–2017). We leveraged autoencoder-based full residual deep networks to model complex nonlinear interrelationships among PM2.5 emission, transport and dispersion factors and other influential features. These included remote sensing data (MAIAC aerosol optical depth (AOD), normalized difference vegetation index, impervious surface), MERRA-2 GMI Replay Simulation (M2GMI) output, wildfire smoke plume dispersion, meteorology, land cover, traffic, elevation, and spatiotemporal trends (geo-coordinates, temporal basis functions, time index). As one of the primary predictors of interest with substantial missing data in California related to bright surfaces, cloud cover and other known interferences, missing MAIAC AOD observations were imputed and adjusted for relative humidity and vertical distribution. Wildfire smoke contribution to PM2.5 was also calculated through HYSPLIT dispersion modeling of smoke emissions derived from MODIS fire radiative power using the Fire Energetics and Emissions Research version 1.0 model. ResultsEnsemble deep learning to predict PM2.5 achieved an overall mean training RMSE of 1.54 μg/m3 (R2: 0.94) and test RMSE of 2.29 μg/m3 (R2: 0.87). The top predictors included M2GMI carbon monoxide mixing ratio in the bottom layer, temporal basis functions, spatial location, air temperature, MAIAC AOD, and PM2.5 sea salt mass concentration. In an independent test using three long-term AQS sites and one short-term non-AQS site, our model achieved a high correlation (>0.8) and a low RMSE (<3 μg/m3). Statewide predictions indicated that our model can capture the spatial distribution and temporal peaks in wildfire-related PM2.5. The coefficient of variation indicated highest uncertainty over deciduous and mixed forests and open water land covers. ConclusionOur method can be generalized to other regions, including those having a mix of major urban areas, deserts, intensive smoke events, snow cover and complex terrains, where PM2.5 has previously been challenging to predict. Prediction uncertainty estimates can also inform further model development and measurement error evaluations in exposure and health studies.

A case study in model failure? COVID-19 daily deaths and ICU bed utilisation predictions in New York state.

Forecasting models have been influential in shaping decision-making in the COVID-19 pandemic. However, there is concern that their predictions may have been misleading. Here, we dissect the predictions made by four models for the daily COVID-19 death counts between March 25 and June 5 in New York state, as well as the predictions of ICU bed utilisation made by the influential IHME model. We evaluated the accuracy of the point estimates and the accuracy of the uncertainty estimates of the model predictions. First, we compared the “ground truth” data sources on daily deaths against which these models were trained. Three different data sources were used by these models, and these had substantial differences in recorded daily death counts. Two additional data sources that we examined also provided different death counts per day. For accuracy of prediction, all models fared very poorly. Only 10.2% of the predictions fell within 10% of their training ground truth, irrespective of distance into the future. For accurate assessment of uncertainty, only one model matched relatively well the nominal 95% coverage, but that model did not start predictions until April 16, thus had no impact on early, major decisions. For ICU bed utilisation, the IHME model was highly inaccurate; the point estimates only started to match ground truth after the pandemic wave had started to wane. We conclude that trustworthy models require trustworthy input data to be trained upon. Moreover, models need to be subjected to prespecified real time performance tests, before their results are provided to policy makers and public health officials.

Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation.

Fully convolutional neural networks (FCNs), and in particular U-Nets, have achieved state-of-the-art results in semantic segmentation for numerous medical imaging applications. Moreover, batch normalization and Dice loss have been used successfully to stabilize and accelerate training. However, these networks are poorly calibrated i.e. they tend to produce overconfident predictions for both correct and erroneous classifications, making them unreliable and hard to interpret. In this paper, we study predictive uncertainty estimation in FCNs for medical image segmentation. We make the following contributions: 1) We systematically compare cross-entropy loss with Dice loss in terms of segmentation quality and uncertainty estimation of FCNs; 2) We propose model ensembling for confidence calibration of the FCNs trained with batch normalization and Dice loss; 3) We assess the ability of calibrated FCNs to predict segmentation quality of structures and detect out-of-distribution test examples. We conduct extensive experiments across three medical image segmentation applications of the brain, the heart, and the prostate to evaluate our contributions. The results of this study offer considerable insight into the predictive uncertainty estimation and out-of-distribution detection in medical image segmentation and provide practical recipes for confidence calibration. Moreover, we consistently demonstrate that model ensembling improves confidence calibration.

Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles

The inaccuracy of neural network models on inputs that do not stem from the distribution underlying the training data is problematic and at times unrecognized. Uncertainty estimates of model predictions are often based on the variation in predictions produced by a diverse ensemble of models applied to the same input. Here we describe Maximize Overall Diversity (MOD), an approach to improve ensemble-based uncertainty estimates by encouraging larger overall diversity in ensemble predictions across all possible inputs. We apply MOD to regression tasks including 38 Protein-DNA binding datasets, 9 UCI datasets, and the IMDB-Wiki image dataset. We also explore variants that utilize adversarial training techniques and data density estimation. For out-of-distribution test examples, MOD significantly improves predictive performance and uncertainty calibration without sacrificing performance on test data drawn from same distribution as the training data. We also find that in Bayesian optimization tasks, the performance of UCB acquisition is improved via MOD uncertainty estimates.

How Sure Can We Be about ML Methods-Based Evaluation of Compound Activity: Incorporation of Information about Prediction Uncertainty Using Deep Learning Techniques.

A great variety of computational approaches support drug design processes, helping in selection of new potentially active compounds, and optimization of their physicochemical and ADMET properties. Machine learning is a group of methods that are able to evaluate in relatively short time enormous amounts of data. However, the quality of machine-learning-based prediction depends on the data supplied for model training. In this study, we used deep neural networks for the task of compound activity prediction and developed dropout-based approaches for estimating prediction uncertainty. Several types of analyses were performed: the relationships between the prediction error, similarity to the training set, prediction uncertainty, number and standard deviation of activity values were examined. It was tested whether incorporation of information about prediction uncertainty influences compounds ranking based on predicted activity and prediction uncertainty was used to search for the potential errors in the ChEMBL database. The obtained outcome indicates that incorporation of information about uncertainty of compound activity prediction can be of great help during virtual screening experiments.

A review and framework for the evaluation of pixel-level uncertainty estimates in satellite aerosol remote sensing

Abstract. Recent years have seen the increasing inclusion of per-retrieval prognostic (predictive) uncertainty estimates within satellite aerosol optical depth (AOD) data sets, providing users with quantitative tools to assist in the optimal use of these data. Prognostic estimates contrast with diagnostic (i.e. relative to some external truth) ones, which are typically obtained using sensitivity and/or validation analyses. Up to now, however, the quality of these uncertainty estimates has not been routinely assessed. This study presents a review of existing prognostic and diagnostic approaches for quantifying uncertainty in satellite AOD retrievals, and it presents a general framework to evaluate them based on the expected statistical properties of ensembles of estimated uncertainties and actual retrieval errors. It is hoped that this framework will be adopted as a complement to existing AOD validation exercises; it is not restricted to AOD and can in principle be applied to other quantities for which a reference validation data set is available. This framework is then applied to assess the uncertainties provided by several satellite data sets (seven over land, five over water), which draw on methods from the empirical to sensitivity analyses to formal error propagation, at 12 Aerosol Robotic Network (AERONET) sites. The AERONET sites are divided into those for which it is expected that the techniques will perform well and those for which some complexity about the site may provide a more severe test. Overall, all techniques show some skill in that larger estimated uncertainties are generally associated with larger observed errors, although they are sometimes poorly calibrated (i.e. too small or too large in magnitude). No technique uniformly performs best. For powerful formal uncertainty propagation approaches such as optimal estimation, the results illustrate some of the difficulties in appropriate population of the covariance matrices required by the technique. When the data sets are confronted by a situation strongly counter to the retrieval forward model (e.g. potentially mixed land–water surfaces or aerosol optical properties outside the family of assumptions), some algorithms fail to provide a retrieval, while others do but with a quantitatively unreliable uncertainty estimate. The discussion suggests paths forward for the refinement of these techniques.

Co-transport of heavy metals in layered saturated soil: Characteristics and simulation

Interest in soil pollution by multiple heavy metals has been growing over the last decades. However, few experiments combining numerical analyses with solute transport in layered soil can be found in the literature. Here, the retention and fate of three coexisting metal ions, Cu, Cd, and Zn, in layered soils were investigated to evaluate soil co-contamination through batch and column experiments. Results showed high amounts of Cu adsorption and retention by soils, followed by Cd and Zn. The partial concentration of Zn in effluent was greater than the input from competition adsorption and the ‘snow plow effect’. These findings indicate the high potential risk of Zn and Cd groundwater pollution when Cu, Cd, and Zn co-exist in the soil. Adsorption isotherms obtained from batch experiments were well described by Freundlich equation. Breakthrough curves (BTCs) obtained from column experiments were well described by standard convection-dispersion equation (CDE) for Br, and Tow-site (TSM) and One-site models (OSM) for metals except for Zn, using the Levenberg-Marquardt nonlinear optimization algorithm. However, the parameters were poorly constrained by the available observational data due to high correlation between parameters, rather than insensitivity to model outputs. The Generalized Likelihood Uncertainty Estimation (GLUE) method did not only qualify the uncertainty of parameters for solute transport in layered medium, but estimate prediction uncertainty. Prediction bounds basically captured the observed Br, Zn and Cd BTCs, while systematically overestimated the effluent Cu concentration.Comparing with the optimization, GLUE method can improve prediction reliability of heavy metal transport in layered soils.

A Bayesian inversion framework to evaluate parameter and predictive inference of a simple soil respiration model in a cool-temperate forest in western Japan

Careful modelling of soil carbon sequestration is essential to evaluate future terrestrial feedback to the earth climate system through atmosphere–surface carbon exchange. Few studies have evaluated, in bio- and geo-applications, parameter and predictive uncertainty of soil respiration models by considering the difference between observations and model predictions; i.e. residual error, which is assumed neither to be independent nor to be described by a normal (i.e. Gaussian) probability distribution with a mean of zero and constant variance. In this paper, we use 2-year observations of soil carbon flux from 2017 to 2018 (hereafter referred to as ‘long-term simulation’) obtained with two open-top chambers to estimate parameter and predictive uncertainty of a simple soil respiration model based on Bayesian statistics in a cool-temperate forest in western Japan. We also use a Gaussian innovative residual error model in which a generalised likelihood uncertainty estimation that accounts for correlated, heteroscedastic, non-normally distributed (i.e. non-Gaussian) residual error flexibly handles statistics varying in skewness and kurtosis. Results show that the effects of correlation and heteroscedasticity were eliminated adequately. Additionally, the posterior distribution of the residuals had a pattern intermediate to those of Gaussian and Laplacian (or double-exponential) distributions. Consequently, the predicted soil respiration rate, and range of uncertainty therein, well-matched the observational data. Furthermore, we compare results of parameter and predictive inference of the soil respiration model from the long-term simulation with those constrained of short-term simulations (i.e. 4-month subsets of the 2-year dataset) to determine the extent to which the approach used affects the estimation of parameter and predictive uncertainty. No significant difference in parameter estimates was found between the long-term simulation versus any of the short-term simulations, whereas short-term simulation analysis of the uncertainty at 50 %—i.e. between the lower (25 %) and upper (75 %) quartiles of the probability range—indicated distinctive variations in model parameters in summer when more vigorous activity of trees and organisms promotes carbon cycling between the atmosphere and ecosystem. Overall we demonstrate that the Bayesian inversion approach is useful as a means by which to evaluate effectively parameter and predictive uncertainty of a soil respiration model with precise representation of residual errors.

Investment Sizing with Deep Learning Prediction Uncertainties for High-Frequency Eurodollar Futures Trading.

In this work we show that prediction uncertainty estimates gleaned from deep learning models can be useful inputs for influencing the relative allocation of risk capital across trades. In this way, consideration of uncertainty is important because it permits the scaling of investment size across trade opportunities in a principled and data-driven way. We showcase this insight with a prediction model and find clear outperformance based on a Sharpe ratio metric, relative to trading strategies that either do not take uncertainty into account, or that utilize an alternative market-based statistic as a proxy for uncertainty. Of added novelty is our modelling of high-frequency data at the top level of the Eurodollar Futures limit order book for each trading day of 2018, whereby we predict interest rate curve changes on small time horizons. We are motivated to study the market for these popularly-traded interest rate derivatives since it is deep and liquid, and contributes to the efficient functioning of global finance -- though there is relatively little by way of its modelling contained in the academic literature. Hence, we verify the utility of prediction models and uncertainty estimates for trading applications in this complex and multi-dimensional asset price space.

Comparison of different quantile regression methods to estimate predictive hydrological uncertainty in the Upper Chao Phraya River Basin, Thailand

AbstractThe estimation of predictive uncertainty and its application as a post‐processor of hydrological model output, such as water level, can provide additional information useful for short‐term hydrological forecasting. In this study, We applied quantile regression models for estimating predictive hydrological uncertainty and used it to derive probabilistic hydrological forecasts. Forecast water levels and associated forecast errors were used as predictor and predictand, respectively, to develop three regression models: (a) linear quantile regression (LQR), (b) weighted LQR and (c) LQR in Gaussian space using Normal Quantile Transformation. These different models for hydrological forecasting were developed for, and applied to, the operational flood forecasting system in the Upper Chao Phraya River, Thailand. The quality of these forecasts in terms of reliability, sharpness and overall skill were assessed using various graphical and numerical verification metrics. Results show that the improvement of forecast in terms of either reliability or sharpness depends upon the configurations used. With comparable overall performance, weighted LQR provided a relatively simple configuration, which can be used for estimating uncertainty in hydrological forecasting.

General-Surrogate Adaptive Sampling Using Interquartile Range for Design Space Exploration

Abstract A surrogate model is a common tool to approximate system response at untested points for design space exploration. Adaptive sampling has been studied for improving the accuracy of surrogates iteratively by introducing additional samples (simulations and experiments). New samples are often selected based on the estimated uncertainty in the design space. While some surrogates such as kriging have readily available uncertainty models for their predictions, other surrogates do not. Consequently, there have been studies of using the process of leaving-samples-out (LSO) used in cross-validation tools to estimate prediction uncertainty, such as the universal prediction distribution (UPD). In this paper, an adaptive sampling scheme for general surrogates is proposed based on LSO, similar to cross-validation and interquartile range (IQR). Multiple submodels are first developed from LSO. The uncertainty is then estimated from the IQR of these surrogates at a given point. New samples are added iteratively at the point with maximum IQR for design space exploration. The proposed scheme is illustrated using kriging, radial basis function, and neural network surrogates. The proposed scheme is evaluated using four algebraic test functions. Multiple sets of initial samples were produced to account for randomness. For these test functions, the proposed scheme was found to be more accurate and robust than kriging with its own uncertainty model. The proposed scheme was more accurate than the UPD for three out of the four test functions. For a fixed number of samples, the IQR-based adaptive sampling also proved to be more accurate than all-at-once sampling in most cases even when the estimated uncertainty was only mildly correlated with prediction errors.

Toward Improved Probabilistic Predictions for Flood Forecasts Generated Using Deterministic Models

AbstractUncertainties in flood forecasts are inevitable, and the key issue is to develop probabilistic predictions so that the predictive uncertainty (PU) bounds can be estimated. We develop and test a general method for probabilistic forecasting and PU estimation that is based on a theoretical and practical analysis of the actual nature of the model residuals, which reveals that the residual mean, standard deviation, and distributional form can all vary with time. Our approach is to condition the nature of the residual distribution on the magnitude of the corresponding streamflow value, but other kinds of conditioning are also possible. Using real data, we illustrate seven progressively more realistic sets of assumptions regarding the model residuals, ranging from homogenous Gaussian to fully heterogeneous non‐Gaussian. Our results show that the estimated probabilistic predictions become progressively better as the assumptions better conform to the actual properties of the residuals. As benchmarks, we compare against results from the state‐of‐the‐art power transformation approach. Our method is generally applicable to any situation where a deterministic model is used to generate predictions, and where empirical probabilistic predictions are required without developing a stochastic version of that model.

Self-Supervised Mixture-of-Experts by Uncertainty Estimation

Learning related tasks in various domains and transferring exploited knowledge to new situations is a significant challenge in Reinforcement Learning (RL). However, most RL algorithms are data inefficient and fail to generalize in complex environments, limiting their adaptability and applicability in multi-task scenarios. In this paper, we propose SelfSupervised Mixture-of-Experts (SUM), an effective algorithm driven by predictive uncertainty estimation for multitask RL. SUM utilizes a multi-head agent with shared parameters as experts to learn a series of related tasks simultaneously by Deep Deterministic Policy Gradient (DDPG). Each expert is extended by predictive uncertainty estimation on known and unknown states to enhance the Q-value evaluation capacity against overfitting and the overall generalization ability. These enable the agent to capture and diffuse the common knowledge across different tasks improving sample efficiency in each task and the effectiveness of expert scheduling across multiple tasks. Instead of task-specific design as common MoEs, a self-supervised gating network is adopted to determine a potential expert to handle each interaction from unseen environments and calibrated completely by the uncertainty feedback from the experts without explicit supervision. To alleviate the imbalanced expert utilization as the crux of MoE, optimization is accomplished via decayedmasked experience replay, which encourages both diversification and specialization of experts during different periods. We demonstrate that our approach learns faster and achieves better performance by efficient transfer and robust generalization, outperforming several related methods on extended OpenAI Gym’s MuJoCo multi-task environments.

Accurate and robust deep learning-based segmentation of the prostate clinical target volume in ultrasound images.

The goal of this work was to develop a method for accurate and robust automatic segmentation of the prostate clinical target volume in transrectal ultrasound (TRUS) images for brachytherapy. These images can be difficult to segment because of weak or insufficient landmarks or strong artifacts. We devise a method, based on convolutional neural networks (CNNs), that produces accurate segmentations on easy and difficult images alike. We propose two strategies to achieve improved segmentation accuracy on difficult images. First, for CNN training we adopt an adaptive sampling strategy, whereby the training process is encouraged to pay more attention to images that are difficult to segment. Secondly, we train a CNN ensemble and use the disagreement among this ensemble to identify uncertain segmentations and to estimate a segmentation uncertainty map. We improve uncertain segmentations by utilizing the prior shape information in the form of a statistical shape model. Our method achieves Hausdorff distance of 2.7 ± 2.3 mm and Dice score of 93.9 ± 3.5%. Comparisons with several competing methods show that our method achieves significantly better results and reduces the likelihood of committing large segmentation errors. Furthermore, our experiments show that our approach to estimating segmentation uncertainty is better than or on par with recent methods for estimation of prediction uncertainty in deep learning models. Our study demonstrates that estimation of model uncertainty and use of prior shape information can significantly improve the performance of CNN-based medical image segmentation methods, especially on difficult images.