Reliable Uncertainty Estimates Research Articles

CFD modelling and grid uncertainty analysis of the free-falling water entry of 2D rigid bodies

The open source Computational Fluid Dynamics (CFD) library, OpenFOAM, is utilized to simulate the water impact of two-dimensional wedges and ship bow sections. For the free falling cases, the predicted results are validated against existing experimental data, and the parameter study is performed regarding the number of cells, time step, and width of the water domain applied in the numerical models. Grid uncertainty analysis for the CFD results is performed with a constant Courant–Friedrichs–Lewy number, to assess the consistency and accuracy of the CFD solver in solving the present problem. Three uncertainty estimation methods are compared on the peak slamming forces and pressures: Correction Factor (Cf), Factor of Safety (Fs), and Least-squares Fit (Lf) based approaches. All three of them can provide reliable uncertainty estimation for CFD simulations, as long as the right mesh configuration has been used. The variations of pressure distribution on wedge sections with various deadrise angles and initial entry velocities are analyzed. To examine the capability of the tool to predict the slamming loads for a section with roll motions two cases are simulated: one considering a free roll motion and another with constant roll angles. The predicted slamming loads are compared with the available experimental data and other numerical results.

Artificial intelligence for melanoma diagnosis.

Convolutional neural networks (CNN) have shown unprecedented accuracy in digital image analysis, which can be harnessed for melanoma recognition through automated evaluation of clinical and dermatoscopic images. In experimental studies, modern CNN architectures perform single image analysis at the level of dermatologists and domain-experts, also for multiclass predictions including a multitude of possible diagnoses. This may not necessarily translate to good clinical performance, and reliable randomized controlled prospective clinical trials for modern CNNs are essentially missing. Weaknesses of CNNs are that limitations of available training image datasets propagate to limitations of CNN predictions, and they cannot provide a reliable estimate of uncertainty. Recent research focuses on human-computer collaboration, where gains in accuracy were measured even with imperfect CNNs. With missing academic and clinical agreement on equivocal melanocytic lesions, fully automating histologic assessment of them with CNNs appear problematic, and applications in the near future are probably limited to supporting, referencing or recommendation roles.

Bayesian uncertainty analysis of inversion models applied to the inference of thermal properties of walls

In this work, we propose a fully Bayesian uncertainty analysis of the indirect measurement of thermal properties of walls from in situ temperature and flux measurements, obtained with an active method, using a one dimensional transient thermal model. We show that this approach is able to take into account the uncertainty of the inputs of the thermal model and the uncertainty of the output observations, for a more reliable uncertainty estimation of the calibration parameters and any derived quantity. For this problem, we improve the classical Bayesian inversion model by taking into account underestimated uncertainty on reported output observations, which is a frequently encountered issue in practice. We provide some recommendations for a wider applicability of the method. We illustrate the principles of uncertainty evaluation of the Guide to the Expression of Uncertainty in Measurement in terms of a real case study to evaluate the thermal resistance of a multilayer wall placed in a climatic chamber. For this application, we compare results of the Bayesian inversion with classical steady-state results in comparable experimental conditions. We perform a sensitivity analysis to study the effect of duration, input uncertainties and excess variance prior, and we make recommendations. R code is made available that enables a Bayesian uncertainty evaluation of inversion models for related applications.

Trajectory prediction of ballistic missiles using Gaussian process error model

Ballistic Missile Trajectory Prediction (BMTP) is critical to air defense systems. Most Trajectory Prediction (TP) methods focus on the coast and reentry phases, in which the Ballistic Missile (BM) trajectories are modeled as ellipses or the state components are propagated by the dynamic integral equations on time scales. In contrast, the boost-phase TP is more challenging because there are many unknown forces acting on the BM in this phase. To tackle this difficult problem, a novel BMTP method by using Gaussian Processes (GPs) is proposed in this paper. In particular, the GP is employed to train the prediction error model of the boost-phase trajectory database, in which the error refers to the difference between the true BM state at the prediction moment and the integral extrapolation of the BM state. And the final BMTP is a combination of the dynamic equation based numerical integration and the GP-based prediction error. Since the trained GP aims to capture the relationship between the numerical integration and the unknown error, the modified BM state prediction is closer to the true one compared with the original TP. Furthermore, the GP is able to output the uncertainty information of the TP, which is of great significance for determining the warning range centered on the predicted BM state. Simulation results show that the proposed method effectively improves the BMTP accuracy during the boost phase and provides reliable uncertainty estimation boundaries.

DIBS: Diversity Inducing Information Bottleneck in Model Ensembles

Although deep learning models have achieved state-of-the art performance on a number of vision tasks, generalization over high dimensional multi-modal data, and reliable predictive uncertainty estimation are still active areas of research. Bayesian approaches including Bayesian Neural Nets (BNNs) do not scale well to modern computer vision tasks, as they are difficult to train, and have poor generalization under dataset-shift. This motivates the need for effective ensembles which can generalize and give reliable uncertainty estimates. In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning the stochastic latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. We evaluate our method on benchmark datasets: MNIST, CIFAR100, TinyImageNet and MIT Places 2, and compared to the most competitive baselines show significant improvements in classification accuracy, under a shift in the data distribution and in out-of-distribution detection. over 10% relative improvement in classification accuracy, over 5% relative improvement in generalizing under dataset shift, and over 5% better predictive uncertainty estimation as inferred by efficient out-of-distribution (OOD) detection.

Robust surface structure analysis with reliable uncertainty estimation using the exchange Monte Carlo method. Corrigendum

An error in the article by Nagai, Anada, Nakanishi-Ohno, Okada & Wakabayashi [J. Appl. Cryst. (2020), 53, 387–392] is corrected.

Pandemic velocity: Forecasting COVID-19 in the US with a machine learning & Bayesian time series compartmental model.

Predictions of COVID-19 case growth and mortality are critical to the decisions of political leaders, businesses, and individuals grappling with the pandemic. This predictive task is challenging due to the novelty of the virus, limited data, and dynamic political and societal responses. We embed a Bayesian time series model and a random forest algorithm within an epidemiological compartmental model for empirically grounded COVID-19 predictions. The Bayesian case model fits a location-specific curve to the velocity (first derivative) of the log transformed cumulative case count, borrowing strength across geographic locations and incorporating prior information to obtain a posterior distribution for case trajectories. The compartmental model uses this distribution and predicts deaths using a random forest algorithm trained on COVID-19 data and population-level characteristics, yielding daily projections and interval estimates for cases and deaths in U.S. states. We evaluated the model by training it on progressively longer periods of the pandemic and computing its predictive accuracy over 21-day forecasts. The substantial variation in predicted trajectories and associated uncertainty between states is illustrated by comparing three unique locations: New York, Colorado, and West Virginia. The sophistication and accuracy of this COVID-19 model offer reliable predictions and uncertainty estimates for the current trajectory of the pandemic in the U.S. and provide a platform for future predictions as shifting political and societal responses alter its course.

Automatic segmentation of organs-at-risk from head-and-neck CT using separable convolutional neural network with hard-region-weighted loss

Accurate segmentation of Organs-at-Risk (OAR) from Head and Neck (HAN) Computed Tomography (CT) images with uncertainty information is critical for effective planning of radiation therapy for Nasopharyngeal Carcinoma (NPC) treatment. Despite the state-of-the-art performance achieved by Convolutional Neural Networks (CNNs) for the segmentation task, existing methods do not provide uncertainty estimation of the segmentation results for treatment planning, and their accuracy is still limited by the low contrast of soft tissues in CT, highly imbalanced sizes of OARs and large inter-slice spacing. To address these problems, we propose a novel framework for accurate OAR segmentation with reliable uncertainty estimation. First, we propose a Segmental Linear Function (SLF) to transform the intensity of CT images to make multiple organs more distinguishable than existing simple window width/level-based methods. Second, we introduce a novel 2.5D network (named as 3D-SepNet) specially designed for dealing with clinic CT scans with anisotropic spacing. Thirdly, we propose a novel hardness-aware loss function that pays attention to hard voxels for accurate segmentation. We also use an ensemble of models trained with different loss functions and intensity transforms to obtain robust results, which also leads to segmentation uncertainty without extra efforts. Our method won the third place of the HAN OAR segmentation task in StructSeg 2019 challenge and it achieved weighted average Dice of 80.52% and 95% Hausdorff Distance of 3.043 mm. Experimental results show that 1) our SLF for intensity transform helps to improve the accuracy of OAR segmentation from CT images; 2) With only 1/3 parameters of 3D UNet, our 3D-SepNet obtains better segmentation results for most OARs; 3) The proposed hard voxel weighting strategy used for training effectively improves the segmentation accuracy; 4) The segmentation uncertainty obtained by our method has a high correlation to mis-segmentations, which has a potential to assist more informed decisions in clinical practice. Our code is available athttps://github.com/HiLab-git/SepNet.

Plasma parameter profile inference from limited data utilizing second-order derivative priors and physic-based constraints

A Bayesian framework has been used to improve the quality of inferred plasma parameter profiles. An integrated data analysis allows for coherent combinations of different diagnostics, and Gaussian process regression provides a reliable regularization process and systematic uncertainty estimation. In this paper, we propose a new profile inference framework that utilizes our prior knowledge about plasma physics, along with integrated data analysis and a Gaussian process. In order to facilitate the use of the Markov chain Monte Carlo sampling, we use a Gaussian process to define quantities corresponding to the second derivatives of the profiles. We validate the analysis technique by using a synthetic one-dimensional plasma, in which the transport properties are known and demonstrate that the proposed analysis technique can infer plasma parameter profiles from line-integrated measurements only. Furthermore, we can even infer unknown parameters in our physics models when our physics knowledge on the system is incomplete. This analysis framework is applicable to laboratory plasmas and provides a means to investigate plasma parameters, to which standard diagnostics are not directly sensitive.

Model-Based Bayesian Deep Learning Architecture for Linear Inverse Problems in Computational Imaging

We propose a neural network architecture combined with specific training and inference procedures for linear inverse problems arising in computational imaging to reconstruct the underlying image and to represent the uncertainty about the reconstruction. The proposed architecture is built from the model-based reconstruction perspective, which enforces data consistency and eliminates the artifacts in an alternating manner. The training and the inference procedures are based on performing approximate Bayesian analysis on the weights of the proposed network using a variational inference method. The proposed architecture with the associated inference procedure is capable of characterizing uncertainty while performing reconstruction with a modelbased approach. We tested the proposed method on a simulated magnetic resonance imaging experiment. We showed that the proposed method achieved an adequate reconstruction capability and provided reliable uncertainty estimates in the sense that the regions having high uncertainty provided by the proposed method are likely to be the regions where reconstruction errors occur.

Negative values and variance functions: Implications for statistical analysis.

When reporting concentrations of substances in biological specimens it has been virtually universal practice to suppress negative results, initially by left-censoring negative results to zero and more recently by left-censoring to values such as limit of blank, limit of detection or even limit of quantification. Negative concentrations are obviously nonsensical and current reporting practices place proper emphasis on assisting the clinician. However, it is easily overlooked that negative concentrations are merely artefacts of data reduction and while adjusting them is sensible clinical practice there are potentially adverse consequences for statistical analysis, in particular for those parametric summaries and analyses which rely on reliable estimates of low-end uncertainty. This article puts a case for the availability of negative results, describes complications with respect to estimating variance functions and discusses practical workarounds.

Crosstalk Effects in the Uncertainty Estimation of Multiplexed Data Acquisition Systems

This article deals with the analysis of multichannel data-acquisition systems with the aim of identifying and combining the main uncertainty contributions according to the guide to the expression of uncertainty in measurement (GUM) framework. Particular attention has been paid toward crosstalk effect, which could be an important uncertainty contribution in multiplexed data-acquisition systems. The uncertainty analysis is described for three commercial data acquisition devices highlighting that crosstalk specifications are often not suitable for reliable uncertainty estimation in operating conditions. For this reason, an experimental setup has been arranged to fully characterize the interchannel effects of the investigated devices. The obtained results have highlighted that proper characterization of a data-acquisition system is effective in estimating the actual performance at the frequency of interest and in the operating conditions for the source resistance and the input-channel configuration. Eventually, a customized procedure has been proposed, which is effective in correcting the crosstalk effects also in very severe conditions of interchannel disturbance.

Reliable Camera Model Identification Using Sparse Gaussian Processes

Identifying the model of a camera that has captured an image can be an important task in criminal investigations. Many methods assume that the image under analysis originates from a given set of known camera models. In practice, however, a photo can come from an unknown camera model, or its appearance could have been altered by unknown post-processing. In such a case, forensic detectors are prone to fail silently. One way to mitigate silent failures is to use a rejection mechanism for unknown examples. In this work, we propose Gaussian processes (GPs), which intrinsically provide such a rejection mechanism. This makes GPs a potentially powerful tool in multimedia forensics, where forensic analysts regularly work on images from unknown origins. We demonstrate that GPs scale well to the task of camera model identification. Probabilistic predictions from a GP classifier achieve high classification accuracy for known camera models while providing reliable uncertainty estimates. The built-in uncertainty estimates effectively tackle open-set camera model identification, outperforming two state-of-the-art methods.

Spatio-temporal Cokriging method for assimilating and downscaling multi-scale remote sensing data

No single satellite remote sensing system is able to provide the observations on the Earth's surface at both high spatial and high temporal resolution due to the general trade-off between orbit revisit frequency and satellite sensor's spatial resolution. This paper presents a spatio-temporal Cokriging (ST-Cokriging) method for assimilating remote sensing data sets acquired by multiple remote sensing systems with different temporal sampling frequencies and different spatial resolutions. By extending the traditional Cokriging technique from a sole spatial domain to a spatio-temporal domain, we derived and implemented ST-Cokriging algorithm that explicitly takes the spatial covariance, temporal covariance and spatio-temporal covariance structures within and between different data sets into account. Compared with previous downscaling methods, such as, STARFM and FSDAF, our ST-Cokriging method produces more accurate and reliable assimilation results for the heterogeneous region, with associated uncertainty estimates. This method has been implemented into a software package using Python language within ArcGIS environment. The advantages and effectiveness of our ST-Cokriging method have been demonstrated through an application example, in which MODIS images (daily, 250 m and 500 m spatial resolution) and Landsat TM/ETM+ images (16 days revisit cycle, 30 m) are assimilated to generate daily spectral bands and NDVI images at 30 m spatial resolution. Our validation and accuracy assessments indicate that our ST-Cokriging method can effectively fill in data gaps due to clouds and generate reliable assimilation results and uncertainty estimates at both high spatial resolution and high temporal frequency

Reduced global warming from CMIP6 projections when weighting models by performance and independence

Abstract. The sixth Coupled Model Intercomparison Project (CMIP6) constitutes the latest update on expected future climate change based on a new generation of climate models. To extract reliable estimates of future warming and related uncertainties from these models, the spread in their projections is often translated into probabilistic estimates such as the mean and likely range. Here, we use a model weighting approach, which accounts for the models' historical performance based on several diagnostics as well as model interdependence within the CMIP6 ensemble, to calculate constrained distributions of global mean temperature change. We investigate the skill of our approach in a perfect model test, where we use previous-generation CMIP5 models as pseudo-observations in the historical period. The performance of the distribution weighted in the abovementioned manner with respect to matching the pseudo-observations in the future is then evaluated, and we find a mean increase in skill of about 17 % compared with the unweighted distribution. In addition, we show that our independence metric correctly clusters models known to be similar based on a CMIP6 “family tree”, which enables the application of a weighting based on the degree of inter-model dependence. We then apply the weighting approach, based on two observational estimates (the fifth generation of the European Centre for Medium-Range Weather Forecasts Retrospective Analysis – ERA5, and the Modern-Era Retrospective analysis for Research and Applications, version 2 – MERRA-2), to constrain CMIP6 projections under weak (SSP1-2.6) and strong (SSP5-8.5) climate change scenarios (SSP refers to the Shared Socioeconomic Pathways). Our results show a reduction in the projected mean warming for both scenarios because some CMIP6 models with high future warming receive systematically lower performance weights. The mean of end-of-century warming (2081–2100 relative to 1995–2014) for SSP5-8.5 with weighting is 3.7 ∘C, compared with 4.1 ∘C without weighting; the likely (66%) uncertainty range is 3.1 to 4.6 ∘C, which equates to a 13 % decrease in spread. For SSP1-2.6, the weighted end-of-century warming is 1 ∘C (0.7 to 1.4 ∘C), which results in a reduction of −0.1 ∘C in the mean and −24 % in the likely range compared with the unweighted case.

Artificial intelligence for melanoma diagnosis: a review of published studies until 2020

Convolutional neural networks (CNN) have shown unprecedented accuracy in digital image analysis, which can be harnessed for melanoma recognition through automated evaluation of clinical and dermatoscopic images. In experimental studies, modern CNN architectures perform single image analysis at the level of dermatologists and domain-experts, also for multi-class predictions including a multitude of possible diagnoses. This may not necessarily translate to good clinical performance, and reliable randomised controlled prospective clinical trials for modern CNNs are essentially missing. Weaknesses of CNNs are that limitations of available training image datasets propagate to limitations of CNN predictions, and they can not provide a reliable estimate of uncertainty. Recent research focuses on human-computer collaboration, where gains in accuracy were measured even with imperfect CNNs. With missing academic and clinical agreement on equivocal melanocytic lesions, fully automating histologic assessment of them with CNNs appear problematic, and applications in the near future are probably limited to supporting, referencing or recommendation roles.

Parameter estimation for the cosmic microwave background with Bayesian neural networks

In this paper, we present the first study that compares different models of Bayesian Neural Networks (BNNs) to predict the posterior distribution of the cosmological parameters directly from the Cosmic Microwave Background temperature and polarization maps. We focus our analysis on four different methods to sample the weights of the network during training: Dropout, DropConnect, Reparameterization Trick (RT), and Flipout. We find out that Flipout outperforms all other methods regardless of the architecture used, and provides tighter constraints for the cosmological parameters. Moreover we compare with MCMC posterior analysis obtaining comparable error correlation among parameters, with BNNs being orders of magnitude faster in inference, although less accurate. Thanks to the speed of the inference process with BNNs, the posterior distribution, outcome of the neural network, can be used as the initial proposal for the Markov Chain. We show that this combined approach increases the acceptance rate in the Metropolis-Hasting algorithm and accelerates the convergence of the MCMC, while reaching the same final accuracy. In the second part of the paper, we present a guide to the training and calibration of a successful multi-channel BNN for the CMB temperature and polarization map. We show how tuning the regularization parameter for the standard deviation of the approximate posterior on the weights in Flipout and RT we can produce unbiased and reliable uncertainty estimates, i.e., the regularizer acts like a hyperparameter analogous to the dropout rate in Dropout. Finally, we show how polarization, when combined with the temperature in a unique multi-channel tensor fed to a single BNN, helps to break degeneracies among parameters and provides stringent constraints.

Direct astrophysical tests of chiral effective field theory at supranuclear densities

Recent observations of neutron stars with gravitational waves and X-ray timing provide unprecedented access to the equation of state (EoS) of cold dense matter at densities difficult to realize in terrestrial experiments. At the same time, predictions for the EoS with reliable uncertainty estimates from chiral effective field theory ($\chi$EFT) bound our theoretical ignorance. In this work, we analyze astrophysical data using a nonparametric representation of the neutron-star EoS conditioned on $\chi$EFT to directly constrain the underlying physical properties of the compact objects. We discuss how the data alone constrain the EoS at high densities when we condition on $\chi$EFT at low densities. We also demonstrate how to exploit astrophysical data to directly test the predictions of $\chi$EFT for the EoS up to twice nuclear saturation density, and estimate the density at which these predictions might break down. We find that the existence of massive pulsars, gravitational waves from GW170817, and NICER observations of PSR J0030+0451 favor $\chi$EFT predictions for the EoS up to nuclear saturation density over a more agnostic analysis by as much as a factor of 7 for the quantum Monte Carlo (QMC) calculations used in this work. While $\chi$EFT predictions using QMC are fully consistent with gravitational-wave data up to twice nuclear saturation density, NICER observations suggest that the EoS stiffens relative to these predictions at nuclear saturation density. Additionally, we marginalize over the uncertainty in the density at which $\chi$EFT begins to break down, constraining the radius of a $1.4\,M_\odot$ neutron star to $R_{1.4}=11.40^{+1.38}_{-1.04}$ ($12.54^{+0.71}_{-0.63}$) km and the pressure at twice nuclear saturation density to $p(2n_\mathrm{sat})=14.2^{+18.1}_{-8.4}$ ($28.7^{+15.3}_{-15.0}$) MeV/fm$^3$ with massive pulsar and gravitational-wave (and NICER) data.

Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation.

ObjectiveCancer is a leading cause of death, but much of the diagnostic information is stored as unstructured data in pathology reports. We aim to improve uncertainty estimates of machine learning-based pathology parsers and evaluate performance in low data settings.Materials and methodsOur data comes from the Urologic Outcomes Database at UCSF which includes 3232 annotated prostate cancer pathology reports from 2001 to 2018. We approach 17 separate information extraction tasks, involving a wide range of pathologic features. To handle the diverse range of fields, we required 2 statistical models, a document classification method for pathologic features with a small set of possible values and a token extraction method for pathologic features with a large set of values. For each model, we used isotonic calibration to improve the model’s estimates of its likelihood of being correct.ResultsOur best document classifier method, a convolutional neural network, achieves a weighted F1 score of 0.97 averaged over 12 fields and our best extraction method achieves an accuracy of 0.93 averaged over 5 fields. The performance saturates as a function of dataset size with as few as 128 data points. Furthermore, while our document classifier methods have reliable uncertainty estimates, our extraction-based methods do not, but after isotonic calibration, expected calibration error drops to below 0.03 for all extraction fields.ConclusionsWe find that when applying machine learning to pathology parsing, large datasets may not always be needed, and that calibration methods can improve the reliability of uncertainty estimates.

Comparing Methods to Constrain Future European Climate Projections Using a Consistent Framework

AbstractPolitical decisions, adaptation planning, and impact assessments need reliable estimates of future climate change and related uncertainties. To provide these estimates, different approaches to constrain, filter, or weight climate model projections into probabilistic distributions have been proposed. However, an assessment of multiple such methods to, for example, expose cases of agreement or disagreement, is often hindered by a lack of coordination, with methods focusing on a variety of variables, time periods, regions, or model pools. Here, a consistent framework is developed to allow a quantitative comparison of eight different methods; focus is given to summer temperature and precipitation change in three spatial regimes in Europe in 2041–60 relative to 1995–2014. The analysis draws on projections from several large ensembles, the CMIP5 multimodel ensemble, and perturbed physics ensembles, all using the high-emission scenario RCP8.5. The methods’ key features are summarized, assumptions are discussed, and resulting constrained distributions are presented. Method agreement is found to be dependent on the investigated region but is generally higher for median changes than for the uncertainty ranges. This study, therefore, highlights the importance of providing clear context about how different methods affect the assessed uncertainty—in particular, the upper and lower percentiles that are of interest to risk-averse stakeholders. The comparison also exposes cases in which diverse lines of evidence lead to diverging constraints; additional work is needed to understand how the underlying differences between methods lead to such disagreements and to provide clear guidance to users.