Estimates Of Prediction Uncertainty Research Articles

Prediction of pathological complete response to chemotherapy for breast cancer using deep neural network with uncertainty quantification.

The I-SPY 2 trial is a national-wide, multi-institutional clinical trial designed to evaluate multiple new therapeutic drugs for high-risk breast cancer. Previous studies suggest that pathological complete response (pCR) is a viable indicator of long-term outcomes of neoadjuvant chemotherapy for high-risk breast cancer. While pCR can be assessed during surgery after the chemotherapy, early prediction of pCR before the completion of the chemotherapy may facilitate personalized treatment management to achieve an improved outcome. Notably, the acquisition of dynamic contrast-enhanced magnetic resonance (DCEMR) images at multiple time points during the I-SPY 2 trial opens up the possibility of achieving early pCR prediction. In this study, we investigated the feasibility of the early prediction of pCR to neoadjuvant chemotherapy using multi-time point DCEMR images and clinical data acquired in the I-SPY2 trial. The prediction uncertainty was also quantified to allow physicians to make patient-specific decisions on treatment plans based on the level of associated uncertainty. The dataset used in our study included 624 patients with DCEMR images acquired at 3 time points before the completion of the chemotherapy: pretreatment (T0), after 3 cycles of treatment (T1), and after 12 cycles of treatment (T2). A convolutional long short-term memory (LSTM) network-based deep learning model, which integrated multi-time point deep image representations with clinical data, including tumor subtypes, was developed to predict pCR. The performance of the model was evaluated via the method of nested 5-fold cross validation. Moreover, we also quantified prediction uncertainty for each patient through test-time augmentation. To investigate the relationship between predictive performance and uncertainty, the area under the receiver operating characteristic curve (AUROC) was assessed on subgroups of patients stratified by the uncertainty score. By integrating clinical data and DCEMR images obtained at three-time points before treatment completion, the AUROC reached 0.833 with a sensitivity of 0.723 and specificity of 0.800. This performance was significantly superior (p<0.01) to models using only images (AUROC=0.706) or only clinical data (AUROC=0.746). After stratifying the patients into eight subgroups based on the uncertainty score, we found that group #1, with the lowest uncertainty, had a superior AUROC of 0.873. The AUROC decreased to 0.637 for group #8, which had the highest uncertainty. The results indicate that our convolutional LSTM network-based deep learning model can be used to predict pCR earlier before the completion of chemotherapy. By combining clinical data and multi-time point deep image representations, our model outperforms models built solely on clinical or image data. Estimating prediction uncertainty may enable physicians to prioritize or disregard predictions based on their associated uncertainties. This approach could potentially enhance the personalization of breast cancer therapy.

Insights into the prediction uncertainty of machine-learning-based digital soil mapping through a local attribution approach

Abstract. Machine learning (ML) models have become key ingredients for digital soil mapping. To improve the interpretability of their predictions, diagnostic tools such as the widely used local attribution approach known as SHapley Additive exPlanations (SHAP) have been developed. However, the analysis of ML model predictions is only one part of the problem, and there is an interest in obtaining deeper insights into the drivers of the prediction uncertainty as well, i.e. explaining why an ML model is confident given the set of chosen covariate values in addition to why the ML model delivered some particular results. In this study, we show how to apply SHAP to local prediction uncertainty estimates for a case of urban soil pollution – namely, the presence of petroleum hydrocarbons in soil in Toulouse (France), which pose a health risk via vapour intrusion into buildings, direct soil ingestion, and groundwater contamination. Our results show that the drivers of the prediction best estimates are not necessarily the drivers of confidence in these predictions, and we identify those leading to a reduction in uncertainty. Our study suggests that decisions regarding data collection and covariate characterisation as well as communication of the results should be made accordingly.

Bayesian Neural Network Variational Autoencoder Inverse Mapper (BNN-VAIM) and its application in Compton Form Factors extraction

We extend the Variational Autoencoder Inverse Mapper (VAIM) framework for the inverse problem of extracting Compton Form Factors (CFFs) from deeply virtual exclusive reactions, such as the unpolarized Deeply virtual exclusive scattering (DVCS) cross section. VAIM is an end-to-end deep learning framework to address the solution ambiguity issue in ill-posed inverse problems, which comprises of a forward mapper and a backward mapper to simulate the forward and inverse processes, respectively. In particular, we incorporate Bayesian Neural Network (BNN) into the VAIM architecture (BNN-VAIM) for uncertainty quantification. By sampling the weights and biases distributions of the BNN in the backward mapper of the VAIM, BNN-VAIM is able to estimate prediction uncertainty associated with each individual solution obtained for an ill-posed inverse problem. We first demonstrate the uncertainty quantification capability of BNN-VAIM in a toy inverse problem. Then, we apply BNN-VAIM to the inverse problem of extracting 8 CFFs from the unpolarized DVCS cross section.

Development and Evaluation of Conformal Prediction Methods for Quantitative Structure-Activity Relationship.

The quantitative structure-activity relationship (QSAR) regression model is a commonly used technique for predicting the biological activities of compounds using their molecular descriptors. Besides accurate activity estimation, obtaining a prediction uncertainty metric like a prediction interval is highly desirable. Quantifying prediction uncertainty is an active research area in statistical and machine learning (ML), but the implementation for QSAR remains challenging. However, most ML algorithms with high predictive performance require add-on companions for estimating the uncertainty of their prediction. Conformal prediction (CP) is a promising approach as its main components are agnostic to the prediction modes, and it produces valid prediction intervals under weak assumptions on the data distribution. We proposed computationally efficient CP algorithms tailored to the most widely used ML models, including random forests, deep neural networks, and gradient boosting. The algorithms use a novel approach to the derivation of nonconformity scores from the estimates of prediction uncertainty generated by the ensembles of point predictions. The validity and efficiency of proposed algorithms are demonstrated on a diverse collection of QSAR data sets as well as simulation studies. The provided software implementing our algorithms can be used as stand-alone or easily incorporated into other ML software packages for QSAR modeling.

Differentiation of glioblastoma from solitary brain metastasis using deep ensembles: Empirical estimation of uncertainty for clinical reliability

Background and ObjectivesTo develop a clinically reliable deep learning model to differentiate glioblastoma (GBM) from solitary brain metastasis (SBM) by providing predictive uncertainty estimates and interpretability. MethodsA total of 469 patients (300 GBM, 169 SBM) were enrolled in the institutional training set. Deep ensembles based on DenseNet121 were trained on multiparametric MRI. The model performance was validated in the external test set consisting of 143 patients (101 GBM, 42 SBM). Entropy values for each input were evaluated for uncertainty measurement; based on entropy values, the datasets were split to high- and low-uncertainty groups. In addition, entropy values of out-of-distribution (OOD) data from unknown class (257 patients with meningioma) were compared to assess uncertainty estimates of the model. The model interpretability was further evaluated by localization accuracy of the model. ResultsOn external test set, the area under the curve (AUC), accuracy, sensitivity and specificity of the deep ensembles were 0.83 (95 % confidence interval [CI] 0.76–0.90), 76.2 %, 54.8 % and 85.2 %, respectively. The performance was higher in the low-uncertainty group than in the high-uncertainty group, with AUCs of 0.91 (95 % CI 0.83–0.98) and 0.58 (95 % CI 0.44–0.71), indicating that assessment of uncertainty with entropy values ascertained reliable prediction in the low-uncertainty group. Further, deep ensembles classified a high proportion (90.7 %) of predictions on OOD data to be uncertain, showing robustness in dataset shift. Interpretability evaluated by localization accuracy provided further reliability in the “low-uncertainty and high-localization accuracy” subgroup, with an AUC of 0.98 (95 % CI 0.95–1.00). ConclusionsEmpirical assessment of uncertainty and interpretability in deep ensembles provides evidence for the robustness of prediction, offering a clinically reliable model in differentiating GBM from SBM.

A multi-model evaluation of probabilistic streamflow predictions via residual error modelling

Probabilistic streamflow predictions are valuable tools for predictive uncertainty estimation, hydrologic risk management, and support for decision-making in water resources. Usually, predictive uncertainty quantification is developed and assessed using only a single hydrological model, making it difficult to generalize to other model configurations. To tackle this issue, we assess changes in the model performance ranking of diverse streamflow models by applying a residual error model post-processing approach to multiple basins and multiple models. This assessment employed 141 basins from the Great Lakes watershed covering the USA and Canada, and 13 diverse streamflow models, which are evaluated using deterministic and probabilistic performance metrics. As the first study to implement probabilistic methods to diverse streamflow models applied to a multitude of basins, the analysis here examines the dependence of probabilistic streamflow estimation quality on model quality. Our findings show that streamflow model choice influences the robustness of probabilistic predictions. It was found that moving from deterministic to probabilistic predictions using a post-processing approach does not change the streamflow model performance ranking for the best and worst deterministic models, but models of intermediate rank in deterministic evaluation do not have consistent ranking when evaluated in probabilistic mode. Post-processing residual errors of long short-term memory (LSTM) network models are consistently the best-performing model in terms of deterministic and probabilistic metrics. This study highlights the significance of combining deterministic streamflow model predictions with residual error models for improving the quality and increasing the value of hydrological predictions, quantifying uncertainty, and facilitating decision-making in operational water management. It also clarifies the degree to which probabilistic predictions depend upon good model performance and can compensate for poor model performance.

Prediction uncertainty estimates elucidate the limitation of current NSCLC subtype classification in representing mutational heterogeneity

The heterogeneous pathogenesis and treatment response of non-small cell lung cancer (NSCLC) has led clinical treatment decisions to be guided by NSCLC subtypes, with lung adenocarcinoma and lung squamous cell carcinoma being the most common subtypes. While histology-based subtyping remains challenging, NSCLC subtypes were found to be distinct at the transcriptomic level. However, unlike genomic alterations, gene expression is generally not assessed in clinical routine. Since subtyping of NSCLC has remained elusive using mutational data, we aimed at developing a neural network model that simultaneously learns from adenocarcinoma and squamous cell carcinoma samples of other tissue types and is regularized using a neural network model trained from gene expression data. While substructures of the expression-based manifold were captured in the mutation-based manifold, NSCLC classification accuracy did not significantly improve. However, performance was increased when rejecting inconclusive samples using an ensemble-based approach capturing prediction uncertainty. Importantly, SHAP analysis of misclassified samples identified co-occurring mutations indicative of both NSCLC subtypes, questioning the current NSCLC subtype classification to adequately represent inherent mutational heterogeneity. Since our model captures mutational patterns linked to clinical heterogeneity, we anticipate it to be suited as foundational model of genomic data for clinically relevant prognostic or predictive downstream tasks.

A review of predictive uncertainty estimation with machine learning

Predictions and forecasts of machine learning models should take the form of probability distributions, aiming to increase the quantity of information communicated to end users. Although applications of probabilistic prediction and forecasting with machine learning models in academia and industry are becoming more frequent, related concepts and methods have not been formalized and structured under a holistic view of the entire field. Here, we review the topic of predictive uncertainty estimation with machine learning algorithms, as well as the related metrics (consistent scoring functions and proper scoring rules) for assessing probabilistic predictions. The review covers a time period spanning from the introduction of early statistical (linear regression and time series models, based on Bayesian statistics or quantile regression) to recent machine learning algorithms (including generalized additive models for location, scale and shape, random forests, boosting and deep learning algorithms) that are more flexible by nature. The review of the progress in the field, expedites our understanding on how to develop new algorithms tailored to users’ needs, since the latest advancements are based on some fundamental concepts applied to more complex algorithms. We conclude by classifying the material and discussing challenges that are becoming a hot topic of research.

COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition.

Automatically recognising apparent emotions from face and voice is hard, in part because of various sources of uncertainty, including in the input data and the labels used in a machine learning framework. This paper introduces an uncertainty-aware multimodal fusion approach that quantifies modality-wise aleatoric or data uncertainty towards emotion prediction. We propose a novel fusion framework, in which latent distributions over unimodal temporal context are learned by constraining their variance. These variance constraints, Calibration and Ordinal Ranking, are designed such that the variance estimated for a modality can represent how informative the temporal context of that modality is w.r.t. emotion recognition. When well-calibrated, modality-wise uncertainty scores indicate how much their corresponding predictions are likely to differ from the ground truth labels. Well-ranked uncertainty scores allow the ordinal ranking of different frames across different modalities. To jointly impose both these constraints, we propose a softmax distributional matching loss. Our evaluation on AVEC 2019 CES, CMU-MOSEI, and IEMOCAP datasets shows that the proposed multimodal fusion method not only improves the generalisation performance of emotion recognition models and their predictive uncertainty estimates, but also makes the models robust to novel noise patterns encountered at test time.

Land Subsidence Model Inversion with the Estimation of Both Model Parameter Uncertainty and Predictive Uncertainty Using an Evolutionary-Based Data Assimilation (EDA) and Ensemble Model Output Statistics (EMOS)

The nonlinearity nature of land subsidence and limited observations cause premature convergence in typical data assimilation methods, leading to both underestimation and miscalculation of uncertainty in model parameters and prediction. This study focuses on a promising approach, the combination of evolutionary-based data assimilation (EDA) and ensemble model output statistics (EMOS), to investigate its performance in land subsidence modeling using EDA with a smoothing approach for parameter uncertainty quantification and EMOS for predictive uncertainty quantification. The methodology was tested on a one-dimensional subsidence model in Kawajima (Japan). The results confirmed the EDA’s robust capability: Model diversity was maintained even after 1000 assimilation cycles on the same dataset, and the obtained parameter distributions were consistent with the soil types. The ensemble predictions were converted to Gaussian predictions with EMOS using past observations statistically. The Gaussian predictions outperformed the ensemble predictions in predictive performance because EMOS compensated for the over/under-dispersive prediction spread and the short-term bias, a potential weakness for the smoothing approach. This case study demonstrates that combining EDA and EMOS contributes to groundwater management for land subsidence control, considering both the model parameter uncertainty and the predictive uncertainty.

A Surrogate Model for Studying Solar Energetic Particle Transport and the Seed Population

AbstractThe high energy particles originating from the Sun, known as solar energetic particles (SEPs), contribute significantly to the space radiation environment, posing serious threats to astronauts and scientific instruments on board spacecraft. The mechanism that accelerates the SEPs to the observed energy ranges, their transport in the inner heliosphere, and the influence of suprathermal seed particle spectrum are open questions in heliophysics. Accurate predictions of the occurrences of SEP events well in advance are necessary to mitigate their adverse effects but prediction based on first principle models still remains a challenge. In this scenario, adopting a machine learning approach to SEP modeling and prediction is desirable. However, the lack of a balanced database of SEP events restrains this approach. We addressed this limitation by generating large data sets of synthetic SEP events sampled from the physics‐based model, Energetic Particle Radiation Environment Module (EPREM). Using this data, we developed neural networks‐based surrogate models to study the seed population parameter space. Our models, EPREM‐S, run thousands to millions of times faster (depending on computer hardware), making simulation‐based inference workflows practicable in SEP studies while providing predictive uncertainty estimates using a deep ensemble approach.

Nonlinear regression of remaining surgery duration from videos via Bayesian LSTM-based deep negative correlation learning

In this paper, we address the problem of estimating remaining surgery duration (RSD) from surgical video frames. We propose a Bayesian long short-term memory (LSTM) network-based Deep Negative Correlation Learning approach called BD-Net for accurate regression of RSD prediction as well as estimation of prediction uncertainty. Our method aims to extract discriminative visual features from surgical video frames and model the temporal dependencies among frames to improve the RSD prediction accuracy. To this end, we propose to train an ensemble of Bayesian LSTMs on top of a backbone network by the way of deep negative correlation learning (DNCL). More specifically, we deeply learn a pool of decorrelated Bayesian regressors with sound generalization capabilities through managing their intrinsic diversities. BD-Net is simple and efficient. After training, it can produce both RSD prediction and uncertainty estimation in a single inference run. We demonstrate the efficacy of BD-Net on publicly available datasets of two different types of surgeries: one containing 101 cataract microscopic surgeries with short durations and the other containing 80 cholecystectomy laparoscopic surgeries with relatively longer durations. Experimental results on both datasets demonstrate that the proposed BD-Net achieves better results than the state-of-the-art (SOTA) methods. A reference implementation of our method can be found at: https://github.com/jywu511/BD-Net.

A self‐evaluated predictive model: A Bayesian neural network approach to colorectal cancer diagnosis

AbstractArtificial intelligence has shown immense potential in cancer prediction, but existing models cannot estimate prediction uncertainty by themselves. Here, we developed a Bayesian neural network (BNN) model, BNN‐CRC15, for colorectal cancer (CRC) prediction while assessing its reliability. The model was trained on routine laboratory data obtained from 27,911 participants and provided quantified prediction uncertainty, allowing identification of a subset of participants in which the model was confident, mimicking the diagnostic process of human practitioners. Our model exhibited superior performance (area under the curve = 0.918) in the confident participant group, which accounted for 46.4% of the patients, indicating that routine laboratory data alone are sufficient for accurate predictions in this subset. For the non‐confident group, further advanced tests, such as colonoscopy, could be recommended to achieve more accurate predictions. In addition, our model demonstrated superior overall accuracy (0.848) in all patients, outperforming other five traditional algorithms (extreme gradient boosting, support vector machine, logistic regression, random forest, and artificial neural network) and fecal immunochemical test in distinguishing CRC from non‐CRC. These findings suggest that our BNN‐CRC15 model could serve as a valuable tool for improving CRC diagnosis and prevention.

Space-dependent turbulence model aggregation using machine learning

Computational models of fluid flows based on the Reynolds-Averaged Navier–Stokes (RANS) equations supplemented with a turbulence model are the gold standard in engineering applications. A plethora of turbulence models and related variants exist, none of which are fully reliable outside the range of flow configurations for which they were calibrated. Thus, the choice of a suitable turbulence closure largely relies on subjective expert judgement and engineering know-how. In this article, we propose a data-driven methodology for combining the solutions of a set of competing turbulence models. The individual model predictions are linearly combined for providing an ensemble solution accompanied by estimates of predictive uncertainty due to the turbulence model choice. First, for a set of training flow configurations we assign high weights to component models in the regions where they perform best, and vice versa, based on a measure of distance between high-fidelity data and individual model predictions. The model weights are then mapped into a space of features, representative of the local flow physics, and regressed using a Random Forests (RF) algorithm. Finally, the RF regressor is used to infer spatial distributions of the model weights for unseen configurations. Predictions of new cases are constructed as a convex linear combination of the underlying models solutions, while the between-model variance provides information about regions of high model uncertainty. The method is demonstrated for a class of flows through the compressor cascade NACA65 V103 at Re≃3×105. The results show that the aggregated solution outperforms the accuracy of individual models for the quantity used to inform the RF regressor, and performs well for other quantities well-correlated with the previous one. The estimated uncertainty intervals are generally consistent with the target high-fidelity data. The present approach then represents a viable methodology for a more objective selection and combination of alternative turbulence models in configurations of interest for engineering practice.

Trustworthy Fault Diagnosis With Uncertainty Estimation Through Evidential Convolutional Neural Networks

Deep neural networks (DNNs) have been widely used for intelligent fault diagnosis under the closed world assumption that any testing data is within classes of the training data. However, in reality, out-of-distribution (OOD) cases such as new fault conditions can happen after the original trained model is deployed. Most of the current DNNs are deterministic which can misclassify with high confidence in the open-world scenario. This overconfident behavior would not guarantee the reliability and robustness of fault diagnosis results in practice. Therefore, trustworthy intelligent fault diagnosis with uncertainty estimation is crucial for real applications. In this paper, we develop a novel convolutional neural network integrating evidence theory to achieve fault classifications with prediction uncertainty estimation. The estimated prediction uncertainty can identify potential OOD samples. This approach allows a minimal modification of the state-of-the-art DNN model by using a risk-calibrated evidential loss function and Dirichlet distribution that replaces the classification probabilities. The experimental results show that the proposed approach can not only achieve accurate classification of known classes but also detect unknown classes effectively. The proposed method shows significant potential in detecting OOD patterns and provides trustworthy fault diagnosis in open and non-stationary environments.

Quantifying Deep Learning Model Uncertainty in Conformal Prediction

Precise estimation of predictive uncertainty in deep neural networks is a critical requirement for reliable decision-making in machine learning and statistical modeling, particularly in the context of medical AI. Conformal Prediction (CP) has emerged as a promising framework for representing the model uncertainty by providing well-calibrated confidence levels for individual predictions. However, the quantification of model uncertainty in conformal prediction remains an active research area, yet to be fully addressed. In this paper, we explore state-of-the-art CP methodologies and their theoretical foundations. We propose a probabilistic approach in quantifying the model uncertainty derived from the produced prediction sets in conformal prediction and provide certified boundaries for the computed uncertainty. By doing so, we allow model uncertainty measured by CP to be compared by other uncertainty quantification methods such as Bayesian (e.g., MC-Dropout and DeepEnsemble) and Evidential approaches.

Estimating daily air temperature and pollution in Catalonia: A comprehensive spatiotemporal modelling of multiple exposures

Environmental epidemiology studies require models of multiple exposures to adjust for co-exposure and explore interactions. We estimated spatiotemporal exposure to surface air temperature and pollution (PM2.5, PM10, NO2, O3) at high spatiotemporal resolution (daily, 250 m) for 2018–2020 in Catalonia. Innovations include the use of TROPOMI products, a data split for remote sensing gap-filling evaluation, estimation of prediction uncertainty, and use of explainable machine learning.We compiled meteorological and air quality station measurements, climate and atmospheric composition reanalyses, remote sensing products, and other spatiotemporal data. We performed gap-filling of remotely-sensed products using Random Forest (RF) models and validated them using Out-Of-Bag (OOB) samples and a structured data split. The exposure modelling workflow consisted of: 1) PM2.5 station imputation with PM10 data; 2) quantile RF (QRF) model fitting; and 3) geostatistical residual spatial interpolation. Prediction uncertainty was estimated using QRF. SHAP values were used to examine variable importance and the fitted relationships. Model performance was assessed via nested CV at the station level.Evaluation of the gap-filling models using the structured split showed error underestimation when using OOB. Temperature models had the best performance (R2 =0.98) followed by the gaseous air pollutants (R2 =0.81 for NO2 and 0.86 for O3), while the performance of the PM2.5 and PM10 models was lower (R2 =0.57 and 0.63 respectively). Predicted exposure patterns captured urban heat island effects, dust advection events, and NO2 hotspots. SHAP values estimated a high importance of TROPOMI tropospheric NO2 columns in PM and NO2 models, and confirmed that the fitted associations conformed to prior knowledge.Our work highlights the importance of correctly validating gap-filling models and the potential of TROPOMI measurements. Moderate performance in PM models can be partly explained by the poor station coverage. Our exposure estimates can be used in epidemiological studies potentially accounting for exposure uncertainty.

Analyse géostatistique des niveaux piézométriques dans un secteur minier avec trois mines en activité

AbstractMining activities can significantly impact groundwater reservoirs in their vicinity. Different approaches have been employed, with varying success, to investigate the spatial variability of groundwater levels in mining areas. Typical problems include the small sample size, the non-Gaussian distribution of the data, and the clustering of sample locations near the mines. These conditions complicate the estimation of spatial dependence. Under sparse and irregular sampling conditions, stochastic methods, which can provide estimates of prediction uncertainty, are preferable to deterministic ones. This research focuses on the comparison of two stochastic methods, stochastic local interactions (SLI) and universal Kriging (UK), using water level data from 72 locations around three mines in Northern Greece. UK is a well-known, variogram-based geostatistical method, while SLI is a computationally efficient kernel-based method that can cope with large spatial datasets. The non-Gaussian distribution of the data is handled by means of a flexible, data-driven Gaussian anamorphosis method that uses kernel functions. The spatial prediction performance of both methods is assessed based on cross-validation. UK performs better than SLI, due to the fact that the former incorporates a linear trend function. On the other hand, a comparison of the two methods using data from a single mine that contains only 28 measurement locations shows that SLI performs slightly better than UK. The prediction uncertainties for both methods are also estimated and compared. The results suggest that SLI can provide better estimates than classical geostatistical methods for small sample sizes that do not allow reliable estimation of the variogram model.

An uncertainty estimator method based on the application of feature density to classify mammograms for breast cancer detection

In the area of medical imaging, one of the factors that can negatively influence the performance of prediction algorithms is the limited number of observations for each class within a labeled dataset. Usually, in order to increase the samples, a second set of unlabeled images is used. However, this set adds two new problems (i) finding patient observations with different pathologies than those observed in the labeled data set and (ii) finding images belonging to a different distribution from the dataset used in the model training process. This way, merging datasets from different sources can have an adverse effect on the distribution of features. Encountering this type of data (better known as out-of-distribution data) within the deployment environments may also lead to varying degrees of performance degradation as can be seen in the different experimental results obtained. In this research, a study of the behavior of Feature Density is made, as a mathematical model for the estimation of predictive uncertainty in supervised classification algorithms, in order to improve the behavior when out-of-distribution data are presented in the dataset. The Feature Density method is based on the estimation of feature density by means of histogram calculation (or Probability Density Function). The advantage of this method over the baseline approach (Mahalanobis distance) is that it does not assume a Gaussian-type distribution of sample characteristics and serves to estimate the uncertainty. This work focuses on the binary classification of mammography X-ray images from three different datasets simulating the condition of a different degree of contamination with out-of-distribution sample. According to the obtained results, the performance of the proposed method depends directly on the architecture of the implemented neural network.

Femtosecond pulse parameter estimation from photoelectron momenta using machine learning

Deep learning models have provided huge interpretation power for image-like data. Specifically, convolutional neural networks (CNNs) have demonstrated incredible acuity for tasks such as feature extraction or parameter estimation. Here we test CNNs on strong-field ionization photoelectron spectra, training on theoretical data sets to ‘invert’ experimental data. Pulse characterization is used as a ‘testing ground’, specifically we retrieve the laser intensity, where ‘traditional’ measurements typically lead to 20% uncertainty. We report on crucial data augmentation techniques required to successfully train on theoretical data and return consistent results from experiments, including accounting for detector saturation. The same procedure can be repeated to apply CNNs in a range of scenarios for strong-field ionization. Using a predictive uncertainty estimation, reliable laser intensity uncertainties of a few percent can be extracted, which are consistently lower than those given by traditional techniques. Using interpretability methods can reveal parts of the distribution that are most sensitive to laser intensity, which can be directly associated with holographic interferences. The CNNs employed provide an accurate and convenient ways to extract parameters, and represent a novel interpretational tool for strong-field ionization spectra.