ImmUQBench: a benchmark on uncertainty quantification of protein immunogenicity prediction

Objective.Deep learning auto-segmentation has greatly advanced contouring in radiotherapy. However, quality assurance remains necessary due to performance fluctuation among individual patients. This manual process reintroduces variability and partially reduces time-saving benefits. As a solution, uncertainty quantification (UQ) is increasingly explored for its ability to estimate output confidence. While numerous methods to quantify uncertainty exist, their comparative reliability remains underexplored. This study compares the reliability of commonly used UQ approaches for auto-segmentation in radiotherapy.Approach.We evaluated the reliability of three popular uncertainty methods (Monte Carlo dropout, deep ensemble modelling and test-time augmentation) and uncertainty metrics (predictive entropy, mutual information and variance). We trained a 3D U-Net within the nnU-Net framework to segment 19 organs at risk (OAR) for head and neck cancer patients. We evaluated the reliability of the UQ methods and metrics on a set of 10 patients using segmentation model accuracy (surface Dice similarity coefficient), confidence calibration (expected calibration error (label)), and error localisation ability (uncertainty-error (U-E) overlap). Both multi-class and class-specific uncertainty maps were assessed.Main results.Segmentation accuracy remained stable without significant deviations across all UQ methods with respect to the baseline model without UQ. The reliability of different UQ methods in terms of confidence calibration and error localisation was also comparable. In contrast, the choice of UQ metric significantly influenced reliability. Multi-class predictive entropy (ECE-label: 0.06-0.06, U-E overlap: 0.45-0.46) consistently outperformed variance (ECE-label: 0.13-0.14, U-E overlap: 0.32-0.40,p< 0.001) and mutual information (ECE-label: 0.13-0.14, U-E overlap: 0.35-0.40,p< 0.001). Predictive entropy also demonstrated superior reliability as a class-specific UQ metric, though variability was observed between OARs.Significance.This study demonstrates that the choice of UQ approach substantially impacts the reliability of uncertainty maps. While different UQ methods performed comparably, the specific UQ metric chosen significantly affected reliability. These findings underscore the importance of careful metric selection and evaluation prior to application.

Background/purposeThe use of artificial intelligence (AI) in radiotherapy (RT) is expanding rapidly. However, there exists a notable lack of clinician trust in AI models, underscoring the need for effective uncertainty quantification (UQ) methods. The purpose of this study was to scope existing literature related to UQ in RT, identify areas of improvement, and determine future directions. MethodsWe followed the PRISMA-ScR scoping review reporting guidelines. We utilized the population (human cancer patients), concept (utilization of AI UQ), context (radiotherapy applications) framework to structure our search and screening process. We conducted a systematic search spanning seven databases, supplemented by manual curation, up to January 2024. Our search yielded a total of 8980 articles for initial review. Manuscript screening and data extraction was performed in Covidence. Data extraction categories included general study characteristics, RT characteristics, AI characteristics, and UQ characteristics. ResultsWe identified 56 articles published from 2015 to 2024. 10 domains of RT applications were represented; most studies evaluated auto-contouring (50 %), followed by image-synthesis (13 %), and multiple applications simultaneously (11 %). 12 disease sites were represented, with head and neck cancer being the most common disease site independent of application space (32 %). Imaging data was used in 91 % of studies, while only 13 % incorporated RT dose information. Most studies focused on failure detection as the main application of UQ (60 %), with Monte Carlo dropout being the most commonly implemented UQ method (32 %) followed by ensembling (16 %). 55 % of studies did not share code or datasets. ConclusionOur review revealed a lack of diversity in UQ for RT applications beyond auto-contouring. Moreover, we identified a clear need to study additional UQ methods, such as conformal prediction. Our results may incentivize the development of guidelines for reporting and implementation of UQ in RT.

Background/purpose:The use of artificial intelligence (AI) in radiotherapy (RT) is expanding rapidly. However, there exists a notable lack of clinician trust in AI models, underscoring the need for effective uncertainty quantification (UQ) methods. The purpose of this study was to scope existing literature related to UQ in RT, identify areas of improvement, and determine future directions.Methods:We followed the PRISMA-ScR scoping review reporting guidelines. We utilized the population (human cancer patients), concept (utilization of AI UQ), context (radiotherapy applications) framework to structure our search and screening process. We conducted a systematic search spanning seven databases, supplemented by manual curation, up to January 2024. Our search yielded a total of 8980 articles for initial review. Manuscript screening and data extraction was performed in Covidence. Data extraction categories included general study characteristics, RT characteristics, AI characteristics, and UQ characteristics.Results:We identified 56 articles published from 2015–2024. 10 domains of RT applications were represented; most studies evaluated auto-contouring (50%), followed by image-synthesis (13%), and multiple applications simultaneously (11%). 12 disease sites were represented, with head and neck cancer being the most common disease site independent of application space (32%). Imaging data was used in 91% of studies, while only 13% incorporated RT dose information. Most studies focused on failure detection as the main application of UQ (60%), with Monte Carlo dropout being the most commonly implemented UQ method (32%) followed by ensembling (16%). 55% of studies did not share code or datasets.Conclusion:Our review revealed a lack of diversity in UQ for RT applications beyond auto-contouring. Moreover, there was a clear need to study additional UQ methods, such as conformal prediction. Our results may incentivize the development of guidelines for reporting and implementation of UQ in RT.

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

It is critical that machine learning (ML) model predictions be trustworthy for high-throughput catalyst discovery approaches. Uncertainty quantification (UQ) methods allow estimation of the trustworthiness of an ML model, but these methods have not been well explored in the field of heterogeneous catalysis. Herein, we investigate different UQ methods applied to a crystal graph convolutional neural network to predict adsorption energies of molecules on alloys from the Open Catalyst 2020 dataset, the largest existing heterogeneous catalyst dataset. We apply three UQ methods to the adsorption energy predictions, namely k-fold ensembling, Monte Carlo dropout, and evidential regression. The effectiveness of each UQ method is assessed based on accuracy, sharpness, dispersion, calibration, and tightness. Evidential regression is demonstrated to be a powerful approach for rapidly obtaining tunable, competitively trustworthy UQ estimates for heterogeneous catalysis applications when using neural networks. Recalibration of model uncertainties is shown to be essential in practical screening applications of catalysts using uncertainties.

Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial

Data-driven machine learning methods for weather forecasting have experienced a steep progress over the last years, with recent studies demonstrating substantial improvements over physics-based numerical weather prediction models. Beyond improved forecasts, the major advantages of purely data-driven models are their substantially lower computational costs and faster generation of forecasts, once a model has been trained. However, in contrast to ensemble forecasts from physical weather models, most efforts in data-driven weather forecasting have been limited to deterministic, point-valued predictions only, making it impossible to quantify forecast uncertainties which is crucial for optimal decision making in applications.Our overarching aim is to evaluate and compare methods for creating probabilistic forecasts from data-driven weather models. The uncertainty quantification (UQ) approaches we compare are either based on generating ensemble forecasts from data-driven weather models via perturbations to the initial conditions, or based on statistical post-hoc UQ methods. The perturbation-based methods either leverage initial conditions from the ECMWF IFS ensemble, add random Gaussian noise to the deterministic initial conditions, or add random field perturbations based on past observations (Magnusson et al., 2009). The post-hoc approaches operate on deterministic forecasts and quantify forecast uncertainty using established post-processing methods, namely distributional regression networks (Rasp and Lerch, 2018) and isotonic distributional regression (Walz et al., 2022; Henzi et al., 2021).Using forecasts from Pangu-Weather (Bi et al., 2023), we evaluate these UQ methods over Europe for selected user-relevant weather variables, such as wind speed at 10 m, temperature at 2 m, and geopotential height at 500 hPa. We focus on daily initialised Pangu-Weather forecasts for 2022 with a forecast horizon of up to 7 days and compare their performance against ECMWF IFS ensemble forecasts. Our results suggest that Pangu-Weather predictions combined with UQ approaches yield improvements over the ECMWF ensemble forecasts for lead times of up to 5 days in terms of the Continuous Ranked Probability Score. However, it strongly depends on the variable of interest which of the UQ methods performs best, none of the different UQ methods performs best over all variables and lead times. Post-hoc UQ methods tend to perform better for shorter lead times, while initial condition perturbations are superior for longer lead times, with in particular the random field method showing promising results.&amp;#160;

When establishing a safety case for a high-level nuclear waste repository, coupled thermal, hydraulic, mechanical and chemical (THMC) processes are modelled to assess radionuclide transport and the integrity of barriers. Modelling these processes requires numerous parameters, which are all subject to uncertainty. However, complex and extensive uncertainty quantification (UQ) methods can come at great computational cost, especially when performed on multi-parametric problems. Therefore, UQ methods are sought with manageable computational effort while still capturing the relevant information to allow a meaningful interpretation of the results. The aim of this study is the comparison of three different UQ methods with different levels of complexity and computational cost in terms of their results. Radionuclide transport and THM-calculations are performed by employing these methods to the French Callovo-Oxfordian claystone as a reference material. The three methods include: (1) a UQ method based on the complete sampling of input parameter distributions by a Monte Carlo approach, (2) a UQ method based on a minimal number of data points, by sampling quantiles of the input parameter distributions, as well as the bounds of the distributions’ intervals, and (3) a UQ method that involves a first-order second-moment reliability approach. The results depict the different limitations and benefits of the UQ methods analysed while highlighting that a comprehensive understanding of parameter sensitivity and modelling approach are key to choosing the appropriate UQ method. • Applicability of UQ methods depends on complexity of problem. • FOSM and quantile method can be appropriate for linear problems and non-coupled processes. • MC approach remains method of choice for complex non-linear analyses.

Machine learning sequence-function models for proteins could enable significant advances in protein engineering, especially when paired with state-of-the-art methods to select new sequences for property optimization and/or model improvement. Such methods (Bayesian optimization and active learning) require calibrated estimations of model uncertainty. While studies have benchmarked a variety of deep learning uncertainty quantification (UQ) methods on standard and molecular machine-learning datasets, it is not clear if these results extend to protein datasets. In this work, we implemented a panel of deep learning UQ methods on regression tasks from the Fitness Landscape Inference for Proteins (FLIP) benchmark. We compared results across different degrees of distributional shift using metrics that assess each UQ method's accuracy, calibration, coverage, width, and rank correlation. Additionally, we compared these metrics using one-hot encoding and pretrained language model representations, and we tested the UQ methods in retrospective active learning and Bayesian optimization settings. Our results indicate that there is no single best UQ method across all datasets, splits, and metrics, and that uncertainty-based sampling is often unable to outperform greedy sampling in Bayesian optimization. These benchmarks enable us to provide recommendations for more effective design of biological sequences using machine learning.

Reservoir uncertainty quantification (UQ) methods are essential for enhancing the reliability of forecasting in oil and gas reservoirs, particularly across varying geological and operational scenarios. As reservoir management and production strategies become increasingly complex, accurate forecasting is crucial for maximizing recovery, minimizing risks, and optimizing resource allocation. Traditional reservoir models often incorporate significant uncertainties due to the inherent heterogeneity of subsurface environments, limited data, and dynamic operational factors. To address these challenges, advanced UQ methods have been developed to quantify and reduce uncertainties in reservoir behavior, providing more reliable forecasts. The application of UQ techniques, including Monte Carlo simulations, Bayesian inference, and machine learning algorithms, enables the integration of multiple sources of uncertainty, from geological heterogeneity and reservoir parameters to operational constraints. By incorporating these uncertainties into reservoir models, UQ methods offer a probabilistic framework that helps decision-makers evaluate a range of potential outcomes, rather than relying on a single deterministic prediction. This probabilistic approach supports more robust risk assessments, facilitating the identification of high-impact scenarios and enabling operators to develop contingency plans. Furthermore, UQ methods enable better integration of real-time data and adaptive reservoir management. As production data and monitoring results become available, UQ models can be updated, refining forecasts and enhancing operational efficiency. This dynamic approach allows for continuous model calibration and real-time decision-making, improving the overall management of reservoirs throughout their lifecycle. In conclusion, the adoption of reservoir uncertainty quantification methods significantly improves the accuracy and reliability of reservoir forecasting. By providing a comprehensive understanding of potential risks and production variability across geological and operational conditions, UQ techniques support informed decision-making, optimized recovery, and more sustainable reservoir management practices.

ABSTRACTBiopharmaceutical products, particularly monoclonal antibodies (mAbs), have gained prominence in the pharmaceutical market due to their high specificity and efficacy. As these products are projected to constitute a substantial portion of global pharmaceutical sales, the application of machine learning models in mAb development and manufacturing is gaining momentum. This paper addresses the critical need for uncertainty quantification in machine learning predictions, particularly in scenarios with limited training data. Leveraging ensemble learning and Monte Carlo simulations, our proposed method generates additional input samples to enhance the robustness of the model in small training datasets. We evaluate the efficacy of our approach through two case studies: predicting antibody concentrations in advance and real‐time monitoring of glucose concentrations during bioreactor runs using Raman spectra data. Our findings demonstrate the effectiveness of the proposed method in estimating the uncertainty levels associated with process performance predictions and facilitating real‐time decision‐making in biopharmaceutical manufacturing. This contribution not only introduces a novel approach for uncertainty quantification but also provides insights into overcoming challenges posed by small training datasets in bioprocess development. The evaluation demonstrates the effectiveness of our method in addressing key challenges related to uncertainty estimation within upstream cell cultivation, illustrating its potential impact on enhancing process control and product quality in the dynamic field of biopharmaceuticals.

Immense effort has been exerted in the materials informatics community towards enhancing the accuracy of machine learning (ML) models; however, the uncertainty quantification (UQ) of state-of-the-art algorithms also demands further development. Most prominent UQ methods are model-specific or are related to the ensembles of models; therefore, there is a need to develop a universal technique that can be readily applied to a single model from a diverse set of ML algorithms. In this study, we suggest a new UQ measure known as the Δ-metric to address this issue. The presented quantitative criterion was inspired by the k-nearest neighbor approach adopted for applicability domain estimation in chemoinformatics. It surpasses several UQ methods in accurately ranking the predictive errors and could be considered a low-cost option for a more advanced deep ensemble strategy. We also evaluated the performance of the presented UQ measure on various classes of materials, ML algorithms, and types of input features, thus demonstrating its universality.

Machine learning (ML) has emerged as a transformative tool in material science, enabling accelerated discovery and design of novel molecules while reducing experimental costs. Uncertainty quantification (UQ) is crucial for enhancing the reliability of ML predictions, particularly in high-stakes applications, such as functional polymer discovery. In this study, we present a comprehensive evaluation of nine UQ methods in ML─ensemble, Gaussian Process Regression (GPR), Monte Carlo Dropout (MCD), mean-variance estimation (MVE), Bayesian Neural Network based on Variational Inference (BNN-VI) and Markov Chain Monte Carlo (BNN-MCMC), evidential deep learning (EDL), quantile regression (QR), natural gradient boosting (NGBoost)─for predicting key polymer properties, including glass transition temperature (Tg), band gap (Eg), melting temperature (Tm) and decomposition temperature (Td). The models are assessed using three independent metrics, including prediction accuracy (R2), Spearman's rank correlation coefficient and calibration area, offering a robust framework for evaluating both mean predictions and uncertainty estimates. Our analysis spans data sets of four properties, out-of-distribution (OOD) experimental and molecular dynamics (MD)-derived data, high-Tg polymers and diverse polymer types, providing a holistic perspective on model performance. Our findings reveal that optimal UQ method selection is highly context-dependent. Ensemble method consistently excelled for general in-distribution predictions across four properties. For challenging OOD scenarios, BNN-MCMC offered a strong balance of predictive accuracy and reliable UQ. NGBoost emerged as the top-performing method for high-Tg polymers, effectively balancing accuracy and uncertainty characterization, with Ensemble method also providing excellent accuracy in this case. Furthermore, BNN-VI demonstrated superior and consistent performance across the nine distinct polymer classes evaluated. This comprehensive benchmark underscores the critical importance of selecting tailored UQ strategies to enhance the trustworthiness of ML predictions, optimize experimental validation efforts, and ultimately accelerate the discovery of advanced functional polymers.

Objective.Deep learning models are increasingly being implemented for automated medical image analysis to inform patient care. Most models, however, lack uncertainty information, without which the reliability of model outputs cannot be ensured. Several uncertainty quantification (UQ) methods exist to capture model uncertainty. Yet, it is not clear which method is optimal for a given task. The purpose of this work was to investigate several commonly used UQ methods for the critical yet understudied task of metastatic lesion segmentation on whole body PET/CT.Approach.59 whole body68Ga-DOTATATE PET/CT images of patients undergoing theranostic treatment of metastatic neuroendocrine tumors were used in this work. A 3D U-Net was trained for lesion segmentation following five-fold cross validation. Uncertainty measures derived from four UQ methods-probability entropy, Monte Carlo dropout, deep ensembles, and test time augmentation-were investigated. Each uncertainty measure was assessed across four quantitative evaluations: (1) its ability to detect artificially degraded image data at low, medium, and high degradation magnitudes; (2) to detect false-positive (FP) predicted regions; (3) to recover false-negative (FN) predicted regions; and (4) to establish correlations with model biomarker extraction and segmentation performance metrics.Mainresults.Test time augmentation and probability entropy respectively achieved the highest and lowest degraded image detection at low (AUC = 0.54 vs. 0.68), medium (AUC = 0.70 vs. 0.82), and high (AUC = 0.83 vs. 0.90) degradation magnitudes. For detecting FPs, all UQ methods achieve strong performance, with AUC values ranging narrowly between 0.77 and 0.81. FN region recovery performance was strongest for test time augmentation and weakest for probability entropy. Performance for the correlation analysis was mixed, where the strongest performance was achieved by test time augmentation for SUVtotalcapture (ρ= 0.57) and segmentation Dice coefficient (ρ= 0.72), by Monte Carlo dropout for SUVmeancapture (ρ= 0.35), and by probability entropy for segmentation cross entropy (ρ= 0.96).Significance.Overall, test time augmentation demonstrated superior UQ performance and is recommended for use in metastatic lesion segmentation task. It also offers the advantage of being post hoc and computationally efficient. In contrast, probability entropy performed the worst, highlighting the need for advanced UQ approaches for this task.

PurposeThis paper aims to provide a comprehensive review of uncertainty quantification methods supported by evidence-based comparison studies. Uncertainties are widely encountered in engineering practice, arising from such diverse sources as heterogeneity of materials, variability in measurement, lack of data and ambiguity in knowledge. Academia and industries have long been researching for uncertainty quantification (UQ) methods to quantitatively account for the effects of various input uncertainties on the system response. Despite the rich literature of relevant research, UQ is not an easy subject for novice researchers/practitioners, where many different methods and techniques coexist with inconsistent input/output requirements and analysis schemes.Design/methodology/approachThis confusing status significantly hampers the research progress and practical application of UQ methods in engineering. In the context of engineering analysis, the research efforts of UQ are most focused in two largely separate research fields: structural reliability analysis (SRA) and stochastic finite element method (SFEM). This paper provides a state-of-the-art review of SRA and SFEM, covering both technology and application aspects. Moreover, unlike standard survey papers that focus primarily on description and explanation, a thorough and rigorous comparative study is performed to test all UQ methods reviewed in the paper on a common set of reprehensive examples.FindingsOver 20 uncertainty quantification methods in the fields of structural reliability analysis and stochastic finite element methods are reviewed and rigorously tested on carefully designed numerical examples. They include FORM/SORM, importance sampling, subset simulation, response surface method, surrogate methods, polynomial chaos expansion, perturbation method, stochastic collocation method, etc. The review and comparison tests comment and conclude not only on accuracy and efficiency of each method but also their applicability in different types of uncertainty propagation problems.Originality/valueThe research fields of structural reliability analysis and stochastic finite element methods have largely been developed separately, although both tackle uncertainty quantification in engineering problems. For the first time, all major uncertainty quantification methods in both fields are reviewed and rigorously tested on a common set of examples. Critical opinions and concluding remarks are drawn from the rigorous comparative study, providing objective evidence-based information for further research and practical applications.