Conformal Prediction Research Articles

A Novel Dual Layer Cascade Reliability Framework for an Informed and Intuitive Clinician-AI Interaction In Diagnosis of Colorectal Cancer Polyps.

We present a novel Cascade Reliability Framework (CRF) that integrates two independent cascade layers of reliability (i.e., variational temperature scaling and conformal prediction) with a pre-trained Machine Learning (ML) model in order to provide clinicians with a more reliable and tunable tool for early-stage diagnosis of Colorectal Cancer (CRC) polyps. The conformal prediction layer generates predictive sets that are guaranteed to contain the true polyp type with an adjustable error rate tuned by clinicians, while the confidence calibration generates meaningful confidence estimates for each predicted label. These two layers provide additional information and an error-tuning-ability for clinicians to assist them in making informed and intuitive decisions considering the outputs of the pre-trained ML model. Utilizing a novel vision-based tactile sensor and unique 3D-printed CRC polyp phantoms, we evaluated the trustworthiness of the proposed architecture and particularly dual outputs of four different types of CRF models, integrated with two different pre-trained ML models (i.e., ResNet18 and Dilated Residual Network) to highlight the model-agnostic feature of the architecture. To thoroughly assess the performance of the proposed approach, we used reliability diagrams and metrics such as accuracy, coverage, and average set size, while also addressing inter-class performance. Results demonstrate that the calibrated CRF models are well capable of handling non-ideal inputs with noise and blur. Moreover, using the conformal prediction with a user-defined error rate and various experiments, we show how clinicians can intuitively interact with a pre-trained ML model to make informed decisions and minimize the risk of CRC polyps misdiagnoses.

Ensemble Predictors: Possibilistic Combination of Conformal Predictors for Multivariate Time Series Classification.

In this article we propose a conceptual framework to study ensembles of conformal predictors (CP), that we call Ensemble Predictors (EP). Our approach is inspired by the application of imprecise probabilities in information fusion. Based on the proposed framework, we study, for the first time in the literature, the theoretical properties of CP ensembles in a general setting, by focusing on simple and commonly used possibilistic combination rules. We also illustrate the applicability of the proposed methods in the setting of multivariate time-series classification, showing that these methods provide better performance (in terms of both robustness, conservativeness, accuracy and running time) than both standard classification algorithms and other combination rules proposed in the literature, on a large set of benchmarks from the UCR time series archive.

A confidence machine for sparse high‐order interaction model

AbstractIn predictive modelling for high‐stake decision‐making, predictors must be not only accurate but also reliable. Conformal prediction (CP) is a promising approach for obtaining the coverage of prediction results with fewer theoretical assumptions. To obtain the prediction set by so‐called full‐CP, we need to refit the predictor for all possible values of prediction results, which is only possible for simple predictors. For complex predictors such as random forests (RFs) or neural networks (NNs), split‐CP is often employed where the data is split into two parts: one part for fitting and another for computing the prediction set. Unfortunately, because of the reduced sample size, split‐CP is inferior to full‐CP both in fitting as well as prediction set computation. In this paper, we develop a full‐CP of sparse high‐order interaction model (SHIM), which is sufficiently flexible as it can take into account high‐order interactions among variables. We resolve the computational challenge for full‐CP of SHIM by introducing a novel approach called homotopy mining. Through numerical experiments, we demonstrate that SHIM is as accurate as complex predictors such as RF and NN and enjoys the superior statistical power of full‐CP.

Federated Inference With Reliable Uncertainty Quantification Over Wireless Channels via Conformal Prediction

Federated Inference With Reliable Uncertainty Quantification Over Wireless Channels via Conformal Prediction

A methodology to correctly assess the applicability domain of cell membrane permeability predictors for cyclic peptides

This work outlines peptide predictive model methodology with conformal prediction, focusing on extrapolation task. Calibrating on the unseen chemical space recovers efficiency and validity enabling reliable predictions without retraining the models.

Prediction of the 3D conformation of a small peptide vaccine targeting Aβ42 oligomers.

The original etiology of Alzheimer's disease (AD) is the deposition of amyloid-beta (Aβ) proteins, which starts from the aggregation of the Aβ oligomers. The optimal therapeutic strategy targeting Aβ oligomer aggregation is the development of AD vaccines. Despite the fact that positive progress has been made for experimental attempts at AD vaccines, the physicochemical and even structural properties of these AD vaccines remain unclear. In this study, through immunoinformatic and molecular dynamics (MD) simulations, we first designed and simulated an alternative of vaccine TAPAS and found that the structure of the alternative can reproduce the 3D conformation of TAPAS determined experimentally. Meanwhile, immunoinformatic methods were used to analyze the physicochemical properties of TAPAS, including immunogenicity, antigenicity, thermal stability, and solubility, which confirm well the efficacy and safety of the vaccine, and validate the scheme reliability of immunoinformatic and MD simulations in designing and simulating the TAPAS vaccine. Using the same scheme, we predicted the 3D conformation of the optimized ACI-24 peptide vaccine, an Aβ peptide with the first 15 residues of Aβ42 (Aβ1-15). The vaccine was verified once to be effective against both full-length Aβ1-42 and truncated Aβ4-42 aggregates, but an experimental 3D structure was absent. We have also explored the immune mechanism of the vaccine at the molecular level and found that the optimized ACI-24 and its analogues can block the growth of either full-length Aβ1-42 or truncated Aβ4-42 pentamer by contacting the hydrophobic residues within the N-terminus and β1 region on the contact surface of either pentamer. Additionally, residues (D1, D7, S8, H13, and Q15) were identified as the key residues of the vaccine to contact either of the two Aβ oligomers. This work provides a feasible implementation scheme of immunoinformatic and MD simulations for the development of AD small peptide vaccines, validating the power of the scheme as a parallel tool to the experimental approaches and injecting molecular-level information into the understanding and design of anti-AD vaccines.

DiffBindFR: an SE(3) equivariant network for flexible protein-ligand docking.

Molecular docking, a key technique in structure-based drug design, plays pivotal roles in protein-ligand interaction modeling, hit identification and optimization, in which accurate prediction of protein-ligand binding mode is essential. Conventional docking approaches perform well in redocking tasks with known protein binding pocket conformation in the complex state. However, in real-world docking scenario without knowing the protein binding conformation for a new ligand, accurately modeling the binding complex structure remains challenging as flexible docking is computationally expensive and inaccurate. Typical deep learning-based docking methods do not explicitly consider protein side chain conformations and fail to ensure the physical plausibility and detailed atomic interactions. In this study, we present DiffBindFR, a full-atom diffusion-based flexible docking model that operates over the product space of ligand overall movements and flexibility and pocket side chain torsion changes. We show that DiffBindFR has higher accuracy in producing native-like binding structures with physically plausible and detailed interactions than available docking methods. Furthermore, in the Apo and AlphaFold2 modeled structures, DiffBindFR demonstrates superior advantages in accurate ligand binding pose and protein binding conformation prediction, making it suitable for Apo and AlphaFold2 structure-based drug design. DiffBindFR provides a powerful flexible docking tool for modeling accurate protein-ligand binding structures.

Efficient Prediction Model of mRNA End-to-End Distance and Conformation: Three-Dimensional RNA Illustration Program (TRIP).

The secondary and tertiary structures of RNA play a vital role in the regulation of biological reactions. These structures have been experimentally studied through in vivo and in vitro analyses, and in silico models have become increasingly accurate in predicting them. Recent technologies have diversified RNA structure predictions, from the earliest thermodynamic and molecular dynamic-based RNA structure predictions to deep learning-based conformation predictions in the past decade. While most research on RNA structure prediction has focused on short non-coding RNAs, there has been limited research on predicting the conformation of longer mRNAs. Our study introduces a computer simulation model called the Three-dimensional RNA Illustration Program (TRIP). TRIP is based on single-chain models and angle restriction of each bead component from previously reported single-molecule fluorescence in situ hybridization (smFISH) experiments. TRIP is a fast and efficient application that only requires up to three inputs to acquire outputs. It can also provide a rough visualization of the 3D conformation of RNA, making it a valuable tool for predicting RNA end-to-end distance.

Adaptive Conformal Prediction Intervals Using Data-Dependent Weights with Application to Seismic Response Prediction

Adaptive Conformal Prediction Intervals Using Data-Dependent Weights with Application to Seismic Response Prediction

Inductive Conformal Prediction Enhanced LSTM-SNN Network: Applications to Birds and UAVs Recognition

Inductive Conformal Prediction Enhanced LSTM-SNN Network: Applications to Birds and UAVs Recognition

Bayesian Optimization With Formal Safety Guarantees via Online Conformal Prediction

Bayesian Optimization With Formal Safety Guarantees via Online Conformal Prediction

Quantum Conformal Prediction for Reliable Uncertainty Quantification in Quantum Machine Learning

Quantum Conformal Prediction for Reliable Uncertainty Quantification in Quantum Machine Learning

DATA ANALYTICS FOR HEALTH-RELEVANT EVENTS DETECTION BASED UPON LONGITUDINAL FITBIT HEART RATE DATA

Abstract The increasing prevalence of wearable devices enables low-cost, long-term collection of health relevant data such as heart rate, exercise, and sleep signals. Currently these data are used to monitor short term changes with limited interpretation of their relevance to health. These data provide an untapped resource to monitor daily and long-term activity patterns. Changes and trends identified from such data can provide insights and guidance to the management of many chronic conditions that change over time. In this study we conducted a machine learning based analysis of longitudinal heart rate data collected over multiple years from Fitbit devices. We built a multi-resolutional pipeline for time series analysis, using model-free clustering methods inspired by statistical conformal prediction framework. With this method, we were able to detect health relevant events, their interesting patterns (e.g., daily routines, seasonal differences, and anomalies), and correlations to acute and chronic changes in health conditions. We present the results, lessons, and insights learned, and how to address the challenge of lack of labels. The study confirms the value of long-term heart rate data for health monitoring and surveillance, as complementary to extensive yet intermittent examinations by health care providers.

Extreme Gradient Boosting Combined with Conformal Predictors for Informative Solubility Estimation.

We used the extreme gradient boosting (XGB) algorithm to predict the experimental solubility of chemical compounds in water and organic solvents and to select significant molecular descriptors. The accuracy of prediction of our forward stepwise top-importance XGB (FSTI-XGB) on curated solubility data sets in terms of RMSE was found to be 0.59-0.76 Log(S) for two water data sets, while for organic solvent data sets it was 0.69-0.79 Log(S) for the Methanol data set, 0.65-0.79 for the Ethanol data set, and 0.62-0.70 Log(S) for the Acetone data set. That was the first step. In the second step, we used uncurated and curated AquaSolDB data sets for applicability domain (AD) tests of Drugbank, PubChem, and COCONUT databases and determined that more than 95% of studied ca. 500,000 compounds were within the AD. In the third step, we applied conformal prediction to obtain narrow prediction intervals and we successfully validated them using test sets' true solubility values. With prediction intervals obtained in the last fourth step, we were able to estimate individual error margins and the accuracy class of the solubility prediction for molecules within the AD of three public databases. All that was possible without the knowledge of experimental database solubilities. We find these four steps novel because usually, solubility-related works only study the first step or the first two steps.

Sample-efficient safety assurances using conformal prediction

When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; that is, of the situations that are unsafe, fewer than ϵ will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an ϵ false negative rate using as few as 1/ϵ data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate the guaranteed false negative rate while also observing a low false detection (positive) rate.

Towards conformal automation in air traffic control: Learning conflict resolution strategies through behavior cloning

A critical factor in achieving conformity of automation tools in performing expert tasks, such as air traffic conflict resolution, is the identification of air traffic controllers’ (ATCOs’) preferences (conflict resolution strategies) and the automation tool’s ability to learn and recommend solutions that incorporate these preferences. We propose a machine learning-based framework to learn and predict ATCOs’ conflict resolution preferences through behavior cloning. This framework is an ensemble of five regressor and classifier models. The conflict resolution data to train the machine learning models was collected from 8 experienced enroute ATCOs. The prediction results demonstrate that the ATCOs’ strategies encoded in the data can be learned by the model with high accuracy for the classification tasks and with low mean absolute error (MAE) for the regression task (for instance, the classification accuracy of above 92.7% for predicting the maneuvering aircraft, MAE for maneuver initiation distance ¡ 5.3 NM, MAE for predicting the heading angle ¡ 5.3°) for the ATCOs’ datasets. A sensitivity analysis performed to test the model robustness demonstrates that the proposed models are robust to up to 7.5% added Gaussian noise (with a mean equal to the value of each feature and varying standard deviation) to the dataset. In addition, we discuss the extent of acceptance of these predictions by the ATCOs through an ATCO acceptance exercise involving two ATCOs who demonstrate different conflict resolution strategies. ATCO A selected the original strategy as one of the resolution preferences for 97% of the scenarios and the predicted strategy as one of the options for 78% of the scenarios. ATCO B selected the conflict resolution strategies depicting ATCO B’s original strategies for 68% of the scenarios. The results from the acceptance exercise demonstrate that the proposed machine learning model can generate ATCO conformal predictions. The presented results and discussions also demonstrate the viability of using behavior cloning with chained predictions to develop individual and group conformal automation assistance tools for ATCOs.

Classification Under Partial Reject Options

In many applications there is ambiguity about which (if any) of a finite number N of hypotheses that best fits an observation. It is of interest then to possibly output a whole set of categories, that is, a scenario where the size of the classified set of categories ranges from 0 to N. Empty sets correspond to an outlier, sets of size 1 represent a firm decision that singles out one hypothesis, sets of size N correspond to a rejection to classify, whereas sets of sizes 2,…,N-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$2,\\ldots ,N-1$$\\end{document} represent a partial rejection to classify, where some hypotheses are excluded from further analysis. In this paper, we review and unify several proposed methods of Bayesian set-valued classification, where the objective is to find the optimal Bayesian classifier that maximizes the expected reward. We study a large class of reward functions with rewards for sets that include the true category, whereas additive or multiplicative penalties are incurred for sets depending on their size. For models with one homogeneous block of hypotheses, we provide general expressions for the accompanying Bayesian classifier, several of which extend previous results in the literature. Then, we derive novel results for the more general setting when hypotheses are partitioned into blocks, where ambiguity within and between blocks are of different severity. We also discuss how well-known methods of classification, such as conformal prediction, indifference zones, and hierarchical classification, fit into our framework. Finally, set-valued classification is illustrated using an ornithological data set, with taxa partitioned into blocks and parameters estimated using MCMC. The associated reward function’s tuning parameters are chosen through cross-validation.

Predicting multiple conformations via sequence clustering and AlphaFold2.

AlphaFold2(ref.1) has revolutionized structural biology by accurately predicting single structures of proteins. However, a protein's biological function often depends on multiple conformational substates2, and disease-causing point mutations often cause population changes within these substates3,4. We demonstrate that clustering a multiple-sequence alignment by sequence similarity enables AlphaFold2 to sample alternative states of known metamorphic proteins with high confidence. Using this method, named AF-Cluster, we investigated the evolutionary distribution of predicted structures for the metamorphic protein KaiB5 and found that predictions of both conformations were distributed in clusters across the KaiB family. We used nuclear magnetic resonance spectroscopy to confirm an AF-Cluster prediction: a cyanobacteria KaiB variant is stabilized in the opposite state compared with the more widely studied variant. To test AF-Cluster's sensitivity to point mutations, we designed and experimentally verified a set of three mutations predicted to flip KaiB from Rhodobacter sphaeroides from the ground to the fold-switched state. Finally, screening for alternative states in protein families without known fold switching identified a putative alternative state for the oxidoreductase Mpt53 in Mycobacterium tuberculosis. Further development of such bioinformatic methods in tandem with experiments will probably have a considerable impact on predicting protein energy landscapes, essential for illuminating biological function.

Improving conformalized quantile regression through cluster-based feature relevance

Conformalized quantile regression, a cutting-edge and model-agnostic algorithm, has emerged as a recent innovation to generate valid prediction intervals on finite samples while addressing heteroscedasticity. It starts by employing quantile regression to estimate conditional quantiles. Subsequently, these estimated conditional quantiles undergo a rectification process using conformal prediction. Under the assumption of exchangeability, a slightly weaker form of independent and identically distributed (i.i.d.) data, the resulting prediction intervals are valid in finite samples. However, a drawback of the proposed conformalization step is identified: it lacks the capacity to adapt to heteroscedasticity due to its independence from the input. To overcome this limitation, we propose an improvement that involves partitioning the covariates space into clusters, assigning higher weights to features with greater predictive power. Following that, within each cluster, a conformal step is applied, leveraging a rectification that is reliant on the input cluster-wise.To demonstrate the superiority of our improved version over the classic version of conformalized quantile regression, we conducted a comprehensive comparison of their respective prediction intervals using synthetic data.

Automated benchmarking of combined protein structure and ligand conformation prediction.

The prediction of protein-ligand complexes (PLC), using both experimental and predicted structures, is an active and important area of research, underscored by the inclusion of the Protein-Ligand Interaction category in the latest round of the Critical Assessment of Protein Structure Prediction experiment CASP15. The prediction task in CASP15 consisted of predicting both the three-dimensional structure of the receptor protein as well as the position and conformation of the ligand. This paper addresses the challenges and proposed solutions for devising automated benchmarking techniques for PLC prediction. The reliability of experimentally solved PLC as ground truth reference structures is assessed using various validation criteria. Similarity of PLC to previously released complexes are employed to judge PLC diversity and the difficulty of a PLC as a prediction target. We show that the commonly used PDBBind time-split test-set is inappropriate for comprehensive PLC evaluation, with state-of-the-art tools showing conflicting results on a more representative and high quality dataset constructed for benchmarking purposes. We also show that redocking on crystal structures is a much simpler task than docking into predicted protein models, demonstrated by the two PLC-prediction-specific scoring metrics created. Finally, we introduce a fully automated pipeline that predicts PLC and evaluates the accuracy of the protein structure, ligand pose, and protein-ligand interactions.