Conformal Prediction Research Articles

Multitask Modeling with Confidence Using Matrix Factorization and Conformal Prediction.

Multitask prediction of bioactivities is often faced with challenges relating to the sparsity of data and imbalance between different labels. We propose class conditional (Mondrian) conformal predictors using underlying Macau models as a novel approach for large scale bioactivity prediction. This approach handles both high degrees of missing data and label imbalances while still producing high quality predictive models. When applied to ten assay end points from PubChem, the models generated valid models with an efficiency of 74.0-80.1% at the 80% confidence level with similar performance both for the minority and majority class. Also when deleting progressively larger portions of the available data (0-80%) the performance of the models remained robust with only minor deterioration (reduction in efficiency between 5 and 10%). Compared to using Macau without conformal prediction the method presented here significantly improves the performance on imbalanced data sets.

Discrimination of Different Species of Dendrobium with an Electronic Nose Using Aggregated Conformal Predictor.

A method using electronic nose to discriminate 10 different species of dendrobium, which is a kind of precious herb with medicinal application, was developed with high efficiency and low cost. A framework named aggregated conformal prediction was applied to make predictions with accuracy and reliability for E-nose detection. This method achieved a classification accuracy close to 80% with an average improvement of 6.2% when compared with the results obtained by using traditional inductive conformal prediction. It also provided reliability assessment to show more comprehensive information for each prediction. Meanwhile, two main indicators of conformal predictor, validity and efficiency, were also compared and discussed in this work. The result shows that the approach integrating electronic nose with aggregated conformal prediction to classify the species of dendrobium with reliability and validity is promising.

The molecular structure of 4-piperidinemethanol in gas, solutions, and solid state: spectral and theoretical investigations

FT-IR (4000–400 cm) spectra of gaseous, solid 4-piperidinemethanol, and its solutions in tetrachloromethane and dichloromethane and Raman spectrum (4000–100 cm) of solid compound have been recorded. Interpretation of spectra has been proposed on a base of joint consideration of experimental data and theoretical predictions of possible 4-piperidinemethanol conformations and its 14 clusters at the B3LYP/6-31+G** and BVP86/TZVP levels of theory. A joint analysis of experimental data with results of potential energy distribution for two conformers and the most stable of 4-piperidinemethanol allowed us to suggest that compound in gaseous state and diluted solutions existed as equilibrium mixture of the most stable 4eNe and the next by stability 4eNa conformations. Solid 4-piperidinemethanol possibly consists of 4eNe chains stabilized by the intermolecular OH…N bonds.

Inductive conformal predictor for convolutional neural networks: Applications to active learning for image classification

Conformal prediction uses the degree of strangeness (nonconformity) of data instances to determine the confidence values of new predictions. We propose an inductive conformal predictor for convolutional neural networks (CNNs), referring to it as ICP-CNN, which uses a novel nonconformity measure that produces reliable confidence values. Furthermore, ICP-CNN is used to improve classification performance through active learning, selecting instances from an unlabeled pool based on the evaluation of three criteria: informativeness, diversity, and information density. Distance metric learning is employed to measure diversity, using a similarity measure that adapts to the database being used. Moreover, information density is considered to filter outliers. Experiments conducted on face and object recognition databases demonstrate that ICP-CNN improves the classification performance of CNNs, outperforming previously proposed active learning techniques, while producing reliable confidence values.

Large scale comparison of QSAR and conformal prediction methods and their applications in drug discovery

Structure–activity relationship modelling is frequently used in the early stage of drug discovery to assess the activity of a compound on one or several targets, and can also be used to assess the interaction of compounds with liability targets. QSAR models have been used for these and related applications over many years, with good success. Conformal prediction is a relatively new QSAR approach that provides information on the certainty of a prediction, and so helps in decision-making. However, it is not always clear how best to make use of this additional information. In this article, we describe a case study that directly compares conformal prediction with traditional QSAR methods for large-scale predictions of target-ligand binding. The ChEMBL database was used to extract a data set comprising data from 550 human protein targets with different bioactivity profiles. For each target, a QSAR model and a conformal predictor were trained and their results compared. The models were then evaluated on new data published since the original models were built to simulate a “real world” application. The comparative study highlights the similarities between the two techniques but also some differences that it is important to bear in mind when the methods are used in practical drug discovery applications.

AdaBoost-based conformal prediction with high efficiency

Conformal prediction presents a novel idea whose error rate is provably controlled by given significant levels. So the remaining goal of conformal prediction is its efficiency. High efficiency means that the predictions are as certain as possible. As we know, ensemble methods are able to obtain a better predictive performance than that obtained from any of the constituent models. Ensemble method such as random forest has been used as underlying method to build conformal predictor. But we do not know the differences of conformal predictors with and without ensemble methods, and how the corresponding performances are improved. In this paper, an ensemble method AdaBoost is used to build conformal predictor, and we introduce another evaluation metric-correct efficiency, which measures the efficiency of correct classification correctly. The good performance of AdaBoost-based conformal predictor (CP-AB) has been validated on seven datasets. The experimental results show that the proposed method has a much higher efficiency.

AdaBoost-based conformal prediction with high efficiency

Conformal prediction presents a novel idea whose error rate is provably controlled by given significant levels. So the remaining goal of conformal prediction is its efficiency. High efficiency means that the predictions are as certain as possible. As we know, ensemble methods are able to obtain a better predictive performance than that obtained from any of the constituent models. Ensemble method such as random forest has been used as underlying method to build conformal predictor. But we do not know the differences of conformal predictors with and without ensemble methods, and how the corresponding performances are improved. In this paper, an ensemble method AdaBoost is used to build conformal predictor, and we introduce another evaluation metric-correct efficiency, which measures the efficiency of correct classification correctly. The good performance of AdaBoost-based conformal predictor (CP-AB) has been validated on seven datasets. The experimental results show that the proposed method has a much higher efficiency.

Valid Model-Free Prediction of Future Insurance Claims

Bias resulting from model misspecification is a concern when predicting insurance claims. Indeed, this bias puts the insurer at risk of making invalid or unreliable predictions. A method that could provide provably valid predictions uniformly across a large class of possible distributions would effectively eliminate the risk of model misspecification bias. Conformal prediction is one such method that can meet this need, and here we tailor that approach to the typical insurance application and show that the predictions are not only valid but also efficient across a wide range of settings.

SHNN-CAD⁺: An Improvement on SHNN-CAD for Adaptive Online Trajectory Anomaly Detection.

To perform anomaly detection for trajectory data, we study the Sequential Hausdorff Nearest-Neighbor Conformal Anomaly Detector (SHNN-CAD) approach, and propose an enhanced version called SHNN-CAD. SHNN-CAD was introduced based on the theory of conformal prediction dealing with the problem of online detection. Unlike most related approaches requiring several not intuitive parameters, SHNN-CAD has the advantage of being parameter-light which enables the easy reproduction of experiments. We propose to adaptively determine the anomaly threshold during the online detection procedure instead of predefining it without any prior knowledge, which makes the algorithm more usable in practical applications. We present a modified Hausdorff distance measure that takes into account the direction difference and also reduces the computational complexity. In addition, the anomaly detection is more flexible and accurate via a re-do strategy. Extensive experiments on both real-world and synthetic data show that SHNN-CAD outperforms SHNN-CAD with regard to accuracy and running time.

Predicting Ames Mutagenicity Using Conformal Prediction in the Ames/QSAR International Challenge Project.

Valid and predictive models for classifying Ames mutagenicity have been developed using conformal prediction. The models are Random Forest models using signature molecular descriptors. The investigation indicates, on excluding not-strongly mutagenic compounds (class B), that the validity for mutagenic compounds is increased for the predictions based on both public and the Division of Genetics and Mutagenesis, National Institute of Health Sciences of Japan (DGM/NIHS) data while less so when using only the latter data source. The former models only result in valid predictions for the majority, non-mutagenic, class whereas the latter models are valid for both classes, i.e. mutagenic and non-mutagenic compounds. These results demonstrate the importance of data consistency manifested through the superior predictive quality and validity of the models based only on DGM/NIHS generated data compared to a combination of this data with public data sources.

Modelling of a surface marine vehicle with kernel ridge regression confidence machine

This paper describes the use of Kernel Ridge Regression (KRR) and Kernel Ridge Regression Confidence Machine (KRRCM) for black box identification of a surface marine vehicle. Data for training and test have been obtained from several manoeuvres typically used for marine system identification. Thus, a 20/20 degrees Zig-Zag, a 10/10 degrees Zig-Zag, and different evolution circles have been employed for the computation and validation of the model. Results show that the application of conformal prediction provides an accurate model that reproduces with large accuracy the actual behaviour of the ship with confidence margins that ensure that the model response is within these margins, making it a suitable tool for system identification.

A Mass‐Spectrometry‐Based Modelling Workflow for Accurate Prediction of IgG Antibody Conformations in the Gas Phase

AbstractImmunoglobulins are biomolecules involved in defence against foreign substances. Flexibility is key to their functional properties in relation to antigen binding and receptor interactions. We have developed an integrative strategy combining ion mobility mass spectrometry (IM‐MS) with molecular modelling to study the conformational dynamics of human IgG antibodies. Predictive models of all four human IgG subclasses were assembled and their dynamics sampled in the transition from extended to collapsed state during IM‐MS. Our data imply that this collapse of IgG antibodies is related to their intrinsic structural features, including Fab arm flexibility, collapse towards the Fc region, and the length of their hinge regions. The workflow presented here provides an accurate structural representation in good agreement with the observed collision cross section for these flexible IgG molecules. These results have implications for studying other nonglobular flexible proteins.

A Mass-Spectrometry-Based Modelling Workflow for Accurate Prediction of IgG Antibody Conformations in the Gas Phase.

Immunoglobulins are biomolecules involved in defence against foreign substances. Flexibility is key to their functional properties in relation to antigen binding and receptor interactions. We have developed an integrative strategy combining ion mobility mass spectrometry (IM‐MS) with molecular modelling to study the conformational dynamics of human IgG antibodies. Predictive models of all four human IgG subclasses were assembled and their dynamics sampled in the transition from extended to collapsed state during IM‐MS. Our data imply that this collapse of IgG antibodies is related to their intrinsic structural features, including Fab arm flexibility, collapse towards the Fc region, and the length of their hinge regions. The workflow presented here provides an accurate structural representation in good agreement with the observed collision cross section for these flexible IgG molecules. These results have implications for studying other nonglobular flexible proteins.

Predicting Off-Target Binding Profiles With Confidence Using Conformal Prediction.

Ligand-based models can be used in drug discovery to obtain an early indication of potential off-target interactions that could be linked to adverse effects. Another application is to combine such models into a panel, allowing to compare and search for compounds with similar profiles. Most contemporary methods and implementations however lack valid measures of confidence in their predictions, and only provide point predictions. We here describe a methodology that uses Conformal Prediction for predicting off-target interactions, with models trained on data from 31 targets in the ExCAPE-DB dataset selected for their utility in broad early hazard assessment. Chemicals were represented by the signature molecular descriptor and support vector machines were used as the underlying machine learning method. By using conformal prediction, the results from predictions come in the form of confidence p-values for each class. The full pre-processing and model training process is openly available as scientific workflows on GitHub, rendering it fully reproducible. We illustrate the usefulness of the developed methodology on a set of compounds extracted from DrugBank. The resulting models are published online and are available via a graphical web interface and an OpenAPI interface for programmatic access.

Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Prediction Errors for Deep Neural Networks.

Deep learning architectures have proved versatile in a number of drug discovery applications, including the modeling of in vitro compound activity. While controlling for prediction confidence is essential to increase the trust, interpretability, and usefulness of virtual screening models in drug discovery, techniques to estimate the reliability of the predictions generated with deep learning networks remain largely underexplored. Here, we present Deep Confidence, a framework to compute valid and efficient confidence intervals for individual predictions using the deep learning technique Snapshot Ensembling and conformal prediction. Specifically, Deep Confidence generates an ensemble of deep neural networks by recording the network parameters throughout the local minima visited during the optimization phase of a single neural network. This approach serves to derive a set of base learners (i.e., snapshots) with comparable predictive power on average that will however generate slightly different predictions for a given instance. The variability across base learners and the validation residuals are in turn harnessed to compute confidence intervals using the conformal prediction framework. Using a set of 24 diverse IC50 data sets from ChEMBL 23, we show that Snapshot Ensembles perform on par with Random Forest (RF) and ensembles of independently trained deep neural networks. In addition, we find that the confidence regions predicted using the Deep Confidence framework span a narrower set of values. Overall, Deep Confidence represents a highly versatile error prediction framework that can be applied to any deep learning-based application at no extra computational cost.

HCF-CRS: A Hybrid Content based Fuzzy Conformal Recommender System for providing recommendations with confidence.

A Recommender System (RS) is an intelligent system that assists users in finding the items of their interest (e.g. books, movies, music) by preventing them to go through huge piles of data available online. In an effort to overcome the data sparsity issue in recommender systems, this research incorporates a content based filtering technique with fuzzy inference system and a conformal prediction approach introducing a new framework called Hybrid Content based Fuzzy Conformal Recommender System (HCF-CRS). The proposed framework is implemented to be used in the domain of movies and it provides quality recommendations to users with a confidence level and an improved accuracy. In our proposed framework, first, a Content Based Filtering (CBF) technique is applied to create a user profile by considering the history of each user. CBF is useful in the situations like: lack of demographic information and the data sparsity problems. Second, a Fuzzy based technique is incorporated to find the similarities and differences between the user profile and the movies in the dataset using a set of fuzzy rules to get a predicted rating for each movie. Third, a Conformal prediction algorithm is implemented to calculate the non-conformity measure between the predicted ratings produced by fuzzy system and the actual ratings from the dataset. A p-value (confidence measure) is computed to give a level of confidence to each recommended item and a bound is set on the confidence level called a significance level ε, according to which the movies only above the specified significance level are recommended to user. By building a confidence centric hybrid conformal recommender system using the content based filtering approach with fuzzy logic and conformal prediction algorithm, the reliability and the accuracy of the system is considerably enhanced. The experiments are evaluated on MovieLens and Movie Tweetings datasets for recommending movies to the users and they are compared with other state-of-the-art recommender systems. Finally, the results confirm that the proposed algorithms perform better than the traditional ones.

Unintended specificity of an engineered ligand-binding protein facilitated by unpredicted plasticity of the protein fold.

Attempts to create novel ligand-binding proteins often focus on formation of a binding pocket with shape complementarity against the desired ligand (particularly for compounds that lack distinct polar moieties). Although designed proteins often exhibit binding of the desired ligand, in some cases they display unintended recognition behavior. One such designed protein, that was originally intended to bind tetrahydrocannabinol (THC), was found instead to display binding of 25-hydroxy-cholecalciferol (25-D3) and was subjected to biochemical characterization, further selections for enhanced 25-D3 binding affinity and crystallographic analyses. The deviation in specificity is due in part to unexpected altertion of its conformation, corresponding to a significant change of the orientation of an α-helix and an equally large movement of a loop, both of which flank the designed ligand-binding pocket. Those changes led to engineered protein constructs that exhibit significantly more contacts and complementarity towards the 25-D3 ligand than the initial designed protein had been predicted to form towards its intended THC ligand. Molecular dynamics simulations imply that the initial computationally designed mutations may contribute to the movement of the helix. These analyses collectively indicate that accurate prediction and control of backbone dynamics conformation, through a combination of improved conformational sampling and/or de novo structure design, represents a key area of further development for the design and optimization of engineered ligand-binding proteins.

Coevolutionary Signals and Structure-Based Models for the Prediction of Protein Native Conformations.

The analysis of coevolutionary signals from families of evolutionarily related sequences is a recent conceptual framework that provides valuable information about unique intramolecular interactions and, therefore, can assist in the elucidation of biomolecular conformations. It is based on the idea that compensatory mutations at specific residue positions in a sequence help preserve stability of protein architecture and function and leave a statistical signature related to residue-residue interactions in the 3D structure of the protein. Consequently, statistical analysis of these correlated mutations in subsets of protein sequence alignments can be used to predict which residue pairs should be in spatial proximity in the native functional protein fold. These predicted signals can be then used to guide molecular dynamics (MD) simulations to predict the three-dimensional coordinates of a functional amino acid chain. In this chapter, we introduce a general and efficient methodology to perform coevolutionary analysis on protein sequences and to use this information in combination with computational physical models to predict the native 3D conformation of functional polypeptides. We present a step-by-step methodology that includes the description and application of software tools and databases required to infer tertiary structures of a protein fold. The general pipeline includes instructions on (1) how to obtain direct amino acid couplings from protein sequences using direct coupling analysis (DCA), (2) how to incorporate such signals as interaction potentials in Cα structure-based models (SBMs) to drive protein-folding MD simulations, (3) a procedure to estimate secondary structure and how to include such estimates in the topology files required in the MD simulations, and (4) how to build full atomic models based on the top Cα candidates selected in the pipeline. The information presented in this chapter is self-contained and sufficient to allow a computational scientist to predict structures of proteins using publicly available algorithms and databases.

Conformational Sampling of Macrocyclic Drugs in Different Environments: Can We Find the Relevant Conformations?

Conformational flexibility is a major determinant of the properties of macrocycles and other drugs in beyond rule of 5 (bRo5) space. Prediction of conformations is essential for design of drugs in this space, and we have evaluated three tools for conformational sampling of a set of 10 bRo5 drugs and clinical candidates in polar and apolar environments. The distance-geometry based OMEGA was found to yield ensembles spanning larger structure and property spaces than the ensembles obtained by MOE-LowModeMD (MOE) and MacroModel (MC). Both MC and OMEGA but not MOE generated different ensembles for polar and apolar environments. All three conformational search methods generated conformers similar to the crystal structure conformers for 9 of the 10 compounds, with OMEGA performing somewhat better than MOE and MC. MOE and OMEGA found all six conformers of roxithromycin that were identified by NMR in aqueous solutions, whereas only OMEGA sampled the three conformers observed in chloroform. We suggest that characterization of conformers using molecular descriptors, e.g., the radius of gyration and polar surface area, is preferred to energy- or root-mean-square deviation-based methods for selection of biologically relevant conformers in drug discovery in bRo5 space.

Predicting skin sensitizers with confidence – using conformal prediction to determine applicability domain of GARD

Lung diseases impose a significant socioeconomic burden and are a leading cause of morbidity and mortality worldwide. Moreover, respiratory medicine, unlike several other therapeutic areas, faces a disappointingly low number of new approved therapies. This is partly due to lack of reliable in vitro or in vivo models that can reproduce organ-level complexity and pathophysiological responses of human lung. Here, we examine new opportunities in application of recently emerged organ-on-chip technology to model human lung alveolus and small airway in preclinical drug development and biomarker discovery. We also discuss challenges that need to be addressed in coming years to further enhance the physiological and clinical relevance of these microsystems, enable their increased accessibility, and support their leap into personalized medicine.