Marginal Coverage Research Articles

Conformal prediction with local weights: randomization enables robust guarantees

Abstract In this work, we consider the problem of building distribution-free prediction intervals with finite-sample conditional coverage guarantees. Conformal prediction (CP) is an increasingly popular framework for building such intervals with distribution-free guarantees, but these guarantees only ensure marginal coverage: the probability of coverage is averaged over both the training and test data, meaning that there might be substantial undercoverage within certain subpopulations. Instead, ideally we would want to have local coverage guarantees that hold for each possible value of the test point’s features. While the impossibility of achieving pointwise local coverage is well established in the literature, many variants of conformal prediction algorithm show favourable local coverage properties empirically. Relaxing the definition of local coverage can allow for a theoretical understanding of this empirical phenomenon. We propose randomly localized conformal prediction (RLCP), a method that builds on localized CP and weighted CP techniques to return prediction intervals that are not only marginally valid but also offer relaxed local coverage guarantees and validity under covariate shift. Through a series of simulations and real data experiments, we validate these coverage guarantees of RLCP while comparing it with the other local conformal prediction methods.

Uncertainty-Aware Deep Learning Characterization of Knee Radiographs for Large-Scale Registry Creation

We present an automated image ingestion pipeline for a knee radiography registry, integrating a multilabel image-semantic classifier with conformal prediction-based uncertainty quantification and an object detection model for knee hardware. Annotators retrospectively classified 26,000 knee images detailing presence, laterality, prostheses, and radiographic views. They further annotated surgical construct locations in 11,841 knee radiographs. An uncertainty-aware multilabel EfficientNet-based classifier was trained to identify the knee laterality, implants, and radiographic view. A classifier trained with embeddings from the EfficientNet model detected out-of-domain images. An object detection model was trained to identify 20 different knee implants. Model performance was assessed against a held-out internal and an external dataset using per-class F1 score, accuracy, sensitivity, and specificity. Conformal prediction was evaluated with marginal coverage and efficiency. Classification Model with Conformal Prediction: F1 scores for each label output > 0.98. Coverage of each label output was >0.99 and the average efficiency was 0.97. The F1 score was 0.99, with precision and recall for knee radiographs of 0.99. Mean average precision across all classes was 0.945 and ranged from 0.695 to 1.000. Average precision and recall across all classes were 0.950 and 0.886. We present a multilabel classifier with domain detection and an object detection model to characterize knee radiographs. Conformal prediction enhances transparency in cases when the model is uncertain.

Conformal Multi‐Target Hyperrectangles

ABSTRACTWe propose conformal hyperrectangular prediction regions for multi‐target regression. We propose split conformal prediction algorithms for both point and quantile regression to form hyperrectangular prediction regions, which allow for easy marginal interpretation and do not require covariance estimation. In practice, it is preferable that a prediction region is balanced, that is, having identical marginal prediction coverage, since prediction accuracy is generally equally important across components of the response vector. The proposed algorithms possess two desirable properties, namely, tight asymptotic overall nominal coverage as well as asymptotic balance, that is, identical asymptotic marginal coverage, under mild conditions. We then compare our methods to some existing methods on both simulated and real data sets. Our simulation results and real data analysis show that our methods outperform existing methods while achieving the desired nominal coverage and good balance between dimensions.

Conformal Autoregressive Generation: Beam Search with Coverage Guarantees

We introduce two new extensions to the beam search algorithm based on conformal predictions (CP) to produce sets of sequences with theoretical coverage guarantees. The first method is very simple and proposes dynamically-sized subsets of beam search results but, unlike typical CP proceedures, has an upper bound on the achievable guarantee depending on a post-hoc calibration measure. Our second algorithm introduces the conformal set prediction procedure as part of the decoding process, producing a variable beam width which adapts to the current uncertainty. While more complex, this procedure can achieve coverage guarantees selected a priori. We provide marginal coverage bounds as well as calibration-conditional guarantees for each method, and evaluate them empirically on a selection of tasks drawing from natural language processing and chemistry.

Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data

Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. In this paper, we study the confidence set prediction problem in the OOD generalization setting. Split conformal prediction (SCP) is an efficient framework for handling the confidence set prediction problem. However, the validity of SCP requires the examples to be exchangeable, which is violated in the OOD setting. Empirically, we show that trivially applying SCP results in a failure to maintain the marginal coverage when the unseen target domain is different from the source domain. To address this issue, we develop a method for forming confident prediction sets in the OOD setting and theoretically prove the validity of our method. Finally, we conduct experiments on simulated data to empirically verify the correctness of our theory and the validity of our proposed method.

How to evaluate uncertainty estimates in machine learning for regression?

As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Conformalized survival analysis with adaptive cut-offs

Summary This paper introduces an assumption-lean method that constructs valid and efficient lower predictive bounds for survival times with censored data. We build on recent work by Candès et al. (2023), whose approach first subsets the data to discard any data points with early censoring times and then uses a reweighting technique, namely, weighted conformal inference (Tibshirani et al., 2019), to correct for the distribution shift introduced by this subsetting procedure. For our new method, instead of constraining to a fixed threshold for the censoring time when subsetting the data, we allow for a covariate-dependent and data-adaptive subsetting step, which is better able to capture the heterogeneity of the censoring mechanism. As a result, our method can lead to lower predictive bounds that are less conservative and give more accurate information. We show that in the Type-I right-censoring setting, if either the censoring mechanism or the conditional quantile of the survival time is well estimated, our proposed procedure achieves nearly exact marginal coverage, where in the latter case we additionally have approximate conditional coverage. We evaluate the validity and efficiency of our proposed algorithm in numerical experiments, illustrating its advantage when compared with other competing methods. Finally, our method is applied to a real dataset to generate lower predictive bounds for users’ active times on a mobile app.

Conformal Prediction for Time Series.

We present a general framework for constructing distribution-free prediction intervals for time series. We establish explicit bounds on the conditional and marginal coverage gaps of estimated prediction intervals, which asymptotically converge to zero under additional assumptions. We also provide similar bounds on the size of set differences between oracle and estimated prediction intervals. To implement this framework, we introduce an efficient algorithm called EnbPI, which utilizes ensemble predictors and is closely related to conformal prediction (CP) but does not require data exchangeability. Unlike other methods, EnbPI avoids data-splitting and is computationally efficient by avoiding retraining, making it scalable for sequentially producing prediction intervals. Extensive simulation and real-data analyses demonstrate the effectiveness of EnbPI compared to existing methods.

Evaluation of a potential umbrella species using favourability models: the case of the endangered little bustard (Tetrax tetrax) and steppe birds

Farmland biodiversity is in alarming decline worldwide due to agriculture intensification. In this context, the umbrella species concept may help in better targeting conservation efforts, focusing on species whose requirements may best cover those of other components of biological communities. We test this idea using the little bustard (Tetrax tetrax), a strongly declining steppe bird depending on extensive agricultural landscapes of south-western Europe, to explore the degree to which its habitat requirements can predict those of other sympatric endangered steppe birds. We use little bustard and other nine species distribution data at 10 × 10 km scale in Castilla-La Mancha (the most important region for the little bustard in Spain and the EU) and habitat favourability models to identify variables explaining little bustard favourability that can robustly predict habitat favourability also for the other nine species. Models fitted with variables explaining little bustard favourability and applied on co-occurring species yielded varying performance results. Models support the role of the little bustard as umbrella species only for a part of the steppe bird community, and more precisely, for species linked to cereal and grassland-dominated landscapes, but not for landscape generalist species, distributed over mosaic landscapes including shrublands and woody crops. Results also highlight the importance of favourable extensive cereal steppes for the conservation of strongly endangered species (little and great bustard, Montagu’s harrier, pin-tailed sandgrouse, calandra lark), some of which are largely understudied (pin-tailed sandgrouse and calandra lark), despite their marginal coverage by the Natura 2000 protected area network.

Sensitivity analysis of individual treatment effects: A robust conformal inference approach

We propose a model-free framework for sensitivity analysis of individual treatment effects (ITEs), building upon ideas from conformal inference. For any unit, our procedure reports the Γ-value, a number which quantifies the minimum strength of confounding needed to explain away the evidence for ITE. Our approach rests on the reliable predictive inference of counterfactuals and ITEs in situations where the training data are confounded. Under the marginal sensitivity model of [Z. Tan, J. Am. Stat. Assoc. 101, 1619-1637 (2006)], we characterize the shift between the distribution of the observations and that of the counterfactuals. We first develop a general method for predictive inference of test samples from a shifted distribution; we then leverage this to construct covariate-dependent prediction sets for counterfactuals. No matter the value of the shift, these prediction sets (resp. approximately) achieve marginal coverage if the propensity score is known exactly (resp. estimated). We describe a distinct procedure also attaining coverage, however, conditional on the training data. In the latter case, we prove a sharpness result showing that for certain classes of prediction problems, the prediction intervals cannot possibly be tightened. We verify the validity and performance of the methods via simulation studies and apply them to analyze real datasets.

Conformalized survival analysis

AbstractIn this paper, we develop an inferential method based on conformal prediction, which can wrap around any survival prediction algorithm to produce calibrated, covariate-dependent lower predictive bounds on survival times. In the Type I right-censoring setting, when the censoring times are completely exogenous, the lower predictive bounds have guaranteed coverage in finite samples without any assumptions other than that of operating on independent and identically distributed data points. Under a more general conditionally independent censoring assumption, the bounds satisfy a doubly robust property which states the following: marginal coverage is approximately guaranteed if either the censoring mechanism or the conditional survival function is estimated well. The validity and efficiency of our procedure are demonstrated on synthetic data and real COVID-19 data from the UK Biobank.

Localized conformal prediction: a generalized inference framework for conformal prediction

Summary We propose a new inference framework called localized conformal prediction. It generalizes the framework of conformal prediction by offering a single-test-sample adaptive construction that emphasizes a local region around this test sample, and can be combined with different conformal scores. The proposed framework enjoys an assumption-free finite sample marginal coverage guarantee, and it also offers additional local coverage guarantees under suitable assumptions. We demonstrate how to change from conformal prediction to localized conformal prediction using several conformal scores, and we illustrate a potential gain via numerical examples.

Selective prediction-set models with coverage rate guarantees.

The current approach to using machine learning (ML) algorithms in healthcare is to either require clinician oversight for every use case or use their predictions without any human oversight. We explore a middle ground that lets ML algorithms abstain from making a prediction to simultaneously improve their reliability and reduce the burden placed on human experts. To this end, we present a general penalized loss minimization framework for training selective prediction-set (SPS) models, which choose to either output a prediction set or abstain. The resulting models abstain when the outcome is difficult to predict accurately, such as on subjects who are too different from the training data, and achieve higher accuracy on those they do give predictions for. We then introduce a model-agnostic, statistical inference procedure for the coverage rate of an SPS model that ensembles individual models trained using K-fold cross-validation. We find that SPS ensembles attain prediction-set coverage rates closer to the nominal level and have narrower confidence intervals for its marginal coverage rate. We apply our method to train neural networks that abstain more for out-of-sample images on the MNIST digit prediction task and achieve higher predictive accuracy for ICU patients compared to existingapproaches.

Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures.

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model’s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications.

Validating Modern Pelvic Nodal and Prostate Bed Contouring Guidelines for Post-Prostatectomy Salvage Radiation: A Secondary Analysis of the LOCATE Trial

<h3>Purpose/Objective(s)</h3> We used the patterns of recurrence on ¹⁸F-fluciclovine PET/CT in post-prostatectomy patients enrolled in the LOCATE trial to evaluate how well the most recent NRG Oncology and Groupe Francophone de Radiothérapie Urologique (GFRU) contouring recommendations encompassed all sites of recurrence in the prostate fossa and pelvic nodes in comparison to former Radiation Therapy Oncology Group (RTOG) recommendations. <h3>Materials/Methods</h3> Patients with biochemically recurrent prostate cancer after radical prostatectomy with a positive finding within the prostate fossa or pelvic nodes on ¹⁸F-fluciclovine PET/CTs were identified from the LOCATE patient population. Areas of gross disease were delineated. Prostate fossa contours were delineated using both the 2010 RTOG consensus guidelines, as well as the recently published 2020 GFRU consensus guidelines. Pelvic nodes were contoured with both the 2009 RTOG consensus guidelines and the 2020 NRG consensus guidelines. The performance of the contouring guidelines was assessed by determining what proportion of gross recurrent lesions were encompassed completely or marginally. <h3>Results</h3> Of the 213 patients within the LOCATE trial, a total of 45 patients were eligible for analysis with positive ¹⁸F-fluciclovine PET findings. Of the 30 total prostate fossa recurrences, the 2010 RTOG contour covered 20 (67%), and missed or marginally covered 10 (33%). The 2020 GFRU contour covered 27 (90%) recurrences, and missed or marginally covered 3 (10%). Of the 43 total nodal recurrences, the 2009 RTOG pelvic nodal contour covered 29 nodes (67%), and missed or marginally covered 14 (32%). The 2020 NRG pelvic nodal contour covered 43 nodes (100%), with no misses or marginal coverage. <h3>Conclusion</h3> This secondary analysis of the LOCATE trial exemplifies the improved coverage of the latest prostate fossa contouring recommendations from the GFRU. Similarly, it also validates the updated 2020 NRG pelvic nodal contouring guidelines by demonstrating improved coverage of recurrent disease in this patient population.

Validating Modern NRG Oncology Pelvic Nodal and Groupe Francophone de Radiothérapie Urologique Prostate Bed Contouring Guidelines for Post-Prostatectomy Salvage Radiation: A Secondary Analysis of the LOCATE Trial

We used the patterns of recurrence on 18F-fluciclovine positron emission tomography (PET)-computed tomography (CT) in patients enrolled in the LOCATE trial after prostatectomy to evaluate how well the most recent NRG Oncology and Groupe Francophone de Radiothérapie Urologique (GFRU) contouring recommendations encompassed all sites of recurrence in the prostate fossa and pelvic nodes in comparison to former Radiation Therapy Oncology Group (RTOG) recommendations. Patients with biochemically recurrent prostate cancer after radical prostatectomy with a positive finding within the prostate fossa or pelvic nodes on 18F-fluciclovine PET/CTs were identified from the LOCATE patient population. Areas of gross disease were delineated. Prostate fossa contours were delineated using both the 2010 RTOG consensus guidelines and the recently published 2020 GFRU consensus guidelines. Pelvic nodes were contoured with both the 2009 RTOG consensus guidelines and the 2020 NRG consensus guidelines. The performance of the contouring guidelines was assessed by determining what proportion of gross recurrent lesions were encompassed completely or marginally. Of the 213 patients within the LOCATE trial, 45 patients were eligible for analysis with positive 18F-fluciclovine PET findings. Of the 30 total prostate fossa recurrences, the 2010 RTOG contour covered 20 (67%) and missed or marginally covered 10 (33%). The 2020 GFRU contour covered 27 recurrences (90%), and missed or marginally covered 3 (10%). Of the 43 total nodal recurrences, the 2009 RTOG pelvic nodal contour covered 29 nodes (67%), and missed or marginally covered 14 (32%). The 2020 NRG pelvic nodal contour covered 43 nodes (100%), with no misses or marginal coverage. This secondary analysis of the LOCATE trial exemplifies the improved coverage of the latest prostate fossa contouring recommendations from the GFRU. Similarly, it also validates the updated 2020 NRG pelvic nodal contouring guidelines by demonstrating improved coverage of recurrent disease in this patient population.

Community-based sanitation as a complementary strategy for the Jakarta Sewerage Development Project: What can we do better?

Indonesian wastewater sector management has been marked as having low coverage and slow in development, even though the system has been initiated since 1969 and the first masterplan for Jakarta Sewerage System and Sanitation Project was developed in 1977. In addition, two other master plans were developed in 1991 and 2012. The long history of planning to develop the citywide sewerage coverage resulted in only marginal coverage. As a complementary and short-term strategy, the Indonesian government also developed community-based sanitation to solve local sanitation problems such as RW Kumuh. In this paper, ideas will be proposed on how to improve the latter strategy and integrate it to the citywide sanitation.

The limits of distribution-free conditional predictive inference

Abstract We consider the problem of distribution-free predictive inference, with the goal of producing predictive coverage guarantees that hold conditionally rather than marginally. Existing methods such as conformal prediction offer marginal coverage guarantees, where predictive coverage holds on average over all possible test points, but this is not sufficient for many practical applications where we would like to know that our predictions are valid for a given individual, not merely on average over a population. On the other hand, exact conditional inference guarantees are known to be impossible without imposing assumptions on the underlying distribution. In this work, we aim to explore the space in between these two and examine what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.

Random Forest Prediction Intervals

Random forests are among the most popular machine learning techniques for prediction problems. When using random forests to predict a quantitative response, an important but often overlooked challenge is the determination of prediction intervals that will contain an unobserved response value with a specified probability. We propose new random forest prediction intervals that are based on the empirical distribution of out-of-bag prediction errors. These intervals can be obtained as a by-product of a single random forest. Under regularity conditions, we prove that the proposed intervals have asymptotically correct coverage rates. Simulation studies and analysis of 60 real datasets are used to compare the finite-sample properties of the proposed intervals with quantile regression forests and recently proposed split conformal intervals. The results indicate that intervals constructed with our proposed method tend to be narrower than those of competing methods while still maintaining marginal coverage rates approximately equal to nominal levels.

Deployment of a charging network based on a cost-benefit model and the CFSFDP algorithm

PurposeThis research aims to focus on the problem of optimizing the layout of charging stations.Design/methodology/approachThe authors estimate the total number of charging stations by the number of existing gas stations using clustering by fast search-and-find of density peaks (CFSFDP) algorithm.FindingsThis paper reveals the relationship between the distance and the population. Priority should be given to the short-distance and high-frequency travel charging needs in the city, followed by low frequency and travel, which can improve the use efficiency of the whole charging network.Originality/valueFirst, the total number of charging stations is estimated by the number of existing gas stations. This method can fully consider the growth potential of charging vehicles instead of the traditional cars. Besides, the cost-benefit method is proposed to explore the marginal coverage distance of the city’s charging network and can be used to reveal the relationship between the distance and the population.