Distribution-free Prediction Research Articles

Selection and Aggregation of Conformal Prediction Sets

Conformal prediction is a generic methodology for finite-sample valid distribution-free prediction. This technique has garnered a lot of attention in the literature partly because it can be applied with any machine learning algorithm that provides point predictions to yield valid prediction regions. Of course, the efficiency (width/volume) of the resulting prediction region depends on the performance of the machine learning algorithm. In the context of point prediction, several techniques (such as cross-validation) exist to select one of many machine learning algorithms for better performance. In contrast, such selection techniques are seldom discussed in the context of set prediction (or prediction regions). In this article, we consider the problem of obtaining the smallest conformal prediction region given a family of machine learning algorithms. We provide two general-purpose selection algorithms and consider coverage as well as width properties of the final prediction region. The first selection method yields the smallest width prediction region among the family of conformal prediction regions for all sample sizes but only has an approximate coverage guarantee. The second selection method has a finite sample coverage guarantee but only attains close to the smallest width. The approximate optimal width property of the second method is quantified via an oracle inequality. As an illustration, we consider the use of aggregation of nonparametric regression estimators in the split conformal method with the absolute residual conformal score. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Uncertainty Quantification of a Machine Learning Model for Identification of Isolated Nonlinearities with Conformal Prediction

Abstract Structural nonlinearities are often spatially localized, such joints and interfaces, localized damage, or isolated connections, in an otherwise linearly behaving system. Quinn and Brink [12] modeled this localized nonlinearity as a deviatoric force component. In other previous work [13], the authors proposed a physics-informed machine learning framework to determine the deviatoric force from measurements obtained only at the boundary of the nonlinear region, assuming a noise-free environment. However, in real experimental applications, the data are expected to contain noise from a variety of sources. In the present work, we explore the sensitivity of the trained network by comparing the network responses when trained on deterministic (“noise-free”) model data and model data with additive noise (“noisy”). As the neural network does not yield a closed-form transformation from the input distribution to the response distribution, we leverage the use of conformal sets to build an illustration of sensitivity. Through the conformal set assumption of exchangeability, we may build a distribution-free prediction interval for both network responses of the clean and noisy training sets. This work will explore the application of conformal sets for uncertainty quantification of a deterministic structure-preserving neural network and its deployment in a structural health monitoring framework to detect deviations from a baseline state based on noisy measurements.

Conformal Prediction for Time Series.

We present a general framework for constructing distribution-free prediction intervals for time series. We establish explicit bounds on the conditional and marginal coverage gaps of estimated prediction intervals, which asymptotically converge to zero under additional assumptions. We also provide similar bounds on the size of set differences between oracle and estimated prediction intervals. To implement this framework, we introduce an efficient algorithm called EnbPI, which utilizes ensemble predictors and is closely related to conformal prediction (CP) but does not require data exchangeability. Unlike other methods, EnbPI avoids data-splitting and is computationally efficient by avoiding retraining, making it scalable for sequentially producing prediction intervals. Extensive simulation and real-data analyses demonstrate the effectiveness of EnbPI compared to existing methods.

Distribution free prediction for geographically weighted functional regression models

Recently, there has been significant interest in distribution-free prediction within the fields of machine learning and statistics. Distribution-free prediction involves techniques that aim to make predictions or create prediction intervals without relying on explicit assumptions about the underlying distribution of the data. In this study, we introduce an inductive conformal prediction strategy specifically designed for spatio-functional data. We define a prediction with a conformity level for the response of two distinct regression models: a Geographically Weighted Functional Regression model and its heteroscedastic version. We propose two novel measures of non-conformity and prediction bands for the functional response variable. The properties of the resulting estimates are examined through simulation and real data analysis on air quality data.

Distribution-free prediction regions of multivariate response PLS models with applications to NIR datasets

Multi-response partial least squares regression (PLS2) is commonly used in analyzing high-dimensional multi-response data, particularly estimating prediction regions. However, calculating prediction regions by local linearization methods requires the assumption of normality to obtain high coverage level. It has been shown that methods under distribution-free assumption outperform local linearization methods. Therefore, in this work, several new methods are proposed to construct distribution-free prediction regions of PLS2 models. The estimated prediction regions do not rely on trivial assumptions and have relatively lower computational cost. Analyses of simulation and real NIR datasets show that the proposed methods have higher predictive coverage and is more computational effective than local linearization methods.

Distribution-Free Prediction Sets for Two-Layer Hierarchical Models

We consider the problem of constructing distribution-free prediction sets for data from two-layer hierarchical distributions. For iid data, prediction sets can be constructed using the method of conformal prediction. The validity of conformal prediction hinges on the exchangeability of the data, which does not hold when groups of observations come from distinct distributions, such as multiple observations on each patient in a medical database. We extend conformal methods to a hierarchical setting. We develop CDF pooling, single subsampling, and repeated subsampling approaches to construct prediction sets in unsupervised and supervised settings. We compare these approaches in terms of coverage and average set size. If asymptotic coverage is acceptable, we recommend CDF pooling for its balance between empirical coverage and average set size. If we desire coverage guarantees, then we recommend the repeated subsampling approach. Supplementary materials for this article are available online.

Ordered maximum ranked set sampling with unequal sample

In this paper, a new sampling method is introduced by ordering the maxima of ranked set samples with unequal sizes. In this new modification of ranked set sampling which is called OMRSSU, the pdf, cdf and joint pdf are obtained. Also, a new distribution-free prediction interval for future order statistics is derived. In addition, the moments of OMRSSU for exponential distribution are computed and then the best linear unbiased estimators of exponential distribution are obtained using OMRSSU. Finally, a comparison is made between BLUE-OMRSSU and the BLUE based on ranked set sampling, ordered ranked set sampling, maximum ranked set sampling and order statistics from a simple random sample.

Recursive Bayesian filtering in circular state spaces

For recursive circular filtering based on circular statistics, we introduce a general framework for estimation of a circular state based on different circular distributions, specifically the wrapped normal distribution and the von Mises distribution. We propose an estimation method for circular systems with nonlinear system and measurement functions. This is achieved by relying on efficient deterministic sampling techniques. Furthermore, we show how the calculations can be simplified in a variety of important special cases, such as systems with additive noise as well as identity system or measurement functions. We introduce several novel key components, particularly a distribution-free prediction algorithm, a new and superior formula for the multiplication of wrapped normal densities, and the ability to deal with non-additive system noise. All proposed methods are thoroughly evaluated and compared to several state-of-the-art solutions.

Prediction Intervals for Future Order Statistics from Two Independent Sequences

In this article, based on observed X-sequence of independent and identically distribution (iid) continuous random variables, we discuss the problem of predicting future order statistics from a Y-sequence of iid continuous random variables from the same distribution. Specifically, distribution-free prediction intervals (PIs) for an order statistic observation based on either progressive Type-II right censoring, or order data from the past X-sequence, as well as outer and inner PIs are derived based on order statistics observations. Such these intervals are exact and do not depend on the sampling distribution. Finally, a real life time data set that given to breakdown of an insulating fluid between electrodes is used to illustrate the proposed procedures.

Outer and inner prediction intervals for order statistics intervals based on current records

This paper considers the largest and smallest observations at the times when a new record of either kind (upper or lower) occurs. These are called the upper and lower current records and are denoted by \({R^l_m}\) and \({R^s_m}\), respectively. The interval \({(R^s_m,R^l_m)}\) is then referred to as the record coverage. The prediction problem in the two-sample case is then discussed and, specifically, the exact outer and inner prediction intervals are derived for order statistics intervals from an independent future Y-sample based on the m-th record coverage from the X-sequence when the underlying distribution of the two samples are the same. The coverage probabilities of these intervals are exact and do not depend on the underlying distribution. Distribution-free prediction intervals as well as upper and lower prediction limits for spacings from a future Y-sample are obtained in terms of the record range from the X-sequence.

Distribution-free prediction intervals for order statistics based on record coverage

In this paper, based on the largest and smallest observations at the times when a new record of either kind (upper or lower) occurs, we discuss the prediction of future order statistics. The proposed prediction intervals are distribution-free in that the corresponding coverage probabilities are known exactly without any assumption about the parent distribution other than that it being continuous. An exact expression for the prediction coefficient of these intervals is derived. Similarly, prediction intervals for future records based on observed order statistics are also obtained. Finally, two real-life data, one involving the average July temperatures in Neurenburg, Switzerland, and the other involving the amount of annual rainfall at the Los Angeles Civic Center, are used to illustrate the procedures developed here.

Prediction intervals for future records

Suppose upper records were observed from a X -sequence of iid continuous random variables, and that another independent Y -sequence of iid variables from the same distribution is to be observed. In this paper, we then derive various exact distribution-free prediction intervals for records from the Y -sequence based on the record values from the X -sequence. Specifically, distribution-free prediction intervals for individual records as well as outer and inner prediction intervals are derived based on X -records, and exact expressions for the coverage probabilities of these intervals are also derived. A data representing the records of temperatures is used to illustrate all the results developed here.

Distribution-free prediction intervals for the future current record statistics

Prediction of records plays an important role in many applications, such as, meteorology, hydrology, industrial stress testing and athletic events. In this paper, based on the observed current records of an iid sequence sample drawn from an arbitrary unknown distribution, we develop distribution-free prediction intervals as well as prediction upper and lower bounds for current records from another iid sequence. We also present sharp upper bounds for the expected lengths of the so obtained prediction intervals. Numerical computations of the coverage probabilities are presented for choosing the appropriate limits of the prediction intervals.

Distribution-Free and other Prediction Intervals

Abstract Saw, Yang, and Mo (1984) gave a distribution-free prediction interval for X based on X 1,…, Xn of the form [[Xbar] -A, [Xbar] + A] with A 2 ≈ λ2(1 + 1/n)S 2. As compared with the range [X (1), X (2)], which has length R(say) and size (minimum coverage probability) (n − 1)/(n + 1), their intervals can have size as high as n/(n +1), a value that is attained when λ2 = n +1. For n = 2, this interval (with λ2 = 3) becomes the “triple range” [X (1) - R, X (2)+ R] and has size 2/3; it coincides with the “normal interval” for n = 2 with coverage probability 2/3 under normality. For all n > 2, the size of their interval (with λ2 = n + 1) equals approximately the coverage probability of the normal interval based on three observations only. A table is given for the value of A required to guarantee a size of at least h' for the distribution-free interval for selected values of h' and for all n ≤ 100. It may also be used when applying a Chebyshev-type inequality for simple random sampling from a finite popula...

Methods for Obtaining a Distribution-Free Prediction Interval for the Median of a Future Sample

In this paper some quality control applications of distribution-free prediction intervals for the median of a future sample are described. Methods for obtaining such intervals are presented, and their use is illustrated by examples. Tables are provided ..

Nonparametric Prediction Intervals for a Future Sample Median

Abstract In this article distribution-free prediction intervals for a future sample median are discussed. Large-sample normal approximations to the coverage probabilities are presented. Asymptotic relative efficiency comparisons are made between the distribution-free procedures and analogous parametric prediction intervals, the resulting values demonstrating the good properties of the nonparametric prediction intervals.