Smooth Receiver Operating Characteristic Research Articles

Smooth ROC curve estimation via Bernstein polynomials.

The receiver operating characteristic (ROC) curve is commonly used to evaluate the accuracy of a diagnostic test for classifying observations into two groups. We propose two novel tuning parameters for estimating the ROC curve via Bernstein polynomial smoothing of the empirical ROC curve. The new estimator is very easy to implement with the naturally selected tuning parameter, as illustrated by analyzing both real and simulated data sets. Empirical performance is investigated through extensive simulation studies with a variety of scenarios where the two groups are both from a single family of distributions (symmetric or right skewed) or one from a symmetric and the other from a right skewed distribution. The new estimator is uniformly more efficient than the empirical ROC estimator, and very competitive to eleven other existing smooth ROC estimators in terms of mean integrated square errors.

The length of the receiver operating characteristic curve and the two cutoff Youden index within a robust framework for discovery, evaluation, and cutoff estimation in biomarker studies involving improper receiver operating characteristic curves.

During the early stage of biomarker discovery, high throughput technologies allow for simultaneous input of thousands of biomarkers that attempt to discriminate between healthy and diseased subjects. In such cases, proper ranking of biomarkers is highly important. Common measures, such as the area under the receiver operating characteristic (ROC) curve (AUC), as well as affordable sensitivity and specificity levels, are often taken into consideration. Strictly speaking, such measures are appropriate under a stochastic ordering assumption, which implies, without loss of generality, that higher measurements are more indicative for the disease. Such an assumption is not always plausible and may lead to rejection of extremely useful biomarkers at this early discovery stage. We explore the length of a smooth ROC curve as a measure for biomarker ranking, which is not subject to directionality. We show that the length corresponds to a divergence, is identical to the corresponding length of the optimal (likelihood ratio) ROC curve, and is an appropriate measure for ranking biomarkers. We explore the relationship between the length measure and the AUC of the optimal ROC curve. We then provide a complete framework for the evaluation of a biomarker in terms of sensitivity and specificity through a proposed ROC analogue for use in improper settings. In the absence of any clinical insight regarding the appropriate cutoffs, we estimate the sensitivity and specificity under a two-cutoff extension of the Youden index and we further take into account the implied costs. We apply our approaches on two biomarker studies that relate to pancreatic and esophageal cancer.

Smoothed time-dependent receiver operating characteristic curve for right censored survival data.

The prediction reliability is of primary concern in many clinical studies when the objective is to develop new predictive models or improve existing risk scores. In fact, before using a model in any clinical decision making, it is very important to check its ability to discriminate between subjects who are at risk of, for example, developing certain disease in a near future from those who will not. To that end, the time-dependent receiver operating characteristic (ROC) curve is the most commonly used method in practice. Several approaches have been proposed in the literature to estimate the ROC nonparametrically in the context of survival data. But, except one recent approach, all the existing methods provide a nonsmooth ROC estimator whereas, by definition, the ROC curve is smooth. In this article we propose and study a new nonparametric smooth ROC estimator based on a weighted kernel smoother. More precisely, our approach relies on a well-known kernel method used to estimate cumulative distribution functions of random variables with bounded supports. We derived some asymptotic properties for the proposed estimator. As bandwidth is the main parameter to be set, we present and study different methods to appropriately select one. A simulation study is conducted, under different scenarios, to prove the consistency of the proposed method and to compare its finite sample performance with a competitor. The results show that the proposed method performs better and appear to be quite robust to bandwidth choice. As for inference purposes, our results also reveal the good performances of a proposed nonparametric bootstrap procedure. Furthermore, we illustrate the method using a real data example.

On Boundary Correction in Kernel Estimation of ROC Curves

The Receiver Operating Characteristic (ROC) curve is a statistical tool for evaluating the accuracy of diagnostics tests. The empirical ROC curve (which is a step function) is the most commonly used non-parametric estimator for the ROC curve. On the other hand, kernel smoothing methods have been used to obtain smooth ROC curves. The preceding process is based on kernel estimates of the distribution functions. It has been observedthat kernel distribution estimators are not consistent when estimating a distribution function near the boundary of its support. This problem is due to “boundary effects” that occur in nonparametric functional estimation. To avoid these difficulties, we propose a generalized reflection method of boundary correction in the estimation problem of ROC curves. The proposed method generates a class of boundary corrected estimators.

Least squares regression methods for clustered ROC data with discrete covariates.

The receiver operating characteristic (ROC) curve is a popular tool to evaluate and compare the accuracy of diagnostic tests to distinguish the diseased group from the nondiseased group when test results from tests are continuous or ordinal. A complicated data setting occurs when multiple tests are measured on abnormal and normal locations from the same subject and the measurements are clustered within the subject. Although least squares regression methods can be used for the estimation of ROC curve from correlated data, how to develop the least squares methods to estimate the ROC curve from the clustered data has not been studied. Also, the statistical properties of the least squares methods under the clustering setting are unknown. In this article, we develop the least squares ROC methods to allow the baseline and link functions to differ, and more importantly, to accommodate clustered data with discrete covariates. The methods can generate smooth ROC curves that satisfy the inherent continuous property of the true underlying curve. The least squares methods are shown to be more efficient than the existing nonparametric ROC methods under appropriate model assumptions in simulation studies. We apply the methods to a real example in the detection of glaucomatous deterioration. We also derive the asymptotic properties of the proposed methods.

CRIB conditional on gender: nonparametric ROC curve.

The purpose of this paper is to use the kernel method to produce a smoothed receiver operating characteristic (ROC) curve and show how baby gender can influence Clinical Risk Index for Babies (CRIB) scale according to survival risks. To obtain the ROC curve, conditioned by covariates, two methods may be followed: first, indirect adjustment, in which the covariate is first modeled within groups and then by generating a modified distribution curve; second, direct smoothing in which covariate effects is modeled within the ROC curve itself. To verify if new-born gender and weight affects the classification according to the CRIB scale, the authors use the direct method. The authors sampled 160 Portuguese babies. The smoothing applied to the ROC curves indicates that the curve's original shape does not change when a bandwidth h = 0.1 is used. Furthermore, gender seems to be a significant covariate in predicting baby deaths. A higher value was obtained for the area under curve (AUC) when conditional on female babies. The challenge is to determine whether gender discriminates between dead and surviving babies. The authors constructed empirical ROC curves for CRIB data and empirical ROC curves conditioned on gender. The authors calculate the corresponding AUC and tested the difference between them. The authors also constructed smooth ROC curves for two approaches.

Mixtures of Receiver Operating Characteristic Curves

Receiver operating characteristic (ROC) curves are ubiquitous in the analysis of imaging metrics as markers of both diagnosis and prognosis. While empirical estimation of ROC curves remains the most popular method, there are several reasons to consider smooth estimates based on a parametric model. A mixture model is considered for modeling the distribution of the marker in the diseased population motivated by the biological observation that there is more heterogeneity in the diseased population than there is in the normal one. It is shown that this model results in an analytically tractable ROC curve which is itself a mixture of ROC curves. The use of creatine kinase-BB isoenzyme in diagnosis of severe head trauma is used as an example. ROC curves are fit using the direct binormal method, ROCKIT software, and the Box-Cox transformation as well as the proposed mixture model. The mixture model generates an ROC curve that is much closer to the empirical one than the other methods considered. Mixtures of ROC curves can be helpful in fitting smooth ROC curves in datasets where the diseased population has higher variability than can be explained by a single distribution.

The Bi-Gamma ROC Curve in a Straightforward Manner

In biomedical research, biomarkers (diagnostic tests) are used in distinguishing healthy and diseased populations. The effectiveness and accuracy of a biomarker generally assessed through the use of a Receiver Operating Characteristic (ROC) curve model, and its functional such as area under the curve (AUC). The parametric (smooth) ROC curves are obtained under the specific distributions assumptions. A resulting ROC curve model is the plot of sensitivity versus 1-specificity for all possible threshold values. Most popular and widely used ROC curve model is bi-normal ROC curve model under the assumptions of normality. When the biomarker results are continuous and positively skewed (non-normal). The gamma distribution is supposed to a flexible model for positively skewed measurements. In practice use of bi-gamma ROC curve model is hindered by the fact that ROC function cannot be written in closed-form. The solution of the problem is to use transformed invariance property of ROC curve model. Which assumes that the test results of both diseased and healthy are normally distributed after some monotone transformation [1]. In this paper we propose a simple approximation solution for the problem mentioned in above lines using a normal approximation due to Wilson and Hilfertys [2]. Which is useful to approximate gamma distribution results with classical normal distribution based results.

Transformation‐invariant and nonparametric monotone smooth estimation of ROC curves

When a new diagnostic test is developed, it is of interest to evaluate its accuracy in distinguishing diseased subjects from non-diseased subjects. The accuracy of the test is often evaluated by receiver operating characteristic (ROC) curves. Smooth ROC estimates are often preferable for continuous test results when the underlying ROC curves are in fact continuous. Nonparametric and parametric methods have been proposed by various authors to obtain smooth ROC curve estimates. However, there are certain drawbacks with the existing methods. Parametric methods need specific model assumptions. Nonparametric methods do not always satisfy the inherent properties of the ROC curves, such as monotonicity and transformation invariance. In this paper we propose a monotone spline approach to obtain smooth monotone ROC curves. Our method ensures important inherent properties of the underlying ROC curves, which include monotonicity, transformation invariance, and boundary constraints. We compare the finite sample performance of the newly proposed ROC method with other ROC smoothing methods in large-scale simulation studies. We illustrate our method through a real life example.

Smooth semiparametric receiver operating characteristic curves for continuous diagnostic tests

We propose a semiparametric kernel distribution function estimator, based on which a new smooth semiparametric estimator of the receiver operating characteristic (ROC) curve is constructed. We derive the asymptotic bias and variance of the newly proposed distribution function estimator and show that it is more efficient than the traditional non-parametric kernel distribution estimator. We also derive the asymptotic bias and variance of our new ROC curve estimator and show that it is more efficient than the smooth non-parametric ROC curve estimator proposed by Zou et al. (Stat. Med. 1997; 16:2143-2156) and Lloyd (J. Am. Stat. Assoc. 1998; 93:1356-1364). For our proposed estimators, we derive data-based methods for bandwidth selection. In addition, we present some results on the analysis of two real data sets. Finally, a simulation study is presented to show that our estimators are better than the non-parametric counterparts in terms of bias, standard error, and mean-square error.

Receiver operating characteristic (ROC) curve: practical review for radiologists.

The receiver operating characteristic (ROC) curve, which is defined as a plot of test sensitivity as the y coordinate versus its 1-specificity or false positive rate (FPR) as the x coordinate, is an effective method of evaluating the performance of diagnostic tests. The purpose of this article is to provide a nonmathematical introduction to ROC analysis. Important concepts involved in the correct use and interpretation of this analysis, such as smooth and empirical ROC curves, parametric and nonparametric methods, the area under the ROC curve and its 95% confidence interval, the sensitivity at a particular FPR, and the use of a partial area under the ROC curve are discussed. Various considerations concerning the collection of data in radiological ROC studies are briefly discussed. An introduction to the software frequently used for performing ROC analyses is also presented.

Original smooth receiver operating characteristic curve estimation from continuous data: Statistical methods for analyzing the predictive value of spiral CT of ureteral stones

Diagnostic studies such as spiral computed tomography (CT) in patients with obstructing ureteral calculi often necessitate the analysis of continuous test data (e.g., stone sizes). The accuracy of a test can be summarized by using a receiver operating characteristic (ROC) curve. The authors developed and compared three methods for constructing a smooth ROC curve from continuous diagnostic data. Nonparametric, semiparametric, and parametric smooth ROC curve analyses were applied to data from 100 unenhanced spiral CT scans of patients with proved obstructing ureteral stones. Accuracy in using stone size to predict the need for intervention was evaluated by means of these methods. Characteristics and summary measures of the resulting ROC curves were estimated. All methods fit the data well. The nonparametric method followed the details of the empiric data. The semiparametric and parametric methods yielded similar estimates of the ROC curve parameters. Areas under the ROC curves were 0.807, 0.821, and 0.814 for nonparametric, semiparametric, and parametric methods, respectively, in comparison with 0.811 for the empiric method. The parametric method is preferred for constructing a smooth ROC curve with available stone-size data derived from spiral CT. The analyses confirm the predictive value of stone size in determining the need for intervention.

Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests.

Receiver operating characteristic (ROC) curves have seen increasing use to assess the accuracy of diagnostic tests that yield continuous test results. We can calculate a fully non-parametric ROC curve from the empirical false positive rate (FPR) and true positive rate (TPR) at each possible decision threshold, but it may be quite jagged. We can obtain a smooth ROC curve by directly fitting a parameteric model, for example, binormal or bilogistic, to the actual test results, but substantial lack-of-fit may result if the distributional assumptions are not valid. A recently proposed alternative algorithm 'LABROC4' is insensitive to such departures, but is not fully flexible in that the form of the resulting ROC curve is restricted to be binormal. We propose a smooth non-parametric ROC curve derived from kernel density estimates of the two test result distributions. We obtain pointwise standard errors for the estimated TPR at each FPR and use them to construct pointwise confidence intervals for the ROC curve, using a logit transformation. We also obtain standard errors for the estimated TPR and FPR at given thresholds and use them to construct confidence rectangles in (FPR, TPR)-space that correspond to a specific threshold. We adapt existing methods for the unsmoothed non-parametric ROC curve to obtain the area under the ROC curve and its standard error; we also give partial areas and corresponding standard errors. We have created a FORTRAN algorithm 'ROC-&-ROL', available upon request. We compare ROC curves and areas obtained by applying our methods and its competitors to two data sets, one of which is fit well by parametric methods and LABROC4, the other of which is not.

Smooth non‐parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests

Receiver operating characteristic (ROC) curves have seen increasing use to assess the accuracy of diagnostic tests that yield continuous test results. We can calculate a fully non-parametric ROC curve from the empirical false positive rate (FPR) and true positive rate (TPR) at each possible decision threshold, but it may be quite jagged. We can obtain a smooth ROC curve by directly fitting a parametric model, for example, binormal or bilogistic, to the actual test results, but substantial lack-of-fit may result if the distributional assumptions are not valid. A recently proposed alternative algorithm ‘LABROC4’ is insensitive to such departures, but is not fully flexible in that the form of the resulting ROC curve is restricted to be binormal. We propose a smooth non-parametric ROC curve derived from kernel density estimates of the two test result distributions. We obtain pointwise standard errors for the estimated TPR at each FPR and use them to construct pointwise confidence intervals for the ROC curve, using a logit transformation. We also obtain standard errors for the estimated TPR and FPR at given thresholds and use them to construct confidence rectangles in (FPR, TPR)-space that correspond to a specific threshold. We adapt existing methods for the unsmoothed non-parametric ROC curve to obtain the area under the ROC curve and its standard error; we also give partial areas and corresponding standard errors. We have created a FORTRAN algorithm ‘ROC-&-ROL’, available upon request. We compare ROC curves and areas obtained by applying our methods and its competitors to two data sets, one of which is fit well by parametric methods and LABROC4, the other of which is not. © 1997 John Wiley & Sons, Ltd.

ROC curves and the binormal assumption

Previous articles in this series have described how receiver operating characteristic (ROC) graphs provide comprehensive graphic representations of the diagnostic performance of non-binary tests and have explained how one constructs "trapezoidal" ROC graphs in which discrete cutoff points are plotted and connected with line segments. In this article, we describe a set of mathematical assumptions that permit the generation of a continuous, smooth ROC curve for a given diagnostic test. These assumptions permit us to characterize a test's performance using a small number of parameters and also to explore properties of diagnostic tests. In this article, we describe a set of mathematical assumptions that can be used to link receiver operating characteristic (ROC) curves to the underlying distribution of values of the diagnostic variable being measured. We will illustrate these assumptions using a diagnostic test that distinguishes alcohol abusers from normal consumers of alcohol and abstainers.