Correlated Receiver Operating Characteristic Curves Research Articles

Linearithmic and unbiased implementation of DeLong’s algorithm for comparing the areas under correlated ROC curves

Receiver operating characteristic (ROC) analysis is a widespread framework in two-group classification problems. The area under the ROC curve (AUC) has been a popular figure of merit to summarize the performance of binary classifiers. Because of its simplicity of implementation, DeLong’s algorithm (DLA) has been widely used in practice. However, DLA suffers two drawbacks of time inefficiency and biasedness. The authors’ previous work has overcome the first drawback. Specifically, the time complexity of DLA is reduced from the quadratic order down to a linearithmic one, by exploiting the relationship between the Heaviside function and the mid-ranks of samples. Unfortunately, the second drawback of DLA remains unaddressed, to the best of our knowledge. Motivated by such unsatisfactory situation, in this paper, the authors correct the bias of DLA with the assistance of an efficient implementation of Kendall’s tau (KT). The overall computational load of the improved DLA (being strictly unbiased) is in linearithmic time. Theoretical analyses and numerical results verified the proposed algorithmic findings in this work.

Test characteristics of rasterstereography for the early diagnosis of adolescent idiopathic scoliosis.

This study aimed to evaluate rasterstereography of the spine as a diagnostic test for adolescent idiopathic soliosis (AIS), and to compare its results with those obtained using a scoliometer. Adolescents suspected of AIS and scheduled for radiographs were included. Rasterstereographic scoliosis angle (SA), maximal vertebral surface rotation (ROT), and angle of trunk rotation (ATR) with a scoliometer were evaluated. The area under the curve (AUC) from receiver operating characteristic (ROC) plots were used to describe the discriminative ability of the SA, ROT, and ATR for scoliosis, defined as a Cobb angle > 10°. Test characteristics (sensitivity and specificity) were reported for the best threshold identified using the Youden method. AUC of SA, ATR, and ROT were compared using the bootstrap test for two correlated ROC curves method. Of 212 patients studied, 146 (69%) had an AIS. The AUC was 0.74 for scoliosis angle (threshold 12.5°, sensitivity 75%, specificity 65%), 0.65 for maximal vertebral surface rotation (threshold 7.5°, sensitivity 63%, specificity 64%), and 0.82 for angle of trunk rotation (threshold 5.5°, sensitivity 65%, specificity 80%). The AUC of ROT was significantly lower than that of ATR (p < 0.001) and SA (p < 0.001). The AUCs of ATR and SA were not significantly different (p = 0.115). The rasterstereographic scoliosis angle has better diagnostic characteristics than the angle of trunk rotation evaluated with a scoliometer, with similar AUCs and a higher sensitivity.

Incorporation of PET Metabolic Parameters With Clinical Features Into a Predictive Model for Radiotherapy-Related Esophageal Fistula in Esophageal Squamous Cell Carcinoma.

PurposeTo determine whether the addition of metabolic parameters from fluorine-18-fluorodeoxyglucose positron emission tomography/computed tomography (18F-FDG PET/CT) scans to clinical factors could improve risk prediction models for radiotherapy-related esophageal fistula (EF) in esophageal squamous cell carcinoma (ESCC).Methods and MaterialsAnonymized data from 185 ESCC patients (20 radiotherapy-related EF-positive cases) were collected, including pre-therapy PET/CT scans and EF status. In total, 29 clinical features and 15 metabolic parameters from PET/CT were included in the analysis, and a least absolute shrinkage and selection operator logistic regression model was used to construct a risk score (RS) system. The predictive capabilities of the models were compared using receiver operating characteristic (ROC) curves.ResultsIn univariate analysis, metabolic tumor volume (MTV)_40% was a risk factor for radiotherapy (RT)-related EF, with an odds ratio (OR) of 1.036 [95% confidence interval (CI): 1.009–1.063, p = 0.007]. However, it was excluded from the predictive model using multivariate logistic regression. Predictive models were built based on the clinical features in the training cohort. The model included diabetes, tumor length and thickness, adjuvant chemotherapy, eosinophil count, and monocyte-to-lymphocyte ratio. The RS was defined as follows: 0.2832 − (7.1369 × diabetes) + (1.4304 × tumor length) + (2.1409 × tumor thickness) – [8.3967 × adjuvant chemotherapy (ACT)] − (28.7671 × eosinophils) + (8.2213 × MLR). The cutoff of RS was set at −1.415, with an area under the curve (AUC) of 0.977 (95% CI: 0.9536–1), a specificity of 0.929, and a sensitivity of 1. Analysis in the testing cohort showed a lower AUC of 0.795 (95% CI: 0.577–1), a specificity of 0.925, and a sensitivity of 0.714. Delong’s test for two correlated ROC curves showed no significant difference between the training and testing sets (p = 0.109).ConclusionsMTV_40% was a risk factor for RT-related EF in univariate analysis and was screened out using multivariate logistic regression. A model with clinical features can predict RT-related EF.

Differences in the Predictive value of Elixhauser Comorbidity Index and the Charlson Comorbidity indices in patients with hand infections

PurposeHand infections are a common source of potentially debilitating morbidity, particularly in patients with comorbid disease. We hypothesize that there is a difference in predictive value between two commonly used comorbidity indices for the prognosis of hand infections, which may have clinical implications in the management of these conditions. MethodsThe Nationwide Inpatient Sample 2001–2013 database was queried for hand infections using International Classification of Diseases, Ninth Revision codes. The Elixhauser (ECI) and Charlson (CCI) comorbidity scores were calculated based on validated sets of ICD-9 codes. Primary outcomes included mortality, prolonged length of stay (LOS, defined as >95 percentile), discharge destination, and postoperative complications. Indices were compared using receiver operating characteristic (ROC) curves and the areas under the curve (AUC). If confidence intervals overlapped, significance was determined using the DeLong method for correlated ROC curves. This is a validated, non-parametric comparison used for the calculation of the difference between two AUCs. ResultsA weighted total of 1,511,057 patients were included in this study. The majority were Caucasian (57.1%) males (61.4%). Complication rates included 0.9% mortality, 5.3% prolonged length of stay, 25.3% discharges to non-home destinations, and 5.3% post-operative complications. The ECI and CCI each demonstrated good predictive value for mortality, but poor predictive value for non-routine discharge, prolonged LOS, and post-operative complications. There was a significantly increased likelihood of each complication with increasing comorbidity score for both indices, with the greatest odds ratio in the ECI ≥4 cohort. ConclusionsThe CCI was superior in predicting mortality while the ECI was superior in predicting non-routine discharge, prolonged length of stay, and postoperative complications, but these indices may not be clinically relevant. While both represent good predictive models, a score specifically designed for patients with hand infections may have superior prognostic value. Level of evidenceLevel IV.

A Permutation Test for Comparing Two Correlated Receiver Operating Characteristic Curves

The area under the Receiver Operating Characteristic (ROC) curve (AUC) is a summary measure when comparing two ROC curves. However, this summary measure is less informative when two ROC curves cross and have the same AUCs. In order to detect differences between ROC curves and to be able to tackle the problem of exchangeability of the labels between two diagnostic tests within subject, an alternative permutation test based on between-subject permutations of the labels of the subjects within each diagnostic test is proposed for assessing a change in the AUCs in a continuous matched pair of data from two diagnostic test procedures having both non-diseased and diseased subject in each of the test. The Wilcoxon signed rank test statistic was modified as a permutation test under the null hypothesis of equality of AUCs. An algorithm for carrying out complete enumeration of all the distinct permutations of the paired test results was developed which provides exact p-values. Using simulated data, the proposed test compares in statistical power to the modified sign test proposed by Braun and Alonzo but the proposed test has better operating characteristics, that is greater statistical power to detect a crossing alternative and is less conservative in test size and in the range of parameters of at least 0.8 of AUCs on the average with a correlation of at least 0.4 and small to moderately large sample sizes. Similarly in applying real life data, the proposed test has the more likelihood of rejecting null hypothesis of equality of AUC1 and AUC2 at nominal level of 0.05 with the proposed test having a p-value of 0.0312 against the Braun and Alonzo’s test with a p-value of 0.0387. This is because the proposed test is modified to adjust for the presence of zero differences in values and considers the signs of values as well as the absolute ranks of values. Also the estimates of AUC1 and AUC2 for the two diagnostic tests are 0.668 and 0.887 respectively showing that AUC2, that is 2hour 100g Oral Glucose Tolerance Test (OGTT) is superior to AUC1 (2hour 70g OGTT) at a time that the specificity is greater than 0.7.

Machine learning to predict occult nodal metastasis in early oral squamous cell carcinoma

ObjectivesTo develop and validate an algorithm to predict occult nodal metastasis in clinically node negative oral cavity squamous cell carcinoma (OCSCC) using machine learning. To compare algorithm performance to a model based on tumor depth of invasion (DOI). Materials and methodsPatients who underwent primary tumor extirpation and elective neck dissection from 2007 to 2013 for clinical T1-2N0 OCSCC were identified from the National Cancer Database (NCDB). Multiple machine learning algorithms were developed to predict pathologic nodal metastasis using clinicopathologic data from 782 patients.The algorithm was internally validated using test data from 654 patients in NCDB and was then externally validated using data from 71 patients treated at a single academic institution. Performance was measured using area under the receiver operating characteristic (ROC) curve (AUC). Machine learning and DOI model performance were compared using Delong’s test for two correlated ROC curves. ResultsThe best classification performance was achieved with a decision forest algorithm (AUC = 0.840). When applied to the single-institution data, the predictive performance of machine learning exceeded that of the DOI model (AUC = 0.657, p = 0.007). Compared to the DOI model, machine learning reduced the number of neck dissections recommended while simultaneously improving sensitivity and specificity. ConclusionMachine learning improves prediction of pathologic nodal metastasis in patients with clinical T1-2N0 OCSCC compared to methods based on DOI. Improved predictive algorithms are needed to ensure that patients with occult nodal disease are adequately treated while avoiding the cost and morbidity of neck dissection in patients without pathologic nodal disease.

Comparison of two correlated ROC curves at a given specificity or sensitivity level

The receiver operating characteristic (ROC) curve is the most popular statistical tool for evaluating the discriminatory capability of a given continuous biomarker. The need to compare two correlated ROC curves arises when individuals are measured with two biomarkers, which induces paired and thus correlated measurements. Many researchers have focused on comparing two correlated ROC curves in terms of the area under the curve (AUC), which summarizes the overall performance of the marker. However, particular values of specificity may be of interest. We focus on comparing two correlated ROC curves at a given specificity level. We propose parametric approaches, transformations to normality, and nonparametric kernel-based approaches. Our methods can be straightforwardly extended for inference in terms of ROC-1 (t). This is of particular interest for comparing the accuracy of two correlated biomarkers at a given sensitivity level. Extensions also involve inference for the AUC and accommodating covariates. We evaluate the robustness of our techniques through simulations, compare them with other known approaches, and present a real-data application involving prostate cancer screening. Copyright © 2016 John Wiley & Sons, Ltd.

Bivariate Test for Testing the EQUALITY of the Average Areas under Correlated Receiver Operating Characteristic Curves (Test for Comparing of AUC’s of Correlated ROC Curves)

Methodology developed for comparing correlated ROC curves are mainly based on nonparametric methods. These nonparametric methods have several disadvantages. In this paper the authors propose an asymptotic bivariate test for comparing pairs of AUCs for independent data based on the Dorfman and Alf maximum likelihood approach. The properties of the test are examined by using simulation studies. The method is illustrated on an example of angiogram results from Sri Lanka. The test applied to the example found that there was a significant difference in the predictive power of three different cut-offs examined.

Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves

Among algorithms for comparing the areas under two or more correlated receiver operating characteristic (ROC) curves, DeLong's algorithm is perhaps the most widely used one due to its simplicity of implementation in practice. Unfortunately, however, the time complexity of DeLong's algorithm is of quadratic order (the product of sample sizes), thus making it time-consuming and impractical when the sample sizes are large. Based on an equivalent relationship between the Heaviside function and mid-ranks of samples, we improve DeLong's algorithm by reducing the order of time complexity from quadratic down to linearithmic (the product of sample size and its logarithm). Monte Carlo simulations verify the computational efficiency of our algorithmic findings in this work.

Empirical Likelihood Confidence Intervals for the Difference of Areas Under Two Correlated ROC Curves

The area under the receiver operating characteristic (ROC) curve, AUC, is commonly used to assess the ability of a diagnostic test to correctly classify individuals into diseased and nondiseased populations. When there are two diagnostic tests available, it is of interest to evaluate and compare their performances. Based on the difference of two placement values, we propose a two-sample empirical likelihood method for comparing AUCs of two ROC curves. The proposed empirical likelihood ratio statistic converges in distribution to a scaled chi-squared random variable. Simulation results show that the proposed empirical likelihood method has a better finite-sample performance than other competitors.

Semiparametric empirical likelihood confidence intervals for the difference of areas under two correlated ROC curves under density ratio model

The area under the receiver operating characteristic (ROC) curve, AUC, is commonly used to assess the ability of a diagnostic test to correctly classify individuals into diseased and nondiseased populations. When there are two diagnostic tests available, it is of interest to evaluate and compare their performances. Based on the difference of two placement values, we propose a two-sample empirical likelihood method for comparing AUCs of two ROC curves. The proposed empirical likelihood ratio statistic converges in distribution to a scaled chi-squared random variable. Simulation results show that the proposed empirical likelihood method has better finite-sample performance than other competitors.

On Efficient Assessment of Image-Quality Metrics Based on Linear Model Observers

This paper is motivated by the problem of image-quality assessment using model observers for the purpose of development and optimization of medical imaging systems. Specifically, we present a study regarding the estimation of the receiver operating characteristic (ROC) curve for the observer and associated summary measures. This study evaluates the statistical advantage that may be gained in ROC estimates of observer performance by assuming that the difference of the class means for the observer ratings is known. Such knowledge is frequently available in image-quality studies employing known-location lesion detection tasks together with linear model observers. The study is carried out by introducing parametric point and confidence interval estimators that incorporate a known difference of class means. An evaluation of the new estimators for the area under the ROC curve establishes that a large reduction in statistical variability can be achieved through incorporation of knowledge of the difference of class means. Namely, the mean 95% AUC confidence interval length can be as much as seven times smaller in some cases. We also examine how knowledge of the difference of class means can be advantageously used to compare the areas under two correlated ROC curves, and observe similar gains.

Wilcoxon‐based group sequential designs for comparison of areas under two correlated ROC curves

Clinical studies to evaluate the relative accuracies of two diagnostic modalities via their receiver operating characteristic (ROC) curves are currently conducted using fixed sample designs: cases are accrued until a predetermined sample size is achieved and, at that point, the areas under the ROC curves are computed and compared (Radiology 1982; 143:29-36; Radiology 1983; 148:839-843). In prospective ROC studies (Radiology 1990; 175:571-575), participants are recruited from a clinically defined cohort and diagnostic test information is obtained and interpreted in advance of ascertaining the definitive proof of diagnosis ('gold standard'). In retrospective studies, cases are selected from a set of patient records and their diagnostic tests are interpreted without knowledge of the 'gold standard'. The conduct of ROC studies requires considerable effort and resources, particularly for the collection of 'gold standard' information. Thus, it is highly desirable to search for designs which are more efficient than using a fixed sample. In this paper, we discuss the formulation and application of group sequential designs (GSDs) to comparative ROC studies based on non-parametric Wilcoxon estimators of the area under the ROC curves. The approach is applicable to comparisons of ROC curve areas of two tests measured on either continuous or ordinal scales on same cases ('paired' designs) with one reader. The adoption of GSDs may lead to substantial savings in the number of required cases, thus resulting in both time and resource use efficiency.

A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets

The receiver operating characteristic (ROC) curve is a statistical tool for evaluating the accuracy of diagnostic tests. Investigators often compare the validity of two tests based on the estimated areas under the respective ROC curves. However, the traditional way of comparing entire areas under two ROC curves is not sensitive when two ROC curves cross each other. Also, there are some cutpoints on the ROC curves that are not considered in practice because their corresponding sensitivities or specificities are unacceptable. For the purpose of comparing the partial area under the curve (AUC) within a specific range of specificity for two correlated ROC curves, a non-parametric method based on Mann–Whitney U-statistics has been developed. The estimation of AUC along with its estimated variance and covariance is simplified by a method of grouping the observations according to their cutpoint values. The method is used to evaluate alternative logistic regression models that predict whether a subject has incident breast cancer based on information in Medicare claims data. Copyright © 2002 John Wiley & Sons, Ltd.

A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets.

The receiver operating characteristic (ROC) curve is a statistical tool for evaluating the accuracy of diagnostic tests. Investigators often compare the validity of two tests based on the estimated areas under the respective ROC curves. However, the traditional way of comparing entire areas under two ROC curves is not sensitive when two ROC curves cross each other. Also, there are some cutpoints on the ROC curves that are not considered in practice because their corresponding sensitivities or specificities are unacceptable. For the purpose of comparing the partial area under the curve (AUC) within a specific range of specificity for two correlated ROC curves, a non-parametric method based on Mann-Whitney U-statistics has been developed. The estimation of AUC along with its estimated variance and covariance is simplified by a method of grouping the observations according to their cutpoint values. The method is used to evaluate alternative logistic regression models that predict whether a subject has incident breast cancer based on information in Medicare claims data.

The Average Area under Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach Based on Generalized Two-Sample Wilcoxon Statistics

SUMMARY It is well known that, when sample observations are independent, the area under the receiver operating characteristic (ROC) curve corresponds to the Wilcoxon statistics if the area is calculated by the trapezoidal rule. Correlated ROC curves arise often in medical research and have been studied by various parametric methods. On the basis of the Mann–Whitney U-statistics for clustered data proposed by Rosner and Grove, we construct an average ROC curve and derive nonparametric methods to estimate the area under the average curve for correlated ROC curves obtained from multiple readers. For the more complicated case where, in addition to multiple readers examining results on the same set of individuals, two or more diagnostic tests are involved, we derive analytic methods to compare the areas under correlated average ROC curves for these diagnostic tests. We demonstrate our methods in an example and compare our results with those obtained by other methods. The nonparametric average ROC curve and the analytic methods that we propose are easy to explain and simple to implement.

Comparison of Correlated Receiver Operating Characteristic Curves Derived from Repeated Diagnostic Test Data

Rational and Objectives It is common to administer the same diagnostic test more than once to the same set of patients. The purpose of this study was to develop two statistical methods for estimating and comparing correlated receiver operating characteristic (ROC) curves for data derived from repeated diagnostic tests. Material and Methods Parametric and semiparametric transformation models were developed. These estimation methods were illustrated with data from 72 pigmented lesions suspected of being malignant melanoma. A diagnostic scoring system based on asymmetry, border irregularity, color variation, and diameter was used repeatedly, with or without a dermoscope. Statistical hypothesis tests were conducted to evaluate whether a dermoscope improved the clarity of the lesion features in the scoring system. The resulting ROC curves were constructed, along with characteristics and summary measures. Results The areas under the ROC curves were 0.885 (parametric method) and 0.893 (semiparametric method) without the dermoscope, and 0.916 (parametric) and 0.912 (semiparametric) with the dermoscope. The statistical hypothesis tests did not yield statistically significant differences between the underlying ROC curves for either estimation method. Conclusion The two transformation models yielded similar results for estimation and comparison of the ROC curves. Although a dermoscope did not add extra information, the scoring system was accurate for diagnosing malignant melanoma.

Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests

We review the principles and practical application of receiver-operating characteristic (ROC) analysis for diagnostic tests. ROC analysis can be used for diagnostic tests with outcomes measured on ordinal, interval or ratio scales. The dependence of the diagnostic sensitivity and specificity on the selected cut-off value must be considered for a full test evaluation and for test comparison. All possible combinations of sensitivity and specificity that can be achieved by changing the test’s cut-off value can be summarised using a single parameter; the area under the ROC curve. The ROC technique can also be used to optimise cut-off values with regard to a given prevalence in the target population and cost ratio of false-positive and false-negative results. However, plots of optimisation parameters against the selected cut-off value provide a more-direct method for cut-off selection. Candidates for such optimisation parameters are linear combinations of sensitivity and specificity (with weights selected to reflect the decision-making situation), odds ratio, chance-corrected measures of association (e.g. kappa) and likelihood ratios. We discuss some recent developments in ROC analysis, including meta-analysis of diagnostic tests, correlated ROC curves (paired-sample design) and chance- and prevalence-corrected ROC curves.

Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach

Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.