Receiver Operating Characteristic Surface Research Articles

Direct estimation of volume under the ROC surface with verification bias

ABSTRACT In practice, the receiver operating characteristic (ROC) curve of a diagnostic test is widely used to show the performance of the test for discriminating two-class events. The area under the ROC curve (AUC) is proposed as an index for the assessment of the diagnostic accuracy of the test under consideration. Due to ethical and cost considerations associated with application of gold standard (GS) tests, only a subset of the patients initially tested have verified disease status. Statistical evaluation of the test performance based only on test results from subjects with verified disease status are typically biased. Various AUC estimation methods for tests with verification biased data have been developed over the last few decades. In this article, we develop new direct estimation methods for the volume under the ROC surface (VUS) by extending the AUC estimation methods for two-class diagnostic tests to three-class diagnostic tests in the presence of verification bias. The proposed methods will provide a comprehensive guide to deal with the verification bias in three-class diagnostic test accuracy studies and lead to a better choice of diagnostic tests.

Construction of joint confidence spaces for the optimal true class fraction triplet in the ROC space using alternative biomarker cutoffs.

Hepatocellular carcinoma (HCC) is the most common primary cancer of the liver. Finding new biomarkers for its early detection is of high clinical importance. As with many other diseases, cancer has a progressive nature. In cancer biomarker studies, it is often the case that the true disease status of the recruited individuals exhibits more than two classes. The receiver operating characteristic (ROC) surface is a well-known statistical tool for assessing the biomarkers' discriminatory ability in trichotomous settings. The volume under the ROC surface (VUS) is an overall measure of the discriminatory ability of a marker. In practice, clinicians are often in need of cutoffs for decision-making purposes. A popular approach for computing such cutoffs is the Youden index and its recent three-class generalization. A drawback of such a method is that it treats the data in a pairwise fashion rather than consider all the data simultaneously. The use of the minimized Euclidean distance from the perfection corner to the ROC surface (also known as closest to perfection method) is an alternative to the Youden index that may be preferable in some settings. When such a method is employed, there is a need for inferences around the resulting true class rates/fractions that correspond to the optimal operating point. In this paper, we provide an inferential framework for the derivation of marginal confidence intervals (CIs) and joint confidence spaces (CSs) around the corresponding true class fractions, when dealing with trichotomous settings. We explore parametric and nonparametric approaches for the construction of such CIs and CSs. We evaluate our approaches through extensive simulations and apply them to a real data set that refers to HCC patients.

Prognostic accuracy for predicting ordinal competing risk outcomes using ROC surfaces.

Many medical conditions are marked by a sequence of events in association with continuous changes in biomarkers. Few works have evaluated the overall accuracy of a biomarker in predicting disease progression. We thus extend the concept of receiver operating characteristic (ROC) surface and the volume under the surface (VUS) from multi-category outcomes to ordinal competing-risk outcomes that are also subject to noninformative censoring. Two VUS estimators are considered. One is based on the definition of the ROC surface and obtained by integrating the estimated ROC surface. The other is an inverse probability weighted U estimator that is built upon the equivalence of the VUS to the concordance probability between the marker and sequential outcomes. Both estimators have nice asymptotic results that can be derived using counting process techniques and U-statistics theory. We illustrate their good practical performances through simulations and applications to two studies of cognition and a transplant dataset.

Estimation of Receiver Operating Characteristic Surface Using Mixtures of Finite Polya Trees (MFPT)

Generalisation of Receiver operating characteristic (ROC) curve has become increasingly useful in evaluating the performance of diagnostic tests that have more than binary outcomes. While parametric approaches have been widely used over the years, the limitations associated with parametric assumptions often make it difficult to modelling the volume under surface for data that do not meet criteria under parametric distributions. As such, estimation of ROC surface using nonparametric approaches have been proposed to obtained insights on available data. One of the common approaches to non-parametric estimation is the use of Bayesian models where assumptions about priors can be made then posterior distributions obtained which can then be used to model the data. This study uses Polya tree priors where mixtures of Polya trees approach was used to model simulated three-way ROC data. The results of VUS estimation which is considered a suitable inference in evaluating performance of a diagnostic test, indicated that the mixtures of Polya trees model fitted well the ROC surface data. Further, the model performed relatively well compared to parametric and semiparametric models under similar assumptions.  

Fast and Unbiased Estimation of Volume Under Ordered Three-Class ROC Surface (VUS) With Continuous or Discrete Measurements

Receiver Operating Characteristic (ROC) surfaces have been studied in the literature essentially during the last decade and are considered as a natural generalization of ROC curves in three-class problems. The volume under the surface (VUS) is useful for evaluating the performance of a trichotomous diagnostic system or a three-class classifier’s overall accuracy when the possible disease condition or sample belongs to one of three ordered categories. In the areas of medical studies and machine learning, the VUS of a new statistical model is typically estimated through a sample of ordinal and continuous measurements obtained by some suitable specimens. However, discrete scales of the prediction are also frequently encountered in practice. To deal with such scenario, in this paper, we proposed a unified and efficient algorithm of linearithmic order, based on dynamic programming, for unbiased estimation of the mean and variance of VUS with unidimensional samples drawn from continuous or non-continuous distributions. Monte Carlo simulations verify our theoretical findings and developed algorithms.

Accurate likelihood inference for the volume under the ROC surface.

With three ordered diagnostic categories, the volume under the receiver operating characteristic (ROC) surface, which is the extension of the area under the ROC curve for binary diagnostic outcomes, is the most commonly used measure for the overall diagnostic accuracy. For a continuous-scale diagnostic test, classical likelihood-based inference about the area under the ROC curve can be inaccurate, in particular when the sample size is small, and higher order inferential procedures have been proposed. The goal of this paper is to illustrate higher order likelihood procedures for parametric inference in small samples, which provide accurate point estimates and confidence intervals for the volume under the ROC surface. Simulation studies are performed in order to illustrate the accuracy of the proposed methodology, and two applications to real data are discussed. We show that likelihood modern inference provide refinements to classical inferential results. Furthermore, the freely available R package likelihoodAsy makes now their use almost automatic. Modern likelihood inference based on higher-order asymptotic methods for the area under the ROC surface provide refinements to classical inferential results. A possible limitation of higher-order asymptotic methods for practical use is that their software implementation can be awkward. Nevertheless, use of the freely available R package likelihoodAsy makes such implementation straightforward.

Accessing the discriminatory performance of FRAIL-NH in two-class and three-class frailty and examining its agreement with the frailty index among nursing home residents in mainland China

BackgroundFRAIL-NH has been commonly used to assess frailty in nursing home residents and validated in many ethnic populations; however, it has not been validated in mainland China, where such an assessment tool is lacking. This study aimed to (1) assess the discriminatory performance of FRAIL-NH in two-class frailty (non-frail+ pre-frail vs. frail) and three-class frailty (non-frail vs. pre-frail vs. frail), based on the Frailty Index (FI), (2) determine the appropriate cutoff points for FRAIL-NH that distinguish two-class and three-class frailty, and (3) examine the agreement in classification between FRAIL-NH and FI.MethodsA cross-sectional study of 302 residents aged 60 years or older from six nursing homes in Changsha was conducted. The FRAIL-NH scale and 34-item FI were used to measure frailty. Two-way and three-way receiver operating characteristic (ROC) curves were used to estimate the performance of FRAIL-NH. Cohen’s Kappa statistics were used to examine the agreement between these two measures.ResultsThe agreement between FRAIL-NH and FI ranged from 0.33 to 0.55. Regardless of what FI cutoff points were based on, the volume under the ROC surface (VUS) for FRAIL-NH from the three-way ROC were higher than the VUS of a useless test (1/6), and the area under the ROC curve (AUC) for FRAIL-NH from the two-way ROC were higher than the clinically meaningless value (0.5). When using FI cutoff points of 0.20 for pre-frail and 0.45 for frail, FRAIL-NH cutoff points of 1 and 9 in classifying three-class frailty had the highest VUS and the largest correct classification rates. Whichever FI was chosen, the performance of FRAIL-NH in distinguishing between pre-frailty and frailty, and between non-frailty and pre-frailty was equivalent. According to FRAIL-NH, the proportion of individuals with frailty misclassified as pre-frailty was higher than that of individuals with non-frailty misclassified as pre-frailty.ConclusionFRAIL-NH can be used as a preliminary frailty screening tool in nursing homes in mainland China. FI should be further used especially for those classified as pre-frailty by FRAIL-NH. It is not advisable to simply combine adjacent two classes of FRAIL-NH to create a new frailty variable in research settings.

Usefulness of the 15-item geriatric depression scale (GDS-15) for classifying minor and major depressive disorders among community-dwelling elders

BackgroundThe 15-item geriatric depression scale (GDS-15) is a short form of GDS and is used to screen, diagnose, and evaluate depression in elderly individuals. Most previous studies evaluated the ability of GDS-15 to discriminate between depressive and non-depressive states. In this study, we investigated the multi-stage discriminating ability of GDS-15. MethodsA total of 774 participants, over 65 years of age were included (normal, n = 650; minor depressive disorder [MnDD], n = 94; major depressive disorder [MDD], n = 30). Multi-category receiver operating characteristic (ROC) surfaces were evaluated to identify three stages of geriatric depression. The optimal cutoff points were selected based on the volume under the ROC surface (VUS) and the Youden index. ResultsIn the results of multi-category classification analyses, VUS of the GDS-15 of 0.61 was obtained, and optimal cutoff points of the GDS-15 for multiple stages of depression of 4 (between normal and MnDD) and 11 (between MnDD and MDD) were derived. The Youden index for the GDS-15 was 0.49, and the derived optimal cutoff points were 5 and 10, for the multiple stages, respectively. The overall diagnostic accuracy based on the Youden index was superior to that based on the VUS in the GDS-15. LimitationsThe participants’ cognitive function has potential to affect the GDS-15 score; nevertheless, the study included those with mild cognitive impairment. ConclusionsGDS-15 was a useful tool to classify stages of geriatric depression into either minor or major depressive disorder.

Bayesian nonparametric estimation of ROC surface under verification bias.

The receiver operating characteristic (ROC) surface, as a generalization of the ROC curve, has been widely used to assess the accuracy of a diagnostic test for three categories. A common problem is verification bias, referring to the situation where not all subjects have their true classes verified. In this paper, we consider the problem of estimating the ROC surface under verification bias. We adopt a Bayesian nonparametric approach by directly modeling the underlying distributions of the three categories by Dirichlet process mixture priors. We propose a robust computing algorithm by only imposing a missing at random assumption for the verification process but no assumption on the distributions. The method can also accommodate covariates information in estimating the ROC surface, which can lead to a more comprehensive understanding of the diagnostic accuracy. It can be adapted and hugely simplified in the case where there is no verification bias, and very fast computation is possible through the Bayesian bootstrap process. The proposed method is compared with other commonly used methods by extensive simulations. We find that the proposed method generally outperforms other approaches. Applying the method to two real datasets, the key findings are as follows: (1) human epididymis protein 4 has a slightly better diagnosis ability compared to CA125 in discriminating healthy, early stage, and late stage patients of epithelial ovarian cancer. (2) Serum albumin has a prognostic ability in distinguishing different stages of hepatocellular carcinoma.

Inference in receiver operating characteristic surface analysis via a trinormal model‐based testing approach

Receiver operating characteristic (ROC) analysis is the methodological framework of choice for the assessment of diagnostic markers and classification procedures in general, in both two‐class and multiple‐class classification problems. We focus on the three‐class problem for which inference usually involves formal hypothesis testing using a proxy metric such as the volume under the ROC surface (VUS). In this article, we develop an existing approach from the two‐class ROC framework. We define a hypothesis‐testing procedure that directly compares two ROC surfaces under the assumption of the trinormal model. In the case of the assessment of a single marker, the corresponding ROC surface is compared with the chance plane, that is, to an uninformative marker. A simulation study investigating the proposed tests with existing ones on the basis of the VUS metric follows. Finally, the proposed methodology is applied to a dataset of a panel of pancreatic cancer diagnostic markers. The described testing procedures along with related graphical tools are supported in the corresponding R‐package trinROC, which we have developed for this purpose.

Fast and Unbiased Estimation of Volume Under Ordered Three-Class ROC Surface (VUS) Based on Dynamic Programming

Receiver operating characteristic (ROC) analysis has been widely used in two-class problems. However, in practice, three-class problems are frequently encountered, especially in the area of medicine. To evaluate the performance of three-class classifiers, researchers have proposed the volume under the three-class ROC surface (VUS) as a figure-of-merit. Unfortunately, to the best of our knowledge, however, all the existing methods suffer heavy computational loads. In this paper, to overcome such an unsatisfactory problem, we develop an efficient dynamic programming-based algorithm for unbiased estimation of the VUS and the corresponding variance. The Monte Carlo simulations verified both the unbiasedness and computing efficiency of our algorithm compared with the state-of-the-art work proposed by Waegeman and co-authors.

Null Distribution of Volume Under Ordered Three-Class ROC Surface (VUS) With Continuous Measurements

Receiver operating characteristic (ROC) analysis has become an indispensable tool in medical care, with a major application to characterizing the performance of binary diagnostic tests in clinical practice. In many circumstances, however, the diagnostic test has three outcomes, that is, the abnormalities are two sided. To deal with this scenario, this letter develops a recursive algorithm for computing the exact null distribution of the volume under the ordered three-class ROC surface (VUS) for samples following continuous distributions. Based on the asymptotic normality, an approximately normal distribution with exact mean and variance is also proposed, which is hoped to be useful for large-sample scenarios. Moreover, an efficient rank-based formula, in linearithmic time, is established for nonparametric estimation of VUS. Monte Carlo simulations verify the usefulness of the theoretical and algorithmic findings in this letter.

Bayesian Semiparametric ROC surface estimation under verification bias

The Receiver Operating Characteristic (ROC) surface is a generalization of the ROC curve and is widely used for assessment of the accuracy of diagnostic tests on three categories. Verification bias occurs when not all subjects have their labels observed. This is a common problem in disease diagnosis since the gold standard test to get labels, i.e., the true disease status, can be invasive and expensive. The same situation happens in the evaluation of semi-supervised learning, where the unlabeled data are incorporated. A Bayesian approach for estimating the ROC surface is proposed based on continuous data under a semi-parametric trinormality assumption. The proposed method is then extended to situations in the presence of verification bias. The posterior distribution is computed under the trinormality assumption using a rank-based likelihood. The consistency of the posterior under a mild condition is also established. The proposed method is compared with existing methods for estimating an ROC surface. Simulation results show that it performs well in terms of accuracy. The method is applied to evaluate the performance of CA125 and HE4 in the diagnosis of epithelial ovarian cancer (EOC) as a demonstration.

Screening for Normal Cognition, Mild Cognitive Impairment, and Dementia with the Korean Dementia Screening Questionnaire.

Objective The Korean Dementia Screening Questionnaire (KDSQ) is an informant-based instrument used to screen for cognitive dysfunction. However, its ability to only dichotomously discriminate between dementia and normal cognition has been previously investigated. This study investigated the ability of the KDSQ to classify not only dichotomous but also multiple stages of cognitive dysfunction. Methods We examined 582 participants. Receiver operating characteristic (ROC) curves were used to determine dichotomous classification parameters. Multi-category ROC surfaces were evaluated to classify the three stages of cognitive dysfunction. Results Dichotomous classification using the ROC curve analyses showed that the area under the curve was 0.92 for dementia for subjects without dementia and 0.96 for dementia in controls. Simultaneous multi-category classification analyses showed that the volume under the ROC surface (VUS) was 0.57 and that the derived optimal cut-off points were 2 and 8 for controls, MCI, and dementia. The estimated Youden index for the KDSQ was 0.48, and the derived optimal cut-off points were 5 and 10. The overall classification accuracy of the VUS and Youden index was 61.2% and 58.6%, respectively. Conclusion The KDSQ is useful for classifying dichotomous and multi-category stages of cognitive dysfunction.

Estimation of the volume under the receiver-operating characteristic surface adjusting for non-ignorable verification bias.

The receiver-operating characteristic surface is frequently used for presenting the accuracy of a diagnostic test for three-category classification problems. One common problem that can complicate the estimation of the volume under receiver-operating characteristic surface is that not all subjects receive the verification of the true disease status. Estimation based only on data from subjects with verified disease status may be biased, which is referred to as verification bias. In this article, we propose new verification bias correction methods to estimate the volume under receiver-operating characteristic surface for a continuous diagnostic test. We assume the verification process is missing not at random, which means the missingness might be related to unobserved clinical characteristics. Three classes of estimators are proposed, namely, inverse probability weighted, imputation-based, and doubly robust estimators. A jackknife estimator of variance is derived for all the proposed volume under receiver-operating characteristic surface estimators. The finite sample properties of the new estimators are examined via simulation studies. We illustrate our methods with data collected from Alzheimer's disease research.

Bayesian bootstrap inference for the receiver operating characteristic surface

Accurate diagnosis of disease is of great importance in clinical practice and medical research. The receiver operating characteristic (ROC) surface is a popular tool for evaluating the discriminatory ability of continuous diagnostic test outcomes when there exist three‐ordered disease classes (e.g., no disease, mild disease, and advanced disease). We propose the Bayesian bootstrap, a fully nonparametric method, for conducting inference about the ROC surface and its functionals, such as the volume under the ROC surface (VUS). The proposed method is based on a simple, yet interesting, representation of the ROC surface in terms of placement variables and has the appealing feature of producing point and interval estimates for the ROC surface and its corresponding VUS in a single integrated framework. Results from a simulation study demonstrate the ability of our method to successfully recover the true ROC surface and to produce valid inferences in a variety of complex scenarios. An application to data from the Trail Making Test to assess cognitive impairment in Parkinson's disease patients is provided.

Informant questionnaire on cognitive decline in the elderly (IQCODE) for classifying cognitive dysfunction as cognitively normal, mild cognitive impairment, and dementia.

The Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) is a reliable, validated informant-based instrument in screening for cognitive dysfunction. However, previous studies have evaluated only the ability to discriminate dichotomously, such as dementia from cognitively normal (CN) individuals or mild cognitive impairment (MCI) from CN. This study investigated the ability of the IQCODE to classify not only dichotomous but also multiple stages of cognitive dysfunction. We examined 228 consecutive participants (76 CN, 76 with MCI, and 76 with dementia). Receiver operating characteristic (ROC) curves determined dichotomous classification parameters. Multi-category ROC surfaces were evaluated to classify three stages of cognitive dysfunction. Dichotomous classification using the ROC curve analyses showed that the area under the ROC curve was 0.91 for dementia from participants without dementia and 0.71 for MCI from CN. Simultaneous multi-category classification analyses showed that the volume under the ROC surface was 0.61 and the derived optimal cut-off points were 3.15 and 3.73 for CN, MCI, and dementia. The Youden index for the IQCODE was estimated as 0.51 and the derived optimal cut-off points were 3.33 and 3.70. The overall classification accuracy by the VUS was 58.3% and that by the Youden index 61.8%. IQCODE is useful to classify the dichotomous and multi-category stages of cognitive dysfunction.

Three-group ROC predictive analysis for ordinal outcomes

ABSTRACTMeasuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning, and credit scoring. The receiver operating characteristic (ROC) surface is a useful tool to assess the ability of a diagnostic test to discriminate among three-ordered classes or groups. In this article, nonparametric predictive inference (NPI) for three-group ROC analysis for ordinal outcomes is presented. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. This article also includes results on the volumes under the ROC surfaces and consideration of the choice of decision thresholds for the diagnosis. Two examples are provided to illustrate our method.

Inverse probability weighting estimation of the volume under the ROC surface in the presence of verification bias.

In diagnostic medicine, the volume under the receiver operating characteristic (ROC) surface (VUS) is a commonly used index to quantify the ability of a continuous diagnostic test to discriminate between three disease states. In practice, verification of the true disease status may be performed only for a subset of subjects under study since the verification procedure is invasive, risky, or expensive. The selection for disease examination might depend on the results of the diagnostic test and other clinical characteristics of the patients, which in turn can cause bias in estimates of the VUS. This bias is referred to as verification bias. Existing verification bias correction in three-way ROC analysis focuses on ordinal tests. We propose verification bias-correction methods to construct ROC surface and estimate the VUS for a continuous diagnostic test, based on inverse probability weighting. By applying U-statistics theory, we develop asymptotic properties for the estimator. A Jackknife estimator of variance is also derived. Extensive simulation studies are performed to evaluate the performance of the new estimators in terms of bias correction and variance. The proposed methods are used to assess the ability of a biomarker to accurately identify stages of Alzheimer's disease.

Bias–corrected methods for estimating the receiver operating characteristic surface of continuous diagnostic tests

Verification bias is a well-known problem that may occur in the evaluation of predictive ability of diagnostic tests. When a binary disease status is considered, various solutions can be found in the literature to correct inference based on usual measures of test accuracy, such as the receiver operating characteristic (ROC) curve or the area underneath. Evaluation of the predictive ability of continuous diagnostic tests in the presence of verification bias for a three-class disease status is here discussed. In particular, several verification bias-corrected estimators of the ROC surface and of the volume underneath are proposed. Consistency and asymptotic normality of the proposed estimators are established and their finite sample behavior is investigated by means of Monte Carlo simulation studies. Two illustrations are also given.