Estimation of AUC for Assessing Its Significance in Classification Models

Okeh Uchechukwu Marius

doi:10.7176/jnsr/9-9-03

Abstract

The assessment of the performance of a diagnostic test when test results are measured on continuous scale can be evaluated using the measures of sensitivity and specificity over the range of possible cut-off points for the predictor variable. This is achieved by the use of a receiver operating characteristic (ROC) curve which is a graph of sensitivity against 1-specificity across all possible decision cut-offs values from a diagnostic test result. This curve evaluates the diagnostic ability of tests to discriminate the true state of subjects especially in classification models. These tasks of assessing the predictive accuracy of classification models is always better achieved using a summary measure of accuracy across all possible ranges of cut-off values called the area under the receiver operating characteristic curve (AUC). In this paper, we propose a simple nonparametric method of calculating AUC from predicted probability of positive response involving multiple prediction rules. This method is based on the knowledge of non-parametric Mann-Whitney U statistic. Based on the predicted outcomes and observed outcomes, the performance of diagnostic tests is assessed for the classification models through the AUC calculated from these outcomes. The proposed method when applied on real data, the significance of AUC for the classification models is assessed. The method offers reliable statistical inferences and circumvents the difficulties of deriving the statistical moments of complex summary statistics seen in the parametric method. The proposed method as a non-parametric estimation is recommended for calculating the AUC as it compares favorably with the existing parametric and non-parametric methods. Keywords: Cut-off value, ROC, Predicted probability, parametric, non-parametric DOI : 10.7176/JNSR/9-9-03 Publication date :May 31 st 2019

Highlights

In medical sciences, the use of diagnostic procedures is based on clinical investigations or laboratory experiments or trials purposely to classify subject into diseased or non-diseased
Receiver operating characteristic curve (ROC) analysis has been used as a popular technique of evaluating the performance or ability of a test to discriminate between alternative health status
It is note worthy that estimates from parametric methods such as the method of maximum-likelihood estimates (MLEs) are inconsistent thereby giving a misleading picture of the regression relationship (Pepe, 2003)

Summary

Introduction

The use of diagnostic procedures is based on clinical investigations or laboratory experiments or trials purposely to classify subject into diseased or non-diseased. The ROC curve was originated in the theory of signal detection in the years 1950-1960 (Green and Swets, 1966; Egan, 1975) to discriminate between signal and noise It has been used in so many areas such as radiology (Metz,1989), psychiatry(Hsiao et al,1989), epidemiology(Aoki et al, 1997), biomedical informatics(Lasko et al,2005). AUC represents the diagnostic accuracy of the test Y, so that the larger the area the better the diagnostic accuracy of Y This means that values closer to 1 indicate that Y optimally discriminates between healthy and diseased subjects, while values near 0.5 indicate that the test is not informative (Zhou et al, 2002). It attempts to calculate AUC based on a simple new method and evaluate its significance in assessing classification models

Objectives

Methods

Discussion

Conclusion