A Novel “Maximizing Kappa” Approach for Assessing the Ability of a Diagnostic Marker and Its Optimal Cutoff Value

Abstract

Threshold-dependent accuracy measures such as true classification rates in ordered multiple-class (k > 3) receiver operating characteristic (ROC) hyper-surfaces have recently been used to assist with medical decision making. However, based on low power performance in some circumstances, we construct a new method that relies on the kappa coefficient to solve such diagnostic problems. Under the approach proposed in the present article, the statistics depend strongly on the cutoff threshold, which can be chosen to maximize the kappa statistics of true disease status and of the new biomarker. The Monte Carlo simulation results confirm the effectiveness of the proposed method in terms of its predictive power. The proposed design is then compared with the volume under the ROC hyper-surface by applying it to intracerebral hemorrhagic patients classified into five stroke classes using the National Institutes of Health Stroke Scale.