A proposal of the evaluation method of the effectiveness of CAD use by means of the bias with variance characteristic (BVC) analysis and the internal structure analysis of the ROC-curve

Abstract

ABSTRACT In this paper, we propose a methodology for evaluating whether the use of CAD is effective for any given reader or case, first analyzing the results of readers’ judgments (0 or 1) by the technique known as analysis of bias-variance characteristics (BVC) 1,2 , then by combining this with ROC analysis, elucidating the internal structure of the ROC curve. The mean and variance are first calculated for the situation when multiple readers examine a medical image for a single case without CAD and with CAD, and assign the values 0 and 1 to their judgment of whether abnormal findings are absent or present or whether the case is normal or abnormal. The mean of these values represents the degree of bias from the true diagnosis for the particular case, and the variance represents the spread of judgments between readers. When the relationship between the two parameters is examined for several cases with differing degrees of diagnostic difficulty, the mean (horizontal axis) and variance (vertical axis) show a bell-shaped relation. We have named this typical phenomenon arising when images are read, the bias-variance characteristic (BVC) of diagnosis. The mean of the 0 and 1 judgments of multiple readers is regarded as a measure of the confidence level determined for the particular case. ROC curves were drawn by usual methods for diagnoses made without CAD and with CAD. From the difference between the TPF obtained without CAD and with CAD for the same FPF on the ROC curve, we were able to quantify the number of cases, the total number of readers, and the total number of cases for which CAD support was beneficial. To demonstrate its usefulness, we applied this method to data obtained in a reading experiment that aimed to evaluate detection performance for abnormal findings and data obtained in a reading experiment that aimed to evaluate diagnostic discrimination performance for normal and abnormal cases. We analyzed the internal structure of the ROC curve produced when all cases were included, and showed that there is a relationship between the degree of diagnostic difficulty of the case and the benefit of CAD support and demonstrated that there are patients and readers for whom CAD is of benefit and those for whom it is not. Keywords : Bias-variance characteristic (BVC), receiver operating characteristic (ROC), computer-aided diagnosis (CAD), decision matrix, double check.