Effect of Case Sample Difficulty and Choice of Rating Scales on Interobserver Variability in Receiver Operating Characteristic Experiments

Abstract

We evaluated the effect of case sample difficulty in receiver operating characteristic (ROC) experiments using both discrete and continuous rating scales, and by analyzing data from the same set of observers at three different levels of image-reading difficulty. Simulated computed radiographic mammograms were used as case samples. Our experimental results show that the averaged Az values were essentially identical with both rating scales. When we used the continuous scale, the significance levels tended to be lower than those with the discrete scale, raising the possibility that tests based on the continuous scale may be more statistically valid.