Comparison of receiver operating characteristic and forced choice observer performance measurement methods

Abstract

The receiver operating characteristic (ROC) method has been successfully used in medical imaging for 20 years. It has been so successful that many people think of it as an end in itself rather than just one of several ways to assess human observer performance of image based decision tasks. Studies of human and ideal observer decision performance are designed to estimate a measure of observer performance (e.g., efficiency, d' or da). The experimenter would like to obtain accurate and precise estimates using a relatively small number of images or decision trials because of a variety of constraints. One purpose of this paper is to introduce medical physicists to another effective psychophysical measurement technique, the forced choice method. The second purpose is to present a comparison of the forced choice and ROC methods, with particular attention to sampling statistics considerations. In brief, the rating scale ROC method is preferable when the limiting constraint is the number of images and at the same time it is not feasible to use the forced choice with more than four alternatives. The forced choice method can be superior for experiments using synthetic images, under some conditions.