<title>Some methodological questions concerning free-response receiver operating characteristic analysis as a method for assessing image quality in radiology</title>

Abstract

McLean, Virginia 22102-3481ABSTRACT'This paper raises methodological questions concerning Free-Response Receiver Operating Characteristic(FROC) analysis as used to judge the quality of images in radiology. This paper has three purposes: describingFROC analysis as used in the literature; discussing whether FROC methods achieve their objectives; andidentifying several methodological shortcomings in FROC at the present state of the art that must be solved ifFROC is to be fully useful.1. INTRODUCTIONThe use of ROC analysis (of which FROC is a subset) to evaluate the quality of medical images is steadilyincreasing.1' li, 14, 22 Today, FROC is the principal technique using human observers for testingcompeting medical imaging modalities. FROC analysis seeks to elicit expert radiological opinion concerningthe quality of radiological images in an objective, valid, replicable way. In one of its most importantapplications, FROC analysis is used to determine which of two or more imaging modalities is superior bymeasuring the ability of observers to identify pathologies using each modality. If observers obtain better FROCcurves (see figure 1) with one modality than with others, the imaging modality used to get the better curve (i.e.,nearer the upper left hand corner of figure 1) produces the beuer images.2. SEPARATING THE OBSERVER FROM THE OBSERVED USING FROCFROC seeks to separate the inherent ability of an imaging modality to portray various states of disease andhealth from the balance struck (consciously or otherwise, by radiologists between false positive and false