Receiver Operating Characteristic Studies Research Articles

Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data.

Receiver operating characteristic (ROC) methodology is widely used in evaluating medical imaging modalities. While appropriate in some cases, it has several drawbacks when the detection task, e.g., nodule detection, involves localizing the abnormality. Free-response receiver operating characteristic (FROC) methodology offers a more natural framework to describe observer performance in such studies and has other advantages. Due to the lack of a statistical analysis procedure comparable to the maximum likelihood procedure (ROCFIT program) available for ROC studies, the FROC method has not gained widespread acceptance. This work presents and solves a two parameter model for the statistical analysis of FROC data. The model assumes that the probability density of the signal stimuli is normally distributed, as is the probability density for producing one or more false positives per image. A program (FROCFIT) is described for estimating the parameters and their uncertainties from experimental data. An index of performance is proposed to quantify observer performance in FROC experiments. Application of this methodology to several FROC data sets produced good to excellent fits.

Some practical issues of experimental design and data analysis in radiological ROC studies.

Receiver operating characteristic (ROC) analysis has been used in a broad variety of medical imaging studies during the past 15 years, and its advantages over more traditional measures of diagnostic performance are now clearly established. But despite the essential simplicity of the approach, workers in the field often find--sometimes only after an ROC study is under way--that a number of subtle issues related to experimental design and data analysis must be confronted in practice. Many of these issues have not been discussed in the literature in detail, and most are not well known. The purposes of this paper are to make users of ROC methodology in medical imaging aware of potential problems that should be confronted before an ROC study is begun and to indicate, at least broadly, how those problems may be dealt with, given the present state of the art. Some of the issues raised here can be addressed adequately by easily prescribed techniques, whereas others remain difficult and will be resolved fully only by new methodologic developments.

Digital Mammography

We investigated the spatial resolution requirement and the effect of unsharp-mask filtering on the detectability of subtle microcalcifications in digital mammography. Digital images were obtained by digitizing conventional screen-film mammograms with a 0.1 X 0.1 mm2 pixel size, processed with unsharp masking, and then reconstituted on film with a Fuji image processing/simulation system (Fuji Photo Film Co., Tokyo, Japan). Twenty normal cases and 12 cases with subtle microcalcifications were included. Observer performance experiments were conducted to assess the detectability of subtle microcalcifications in the conventional, the unprocessed digital, and the unsharp-masked mammograms. The observer response data were evaluated using receiver operating characteristic (ROC) and LROC (ROC with localization) analyses. Our results indicate that digital mammograms obtained with 0.1 X 0.1 mm2 pixels provide lower detectability than the conventional screen-film mammograms. The detectability of microcalcifications in the digital mammograms is improved by unsharp-mask filtering; the processed mammograms still provide lower accuracy than the conventional mammograms, however, chiefly because of increased false-positive detection rates for the processed images at each subjective confidence level. Viewing unprocessed digital and unsharp-masked images in pairs resulted in approximately the same detectability as that obtained with the unsharp-masked images alone. However, this result may be influenced by the fact that the same limited viewing time was necessarily divided between the two images.