Perceptually Based FROC Analysis 1

Abstract

Analysis of reading data when cases have multiple targets and/or the reader is required to localize targets is difficult. One approach to this free-response operating characteristic (FROC) problem is for images to be segmented (eg, with quadrants) by the investigator and a segment-level analysis be conducted with the case as a nesting factor. In this report, we introduce an alternative method that uses the visual scan path of the reader to segment the image. We evaluate the new method by applying it to data from a mammography reading experiment. The gaze scan path of one radiologist was recorded as she scanned 40 mammograms for masses and microcalcifications. The observer is an experienced mammographer and was not one of the authors. In addition, the reader provided a rating indicating the degree of suspicion for any suspected targets she identified and localized. We then established "perceptual regions" by using a clustering algorithm on the visual fixations. We combined ratings given to specific locations indicated by the reader with the segmentation from the visual scan to generate a series of ratings classified for whether the perceptually based region associated with the rating contained or did not contain a known target. We analyzed data generated by our method from all 40 cases by using the conventional maximum-likelihood method based on the binormal model. Finally, we tested goodness-of-fit of the binormal model to the data by using chi-square. Maximum-likelihood estimation led to a model that did not fit the data (P < .001). However, examination of the observed and expected counts suggests that the binormal assumption does not hold for segments that contain targets and a bimodal distribution model might be preferred. Our new method provides an alternative approach to analysis of the FROC experiment. It needs to be developed further. Specifically, we propose that a mixture model extension of the binormal model be developed for ratings data arising from perceptually based FROC experiments. A disadvantage to our method is the requirement to record the scan path of the reader. However, we believe that adding such information to receiver operating characteristic (ROC) curve analysis will pay off when appropriate statistical models have been identified because we believe our data support our hypothesis that the perceptual scanning of images by humans deconvolves interpretation correlation. If true, this hypothesis implies that conventional statistical methods for ROC analysis based on independent data can be applied to the analysis of FROC data after conditioning on the scan path of the observer.