SU‐E‐I‐117: Quantifying Search and Classification Performance in Lesion Localization Tasks

Abstract

Purpose: Search involves identifying the locations of potential lesions. Classification involves determining if an identified region is a lesion. The area (AUC) under the ROC curve is affected by both search and classification performance. The purpose of this work is to suggest a method for quantifying the two distinct contributions. Methods: The search model for free‐response data involves sampling from noise and signal site unit variance normal distributions. It predicts ROC curves that extend from (0,0) to (x,y), where (x,y) is to the lower left of (1,1). The quantity (y‐x) is proposed as a measure of search performance. It equals the perpendicular distance from (x,y) to the chance diagonal multiplied by the square root of 2. The probability that a sample from the signal distribution is greater than a sample from the noise distribution is proposed as a measure of classification performance. AUC performance is the probability that the highest sample from an abnormal image is greater than the highest sample from a normal image. Search, classification and AUC performance were calculated for mammographer data from a FROC study in which 5 radiologists interpreted 96 normal and 89 disease‐present cases in breast tomosynthesis (BT) and digital mammography (DM) modalities. Results: Search performance ranged from zero to 0.65, classification performance ranged from 0.64 to 0.98 and AUC performance ranged from 0.68 to 0.88. AUC correlated positively with search performance while classification performance was inversely correlated with search performance. All three measures were greater for the BT modality. Conclusions: The method, which is applicable to FROC or ROC data, allows separation of physically distinct contributions to AUC, and consequent identification of the weak link. The preliminary results suggest that search is generally the weak link but that radiologists may learn to compensate for poor search performance with better classification performance.This work was supported by grants from the Department of Health and Human Services, National Institutes of Health, R01‐EB005243 and R01‐ EB008688.