WE‐E‐304A‐07: Clinically Relevant FROC Analysis

Abstract

Purpose: While it has been established that the free‐response receiver operating characteristic (FROC) method generally yields more statistical power than the receiver operating characteristic (ROC) method in localization tasks, it is not clear if the higher power is clinically‐relevant. Clinically‐relevant power is defined as the sensitivity of an analysis method to a modality induced change in the detectability (e.g., contrast) of clinically‐relevant lesions. The purpose of this work was to develop a new clinically‐relevant figure or merit and compare the powers of ROC, traditional FROC and the new FROC method. Method and Materials: In traditional lesion‐based FROC the y‐axis is normalized to the total number of lesions and the different clinical‐relevance of lesions is not taken into account. In case‐based analysis the y‐axis is normalized to the total number of abnormal cases and the method accounts for different clinical‐relevance of lesions. A case‐based figure of merit was developed that weights the lesions according to their clinical importance, and the search‐model simulator was used to compare the statistical powers of ROC and lesion‐based and case‐based FROC for case sets with two lesions per abnormal case, one with zero weight (e.g., a benign lesion) and the other with unit weight (e.g., a high‐morbidity lesion). The modality effect was modeled by changing the contrast and probability of the lesion being found during search. Results: Unless the modality effect decreased the contrast of the unit‐weight lesions, the power for case‐based analysis was comparable to lesion‐based FROC. Either FROC method exceeded the power of ROC. Conclusion: Case‐based analysis is expected to better correlate with clinical outcome. It does not sacrifice the statistical power of traditional FROC.