JAFROC analysis revisited: figure-of-merit considerations for human observer studies

Abstract

ABSTRACT Jackknife alternative free-response receiver operating characteristic (JAFROC) is a method for measuring human observer performance in localization tasks. JAFROC is be ing increasingly used to evaluate imaging modalities because it has been shown to have greater statistical power than conventional receiver operating characteristic (ROC) analysis, which neglects location information. JAFROC neglects the no n-lesion localization marks (false positives) on abnormal images. JAFROC1 is an alternative method that includes these marks. Both methods are lesion-centric in the sense that they assign equal importance to all lesions; an image with many lesions would tend to dominate the performance metric, and clinically less significant lesions are treated identically as more significant ones. In this paper weighted JAFROC and JAFROC1 analyses are described that treat each abnorm al image (not each lesion) as a unit of measurement and account for different lesion clinical significances (weights). Lesion-centric and weighted methods were tested using a simulator that includes multiple-reader multiple-case multiple-modality location level correlations. For comparison, analysis was also tested where the rating of the highest rated mark on an image was assumed to be its ROC rating. The testing involved random numbers of lesions per image, random weights, case-mixes (ratio of normal to abnormal images) and different correlation structures. We found that for either JAFROC or JAFROC1, both lesion-centric and weighted analyses had correct NH behavior and comparable statistical powers. For either lesion-centric or weighted analyses JAFROC1 yielded the highest power, followed by JAFROC and yielded the least power, confirming a recent study using a less flexible single-reader dual-modality simulator. Provided the number of normal cases is not too small, JAFROC1 is the preferred method fo r analyzing human observer free- response data. For either JAFROC or JAFROC1 weighted analysis is preferable. Keywords: Free-response, imaging system assessment, JAFROC, JAFROC1, simulators, lesion localization, lesion-centric analysis, weighted analysis