Comparative Analysis of Data Collection Methods for Individualized Modeling of Radiologists' Visual Similarity Judgments in Mammograms

Abstract

We conducted an observer study to investigate how the data collection method affects the efficacy of modeling individual radiologists' judgments regarding the perceptual similarity of breast masses on mammograms. Six observers of varying experience levels in breast imaging were recruited to assess the perceptual similarity ofmammographic masses. The observers' subjective judgments were collected using (i) a rating method, (ii) a preference method, and (iii)ahybrid method combining rating and ranking. Personalized user models were developed with the collected data to predict observers' opinions. The relative efficacy of each data collection method was assessed based on the classification accuracy of the resulting usermodels. The average accuracy of the user models derived from data collected with the hybrid method was 55.5±1.5%. The models were significantly more accurate (P < .0005) than those derived from the rating (45.3±3.5%) and the preference (40.8±5%) methods. Onaverage, the rating data collection method was significantly faster than the other two methods (P < .0001). No time advantage was observed between the preference and the hybrid methods. A hybrid method combining rating and ranking is an intuitive and efficient way for collecting subjective similarity judgments to model human perceptual opinions with a higher accuracy than other, more commonly used data collection methods.