Teaching Receiver Operating Characteristic Analysis: An Interactive Laboratory Exercise

Abstract

Despite its fundamental importance in the evaluation of diagnostic tests, receiver operating characteristic (ROC) analysis is not easily understood. The purpose of this project was to create a learning experience that resulted in an intuitive understanding of the basic principles of ROC analysis. An interactive laboratory exercise was developed for a class about radiology testing taught within a clinical epidemiology course between 2000 and 2009. The physician students in the course were clinical fellows from various medical specialties who were enrolled in a graduate degree program in clinical investigation. For the exercise, the class was divided into six groups. Each group interpreted radiographs from a set of 50 exams of the peripheral skeleton to determine the presence or absence of an acute fracture. Data from the class were pooled and given to each student. Students calculated the area under the ROC curve (AUC) corresponding to overall class performance. A binormal ROC curve was also fitted to the data from each class year. The laboratory exercise was conducted for 8 years with approximately 20-30 students per year. The mean AUC over the eight laboratory classes was 0.72 with a standard deviation of 0.08 (range, 0.60-0.85). With some simplifications in design, an observer study can be conducted in a laboratory classroom setting. Participatory data collection promotes the intuitive understanding of ROC analysis principles.