Hierarchical models for ROC curve summary measures: Design and analysis of multi‐reader, multi‐modality studies of medical tests

Abstract

Comparative studies of the accuracy of diagnostic tests often involve designs according to which each study participant is examined by two or more of the tests and the diagnostic examinations are interpreted by several readers. Tests are then compared on the basis of a summary index, such as the (full or partial) area under the receiver operating characteristic (ROC) curve, averaged over the population of readers. The design and analysis of such studies naturally need to take into account the correlated nature of the diagnostic test results and interpretations. In this paper, we describe the use of hierarchical modelling for ROC summary measures derived from multi-reader, multi-modality studies. The models allow the variance of the estimates to depend on the actual value of the index and account for the correlation in the data both explicitly via parameters and implicitly via the hierarchical structure. After showing how the hierarchical models can be employed in the analysis of data from multi-reader, multi-modality studies, we discuss the design of such studies using the simulation-based, Bayesian design approach of Wang and Gelfand (Stat. Sci. 2002; 17(2):193-208). The methodology is illustrated via the analysis of data from a study conducted to evaluate a computer-aided diagnosis tool for screen film mammography and via the development of design considerations for a multi-reader study comparing display modes for digital mammography. The hierarchical model methodology described in this paper is also applicable to the meta-analysis of ROC studies.