Simulation of multi-reader multi-case study data with realistic ROC performance characteristics

Abstract

Multi-reader multi-case (MRMC) studies are widely used in assessing medical imaging and computer-aided diagnosis devices to demonstrate the generalizability of diagnostic performance to both the population of patient cases and the population of physician readers. Simulation of MRMC study data plays an important role in validating MRMC data analysis methods or sizing a pivotal study based on pilot data. The popular Roe and Metz simulation model is a linear mixed-effect model that models a human reader's latent decision variable in assessing patient's likelihood of disease as a sum of fixed modality effect, random reader effect, random case effect, random interaction effects, and a random error. The fixed effect is represented by a mean parameter and each random effect is represented by the variance parameter of a zero-mean Gaussian distribution. The purpose of this paper is to develop a method to set these parameters such that the simulated data have realistic ROC performance characteristics (mean AUC, variance components of AUC, and inter-reader/inter-modality correlations). To this end, we derived quasi-closed-form expressions to express the mean AUC and its U-statistic variance components as functions of the simulation model parameters. We then developed a numerical algorithm to solve the simulation model parameters from the mean AUC and its U-statistic variance components. Since the mean AUC and its U-statistic variance components can be estimated from real-world reader study data, simulated data have similar performance characteristics with the real-world data. Simulation studies were conducted to verify our parameter transformation algorithm.