Reader disagreement index (RDI) as an indicator of reader performance.

Abstract

e18592 Background: Blinded independent central review utilizing double read with adjudication is a preferred model by regulatory authorities to minimize reader bias in clinical trials. Monitoring of reader performance is critical to trial outcome. Adjudication rate, percent cases triggering adjudication, is commonly used for monitoring read quality. However, this does not consider how often the adjudicator agreed with a given reader (adjudicator agreement rate) or when no adjudication is required. We propose an innovative RDI to more accurately monitor reader quality. Methods: A retrospective review of adjudication data was performed for 20 clinical trials with a total of 7163 subjects (32,536 visits) using Response Evaluation Criteria in Solid Tumors (RECIST) 1.0 or 1.1. The adjudication rate, adjudication agreement rate and RDI were generated per reader per study. RDI for each reader was calculated as below, with high RDI indicating high % disagreement overall: RDI = (# cases where adjudicator disagreed with given reader ÷ Total # of all cases read) × 100 Each RDI was used to identify the discordant reader (ie, reader with the highest level of cases disagreed by the adjudicator). Results: RDI identified the discordant reader in all 20 studies, whereas adjudication rate and adjudication agreement rate identified the discordant reader in 13 and 12 of the 20 studies, respectively. In 3 studies, the reader with highest % adjudicator disagreement had neither the highest adjudication rate nor lowest adjudicator agreement rate. This reader could have been missed without the RDI. Conclusions: Though adjudication and adjudicator agreement rates are widely used to assess reader quality, these rates, individually, do not give full insight into reader performance. A high adjudication rate may not always mean poor reader performance if associated with high adjudicator agreement. Similarly a low adjudicator agreement rate may not always mean poor reader performance if associated with low adjudication rate. RDI proves to be a composite quality indicator by effectively combining adjudication and adjudication agreement rates in identifying potential outliers, and can serve as an excellent tool for identifying the discordant reader for timely intervention.