Reader Disagreement Index: a better measure of overall review quality monitoring in an oncology trial compared to adjudication rate

Abstract

Purpose: Blinded Independent Central Review (BICR) is highly recommended by regulatory authorities for oncology registration trials. “Adjudication rate” provided by “Two Reviewers and Adjudicator Paradigm” of BICR has been part of reviewer performance metrics and trial efficacy. However, adjudication rate does not consider the adjudicator agreement or disagreement rate of a reviewer. We analyzed that Reader Disagreement Index (RDI) is a better measure than adjudication rate to monitor reviewer performance in BICR. Methods: BICR adjudication data from 3 different clinical trials including 10 board-certified radiologist reviewers using Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria was analyzed. RDI for each reviewer was calculated using the below mentioned formula. Reviewer adjudication rate and adjudicator agreement rate was calculated for each reviewer along with RDI. RDI was used to identify the discordant reviewer with highest disagreement rate. Number of cases where adjudicator disagreed with given reader RDI (%) = Total number of cases read ×100 Results: RDI identified the discordant reviewer in all 3 studies. Discordant reviewers identified using RDI were not the reviewers with highest adjudication or lowest agreement rates. Adjudication rate identified the discordant reviewer in 1 of the 3 studies. Reviewer with lowest adjudicator agreement could not have been identified as discordant reviewer using only adjudication rate in monitoring reviewer performance. RDI is more robust in identifying a discordant reviewer who neither has highest adjudication nor lowest agreement rate. Conclusions: RDI is more reliable measure of reviewer performance as compared to adjudication rate and could be effectively used to monitor reviewer performance as it combines both reviewer adjudication percentage and adjudication agreement percentage.