Local Evaluation and Blinded Central Review Comparison: A Victim of Meta-Analysis Shortcomings.

Abstract

The meta-analysis conducted by Zhang et al describes an important debate on the validity of replacing blinded independent central reviews (BICRs) with local evaluations (LEs). The meta-analysis uses published and publicly available summary data to analyze LE and BICR agreement, but the studies used are inappropriate for testing their hypothesis. The analysis is based on the strong premise that agreement was due to the equivalent and independent LE and BICR evaluations. Although the studies were not identified, it is understood that many of the studies from Amit et al were also used in this meta-analysis. A review of those studies revealed that only 11 of the 27 studies could reasonably be concluded to be independent LE-BICR comparisons. Five studies described shared information between LE and BICR readers (eg, ‘‘ . . . blinded clinical tumor assessments were provided to the IRC.’’), and 8 studies described the BICR as confirming LE progression. In confirmation of progression studies, when an LE evaluates a patient as Progressive Disease, the images are sent to a BICR radiologist to confirm progression or to return the patient to the study until confirmation or until the local oncologist removes the patient for clinical deterioration (ie, censored). Agreement is then mandated by the protocol, not by equivalent performance. Other studies compared BICR radiologists to highly experienced academic radiologists at a major center whose performance is almost certainly not typical of local radiologists who likely receive no training in the use of formal response. One study, Trial 28, the C2325 (carcinoid) everolimus study, had such discordant results between LEs and BICRs at the second interim analysis that the study protocol was amended due to what was described by the FDA as a probable LE bias toward efficacy, ‘‘making (the results) uninterpretable.’’ The LE hazard ratio was dramatically closer to the BICR at the end of the trial (0.78 vs 0.93, respectively) and almost certainly due to the regulatory concern than to a sudden regression to the mean. A comparison at the second interim clearly tells a much more accurate story. For the 11 studies in which the BICR versus LE comparison appeared to be valid, there was good agreement over a broad range of hazard ratios. However, the other studies are misleading, with agreement based on the strong dependence of one reader on the other in which agreement was defined as the end point. Additionally, study sponsors are unlikely to publish the results of a study in which the LE and BICR results are contradictory, and valid conclusions on LE and BICR agreement cannot assume that a publication bias can just be ignored, a common criticism of meta-analyses. The comparison of LEs and BICRs needed to be revisited following the regulatory decisions from 2 decades ago, designed to prevent fraudulent data submission, but the conclusions so far are founded on a set of studies that, by their very nature, ensure agreement. Although this meta-analysis is thorough, well done, and interesting, it draws the wrong conclusion and cannot be the foundation for any clinical trial design decisions.