Meta-Analyses Based on Abstracted Data: A Step in the Right Direction, but Only a First Step

Abstract

“Only what is exhaustive is of interest. The truth comes from an accumulation of details.” Thomas Mann (1875-1955) Despite the development of new drugs, platinum compounds remain a major component of most chemotherapy regimens in patients with advanced non–small-cell lung cancer (NSCLC). In this issue of the Journal of Clinical Oncology, Hotta et al use a meta-analytic approach to compare cisplatin and carboplatin in NSCLC. In their analysis of eight trials and 3,000 patients, they find cisplatin to be superior to carboplatin in terms of tumor response rates (odds ratio 1.36; 95% CI, 1.15 to 1.61; P .001), but not in terms of survival (hazard ratio 1.05; 95% CI, 0.91 to 1.22; P .51). In a subgroup analysis of five trials, they report that—when combined with taxanes or gemcitabine—cisplatin may be superior to carboplatin in terms of overall survival (hazard ratio 1.11; 95% CI, 1.01 to 1.22; P .039). NSCLC is one of the most common cancers in the world, and the choice of platinum agents deserves particular attention, since the substitution of cisplatin by carboplatin may have important implications, not only in terms of drug administration, but also in cost. To address this very important issue, Hotta et al have performed a metaanalysis based on abstracted data, without retrieval of individual patient data (IPD). In other words, their work is based on a re-analysis of published data, tables, and survival curves rather than on an exhaustive collection, checking, and analysis of data of individual patients included in all trials. In addition to not restricting the trial search to full papers, the approach they used, which consists of extracting data from publications, has severe limitations. By definition, meta-analyses based on abstracted data must deal with already-performed analyses. The reliability of the randomization methods cannot be evaluated, the data cannot be checked, the original analyses cannot be redone, and additional analyses cannot be performed. Authors of meta-analyses based on abstracted data often assess the quality of individual studies included in their analyses, and weigh each trial in the meta-analysis in proportion to its quality. However, no quality assessment system can substitute for the use of individual patient data in guaranteeing the absence of bias in the original analyses. Deliberate errors are quite rare, especially in multicenter randomized clinical trials, but random errors and biases in the reported results are not. Randomization procedure and absence of patient exclusion are key features for the reliability of trial results. In the set of trials considered by Hotta et al, for example, 5% of the patients were excluded from the tumor response analysis, but this proportion reached 15% in one trial, and in four of eight trials, survival analyses were not based on intent to treat. Exclusion of patients from analyses may be related to patient ineligibility (which creates no bias), but may also be related to treatment outcome (in which case bias may be present). Such bias can be more serious when a report does not even mention randomized but not treated or ineligible patients who have been excluded from all analyses. By contrast, the collection of IPD allows including all randomly assigned patients in the analyses. IPD meta-analyses may also reveal inadequate randomization procedures, such as nonrandomized patients included in the analysis of a randomized trial, a practice that has been documented in some IPD meta-analyses. Meta-analyses based on abstracted data do not allow updating of patient outcomes, which are frozen in time. Such an update, usually done in IPD meta-analyses, is crucial in the adjuvant setting, but can also be important in advanced stage studies when a small but clinically and statistically significant survival difference may matter. Analyses based on IPD represent both the only reliable and the most powerful method to calculate and compare times to tumor recurrence or death. Extrapolations based on JOURNAL OF CLINICAL ONCOLOGY E D I T O R I A L VOLUME 22 NUMBER 19 OCTOBER 1 2004