Meta-Analyses Do Not Establish Improved Mortality With Ivermectin Use in COVID-19

Abstract

We thank Bryant et al1 for their reply and appreciate their intent to study an inexpensive drug with potential to be repurposed during this pandemic. We recognize their self-published protocol and attempt to register with the Cochrane Collaboration.2 Inclusion of a preferred reporting items for systematic reviews and meta-analyses checklist would have obviated mention of that item. The authors seem to be mistaken about media reports within our commentary.1 Our citations regarding off-label ivermectin use only included the New England Journal of Medicine, the Center for Disease Control Health Advisory Network, the National Institutes of Health, and the Food and Drug Administration.3–5 These acknowledgments aside, Bryant's letter serves as an opportunity to highlight important and potentially controversial considerations when performing meta-analyses.1 As we demonstrate, decisions regarding these issues can dramatically influence the results of systematic reviews and meta-analyses. INDIVIDUAL STUDY CONDUCT AND QUALITY Individual study evaluation Critical evaluation of individual studies is an important step in assessing validity of a meta-analysis. This includes identifying whether studies used the appropriate reporting guidelines (eg, CONSORT/STROBE).6 Guideline use is associated with increased transparency and improved methodological and reporting quality and has been described as “central to application of findings from research to clinical practice.” 7–11 Only 2 of 14 mortality randomized controlled trials (RCTs) within Bryant's meta-analysis used the reporting guidelines subjecting these studies to potential biases related to allocation, randomization, confounding, and blinding issues described below.6,7,9,12–16 Ideally, poor quality articles are identified during risk of bias (RoB) assessment. Two studies (ie, Elgazzar and Niaee) with data irregularities and statistical errors in the initial meta-analysis by Bryant et al were not identified as having a high RoB within any domain.12,17,18 RoB quality assessment can be subjective as evidenced by the 4 overlapping studies within Bryant's and Popp's Cochrane mortality meta-analyses having poor interrater agreement among RoB domains (raw agreement 40%, kappa 0.02, and 95% confidence interval −0.23 to 0.26).12,19 Others have noted similar RoB appraisal inconsistencies within Cochrane systematic reviews with only poor-to-fair interrater reliability (kappa 0.13–0.34) for all domains except allocation concealment and sequence generation.20,21 It is not clear that RoB assessment alone can adequately detect poor quality articles. Several studies in Bryant's meta-analysis had deficiencies not identified during RoB quality assessment.13 Previously, we noted that none were constructed to detect mortality differences, 8 failed to describe basic comorbidity differences between groups, 7 had inadequate random sequence generation or allocation concealment, and many contained inappropriate statistics.3 Although these study-level issues did not always affect RoB quality assessment, they indicate that many studies had an inadequate design, conduct, or methodology. As noted below, many of these deficiencies could be prevented using a strict population, intervention, comparison, outcome, and study design (PICOS) format. Active controls We disagree with Bryant's opinion that active controls might “under-stat(e) ivermectin's efficacy” and not skew results in favor of ivermectin.1 Three studies comparing hydroxychloroquine/chloroquine (HCQ/CQ) as active controls with ivermectin were included in their repeat meta-analysis.13,18,22,23 Those studies contained 65% (42/65) of control deaths despite comprising only 23% (212/938) of control cases.13 We have identified 17 meta-analyses concluding HCQ/CQ increased mortality in COVID-19, potentially worsening control outcomes, and overstating ivermectin's efficacy.24–40 Only 3 meta-analyses indicated reduced mortality with HCQ/CQ, potentially favoring control outcomes.41–43 Importantly, the Food and Drug Administration and European Medicines Agency recommend using active controls that are standard of care and already approved for the studied indication which is not the case for HCQ/CQ.44–46 Blinding Cochrane Collaboration authors and Bryant et al suggest blinding is less important in trials analyzing mortality.1,47 However, inadequate blinding can affect mortality rates with treatment effects significantly larger in studies that are unblinded or have an unclear or high RoB related to blinding.48–51 An exaggerated mortality benefit, favoring treatment, has been noted across Cochrane intervention systematic reviews and in studies analyzing treatment for acute pulmonary emboli, acute myocardial infarction, and critical illness when blinding is inadequate.48–51 Ten studies in Bryant's meta-analysis had an unclear or high blinding RoB potentially favoring ivermectin.12 Removing the 2 studies with a high RoB (Bryant, Figure 2) within blinding domains renders mortality differences between ivermectin and controls insignificant, risk ratio (RR) 0.5 (0.22–1.16).12,13 Children Bryant et al1 noted that “around” 0.4% of study subjects were children. Children comprised 13% of the patients in one study and an unknown number in another study potentially suppressing mortality rates in these studies.52,53 The preferred reporting items for systematic reviews and meta-analyses for children (PRISMA-C) guidelines recommend separately reporting pediatric events with consideration of unique developmental aspects, metabolism, pharmacology & safety, result applicability to pediatric age groups, and performance of pediatric subgroup analyses, none of which was performed in this meta-analysis.12,13,54–56 Bryant et al combined adult and pediatric outcomes into the same summary findings without mentioning different ages within results, discussion, or conclusion. This might lead readers to interpret their meta-analysis as supporting a treatment in children, despite children comprising a small minority of total subjects.12,13 META-ANALYSIS ISSUES Article selection Article selection and framing of a study question (PICOS format) is an important determinant of the magnitude and direction of meta-analysis results. In their meta-analysis, Popp et al19 used more restrictive PICOS criteria: P (no COVID-19 negative patients and no children), I (no coadministered medicines), C (no active controls), O (28-day mortality vs. variable time limit), and S (no quasi-RCTs). Because they used a more selective PICOS framework to select articles and address ivermectin's effect on mortality, it is not surprising that Popp's meta-analysis came to a different conclusion than Bryant's meta-analysis.12,13,19 Small studies Although Bryant et al believed that small studies do not constitute a bias in meta-analyses, others have described small-study effects as a “major threat” to their validity.1,57 Small studies have more exaggerated effects on results and are more likely to be selected for publication if statistically significant and more frequently exhibit methodological flaws.57–61 The funnel plot constructed to analyze publication bias was described as symmetric in Bryant's meta-analysis.12 Visual funnel plot inspection is subjective with trained individuals incorrectly interpreting 47.5% of all funnel plots.62 Contrary to Bryant's interpretation, smaller studies seem clustered to the left in their funnel plot (Figure 1A, and Bryant Figure 7) indicating unpublished small studies showing no benefit to ivermectin, spurious inflated effects due to poor methodology, data irregularities, chance, or true heterogeneity.12,60,63 Egger regression of their initial meta-analysis suggested potential small study bias (Figure 1A, B). After deleting studies by Elgazzar and Niaee, this bias persisted (Figure 2A). Trim and fill analysis excluding studies by Elgazzar and Niaee indicated 3 missing small studies with a mortality adjusted RR (0.84, 95% confidence interval 0.48–1.46) (Figure 2B). Although the true RR may differ from the adjusted result, this calculation serves as a sensitivity analysis indicating publication bias may be substantial.64FIGURE 1.: (A) Funnel plot of original data, ivermectin versus control all-cause mortality, with no exclusion of studies, using log risk ratio (logRR) for x axis instead of RR. Note that no continuity correction for zero cell events made in the original figure (Figure 7) and studies and subsets with zero events in both study arms appear to be excluded leaving 11 studies within this figure and original figure.12,52,71–75 (B) Funnel plot with trim and fill for original data set excluding no studies. Four imputed missing studies to the right (•). Adjusted RR 0.59 (0.30–1.16). Egger regression intercept −1.4 (−2.97 to 0.17), P = 0.074 (2 tailed). P < 0.1 considered significant by Egger and others analyzing heterogeneity in meta-analyses.60,76–79FIGURE 2.: (A) Funnel plot of original data excluding studies by elgazzar and niaee.12 (B) Funnel plot with trim and fill excluding studies by Elgazzar and Niaee.12 Three imputed missing studies to the right (•). Adjusted RR 0.84 (0.48–1.46). Egger regression intercept −1.07 (−2.3 to 0.15), P = 0.076 (2 tailed). No continuity correction for zero cell events was made. If a continuity correction adding 0.5 to each zero event cell for studies with zero events in each arm is made, 3 missing studies would still be imputed and Egger regression intercept would be −0.55 (−1.14 to 0.04), P = 0.067 (2 tailed). P < 0.1 considered significant by Egger and others analyzing heterogeneity in meta-analyses.60,76–79Independent patient data Bryant et al1 make an important point that independent patient data (IPD) rely on responsiveness of individual authors and can take more time. Despite these drawbacks, IPD meta-analyses are the “gold standard” of meta-analysis, “improve the quality of data,” and “produce more reliable results.”65 An IPD meta-analysis could have detected “apparent scientific fraud” and avoided publication delays while reworking their meta-analysis.17,66 Another study with impossible data points and miscalculated statistical tests might also have been identified.3,18 A third study described patients with illness duration subsets adding up to more patients than enrolled in the entire study, whereas a fourth study miscounted/miscalculated patient's sex, ethnicity, covariates, and treatments.23,67 These discrepancies indicate potential for more serious issues identifiable using IPD. Recently, Hill et al68 acknowledged “potential fraud” in 2 studies and re-evaluated others with a high RoB from their initial meta-analysis. Their reworked meta-analysis found no effect of ivermectin on mortality after deleting problematic studies.68 Because of these issues, Hill et al concluded “it is essential that access to patient-level databases is provided.” 68 Others recommend reviewing IPD in ivermectin meta-analyses “by default” after finding studies “contain(ed) impossible numbers,” “unexplainable (data) mismatches,” “timelines not consistent with veracity of data,” and “duplication of blocks of patient records.”69 FUTURE DIRECTION An important question regarding meta-analyses remains: how can researchers ensure the quality and trustworthiness of these types of studies in the future? The Cochrane Collaboration has provided detailed instructions on the performance of meta-analyses and review of composite articles.70 Further steps are needed to obtain reliable results from systematic reviews and meta-analyses for interventions. Ideally, included studies should follow the reporting guidelines. Double blinding should be mandatory for RCTs evaluating mortality, active controls should only comprise drugs approved by regulatory agencies for the studied disorder, adult and pediatric cases should be evaluated separately, stricter PICOS format is needed, and a statistical evaluation for small study effects should be performed. To improve transparency, included studies should make deidentified data available for independent review and if not provided “should be considered at high RoB or excluded.”69 This would allow for investigation of data irregularities and performance of IPD meta-analyses. SUMMARY The 20 meta-analyses highlighted in our prior commentary contained studies with multiple defects previously outlined.3 Data irregularities have led to retraction of several of these studies.68,69 Without access to data contained within any study, it is impossible to know whether that data are reliable. Removing the retracted Elgazzar article and the flawed article by Niaee et al from the meta-analysis by Bryant et al and several other meta-analyses alters their results with ivermectin no longer showing a mortality benefit in COVID-19.3 We believe this should be a wake-up call to more rigorously evaluate and select individual studies, insist authors allow more in-depth access to data contained within individual studies, and promote IPD analyses as the gold standard for meta-analyses of interventions in medicine.