A Critical Appraisal of the Risk of Bias in Systematic Reviews and Metaanalyses Pertaining to COVID-19

Abstract

Background: We critically evaluated the risk of bias in published systematic reviews (SRs) and meta-analyses (MAs) pertaining to COVID-19 using ROBIS tool. Materials And Methods: MEDLINE and Cochrane Central Library were searched for SRs/MAs on 14th May 2020, including studies of all designs describing various facets of COVID-19 in humans; no restrictions were applied for interventions, comparators, and outcomes. Two reviewers independently assessed all the SRs/MAs with ROBIS. Results: Out of 204 identified records, 48 SRs/MAs were included. The most frequently reviewed topics were therapy outcomes, diagnosis, and comorbidities (15, 8, and 6 papers respectively). Only 29/48(60.41%) papers had made a mention of using PRISMA or other guidelines for drafting the SR/MA. Only 5/48(10.42%) of all included SRs/MAs had low overall risk of bias as per ROBIS tool; 41/48(85.42%) had high risk of bias, 2/48(4.17%) had unclear risk of bias. The highest proportion of bias was found in data synthesis and findings (30/48, 62.50% of studies had high risk of bias), followed by study identification and selection (29/48, 60.42%). The IRR for methodological quality assessment was substantial, with the Cohen’s kappa values being 0.64, 0.68, 0.62, and 0.75 for domains 1-4 of ROBIS tool, and 0.66 for overall risk of bias assessment. Conclusion: There are serious concerns about the methodology employed to generate the results of the SRs/ MAs pertaining to COVID-19, with ‘quantity’ seemingly being given more importance than ‘quality’ of the paper.