The application of AMSTAR 2 in overviews of systematic reviews shows a need for improvement

Abstract

Abstract Background Overviews of systematic reviews (SR) summarise and appraise the quality of research relevant for evidence-based decision-making. ‘A Measurement Tool to Assess Systematic Reviews' (AMSTAR 2) is a valid and moderately reliable appraisal instrument with 16 items that can be used to derive the overall confidence rating (OCR). OCR (high, moderate, low or critically low) can be derived based on the original algorithm with 7/9 critical/non-critical items. We aimed to assess the methods for deriving OCR in overviews of SRs of clinical interventions. Methods Our cross-sectional study was conducted using 45 overviews of SRs with at least 20 AMSTAR 2 appraisals selected by two authors from a title/abstract search for ‘AMSTAR 2' in Medline, Epistemonikos and CINAHL in 09/2020. One author coded the data. OCR methods and outcomes were compared using descriptive and univariate statistics. Results The 45 overviews were published in 2018-2020 by 2-16 authors. The overviews appraised on average 44 SRs (range: 20-158). OCRs of 1.412 SRs were derived in 41 overviews (91%). OCR were derived according to the original algorithm (22; 54%), own algorithm (9; 22%), unspecified algorithm (9; 22%) or AMSTAR 2 website (1; 2%). The OCR were high (190; 13%), moderate (182; 13%), low (267, 19%) or critically low (773; 55%). The original algorithm was more likely to generate the critically low+low ratings than either own algorithms (OR = 2.28, 95% CI [1.66-3.13]) or unclear algorithms (OR = 3.78 [2.81-5.09]). Conclusions High quality SRs are needed for making public health decisions. AMSTAR 2 assigns mostly low and critically low OCR to SRs in various clinical areas. These SRs should not be relied on as a source of accurate evidence. Furthermore, our results show that SR reporting guidelines need to be better followed and that AMSTAR 2 users should explicitly report their OCR methods. Key messages The methodological quality of the majority of systematic reviews (SRs) in various clinical fields is disappointing, highlighting the requirement for better adherence to reporting guidelines. There is a need for improvement on reporting how AMSTAR 2 is applied in overviews of SRs.