PRM225 - Evaluation of Quality Assessment Tools for Non-Randomised Controlled Trials Assessing Surgical Interventions: A Systematic Review of Systematic Reviews

Abstract

Evaluating the effectiveness and safety of surgical techniques, operations, and involved medical devices is relatively difficult, sometimes unethical, with randomised controlled trials (RCTs). Non-randomised designs are commonly applied and used to inform decision making. Quality assessment (QA) methods for these studies have previously been reviewed, but not specifically for their applicability to non-RCTs in surgical interventions. The objectives of this systematic review were to evaluate which QA tools have been used in this research field and to critically appraise these tools. We systematically searched three electronic databases (MEDLINE, Embase and Cochrane Library) and Health Technology Assessments. Systematic reviews appraising the quality of non-RCTs on surgical interventions were included. In total, 1,741 potentially relevant citations were identified. After removing duplicates, 1,525 citations were screened. Of these, 159 full text references were reviewed and 85 systematic reviews met predefined inclusion criteria. Five QA methods were most commonly employed: Newcastle-Ottawa Quality Assessment Scale (NOS) or modified NOS (28%); checklists developed by authors (15%); the Cochrane checklist or modified version (11%); modified checklists from other authors (5%); applying disease-specific QA tools (5%). The reliability and applicability of the most commonly employed tool in this research field, NOS, were questioned in included reviews, corresponding with concerns on the validity of the NOS reported in recently published literature of research methodology. The available evidence demonstrates a lack of consensus on the use of QA tools for non-RCTs assessing surgical interventions. Various methods have been adapted or newly developed by researchers, and the most commonly applied QA tool (NOS) may not be fit for purpose in this field of research. There is an urgent need for a validated QA tool to appraise the quality of evidence to help inform evidence-based decision making on the use of surgical devices and types of surgical approaches.