The classification, prevalence and impact of selective inclusion and reporting of results in systematic reviews

Abstract

BACKGROUND Systematic reviews of randomised controlled trials (RCTs) of health care interventions can have a major impact on clinical practice, research agendas and health care policy-making. However, the value of systematic reviews is compromised when results of the review are biased. Two sources of bias that have received limited attention are selective inclusion and selective reporting of results in systematic reviews. Selective inclusion of results occurs when the selection of trial effect estimates to include in a meta-analysis is based on the nature of the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting of results occurs when the reporting of a subset of outcomes or analyses in a systematic review is based on the nature of the results. AIM The aim of this thesis was to collate and classify examples of selective inclusion and reporting of results in systematic reviews of RCTs of health care interventions, investigate the prevalence of these practices, and explore their potential impact on meta-analytic effects. METHODS Several methods were used in this thesis. Systematic review methods were used to collate examples of selective inclusion and reporting of results. Categorisation of examples was undertaken to generate a taxonomy of scenarios. Systematic review methods and meta-analysis were used to synthesise results of empirical studies that have investigated different types of selective reporting of results in systematic reviews. An empirical study was undertaken to investigate, in a sample of 44 systematic reviews of RCTs, (i) the extent to which systematic reviewers encountered multiple trial effect estimates that were available for inclusion in a particular meta-analysis (for example, from multiple scales, time points); (ii) whether there was evidence of selective inclusion of trial effect estimates; and (iii) the potential impact of selective inclusion on meta-analytic effects. RESULTS From an analysis of 290 methodological reports, eight categories (27 examples) of selective inclusion of results and eight categories (33 examples) of selective reporting of results in systematic reviews of RCTs were generated. Synthesis of seven empirical studies identified that (1) systematic review outcomes were often changed between the protocol and published review, (2) it was unclear whether the decision to change outcomes was related to statistical significance of the results, and (3) statistically significant results were more likely to be completely reported in systematic review abstracts than statistically non-significant results. The empirical study revealed that systematic reviewers frequently encountered multiple effect estimates in trial reports that could be included in a particular meta-analysis, but methods to select results to include in meta-analyses were underreported in systematic review protocols and reviews. There was uncertainty as to whether selective inclusion of trial effect estimates occurred in this sample of meta-analyses. However, any potential selective inclusion did not have an important impact on the meta-analytic effects. DISCUSSION The overarching findings from this thesis are that there are many potential scenarios for selective inclusion and reporting of results in systematic reviews of RCTs, though few empirical studies have investigated the prevalence and impact of these practices. Further, existing empirical studies have some limitations and there would be advantage in future studies being designed to avoid these limitations. Several recommendations for systematic reviewers arose from this research. It is recommended that systematic reviewers pre-define in a review protocol the outcomes of the review, analysis plans, and methods to select results to include in meta-analyses. Reviewers are also encouraged to report in the review any post-hoc modifications to outcomes and analytic strategies, and provide justification for these modifications. Further, reviewers are encouraged to report in the review whether they encountered multiple results that were available for inclusion in particular meta-analyses, and any methods they used to select results to include in meta-analyses.