Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review.

Elizabeth Gargon,Mike Clarke,Paula R Williamson

doi:10.1186/s12874-015-0019-9

Abstract

BackgroundThe COMET (Core Outcome Measures in Effectiveness Trials) Initiative is developing a publicly accessible online resource to collate the knowledge base for core outcome set development (COS) and the applied work from different health conditions. Ensuring that the database is as comprehensive as possible and keeping it up to date are key to its value for users. This requires the development and application of an optimal, multi-faceted search strategy to identify relevant material. This paper describes the challenges of designing and implementing such a search, outlining the development of the search strategy for studies of COS development, and, in turn, the process for establishing a database of COS.MethodsWe investigated the performance characteristics of this strategy including sensitivity, precision and numbers needed to read. We compared the contribution of databases towards identifying included studies to identify the best combination of methods to retrieve all included studies.ResultsRecall of the search strategies ranged from 4% to 87%, and precision from 0.77% to 1.13%. MEDLINE performed best in terms of recall, retrieving 216 (87%) of the 250 included records, followed by Scopus (44%). The Cochrane Methodology Register found just 4% of the included records. MEDLINE was also the database with the highest precision. The number needed to read varied between 89 (MEDLINE) and 130 (SCOPUS).ConclusionsWe found that two databases and hand searching were required to locate all of the studies in this review. MEDLINE alone retrieved 87% of the included studies, but actually 97% of the included studies were indexed on MEDLINE. The Cochrane Methodology Register did not contribute any records that were not found in the other databases, and will not be included in our future searches to identify studies developing COS. SCOPUS had the lowest precision rate (0.77) and highest number needed to read (130). In future COMET searches for COS a balance needs to be struck between the work involved in screening large numbers of records, the frequency of the searching and the likelihood that eligible studies will be identified by means other than the database searches.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-015-0019-9) contains supplementary material, which is available to authorized users.

Highlights

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative is developing a publicly accessible online resource to collate the knowledge base for core outcome set development (COS) and the applied work from different health conditions
This confirmed that the search strategy was identifying relevant studies, that the likely yield of such studies was likely to be high and that a formal, systematic review was necessary to identify papers that were not yet known to COMET if we were to create a comprehensive resource that others could use to determine whether or not a COS had already been developed in an area of interest to them
The decision to search SCOPUS may have increased the de-duplication burden due to the lack of flexibility in its interface, for example de-duplicating against MEDLINE within OVID. This could be an added advantage of searching EMBASE and may be considered in any future updates to this search. In considering how this analysis might inform future decisions about the COMET searches for COS, a balance needs to be struck between the work involved in screening large numbers of records, the frequency of the searching and the likelihood that eligible studies will be identified by means other than the database searches

Summary

Introduction

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative is developing a publicly accessible online resource to collate the knowledge base for core outcome set development (COS) and the applied work from different health conditions. There are often differences in how outcomes are defined and measured making it difficult, sometimes impossible, to Gargon et al BMC Medical Research Methodology (2015) 15:26 with a statistically significant result, it was found that 19% would not have remained significant had all studies contributed to the analyses and 26% had overestimated the treatment effect by more than 20% [4] These issues of inconsistency and outcome reporting bias could be addressed with the development and application of agreed standardised sets of outcomes, known as COS, that should be measured and reported in all trials for a specific clinical area [5].

Objectives

Methods

Results

Discussion

Conclusion