Reproducibility of search strategies of non-Cochrane systematic reviews published in anaesthesiology journals is suboptimal: primary methodological study

Abstract

Editor—Many systematic reviews (SRs) suffer from incomplete reporting despite availability of relevant reporting guidelines.1Page M.J. Moher D. Evaluations of the uptake and impact of the preferred reporting Items for systematic reviews and meta-analyses (PRISMA) statement and extensions: a scoping review.Syst Rev. 2017; 6: 263Crossref PubMed Scopus (267) Google Scholar As a crucial step in SR methodology, search strategies should be extensively reported to be reproducible; however, it has not been explored whether SRs in the field of anaesthesiology have reproducible search strategies. We conducted a study in which we analysed reproducibility of search strategies of non-Cochrane SRs in the field of anaesthesiology, and analysed instructions for authors in anaesthesiology journals regarding reporting of SR search strategies. This was a primary methodological study where the unit of analysis was a published SR; therefore, approval of the study protocol by a research ethics committee was not needed. We included non-Cochrane SRs of randomised controlled trials about interventions with meta-analysis, published in English in journals from the Journal Citation Reports category Anaesthesiology from 2012 to 2016. We defined SRs as studies that searched more than one database and had more than one author.2Puljak L. If there is only one author or only one database was searched, a study should not be called a systematic review.J Clin Epidemiol. 2017; 91: 4-5Abstract Full Text Full Text PDF PubMed Scopus (15) Google Scholar We defined reproducible search according to Koffel and Rethlefsen.3Koffel J.B. Rethlefsen M.L. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study.PLoS One. 2016; 11: e0163309Crossref PubMed Scopus (59) Google Scholar We searched PubMed on April 6, 2017 using a filter for meta-analysis, date, and journal name. Two authors independently screened abstracts and full texts of retrieved manuscripts. We extracted the following data: date of the search and was it reported for one information source or all, date of online publication on PubMed, presence of search terms or search strategy in the manuscript, use of keywords or full search strategy, presentation of search strategy as supplementary material/appendix, presentation of full search strategy for one database only or for all information sources used, use of language limitations in the search strategy, mention of SR reporting standards, citation of reference for SR reporting standards, report of involvement of a librarian or information specialist, report of peer review of search strategy, report of provider/interface for the database, and presence of a Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flowchart. We analysed instructions for authors of all included journals to determine whether they mentioned a SR reporting guideline, or any other instructions for reporting information sources used and search strategies. We calculated descriptive statistics (frequencies, percentages) and used the χ2 test to examine differences in reproducibility between SRs that did or did not cite reporting guidelines. After searching PubMed, we retrieved 376 records; after screening those records for eligibility, we included 287 SRs in our study. Table 1 presents study results in detail. In 265 (92.3%) SRs, authors reported the search date for all searched databases; in three (1%) SRs, authors reported the search date for just one database of more databases that were searched, and 19 (6.6%) SRs did not report any search date.Table 1Reporting search elements and search strategies in analysed systematic reviews (N=287). *Some manuscripts had multiple language limits. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analysis; QUOROM, Quality of Reporting of Meta-AnalysesInformation about search elements and search strategiesN (%)Reported search elements in the manuscript220 (76.7) Full search strategy23 (10.5) Search terms2198 (90)Reported search elements in supplemental material101 (35.2) Full search strategy94 (93.1) Search terms10 (9.9)Presenting full search strategy for searched information sources anywhere115 (40.1) All information sources66 (57.4) One information source out of more that were searched46 (40) Two information sources out of more that were searched3 (2.6)Did not present full search strategy for searched information sources anywhere172 (60)Citing reference for systematic review reporting standards182 (63.4)Explicitly mentioning systematic review reporting standard PRISMA167 (92) Cochrane Collaboration8 (4.3) QUOROM14 (7.5)Involvement of a librarian or information specialist26 (8.8)Reporting peer review of a search strategy0 (0)Reporting provider or interface for the database46 (16)Presenting PRISMA flowchart268 (93.4)Limitations to a certain languageYes80 (27.9)No168 (58.5)Contradictory information about language limitations6 (2.1)Not reported33 (11.5)Languages were limited* to (N=85):Chinese2 (2.5)Dutch3 (3.8)English80 (100)French4 (5)German4 (5)Portuguese2 (2.5)Spanish2 (2.5) Open table in a new tab In 280 (97.6%) SRs we found search elements (search strategy, search terms) in the manuscript or supplementary material. More SRs presented search terms or search strategy in the text of the manuscript than in a supplemental material. Among SRs that presented search terms or strategy in the manuscript, the majority presented only search terms. Among SRs that presented search strategies in supplemental material, the majority presented full search strategies while only ten SRs presented search terms. Full search strategies were presented in less than half of the analysed SRs. In the majority of SRs that presented full search strategies in the manuscript or in supplemental material, search strategies were presented for all information sources. More than half of the analysed SRs cited references for SR reporting standards. The most commonly cited SR reporting guidelines were: PRISMA (91.8%), Quality of Reporting of Meta-Analyses (QUOROM) (7.7%), and guidelines from the Cochrane (4.4%). More than half of the analysed SRs reported that they did not have any language limitations in their search strategies. In six (2.1%) SRs, contradictory information was reported in the manuscript, while 33 (11.5%) SRs did not report any information regarding language limitation. The majority of SRs limited their searches to the English language. Less than 10% of all analysed SRs reported involvement of a librarian or information specialist and none of the SRs reported peer review of search strategies. Less than half reported the provider or interface for the database. The majority of analysed SRs presented a PRISMA flowchart. We compared SRs that cited reporting guidelines and SRs that did not cite any reporting guideline to determine whether there were significant differences between those two groups in reproducibility. We found that a higher number of SRs that cited reporting guidelines did not limit searches to specific languages than SRs that did not cite any guideline (χ2=20.981; df=1; P<0.0001). We did not find significant difference between the two groups in other variables. Full data for this analysis and a more detailed description of study methods are available on Open Science Framework (https://osf.io/rfgj6/). Twelve (46.2%) of the 26 analysed instructions for authors mentioned guidelines for reporting standards; all 12 mentioned PRISMA, and one journal mentioned both PRISMA and Meta-analyses Of Observational Studies in Epidemiology (MOOSE) guidelines. Five (19.2%) journals gave instructions for reporting information sources and search strategies. In this study, we found that the majority of analysed anaesthesiology non-Cochrane SRs did not have reproducible search strategies because of missing methodological details. Earlier studies have recognised a wide range of problems in search strategies published in SRs, from inadequate designs to poor reporting of search strategies.4Franco J.V.A. Garrote V.L. Escobar Liquitay C.M. Vietto V. Identification of problems in search strategies in Cochrane Reviews.Res Synth Methods. 2018; 9: 408-416Crossref PubMed Scopus (27) Google Scholar, 5Faggion Jr., C.M. Atieh M.A. Park S. Search strategies in systematic reviews in periodontology and implant dentistry.J Clin Periodontol. 2013; 40: 883-888Crossref PubMed Scopus (36) Google Scholar, 6Faggion Jr., C.M. Wu Y.C. Tu Y.K. Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery.Br J Radiol. 2016; 89: 20150878Google Scholar, 7Yoshii A. Plaut D.A. McGraw K.A. Anderson M.J. Wellik K.E. Analysis of the reporting of search strategies in Cochrane systematic reviews.J Med Libr Assoc. 2009; 97: 21-29Crossref PubMed Scopus (90) Google Scholar We have shown that the anaesthesiology field has a high prevalence of the same problems. Interventions for ensuring better reporting and reproducibility of SR search strategies are warranted. Such interventions could be the inclusion of a librarian or information specialist in an SR team8Townsend W.A. Anderson P.F. Ginier E.C. et al.A competency framework for librarians involved in systematic reviews.J Med Libr Assoc. 2017; 105: 268-275Google Scholar reviewing search strategies,9Sampson M. McGowan J. Cogo E. Grimshaw J. Moher D. Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies.J Clin Epidemiol. 2009; 62: 944-952Abstract Full Text Full Text PDF PubMed Scopus (418) Google Scholar fostering adherence with reporting guidelines,10Lee S.Y. Sagoo H. Whitehurst K. et al.Compliance of systematic reviews in plastic surgery with the PRISMA Statement.JAMA Facial Plast Surg. 2016; 18: 101-105Crossref PubMed Scopus (21) Google Scholar,11Akhigbe T. Zolnourian A. Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature.J Clin Neurosci. 2017; 39: 45-48Abstract Full Text Full Text PDF PubMed Scopus (19) Google Scholar and emphasising detailed reporting requirements in journals' instructions for authors. We showed that citing reporting guidelines was not sufficient. In conclusion, suboptimal reporting of search strategies in SRs in the field of anaesthesiology and pain medicine hinders reproducibility of search strategies. This finding was in spite of the fact that the majority of analysed SRs referenced relevant reporting guidelines. SRs published in anaesthesiology journals need better reporting of search strategies. Novel interventions should be explored to improve transparency and reproducibility of SRs.12Heesen M. Klimek M. Imberger G. Hoeks S.E. Rossaint R. Straube S. On differences between systematic reviews.Br J Anaesth. 2018; 120: 1133-1134Abstract Full Text Full Text PDF Scopus (2) Google Scholar The authors thank Karla Cikes for conducting the initial screening of SRs. The authors declare that they have no conflicts of interest.