Abstract
ObjectiveHypothetically, content in MEDLINE records is consistent across multiple platforms. Though platforms have different interfaces and requirements for query syntax, results should be similar when the syntax is controlled for across the platforms. The authors investigated how search result counts varied when searching records among five MEDLINE platforms.MethodsWe created 29 sets of search queries targeting various metadata fields and operators. Within search sets, we adapted 5 distinct, compatible queries to search 5 MEDLINE platforms (PubMed, ProQuest, EBSCOhost, Web of Science, and Ovid), totaling 145 final queries. The 5 queries were designed to be logically and semantically equivalent and were modified only to match platform syntax requirements. We analyzed the result counts and compared PubMed’s MEDLINE result counts to result counts from the other platforms. We identified outliers by measuring the result count deviations using modified z-scores centered around PubMed’s MEDLINE results.ResultsWeb of Science and ProQuest searches were the most likely to deviate from the equivalent PubMed searches. EBSCOhost and Ovid were less likely to deviate from PubMed searches. Ovid’s results were the most consistent with PubMed’s but appeared to apply an indexing algorithm that resulted in lower retrieval sets among equivalent searches in PubMed. Web of Science exhibited problems with exploding or not exploding Medical Subject Headings (MeSH) terms.ConclusionPlatform enhancements among interfaces affect record retrieval and challenge the expectation that MEDLINE platforms should, by default, be treated as MEDLINE. Substantial inconsistencies in search result counts, as demonstrated here, should raise concerns about the impact of platform-specific influences on search results.
Highlights
The replication and reproduction of research, or lack thereof, is a perennial problem among research communities [1,2,3]
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Guidelines and the Cochrane Handbook for Systematic Reviews and Interventions are examples of how scholars recognize the need for systematic reporting of methods and the organization of review research [4, 5]
A reply to this study suggested that the differences could be explained by basic problems with bibliographic and MEDLINE searching and concluded that “database and search interface providers should agree on common standards in terminology and search semantics and soon make their professional tools as useful as they are intended to” [25]
Summary
The replication and reproduction of research, or lack thereof, is a perennial problem among research communities [1,2,3]. For systematic reviews and other research that relies on citation or bibliographic records, the evaluation of scientific rigor is partly based on the reproducibility of search strategies. Differences in search interfaces, article indexing, and retrieval algorithms impact reproducibility and replication, which are important aspects of the scientific process, evidence-based medicine, and the creation of systematic reviews [6,7,8,9,10,11,12]. Even if search strategies are methodical and well documented, searches might not be reproducible because many platforms are proprietary products, and the code, algorithms, and, in general, the software that drives these products are not available for public. One can only speculate how such systems work by inference from use; for example, by comparing them to similar products [13, 14]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
