Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews

Abstract

A review based on a biased collection of studies is likely to produce biased conclusions. As such, a central component of a quality systematic review is a comprehensive document search. Three issues are explored in this paper that underpin the importance of such a search. First, the evidence of publication selection bias clearly establishes that peer-reviewed journal articles are more likely to show statistically significant results than less formally published studies. Second, the presence of grey literature studies that cannot be discovered through a search of bibliographic databases is demonstrated using Campbell Collaboration systematic reviews, and third, the logical and empirical problems with using publication status as a proxy for methodological quality are examined. More robust methods of identifying research conducted in the social sciences are needed if we are to take seriously the accumulation of knowledge, such as the development of a priori registries of research studies.