Assessing the robustness of meta-analytic results in information systems: publication bias and outliers

Abstract

Meta-analytic studies serve to generate cumulative knowledge and guide evidence-based practice. However, publication bias and outliers threaten the accuracy and robustness of meta-analytic results. Unfortunately, most meta-analytic studies in information systems (IS) research do not assess the presence of these phenomena. Furthermore, some methods commonly used for the detection of publication bias are now recognised as inappropriate. We conduct a comprehensive assessment of four previously published meta-analytic studies in IS. We use multiple methods to assess the effects of publication bias and outliers on the meta-analytic results. Our findings indicate that publication bias and/or outliers have affected the results of three of the four meta-analytic studies. Some methods indicate that select meta-analytic means were misestimated by potentially more than 100%. Our analyses offer methodological exemplars that can be followed to assess the potential adverse effects of publication bias and outliers on meta-analytic results, including their combined effects. We make additional contributions to scientific knowledge by evaluating the performance of different publication bias assessment methods used across scientific disciplines. In brief, we highlight the importance of a rigorous assessment of publication bias and outliers on meta-analytic results to improve the trustworthiness of our cumulative knowledge and evidence-based practice.