Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation

Balazs Aczel,Peter Szecsi,Mark Zrubka,Don Van Den Bergh,Aba Szollosi,Bence Palfi,Marton Kovacs,Barnabas Szaszi,Quentin F Gronau,Eric-Jan Wagenmakers

doi:10.1177/2515245918773742

Abstract

In the traditional statistical framework, nonsignificant results leave researchers in a state of suspended disbelief. In this study, we examined, empirically, the treatment and evidential impact of nonsignificant results. Our specific goals were twofold: to explore how psychologists interpret and communicate nonsignificant results and to assess how much these results constitute evidence in favor of the null hypothesis. First, we examined all nonsignificant findings mentioned in the abstracts of the 2015 volumes of Psychonomic Bulletin & Review, Journal of Experimental Psychology: General, and Psychological Science ( N = 137). In 72% of these cases, nonsignificant results were misinterpreted, in that the authors inferred that the effect was absent. Second, a Bayes factor reanalysis revealed that fewer than 5% of the nonsignificant findings provided strong evidence (i.e., BF01 > 10) in favor of the null hypothesis over the alternative hypothesis. We recommend that researchers expand their statistical tool kit in order to correctly interpret nonsignificant results and to be able to evaluate the evidence for and against the null hypothesis.

Highlights

The interpretation of statistically nonsignificant findings is a vexing point of traditional psychological research.1 Within the framework of null-hypothesis significance testing (NHST; Fisher, 1925; Neyman & Pearson, 1933), decisions about the null hypothesis are based on the p value
We examine the degree of miscommunication of nonsignificant findings in current psychological publications; in addition, we use Bayes factors to assess how much these findings support the null hypothesis relative to a composite alternative hypothesis (e.g., Etz & Vandekerckhove, 2017)
P values larger than the threshold indicate only that the test was incapable of rejecting the null hypothesis; this could have occurred because the effect does not exist, but it could have occurred because the power of the test was insufficient to detect a true effect (Dienes, 2014, 2016)

Summary

Introduction

The interpretation of statistically nonsignificant findings is a vexing point of traditional psychological research. Within the framework of null-hypothesis significance testing (NHST; Fisher, 1925; Neyman & Pearson, 1933), decisions about the null hypothesis are based on the p value. When confronted with nonsignificant findings, researchers may seek refuge in a description of the sample rather than inference concerning the population; such a tendency is revealed by expressions such as “no difference between the groups was observed.”. Such statements about the sample are problematic, as the observed difference is never exactly zero in the case of continuous data, even when the null hypothesis holds exactly. When the necessary information was available, we computed Bayes factors for all reported nonsignificant t-test results in our sample This allowed us to explore the degree to which reported nonsignificant results provide support for the null hypothesis

Objectives

Methods

Results

Discussion

Conclusion