A Bayesian Approach to Measuring Evidence in L2 Research: An Empirical Investigation

Abstract

AbstractNull hypothesis testing has long since been the ‘go‐to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open‐science framework, the present study (a) re‐evaluates the empirical findings of 418 t‐tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p‐values fell between .01 and .05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field‐wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.