Study Quality in Quantitative L2 Research (1990–2010): A Methodological Synthesis and Call for Reform

Abstract

This article builds on the growing line of inquiry into methodological practices in quantitative second language (L2) research. Specifically, the study uses synthetic techniques to examine changes over time in research and reporting practices. 606 primary reports of quantitative L2 research from two journals—Language Learning and Studies in Second Language Acquisition—were surveyed on different design features, statistical analyses, and data reporting practices. Frequencies and percentages of each feature were then calculated and compared across the 1990s and the first decade of the 2000s to examine changes taking place in the field. The results, while not necessarily representative of all substantive and methodological approaches within the domain of L2 research, indicate numerous changes including increases in sample sizes, delayed posttesting, and the availability of critical data such as effect sizes, reliability estimates, and standard deviations to accompany means. With respect to statistical procedures, the range of analyses has not changed, and the field continues its unfortunate reliance on statistical significance. The findings are grouped according to three themes, which are discussed in light of previous reviews in this and other fields: (a) means‐based analyses, (b) missing data, null hypothesis significance testing, and the “power problem,” and (c) design preferences. The article concludes with an extended call for reform targeting six groups of stakeholders in the field. Most notably, an argument is made for field‐specific methodological standards and enhancements to graduate curricula and training.