A review of the methodological quality of quantitative mobile-assisted language learning research

Abstract

Mobile-assisted language learning (MALL) is a novel approach to language learning and teaching. The present study aims to review the methodological quality of quantitative MALL research by focusing on the applications of statistical techniques and instrument reliability and validity. A total of 174 papers within 41 journals identified using the Scopus database were screened and coded. Of these, 77 quantitative MALL studies that investigated English as a foreign or second language using mobile devices met the inclusion criteria. In the full-text screening, each study was coded for the statistical techniques applied, assumptions reported, reliability and validity investigation of the instruments and coding practices used. The results show the ubiquity of the general linear model (GLM) (i.e., mean-based data analysis, such as t-test, univariate analysis of variances (ANOVA), multivariate analysis of variances (MANOVA)), with 61.40% of the analyzed studies using this statistical method. Notably, the majority of studies that used GLM did not report confirmation of the fundamental assumptions (i.e., normality, homogeneity of variance, and linearity) of such analysis. In addition, a reliance of null hypothesis significance testing was observed without reporting of the practical significance of the investigated effect or relations (effect size). Lastly, less than half of MALL studies reported reliability and even fewer studies reported validity evidence, indicating a lack of evidence of the precision, meaningfulness of data, and accuracy of many of the measurement instruments used. Implications of these findings for MALL research are discussed, with several suggestions for future research.