The quality of experimental designs in mobile learning research: A systemic review and self-improvement tool

Abstract

The intervention effects of mobile learning have become one of the most popular topics in education research. Despite scholars critiquing the quality of mobile learning research, there is an obvious lack of quantitative studies exploring the design quality of experimental studies of mobile learning as well as tools for self-checking and improving experimental design. Based on the newly developed CREED (Checklist for the Rigor of Education-Experiment Designs), which focuses on the internal, construct, and statistical-conclusion validities of research, this study investigated the experimental research designs of mobile-learning studies. By analyzing 342 experimental studies published in refereed journals from 2006 to 2016, this study found that 72% of the mobile-learning studies exhibited experimental designs with a low or medium-low level of rigor. The mobile-learning research suffered from methodological limitations such as a lack of baseline equivalence between the experimental and control groups, not examining the fulfillment of statistical assumptions for specific statistical methods, poor quality of outcome assessment tools, and insufficient statistical powers. This study concluded that while utilizing advanced technologies, mobile-learning research employs laggard methodologies compared with standards proposed in educational and psychological research. Recommendations for overcoming these limitations are provided for use in further research and practices.