Research Synthesis and Meta-Analysis in Second Language Learning and Testing

Abstract

Research synthesis and meta-analysis are useful techniques for summarizing studies to understand what is known/unknown and to accumulate knowledge in an academic domain. In this paper, we attempt to provide an overview of the status quo of research synthesis and meta-analysis in second language learning and testing. We compare research synthesis with traditional methods of literature review (i.e., narrative and vote-counting methods), describe qualitative and quantitative research syntheses, and report a quantitative synthetic approach that simply counts the number of studies for each category without using any statistical significance. Also included are the strengths (e.g., transparency, effect sizes, moderator variable analysis) and limitations (e.g., apples-and-oranges, study quality, and file-drawer problems) of research synthesis. Finally, we review whether and how the four challenges Norris and Ortega (2006) raised for synthesists to overcome have been since addressed (i.e., reporting inconsistency, study quality, non-English-language studies, and integration of qualitative and quantitative syntheses). We find that while steady progress has been made, these challenges have not yet been fully handled.