Abstract
BackgroundLarge-scale cross-national studies designed to measure student achievement use different social, cultural, economic and other background variables to explain observed differences in that achievement. Prior to their inclusion into a prediction model, these variables are commonly scaled into latent background indices. To allow cross-national comparisons of the latent indices, measurement invariance is assumed. However, it is unclear whether the assumption of measurement invariance has some influence on the results of the prediction model, thus challenging the reliability and validity of cross-national comparisons of predicted results.MethodsTo establish the effect size attributed to different degrees of measurement invariance, we rescaled the ‘home resource for learning index’ (HRL) for the 37 countries (n=166,709 students) that participated in the IEA’s combined ‘Progress in International Reading Literacy Study’ (PIRLS) and ‘Trends in International Mathematics and Science Study’ (TIMSS) assessments of 2011. We used (a) two different measurement models [one-parameter model (1PL) and two-parameter model (2PL)] with (b) two different degrees of measurement invariance, resulting in four different models. We introduced the different HRL indices as predictors in a generalized linear mixed model (GLMM) with mathematics achievement as the dependent variable. We then compared three outcomes across countries and by scaling model: (1) the differing fit-values of the measurement models, (2) the estimated discrimination parameters, and (3) the estimated regression coefficients.ResultsThe least restrictive measurement model fitted the data best, and the degree of assumed measurement invariance of the HRL indices influenced the random effects of the GLMM in all but one country. For one-third of the countries, the fixed effects of the GLMM also related to the degree of assumed measurement invariance.ConclusionThe results support the use of country-specific measurement models for scaling the HRL index. In general, equating procedures could be used for cross-national comparisons of the latent indices when country-specific measurement models are fitted. Cross-national comparisons of the coefficients of the GLMM should take into account the applied measurement model for scaling the HRL indices. This process could be achieved by, for example, adjusting the standard errors of the coefficients.
Highlights
Large-scale cross-national studies designed to measure student achievement use different social, cultural, economic and other background variables to explain observed differences in that achievement
The results support the use of country-specific measurement models for scaling the home resources for learning (HRL) index
We considered that reanalyzing these data sets by allowing different degrees of cross-national measurement invariance could help to answer the question of whether this assumption has an influence on (1) the cross-national comparisons performed by Martin et al (2013) in particular, and (2) the results of large-scale assessment studies that use a design comparable to the one Martin and his colleagues employed in general
Summary
Large-scale cross-national studies designed to measure student achievement use different social, cultural, economic and other background variables to explain observed differences in that achievement Prior to their inclusion into a prediction model, these variables are commonly scaled into latent background indices. Thirty-four countries and three benchmark participants collected data on Grade 4 students’ educational achievement in three competence domains: reading comprehension, mathematics, and science (Martin and Mullis 2013) In their efforts to explain observed achievement differences in the data from largescale assessment studies, researchers have increasingly combined different background indicators (Bos et al 2012; Martin et al 2008; Mullis et al 2007, 2008; OECD 2014a) by scaling them into latent background variables. Controlling for measurement error is especially important in educational research studies because the multilevel prediction models commonly used in this area are very sensitive to these errors (Lüdtke et al 2011)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
