The Effect of Sample Weighting on Hierarchical Linear Modeling in the Large-Scale Assessment Data

Abstract

This study examines how the different uses of sampling weights in the analysis of TIMSS 2019 data affect the ratio of variance in student achievement explained by schools and the estimation of standard errors. The research sample comprises 227,345 8th grade students from 7,636 schools in 39 countries. Mathematics achievement and science achievement are considered separately as dependent variables in all 39 countries. All plausible values are included in the analysis. Four weighting scenarios are examined: no weighting, weighting at only level 1, weighting at only level 2, and weighting at both levels. In total, 312 models are established and examined. According to the research results, the coefficients, standard errors, reliabilities, and χ^2 estimations change depending on how the weighting variable is handled in the models, and as a result, the ratio of variance in the dependent variable arising from the differences between schools also changes. The ratio attributable to between-school differences can reach up to 20% in some countries. Therefore, researchers modeling hierarchical data using HLM are suggested to plan how they handle the weighting variable prior to conducting the study.