General or specific abilities? Evidence from 33 countries participating in the PISA assessments

Abstract

Psychometricians working on International Large Scale Assessments (ILSAs) typically specify latent ability factors with distinct and correlated constructs for test domains, such as reading, mathematics and science. A construct for general ability is not specified. However, several country-specific studies conclude that ILSAs largely reflect general ability. We extend such studies and examine the dimensionality of the 2018 PISA assessment in 33 OECD countries examining three models: three-dimensional IRT model, the bifactor IRT model and the bifactor (S-1) IRT model. A four-tiered approach was adopted. First, models were compared using an information criterion (AIC). Second, the correlations from the multidimensional model were estimated to assess in which countries the three dimensions are sufficient discriminant validity. Third, a variety of bifactor indices were utilized to establish the explanatory power and reliabilities of the latent dimensions generated by the three models. Finally, the statistical relationships between the latent factors derived from the three models and educationally relevant covariates were estimated. The bifactor model fits the data better than standard multidimensional model or S-1 model in every country investigated. The correlations in the correlated factor model are above 0.8 in all 33 countries. The symmetrical bifactor general ability model shows that 80%, or more, of the common variance in student responses to the PISA instruments is accounted for by a general ability factor. On average, 27% of variance in the mathematics items is independent of the general factor and can be attributed to a specific mathematics ability factor. The respective estimates for reading are 12% and science is 17%. Relationships for selected covariates with the PISA domains follow the same pattern as general ability in the bifactor model.