Psychometric models of individual differences in reading comprehension: A reanalysis of Freed, Hamilton, and Long (2017)

Abstract

• Advocate network models complement latent variable models of individual differences. • Reviews best practices for data cleaning and diagnostics for psychometric modeling. • Reanalysis of Freed et al. (2017) reveals problematic measurement model. • Measures of reading comprehension and language experience load on one latent factor. • Network models corroborate findings and alternative conclusions are proposed. Individual differences in reading comprehension have often been explored using latent variable modeling (LVM), to assess the relative contribution of domain-general and domain-specific cognitive abilities. However, LVM is based on the assumption that the observed covariance among indicators of a construct is due to a common cause (i.e., a latent variable; Pearl, 2000). This is a questionable assumption when the indicator variables are measures of performance on complex cognitive tasks. According to Process Overlap Theory (POT; Kovacs & Conway, 2016), multiple processes are involved in cognitive task performance and the covariance among tasks is due to the overlap of processes across tasks. Instead of a single latent common cause, there are thought to be multiple dynamic manifest causes, consistent with an emerging view in psychometrics called network theory (Barabási, 2012; Borsboom & Cramer, 2013). In the current study, we reanalyzed data from Freed et al. (2017) and compared two modeling approaches: LVM (Study 1) and psychometric network modeling (Study 2). In Study 1, two exploratory LVMs demonstrated problems with the original measurement model proposed by Freed et al. Specifically, the model failed to achieve discriminant and convergent validity with respect to reading comprehension, language experience, and reasoning. In Study 2, two network models confirmed the problems found in Study 1, and also served as an example of how network modeling techniques can be used to study individual differences. In conclusion, more research, and a more informed approach to psychometric modeling, is needed to better understand individual differences in reading comprehension.