How reliable are individual differences in eye movements in reading?

Abstract

• Assessed reliability of individual differences in readers’ eye movement effects. • Reliability was poor when based on split-half correlation of simple effects. • Reliability was higher, for a subset of effects, using a model-based method. • Implications for individual differences research are discussed. This study assessed the reliability of individual differences among fluent adult readers in the effects of four variables - word frequency, predictability, visual contrast, and font difficulty – on eye fixation duration measures, word skipping probability, and regression probability. Split-half reliability was computed in a reanalysis of data from two large, previously published experiments (Staub, 2020) by correlating simple effects in two halves of each experiment (e.g., Hedge, Powell, & Sumner, 2018) and by estimating, in the context of mixed-effects models, a correlation parameter between by-subject slopes for each half (Rouder & Haaf, 2019). The reliability of the effects was generally low, though the second of these methods revealed a few notable exceptions. First, the effects of visual contrast were quite reliable, as expected based on presumed individual differences in contrast sensitivity. Second, the frequency effect on gaze duration was also reliable, but only when raw (as opposed to log) gaze duration was used as the dependent measure. The effect of predictability demonstrated poor reliability for all dependent measures. Model comparison confirmed that model fit was improved by inclusion of by-subject slopes for those effects that showed substantial reliability. These results have implications for the feasibility of studies on individual differences in eye movements in reading, as only experimental effects that demonstrate substantial reliability are good candidates to be explored in individual difference studies.