Abstract
This paper reports on a pilot study that used eye tracking techniques to make detailed observations of item response processes in the OECD Programme for the International Assessment of Adult Competencies (PIAAC). The lab-based study also recorded physiological responses using measures of pupil diameter and electrodermal activity. The study tested 14 adult respondents as they individually completed the PIAAC computer-based assessment. The eye tracking observations help to fill an ‘explanatory gap’ by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. The data on fixations and saccades provided detailed information on test item response strategies, enabling profiling of respondent engagement and response processes associated with successful performance. Much of that activity does not include the use of the keyboard and mouse, and involves ‘off-screen’ use of pen and paper (and calculator) that are not captured by assessment log-files. In conclusion, this paper points toward an important application of eye tracking in large-scale assessments. This includes insights into response processes in new domains such as adaptive problem-solving that aim to identify individuals’ ability to select and combine resources from the digital and physical environment.
Highlights
This paper reports on a pilot study that used eye tracking techniques to make detailed observations of item response processes in the OECD Programme for the International Assessment of Adult Competencies (PIAAC)
We found that the eye tracking glasses captured the on-screen activity of participants as they completed the PIAAC assessment, and the content of off-screen behaviour, such as the use of pen and paper and the calculator
This paper contributes to the recent expansion of interest in response processes in large-scale assessments (Zumbo and Hubley, 2017; Ercikan and Pellegrino, 2017)
Summary
This paper reports on a pilot study that used eye tracking techniques to make detailed observations of item response processes in the OECD Programme for the International Assessment of Adult Competencies (PIAAC). Eye tracking can provide important clues to visual attention (Ferreira and and Henderson 2004) as well as insightful data regarding assessment response processes These techniques are extensively used in psychological research in areas such as reading, joint gaze and scene perception Eye tracking technology enables the researcher to observe and record detail of respondent’s eye movements and the precise area of their attentional focus. They capture detail that is not obtained from assessment log files, think aloud protocols or administrator observations of assessment. We wanted to examine what additional insights eye tracking would provide about assessment response processes
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call