Item wording effects in self-report measures and reading achievement: Does removing careless respondents help?

Abstract

This study examined the impact of item wording effects on the factor structure of the Student Confidence in Reading (SCR) scale in the 2016 Progress in International Reading Literacy Study (PIRLS). First, using a large sample of students from seven English-speaking countries, item wording effects in the SCR scale were investigated for the entire sample and the subgroups of students based on their reading achievement levels in PIRLS 2016. Second, the presence of item wording effects was re-examined after identifying and removing students who answered negatively worded items inconsistently. The results showed that the use of negatively worded items distorted the factor structure of the SCR scale, leading to an unintentionally multidimensional factor structure. Removing students with careless responses had a negligible impact on the model fit for students at the lowest achievement level, whereas it considerably improved the model fit for other students with moderate to high reading achievement.