Textual strategies for answering multiple choice questions among South African learners: what can we learn from TIMSS 2003?

Abstract

This study investigated whether readability factors and/or misconceptions could account for the patterns of choice evidenced by South African learners in multiple-choice items in TIMSS 2003. The investigation focused on 20 items in which learners showed a strong preference for an incorrect answer. The trend was particularly noticeable among learners who attended schools in which all the learners and teachers speak an African language as their home language. The study arose from a concern about the readability of TIMSS items as a valid instrument for assessing the scientific knowledge of learners who lack proficiency in English.Text-based strategies could explain the pattern of choice in 14 of the 20 items in which over 40% of learners in African schools selected one incorrect answer. The most common strategy (eight items) consisted of favouring the answer that contained the greatest number of familiar words. Learners rejected answers containing unfamiliar words. Other strategies included selecting answers that contained words that were also in the question (three items), choosing an answer that contained all the options (one item), or misunderstanding the question (two items). Five items were consistent with misconceptions, and one item could not be classified.The strategies indicated that learners did not understand the 20 items clearly, and resorted to strategies based on ‘textual tricks’ that, with luck, might lead to the correct answer. Learners attending non-African schools may also have used ‘textual tricks’, although more learners in this group favoured the correct answer than the incorrect answer. The results of this analysis raise further concerns about the level of content knowledge among South African Grade 8 learners, and therefore the content validity of TIMSS in South Africa. The lack of appropriate content knowledge as well as misconceptions is compounded by low levels of proficiency in English, coupled with unclear wording of some TIMSS items.