Abstract
BackgroundItem position effects have been a common concern in large-scale assessments as changing the order of items in booklets may have an undesired effect on test performance. If every test taker would be affected by the effect in the very same way, comparisons between groups of individuals would still be valid. However, research has shown that in addition to a general fixed effect of item positions, the extent of the effect varies considerably across individuals. These individual differences are referred to as persistence. Test takers with a high level of persistence are able to keep up their performance better throughout the test administration, whereas those with a lower level of persistence show a larger decline in their test performance.MethodsThe present study applied a multilevel extended item response theory (IRT) framework and used the data from the PISA 2006 science, 2009 reading, and 2012 mathematics assessments. The first objective of this study is to provide a systematic investigation of item position effects across the three PISA domains, partially replicating the previous studies on PISA 2006 and 2009. Second, this study aims to gain a better understanding of the nature of individual differences in position effects by relating them to student characteristics. Gender, socio-economic status, language spoken at home, and three motivational scales (enjoyment of doing the subject being assessed, effort thermometer, perseverance) were used as person covariates for persistence.ResultsThis study replicated and extended the results found in previous studies. An overall negative item cluster position effect and significant individual differences in this effect were found in all the countries in the three PISA domains. Furthermore, the most frequently observed effect of person covariates on persistence is gender, with girls keeping up their performance better than boys. Other predictors showed little or inconsistent effects on persistence.ConclusionsOur study demonstrated inter-individual differences as well as group differences in item position effects, which may threaten the comparability between persons and groups. The consequences and implications of item position effects and persistence for the interpretation of PISA results are discussed.
Highlights
In large-scale assessments, items are often presented in different positions within a test using a booklet design
Our study demonstrated inter-individual differences as well as group differences in item position effects, which may threaten the comparability between persons and groups
This study investigates the relationship of persistence in Programme of International Student Achievement (PISA) assessments with (a) student background characteristics that have been known as stable predictors of test performance and (b) motivational variables that have been included in the PISA context questionnaires
Summary
In large-scale assessments, items are often presented in different positions within a test using a booklet design. That is, when presented at the end of a test, an item may become less or more difficult than when presented at the beginning of the test, resulting in an increase or a decrease in the performance during testing, respectively. Under such circumstances, the responses to test items are not solely dependent on the latent trait(s) of interest, and on context factors that may be construct-irrelevant (Messick 1995). Item position effects have been a common concern in large-scale assessments as changing the order of items in booklets may have an undesired effect on test performance. Test takers with a high level of persistence are able to keep up their performance better throughout the test administration, whereas those with a lower level of persistence show a larger decline in their test performance
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
