Low-stakes Tests Research Articles

Investigating the impact of multiple priming questions on examinee effort during low-stakes testing

Priming examinees with questions about intended effort prior to testing has been shown to significantly increase examinee expended effort via self-reported effort and response-time effort. However, this question-behavior effect seems to wear off later in a testing session. We examined whether administering a second “dose” of the question-behavior effect could combat the decrease in examinee effort later in a testing session. We randomly assigned examinees to one of three question conditions prior to completing two low-stakes tests: answering three questions about intended effort directly before the first test in a session, answering three questions about intended effort directly before each test in a session, and answering no priming questions (control). Administering a second dose of questions directly before the second test in a session significantly increased response-time effort and self-reported effort for the more difficult test. This simple administration of multiple sets of questions throughout a testing session appears to combat issues with low effort on difficult tests that are administered later in a testing session.

Test anxiety fluctuations during low-stakes secondary school assessments: The role of the needs for autonomy and competence over and above the number of tests

Test anxiety poses a fundamental educational challenge as it is associated with lower academic performance and well-being. Grounded in the Self-Determination Theory, this study will focus on test anxiety fluctuations in relation to low-stakes assessments and investigates whether fluctuations in students’ experiences of autonomy and competence satisfaction and frustration relate to their test anxiety. For this purpose, 253 secondary school students completed a survey at three different times throughout the second semester. Students' feelings of autonomy and competence in the classroom were administered as well as their test anxiety. Each student completed the same two test anxiety scales at each measurement occasion, with one scale consistently administered to all students and the other two scales randomly assigned between classes. Multilevel analyses revealed that students showed higher test anxiety in weeks in which their need for competence was more frustrated and when they had to take more low-stakes tests. This association was robust across the three test anxiety instruments and after considering important test anxiety covariates (e.g., gender and prior achievement). These findings imply that competence frustration is an important underlying mechanism of test anxiety that should be taken into account when designing anxiety-reducing interventions.

The effects of private and social incentives on students’ test-taking effort

Despite widespread educational reforms, the challenge of students’ suboptimal academic effort persists across numerous countries. We investigate the role of private and social incentives on test performance in real, low-stakes test settings. We implemented a multifaceted incentive program for three cohorts of Grade 7 students participating in a low-stakes national standardized test in Australia. Results indicate that incentives can improve test performance. Our experimental design ensures that performance gains are attributable to increased test-taking effort only. Performance gains in three subject domains are measured against a baseline domain with no incentives in a within-student design. Although the private incentives that targeted individual performance elicited larger improvements than a social incentive targeting group outcomes, the latter proved more cost efficient. Our intervention suggests valuable strategies to enhance assessment of student performance that may be biased due to suboptimal effort, especially in low-stakes scenarios that are often employed for educational policy evaluation.

Priming Examinees to Give Good Effort: Differential Utility Across Gender Identity

We examined the utility of a question-behavior effect (QBE) intervention on test-taking effort. Questioning individuals about future behavior influences the subsequent performance of that behavior. Previous research found this intervention to significantly increase test-taking effort for incoming university students but it was unknown if it would be effective for more advanced college students or differentially effective across gender identities. Thus, 503 more advanced college students were randomly assigned to one of three question conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to testing, answering five questions regarding intended effort that referenced positive self-identity prior to testing, or a no-question control condition. We then administered a multiple-choice test and collected two measures of test-taking effort: self-reported effort and response time. We found that students identifying as female had significantly and practically higher self-reported effort and response time when in the question conditions compared to the no question condition. There was no effect of condition for students identifying as male for either measure of test-taking effort. Implications for practically increasing motivation on low-stakes tests and surveys, especially for more advanced students, are discussed. Implications for QBE theory, including differential effects related to gender, are discussed.

Comparing Examinee-Based and Response-Based Motivation Filtering Methods in Remote Low-Stakes Testing

ABSTRACT Large-scale educational assessments are sometimes considered low-stakes, increasing the possibility of confounding true performance level with low motivation. These concerns are amplified in remote testing conditions. To remove the effects of low effort levels in responses observed in remote low-stakes testing, several motivation filtering methods can be used to purify the data. We estimated scores from assessment data collected remotely in Spring 2021 six ways, applying examinee-based filtering methods (filtering examinees based on total time) and response-based filtering methods (filtering responses using the effort-moderated IRT model), varying the thresholds selected to separate effortful and non-effortful responses. We compared the 2021 scores (estimated six ways) to those obtained in previous cohorts tested in person. The results support the use of motivation filtering regardless of which method is used. Practical implications regarding method and threshold selection are discussed.

English High-stakes Testing in Indonesia: Friend or Foe?

The Education Ministry has eliminated standardised high-stakes testing called Ujian Nasional (UN) since 2021 throughout schools all around Indonesia. However, it might be practical to look backwards at the UN and its implementation with the impact it gives on the teachers. This article has a concern about the washback consequences of a high-stakes exam on teachers. The effects were investigated within the specific context of the 2016 UN for students in Indonesia at the junior secondary level. A semi-structured interview with eight English teachers was set up to explore the teachers' perceptions towards the UN. Regarding feelings, all participants shared divergent feelings ranging from disinterested to excessive levels of fear or anxiety and mixed feelings. The data also showed mixed opinions, both positive and negative, regarding the examination. The students were thought to be motivated to study because the examination was used for selection purposes. The results did, however, also imply that the students' performance on the national exam was thought to have been assessed unfairly. The instructors demonstrated how the exam affected their teaching roles and methods. Making the national exam a low-stakes test was one of the suggestions made by attendees to enhance it.

Measurements of performance gaps are sensitive to the level of test stakes: Evidence from PISA and a Field Experiment

Educational performance gaps are a long-lasting issue of concern in many countries. Many times, these gaps are measured using low-stakes tests that are especially sensitive to motivation and test engagement. I conduct a field experiment in 7 Israeli Jewish and Arab middle schools and use data from PISA 2015 to investigate whether part of the performance gaps between ethnic minority and majority groups could be attributed to test engagement rather than proficiency. I find that test engagement can account for more than 50% of the performance gaps measured in low-stakes assessments. Test engagement is also important, as it correlates with several personality traits important to life outcomes. Therefore, considering the importance of both test engagement, which influences low-stakes assessments, and proficiency, which is more accurately discerned in high-stakes assessments, policymakers should make deliberate decisions regarding the stakes assigned to assessment tests, taking into account the specific evaluation criteria that they aim to emphasize.

Teachers’ perceptions of national large-scale assessment: the pedagogical dimension

ABSTRACT Large-scale assessments have become a basic national policy for educational improvement encouraging standards, decentralisation and school accountability. The current study focuses on the pedagogical dimension of large-scale assessments, examining its uses as a policy instrument for effecting pedagogical change. The paper presents and discusses the case of the Israeli NLSA (national large-scale assessment) regime – the Meitzav (Hebrew acronym for: Growth and Effectiveness Measures for Schools) tests. Although it aimed to design a low-stakes testing regime, its implementation was a top-down procedure which, in practice, restricts principals’ and teachers’ autonomy. Using qualitative and quantitative methods, the findings showed that the Meitzav test results are barely used as a means leading to pedagogical change to improve learning. Teachers considered these tests an unreliable assessment tool that in the main, does not reflect the school curriculum or student learning, while producing a high level of pressure on the teaching routine. In consequence of the Meitzav test results, the most common pedagogical change in practice chosen by teachers was: “teaching to the test”. Other pedagogical changes following the Meitzav were implemented to a minor extent. Policy implications are discussed.

An Illustration of an IRTree Model for Disengagement.

Low-stakes test performance commonly reflects examinee ability and effort. Examinees exhibiting low effort may be identified through rapid guessing behavior throughout an assessment. There has been a plethora of methods proposed to adjust scores once rapid guesses have been identified, but these have been plagued by strong assumptions or the removal of examinees. In this study, we illustrate how an IRTree model can be used to adjust examinee ability for rapid guessing behavior. Our approach is flexible as it does not assume independence between rapid guessing behavior and the trait of interest (e.g., ability) nor does it necessitate the removal of examinees who engage in rapid guessing. In addition, our method uniquely allows for the simultaneous modeling of a disengagement latent trait in addition to the trait of interest. The results indicate the model is quite useful for estimating individual differences among examinees in the disengagement latent trait and in providing more precise measurement of examinee ability relative to models ignoring rapid guesses or accommodating it in different ways. A simulation study reveals that our model results in less biased estimates of the trait of interest for individuals with rapid responses, regardless of sample size and rapid response rate in the sample. We conclude with a discussion of extensions of the model and directions for future research.

Examining the Question-Behavior Effect in Low-Stakes Testing Contexts: A Cheap Strategy to Increase Examinee Effort

ABSTRACT Questioning individuals about future behavior influences the subsequent performance of that behavior. This question-behavior effect may be enhanced when questions include positive self-identity prompts. To evaluate the effectiveness of this intervention in a testing context, we randomly assigned examinees to one of three conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to the test, answering five questions regarding intended effort that referenced positive self-identity prior to the test, or a control condition. We then administered two multiple-choice tests and collected two measures of test-taking effort: self-reported effort and response time effort. Answering questions about effort prior to completing the tests resulted in higher self-reported effort and perceived test importance. Moreover, in the question conditions, significantly fewer examinees were identified to be filtered from the dataset due to low effort. The easy strategy of asking examinees to report their intended effort appears to increase test-taking effort.

The role of student effort on performance in PISA: revisiting the gender gap in achievement

Abstract International assessments are important to benchmark the quality of education internationally. However, ignoring students’ efforts in low-stakes tests can lead to biased interpretations of test performance. We use data from the PISA to analyse the potential role of student effort in explaining gender achievement gaps across countries. We build two effort measures based on students’ response times to test questions (i.e. rapid-guessing rates) and the student effort on the post-test survey (i.e. item non-response rates). We find a strong association between student efforts and test scores. After accounting for differences in effort, the estimated gender achievement gap in mathematics and science could be up to 36% and 40% of an standard deviation wider, respectively, in favour of boys. Altogether, our effort measures on average explain between 51% and 56% of the cross-country variation in test scores.

Effects of response format on achievement and aptitude assessment results: multi-level random effects meta-analyses.

Psychological achievement and aptitude tests are fundamental elements of the everyday school, academic and professional lives of students, instructors, job applicants, researchers and policymakers. In line with growing demands for fair psychological assessment tools, we aimed to identify psychometric features of tests, test situations and test-taker characteristics that may contribute to the emergence of test bias. Multi-level random effects meta-analyses were conducted to estimate mean effect sizes for differences and relations between scores from achievement or aptitude measures with open-ended (OE) versus closed-ended (CE) response formats. Results from 102 primary studies with 392 effect sizes revealed positive relations between CE and OE assessments (mean r = 0.67, 95% CI [0.57; 0.76]), with negative pooled effect sizes for the difference between the two response formats (mean d av = -0.65; 95% CI [-0.78; -0.53]). Significantly higher scores were obtained on CE exams. Stem-equivalency of items, low-stakes test situations, written short answer OE question types, studies conducted outside the United States and before the year 2000, and test-takers' achievement motivation and sex were at least partially associated with smaller differences and/or larger relations between scores from OE and CE formats. Limitations and the results' implications for practitioners in achievement and aptitude testing are discussed.

Improved Behavioural Engagement of Students through Low-stakes Online Tests and Immediate Dialogic Feedback

The engagement of students is a recognised challenge for teachers. Technology offers some practical student engagement tools, and this paper examines the use of low-stakes online tests and immediate dialogic feedback to improve behavioural engagement. The academic exploration of low-stakes tests and dialogic feedback has been extensive, and they are credible teaching tools. In this study, we explore the learning benefit of their combination. Postgraduate engineering students' self-reported and learning analytics data shows conclusive evidence of improved behavioural engagement. We measured a 500% increase in the Learning Management System (LMS) page views on the days when we ran the low-stakes tests (each worth 2% of the marks for the subject) and engaged in immediate dialogic feedback. To interpret these results, we draw on theories of behavioural engagement, low-stakes tests, and feedback. We conclude that the combination of low-stakes tests and immediate feedback improves student behavioural engagement.

Non-cognitive skills and social gaps in digital skills: Evidence from ICILS 2018

Using data from the 2018 round of the International Computer and Literacy Survey (ICILS), this study looks at the effect of non-cognitive skills (e.g., motivation, ambition, and conscientiousness) on digital competences as measured by the Computer and Information Literacy (CIL) test score. Non-cognitive skills may be especially important in low-stakes tests such as ICILS, where students face no consequences — positive or negative — as a result of their performance. The empirical results show that several non-self-reported measures acting as proxies for non-cognitive skills are significant determinants of CIL test scores. Furthermore, the findings point at differences in non-cognitive skills across gender, immigrant background, and socioeconomic status. This suggests that one should be cautious when inferring about inequality in digital competences along these dimensions using low-stakes test scores, and underscores the importance of controlling for non-cognitive skills.

Predictors of Accessibility Tool Use on a Low-Stakes Computer-Based Math Test

One perceived advantage of computer-based testing is that accessibility tools can be embedded within the testing format, allowing students with disabilities to use them when necessary to remove unique barriers within testing. However, an important assumption is that students activate and use the tools when needed. Initial data from large-scale computer-based testing suggest many students with disabilities are not using them; information is needed to understand why. Both computer skills and motivation are likely necessary for students to use accessibility tools; therefore, we explored whether prior computer use, math motivation, and test motivation predicted accessibility tool use on a national math test. We further explored the relationship between accessibility tool use and test performance. Accessibility tool use was relatively infrequent. Test motivation was weakly associated with text-to-speech use. Use of eliminate choice and scratchwork tools were weakly associated with performance. When combined with related empirical work, findings suggest a potential need to improve student test motivation and corresponding use of accessibility tools to improve validity of low-stakes test scores. However, given the weak relationships identified between tool use and performance, evidence-based math interventions are anticipated to be more helpful for improving math performance than mere promotion of accessibility tool use.

Are Large Admissions Test Coaching Effects Widespread? A Longitudinal Analysis of Admissions Test Scores

ABSTRACT We examine longitudinal data from 120,384 students who took a version of the PSAT/SAT in the 9th, 10th, 11th, and 12th grades. We investigate score changes over time and show that socioeconomic status (SES) is related to the degree of score improvement. We note that the 9th and 10th grade PSAT are low-stakes tests, while the operational SAT is a high-stakes test. We posit that investments in coaching would be uncommon for early PSAT administrations, and would be concentrated on efforts to prepare for the operational SAT. We compare score improvements between 9th and 10th grade with improvements between 10th and 12th grade, examining results separately by level of SES. We find similar levels of score improvement in low-stakes and high-stakes settings, with 3.4% of high-SES and 1.1% of low-SES students showing larger-than-expected score improvements, which is inconsistent with claims that high-SES students have routine access to highly effective coaching.

Affective-motivational effects of performance feedback in computer-based assessment: Does error message complexity matter?

The impact of computer-based performance feedback on students’ affective-motivational state may be very different, depending on the positive or negative direction of the feedback message and its specific content. This experiment investigated whether more elaborated error messages improve students’ affective-motivational response to negative (i.e., corrective) feedback. We systematically varied the presence and complexity of corrective feedback messages (1 × 4 between-subjects design) and analyzed the effects of the provided feedback on students’ emotions, task-related perceived usefulness, and expectancy-value beliefs. University students (N = 439) worked on a low-stakes test with 12 constructed-response geometry tasks. They received either no feedback or different complexities of immediate corrective feedback after incorrect responses (i.e., Knowledge of Results [KR], Knowledge of Correct Response [KCR], or Elaborated Feedback [EF]), paired with immediate confirmatory KCR feedback after correct responses (i.e., confirming their response). Our data showed that students’ task-level performance moderated the emotional impact of feedback (i.e., beneficial effects after correct responses; detrimental effects after incorrect responses). Students’ performance further moderated several feedback effects on students’ expectancy-value beliefs. Regarding error message complexity, we found that students reported higher levels of positive emotions after receiving EF or KCR compared to KR, while only EF decreased students' level of negative emotions compared to KR and increased students' task-related perceived usefulness compared to all other groups. Overall, our results suggest that performance feedback is likely to improve students’ affective-motivational state when the feedback confirms a correct response. Moreover, when reporting an error, EF (or KCR messages) were more beneficial to affective-motivational outcomes than simple KR notifications.

Development and Internal Validity of the Student Opinion Scale: A Measure of Test-Taking Motivation

In low-stakes testing contexts, there are minimal personal consequences associated with examinee performance. Examples include assessments administered for research, program evaluation, test development, and international comparisons (e.g., Programme for International Student Assessment [PISA]). Because test-taking motivation can suffer in low-stakes conditions, the Student Opinion Scale (SOS) is commonly administered to measure test-taking effort and how personally important the examinee feels it is to do well on the test. Although popular, studies of the scale’s internal validity yield conflicting findings. The present study critically evaluates the creation of the SOS and considers its factor structure across six samples of college students differing in their college experience level and version of the SOS administered. Because findings only support the internal validity of the effort subscale, further study and development of the importance subscale is recommended.

Math Attitudes, Engagement, and Performance of High School Students on High and Low-stakes Tests of Statistics Knowledge

Understanding the extent engagement and math attitudes predict performance in statistics courses could inform educational interventions in this subject area, which has growing demand. We examined direct and indirect associations between course engagement-related constructs, math attitudes, and learning outcomes. Confirmatory factor analysis was used to validate scores from measures of these constructs with a sample of high school students enrolled in Advanced Placement (AP) statistics (N = 720, mean age = 16.8 years, SD = 0.82). Structural equation models were fitted to the data to examine relations between these constructs on a subsample (N = 220). A greater proportion of variation was explained in a high-stakes learning outcome (R2 = 0.54) than a low-stakes learning outcome (R2 = 0.24). We found some evidence of indirect effects of academic procrastination and course engagement on the learning outcome by way of math attitudes. The findings shed light on opportunities for intervention on academic maladaptive behaviors, such as procrastination, which could lessen negative effects on math attitudes and learning. These findings highlight the importance of testing stakes when examining associations between engagement, math attitudes, and learning, particularly in the context of high school statistics, a growing and yet understudied STEM learning context.

Uncovering the Complexity of Item Position Effects in a Low-Stakes Testing Context.

Previous researchers have only either adopted an item or examinee perspective to position effects, where they focused on exploring the relationships among position effects and item or examinee variables separately. Unlike previous researchers, we adopted an integrated perspective to position effects, where we focused on exploring the relationships among position effects, item variables, and examinee variables simultaneously. We evaluated the degree to which position effects on two separate low-stakes tests administered to two different samples were moderated by different item (item length, number of response options, mental taxation, and graphic) and examinee (effort, change in effort, and gender) variables. Items exhibited significant negative linear position effects on both tests, with the magnitude of the position effects varying from item to item. Longer items were more prone to position effects than shorter items; however, the level of mental taxation required to answer the item, the presence of a graphic, and the number of response options were not related to position effects. Examinee effort levels, change in effort patterns, and gender did not moderate the relationships among position effects and item features.