Low-stakes Tests Research Articles

An Investigation of Examinee Test-Taking Effort on a Large-Scale Assessment

Assessment results collected under low-stakes testing situations are subject to effects of low examinee effort. The use of computer-based testing allows researchers to develop new ways of measuring examinee effort, particularly using response times. At the item level, responses can be classified as exhibiting either rapid-guessing behavior or solution behavior based on the item response time. Most previous research involving the study of response times has been conducted using locally developed instruments. The purpose of the current study was to examine the amount of rapid-guessing behavior within a commercially available, low-stakes instrument. Results indicate that smaller amounts of rapid-guessing behavior exist within the data compared to published results using other instruments. Additionally, rapid-guessing behavior varied by item and was significantly related to item length, item position, and presence of ancillary reading material. The amount of rapid-guessing behavior was consistently very low among various demographic subpopulations. On average, rapid-guessing behavior was observed on only 1% of item responses. Also found was that a small amount of rapid-guessing behavior can impact institutional rankings.

Management Insights

Management Insights

Working When No One Is Watching: Motivation, Test Scores, and Economic Success

This paper provides evidence that scores on simple, low-stakes tests are associated with future economic success because the scores also reflect test takers' personality traits associated with their level of intrinsic motivation. To establish this, I use the coding speed test that was administered without incentives to participants in the National Longitudinal Survey of Youth (NLSY). I show that, controlling for cognitive ability, the coding speed scores are correlated with future earnings of male NLSY participants. I provide evidence that the coding speed scores relate to intrinsic motivation. I show that the scores of the highly motivated, though less educated, group (potential recruits to the U.S. military), are higher than the NLSY participants' scores. I use controlled experiments to show directly that intrinsic motivation is an important component of the unincentivized coding speed scores and that it relates to test takers' personality traits. This paper was accepted by Teck Ho, behavioral economics.

Achievement Goal Orientation and Situational Motivation for a Low-Stakes Test of Content Knowledge

To determine the extent of the relationship between students' inherent motivation to achieve in a doctor of pharmacy program and their motivation to achieve on a single low-stakes test of content knowledge. The Attitude Toward Learning Questionnaire (ATL) was administered to 66 third-year pharmacy students at the beginning of the spring 2011 semester, and the Student Opinion Scale (SOS) was administered to the same group immediately following completion of the Pharmacy Curricular Outcomes Assessment (PCOA). Significant differences were found in performance approach and work avoidance based on situational motivation scores. Situational motivation was also found to be directly correlated with performance and mastery approaches and inversely correlated with work avoidance. Criteria were met for predicting importance and effort from performance and mastery approaches and work avoidance scores of pharmacy students. The ability to predict pharmacy students' motivation to perform on a low-stakes standardized test of content knowledge increases the test's usefulness as a measure of curricular effectiveness.

Statewide low-stakes tests and a teaching to the test effect? An analysis of teacher survey data from two German states

Governments worldwide are changing their educational policies to improve student performance. Some of these reforms involve the introduction of high-stakes testing, which often creates negative effects such as ‘teaching to the test’. This article deals with the use of low-stakes testing in the German states of Hesse and Bremen, and analyses whether statewide low-stakes testing produces comparable results to the use of high-stakes tests from the perspective of teachers. Moreover, it seeks to identify the factors responsible for the observable teaching to the test effect, above and beyond the use of statewide exams. The data come from surveys carried out in the framework of a longitudinal study funded by the German Research Foundation. Hesse and Bremen introduced statewide exit exams in several subjects in 2007 and 2008 as low-stakes tests. Our quasi-experimental study has been evaluating these implementations since 2007.

Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment:

Abstract This article examines how the amount of time students spend on a low-stakes test, motivation, and local test administration procedures represent problematic intervening variables in the measurement of student learning. Results from successive administrations of the Collegiate Learning Assessment at a public comprehensive university reveal wide year-to-year variations in student performance that may be related to effort rather than cognitive ability. Findings call into question the efficacy of low-stakes testing to measure educational achievement or institutional effectiveness for accountability purposes.

Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment:

Abstract This article examines how the amount of time students spend on a low-stakes test, motivation, and local test administration procedures represent problematic intervening variables in the measurement of student learning. Results from successive administrations of the Collegiate Learning Assessment at a public comprehensive university reveal wide year-to-year variations in student performance that may be related to effort rather than cognitive ability. Findings call into question the efficacy of low-stakes testing to measure educational achievement or institutional effectiveness for accountability purposes.

The Validation Process in Developing a Web-based English Speaking and Writing Test

This article summarizes findings from investigations into the development and use of a Web-based English speaking and writing test for local use as a low-stakes test. The study focused on the validation process in language test development of the test following Kim’s (2006a) testing model. Two research questions were proposed to examine test validity and task difficulty of the test. 53 test takers at a Korean university participated in the test implementation and a post-survey. Validity arguments were gathered from design and validation implications in test development, test results, and survey data, which built validity evidence in the Usefulness Analysis Table (Bachman & Palmer, 1996). They evidenced validity of the test and its development. Research findings from the statistical analysis of the test results and survey reported that test takers need more preparation time and response time on writing and familiarity enhancement with task content. The findings on task difficulty across proficiency levels indicated that both groups showed similar perceptions of task difficulty while the test takers in the low level group revealed that their difficulty perceptions were inappropriate for overall and writing scores.

Low-Stakes Testing and Psychological Reactance: Using the Hong Psychological Reactance Scale to Better Understand Compliant and Non-Compliant Examinees

The current study examined whether psychological reactance differs across compliant and non-compliant examinees. Given the lack of consensus regarding the factor structure and scoring of the Hong Psychological Reactance Scale (HPRS), its factor structure was evaluated and subsequently tested for measurement invariance (configural, metric, and scalar) across two types of examinees: examinees that attended university assessments (i.e., compliant examinees) and examinees that skipped these assessments (i.e., non-compliant examinees). Measurement invariance of the HPRS across compliant and non-compliant examinees was supported, enabling the testing of latent mean differences, which provided known-groups validity evidence: non-compliant examinees reported significantly higher levels of reactance than did compliant examinees. Implications for low-stakes testing internationally, including strategies to increase compliance, are discussed.

Strategies to Motivate Students for Program Assessment

Accountability in higher education has increased, with more institutions requiring standardized tests. These tests are high stakes for institutions, but low-stakes test for students, who seldom experience consequences for their performance. This study describes how one psychology department improved students' scores on the Psychology Area Concentration Achievement Test. Results were compared between three motivation conditions: no incentive, a monetary incentive, and a motivational Microsoft PowerPoint presentation. The presentation gave students information about the assessment, encouraged them to do well, and informed them that faculty would discuss scores while evaluating the psychology program. Results showed that test scores were significantly higher and correlated significantly with grade point average for students exposed to the motivational presentation. The motivational PowerPoint presentation seemed to have reduced the number of underachieving students and provided more accurate assessment data, with minimal investment in time and effort on the part of faculty.

Self-monitoring and its relationship to medical knowledge

In the domain of self-assessment, researchers have begun to draw distinctions between summative self-assessment activities (i.e., making an overall judgment of one's ability in a particular domain) and self-monitoring processes (i.e., an "in the moment" awareness of whether one has the necessary knowledge or skills to address a specific problem with which one is faced). Indeed, previous research has shown that, when responding to both short answer and multiple choice questions, individuals are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test. These studies, however, were conducted in the context of low-stakes tests of general "trivia". The purpose of the present study was to further this line of research by investigating the relationship between self-monitoring and performance in the context of a high stakes test assessing medical knowledge. Using a recent administration of the Medical Council of Canada Qualifying Examination Part I, we examined three measures intended to capture self-monitoring: (1) the time taken to respond to each question, (2) the number of questions a candidate flagged as needing to be considered further, and (3) the likelihood of changing one's initial answer. Differences in these measures as a function of the accuracy of the candidate's response were treated as indices of each candidate's ability to judge his or her likelihood of responding correctly. The three self-monitoring indices were compared for candidates at three different levels of overall performance on the exam. Relative to correct responses, when examinees initially responded incorrectly, they spent more time answering the question, were more likely to flag the question for future consideration, and were more likely to change their answer before committing to a final answer. These measures of self-monitoring were modulated by candidate performance in that high performing examinees showed greater differences on these indices relative to poor performing examinees. Furthermore, reliability analyses suggest that these difference measures hold promise for reliably differentiating self-monitoring at the level of individuals, at least within a given content area. The results suggest that examinees were self-monitoring their knowledge and skills on a question by question basis and altering their behavior appropriately in the moment. High performing individuals showed stronger evidence of accurate self-monitoring than did low performing individuals and the reliability of these measures suggests that they have the potential to differentiate between individuals. How these findings relate to performance in actual clinical settings remains to be seen.

Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement.

Typically, teachers use tests to evaluate students’ knowledge acquisition. In a novel experimental study, we examined whether low-stakes testing (quizzing) can be used to foster students’ learning of course content in 8th grade science classes. Students received multiple-choice quizzes (with feedback); in the quizzes, some target content that would be included on the class summative assessments was tested, and some of the target content was not tested. In Experiment 1, three quizzes on the content were spaced across the coverage of a unit. Quizzing produced significant learning benefits, with between 13% and 25% gains in performance on summative unit examinations. In Experiments 2a and 2b, we manipulated the placement of the quizzing, with students being quizzed on some content prior to the lecture, quizzed on some immediately after the lecture, and quizzed on some as a review prior to the unit exam. Review quizzing produced the greatest increases in exam performance, and these increases were only slightly augmented when the items had appeared on previous quizzes. The benefits of quizzing (relative to not quizzing) persisted on cumulative semester and end-of-year exams. We suggest that the present effects reflect benefits accruing to retrieval practice, benefits that are well established in the basic literature.

Pharmacy Students' Test-Taking Motivation-Effort on a Low-Stakes Standardized Test

To measure third-year pharmacy students' level of motivation while completing the Pharmacy Curriculum Outcomes Assessment (PCOA) administered as a low-stakes test to better understand use of the PCOA as a measure of student content knowledge. Student motivation was manipulated through an incentive (ie, personal letter from the dean) and a process of statistical motivation filtering. Data were analyzed to determine any differences between the experimental and control groups in PCOA test performance, motivation to perform well, and test performance after filtering for low motivation-effort. Incentivizing students diminished the need for filtering PCOA scores for low effort. Where filtering was used, performance scores improved, providing a more realistic measure of aggregate student performance. To ensure that PCOA scores are an accurate reflection of student knowledge, incentivizing and/or filtering for low motivation-effort among pharmacy students should be considered fundamental best practice when the PCOA is administered as a low-stakes test.

Rise to the Challenge or Not Give a Damn: Differential Performance in High vs. Low Stakes Tests

This paper studies how different demographic groups respond to incentives by comparing performance in the GRE examination in and stakes situations. The high stakes situation is the real GRE examination and the low stakes situation is a voluntary experimental section of the GRE that examinees were invited to take immediately after they finished the real GRE exam. We show that males exhibit a larger difference in performance between the high and low stakes examinations than females, and that Whites exhibit a larger difference in performance between the high and low stakes examinations relative to Asians, Blacks, and Hispanics. We find that the larger differential performance between high and low stakes tests among men and whites can be partially explained by the lower level of effort invested by these groups in the low stake test.

Ecological rationality in teachers' conceptions of assessment across samples from Cyprus and New Zealand

Teacher conceptions of assessment are influential mediators of how assessment policy initiatives are implemented in schools. Four hierarchical, intercorrelated factors (i.e., assessment for improvement, school accountability, and student accountability, and assessment as irrelevant) of how teachers' conceive of assessment have been reported. However, most studies have been conducted only in English in jurisdictions with policies of low-stakes testing. This paper extends the research by surveying 249 Greek-Cypriot teachers with a Greek translation of the Teachers' Conceptions of Assessment inventory. Cyprus has a relatively low-stakes assessment policy during the compulsory school years, suggesting, under the assumption of ecological rationality, that conceptions would be similar to previous English-language studies. Exploratory factor analysis of the Cyprus data led to a five-factor solution with 24 items within two inversely correlated second-order factors (i.e., assessment is positive and negative; r = −0.49). A multigroup nested invariance confirmatory factor analysis found statistical invariance between the Cyprus and the New Zealand data. Mean score differences were small for two improvement-oriented conceptions, moderate for assessment that is bad, and large for school accountability and ignoring assessment factors. Similarities and differences in conceptions appear to reflect commonalities and discrepancies in educational system policies and practices.

Do Examinees Have Similar Test-Taking Effort? A High-Stakes Question for Low-Stakes Testing

Given the prevalence of low-stakes testing internationally (e.g., NAEP, TIMSS, PIRLS), it is crucial to try to better understand examinee motivation in these contexts. In the current study, mixture modeling results supported three different profiles of test-taking effort over the course of five tests. Classes 1 and 2 had varying levels of effort whereas Class 3 had steady levels of effort across the five tests. Examination of relationships between class membership and external variables suggested the classes could be differentiated by achievement goals, personality, and ability. Implications of these results for international assessment and directions for future research are discussed.

Can Differential Rapid-Guessing Behavior Lead to Differential Item Functioning?

This investigation examined whether different rates of rapid guessing between groups could lead to detectable levels of differential item functioning (DIF) in situations where the item parameters were the same for both groups. Two simulation studies were designed to explore this possibility. The groups in Study 1 were simulated to reflect differences between high-stakes and low-stakes conditions, with no rapid guessing in the high-stakes condition. Easy, discriminating items with high rates of rapid guessing by the low-stakes group were detected as showing DIF favoring the high-stakes group when using the Mantel-Haenszel index. The groups in Study 2 were simulated to reflect gender differences in rapid guessing on a low-stakes test. Both groups had some rapid guessing, but the focal group guessed more. Easy items with greater differences in rapid guessing were more likely to be detected as showing DIF. When the group with more rapid guessing had lower mean proficiency, the overall proportion of flagged items was lower but the effect of difference in rapid guessing remained. Our results suggest that there likely are instances in which statistically identified DIF is observed due to the behavioral characteristics of the studied subgroups rather than the content of the items.

Examinee Noneffort and the Validity of Program Assessment Results

Educational program assessment studies often use data from low-stakes tests to provide evidence of program quality. The validity of scores from such tests, however, is potentially threatened by examinee noneffort. This study investigated the extent to which one type of noneffort—rapid-guessing behavior—distorted the results from three types of commonly used program assessment designs. It was found that, for each design, a modest amount of rapid guessing had a pronounced effect on the results. In addition, motivation filtering was found to be successful in mitigating the effects caused by rapid guessing. It is suggested that measurement practitioners routinely apply motivation filtering whenever the data from low-stakes tests are used to support program decisions.

The relationship between motivation and achievement in low-stakes examinations

Achievement tests are used in several settings in applied measurement. A serious problem inherent in low-stakes tests is that it cannot be assumed that all examinees will make their best effort, which suggests that test scores of some examinees may underrate the level of academic performance of which they are capable. In this study the relationship between motivation and achievement in low-stakes examinations was examined. A sample (N = 797) of ninth-grade students sat either a mathematics or a science examination. It was found that a high level of motivation to take the examination invariably increased mean performance in both examinations and the correlation was significant.

Correlates of Rapid-Guessing Behavior in Low-Stakes Testing: Implications for Test Development and Measurement Practice

Previous research has shown that rapid-guessing behavior can degrade the validity of test scores from low-stakes proficiency tests. This study examined, using hierarchical generalized linear modeling, examinee and item characteristics for predicting rapid-guessing behavior. Several item characteristics were found significant; items with more text or those occurring later in the test were related to increased rapid guessing, while the inclusion of a graphic in a item was related to decreased rapid guessing. The sole significant examinee predictor was SAT total score. Implications of these results for measurement professionals developing low-stakes tests are discussed.