Test-taking Effort Research Articles

Student characteristics and effort during test-taking

BackgroundAchievement tests are designed as measurement tools for student knowledge and learning, but also reflect student effort during the test. Understanding better what determines differences in (different dimensions of) effort can help in understanding what measured achievement differences reflect. AimWe analyze how test-taking effort relates to students’ demographic characteristics, past attainment and personality traits. Data13,791 9th grade students in the Netherlands, administered in 2012, 2014 and 2016, answering a total of 449,956 observations. MethodsWe distinguish between two measures of effort: solution behavior and response time given solution behavior. We estimate multi-level cross-classified models that include individual and test characteristics as predictors. We further include interaction terms between question position and individual characteristics, to identify how effort decline across the test differs by student type. ResultsGirls, high achievers, more agreeable, more conscientious and less extravert students exert more effort. Differences by past achievement are especially large and further increase along the test, while differences in other characteristics tend to be more stable. Effort differences by socioeconomic status are relatively small. ConclusionsSystematic differences between different types of students can partly reflect differences in test effort across these groups. Thus, test effort should be considered when analyzing achievement gaps and differences across learning outcomes. Tests with different test lengths imply different measures of educational inequality through differential effort declines.

The Impact of Non-Effortful Responding on Item and Person Parameters in Item-Pool Scaling Linking

ABSTRACT Low test-taking effort of examinees has been considered a source of construct-irrelevant variance in item response modeling, leading to serious consequences on parameter estimation. This study aims to investigate how non-effortful response (NER) influences the estimation of item and person parameters in item-pool scale linking (IPSL) and whether differential NER for two groups has a different effect on parameter estimation under the equivalent and non-equivalent group designs. Three simulation studies and an empirical study were conducted to answer these research questions. Results show that NER on items in a certain item set (e.g. common new items, exclusive new items) have a dominant negative effect on parameter estimation of items from the same item set, while NER on items from the item pool would seriously impact the accuracy of ability parameters. Discussions and recommendations for item set orders are provided.

Does 'summative' count? The influence of the awarding of study credits on feedback use and test-taking motivation in medical progress testing.

Despite the increasing implementation of formative assessment in medical education, its' effect on learning behaviour remains questionable. This effect may depend on how students value formative, and summative assessments differently. Informed by Expectancy Value Theory, we compared test preparation, feedback use, and test-taking motivation of medical students who either took a purely formative progress test (formative PT-group) or a progress test that yielded study credits (summative PT-group). In a mixed-methods study design, we triangulated quantitative questionnaire data (n = 264), logging data of an online PT feedback system (n = 618), and qualitative interview data (n = 21) to compare feedback use, and test-taking motivation between the formative PT-group (n = 316), and the summative PT-group (n = 302). Self-reported, and actual feedback consultation was higher in the summative PT-group. Test preparation, and active feedback use were relatively low and similar in both groups. Both quantitative, and qualitative results showed that the motivation to prepare and consult feedback relates to how students value the assessment. In the interview data, a link could be made with goal orientation theory, as performance-oriented students perceived the formative PT as not important due to the lack of study credits. This led to low test-taking effort, and feedback consultation after the formative PT. In contrast, learning-oriented students valued the formative PT, and used it for self-study or self-assessment to gain feedback. Our results indicate that most students are less motivated to put effort in the test, and use feedback when there are no direct consequences. A supportive assessment environment that emphasizes recognition of the value of formative testing is required to motivate students to use feedback for learning.

The effects of private and social incentives on students’ test-taking effort

Despite widespread educational reforms, the challenge of students’ suboptimal academic effort persists across numerous countries. We investigate the role of private and social incentives on test performance in real, low-stakes test settings. We implemented a multifaceted incentive program for three cohorts of Grade 7 students participating in a low-stakes national standardized test in Australia. Results indicate that incentives can improve test performance. Our experimental design ensures that performance gains are attributable to increased test-taking effort only. Performance gains in three subject domains are measured against a baseline domain with no incentives in a within-student design. Although the private incentives that targeted individual performance elicited larger improvements than a social incentive targeting group outcomes, the latter proved more cost efficient. Our intervention suggests valuable strategies to enhance assessment of student performance that may be biased due to suboptimal effort, especially in low-stakes scenarios that are often employed for educational policy evaluation.

Examining successful and unsuccessful time management through process data: two novel indicators of test-taking behaviors

Test-taking behavior is a potential source of construct irrelevant variance for test scores in international large-scale assessments where test-taking effort, motivation, and behaviors in general tend to be confounded with test scores. In an attempt to disentangle this relationship and gain further insight into examinees’ test-taking processes, researchers can now utilize process and timing data to obtain a more comprehensive view of test-taking behaviors, such as test-taking effort. The purpose of this study is to propose and evaluate two novel response-based, standardized indicators of test-taking behaviors that utilize a combination of examinee response and process (timing) data to better understand and describe test-taking effort in ILSAs. These indices were empirically estimated with USA data from two booklets from e-TIMSS 2019 in mathematics for grade 4. In addition, their predictive validity was examined with respect to achievement estimates. Their network of associations with other relevant variables such as motivation, interest in the subject, as well as across subjects were also examined to test their intra-individual stability in e-TIMSS.

Priming Examinees to Give Good Effort: Differential Utility Across Gender Identity

We examined the utility of a question-behavior effect (QBE) intervention on test-taking effort. Questioning individuals about future behavior influences the subsequent performance of that behavior. Previous research found this intervention to significantly increase test-taking effort for incoming university students but it was unknown if it would be effective for more advanced college students or differentially effective across gender identities. Thus, 503 more advanced college students were randomly assigned to one of three question conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to testing, answering five questions regarding intended effort that referenced positive self-identity prior to testing, or a no-question control condition. We then administered a multiple-choice test and collected two measures of test-taking effort: self-reported effort and response time. We found that students identifying as female had significantly and practically higher self-reported effort and response time when in the question conditions compared to the no question condition. There was no effect of condition for students identifying as male for either measure of test-taking effort. Implications for practically increasing motivation on low-stakes tests and surveys, especially for more advanced students, are discussed. Implications for QBE theory, including differential effects related to gender, are discussed.

Comparing Test-Taking Effort Between Paper-Based and Computer-Based Tests

The article compares the trajectories of students’ self-reported test-taking effort during a 120 minutes low-stakes large-scale assessment of English comprehension between a paper-and-pencil (PPA) and a computer-based assessment (CBA). Test-taking effort was measured four times during the test. Using a within-subject design, each of the N = 2,676 German ninth-grade students completed half of the test in PPA and half in CBA mode, where the sequence of modes was balanced between students. Overall, students’ test-taking effort decreased considerably during the course of the test. On average, effort was lower in CBA than in PPA. While on average, effort was lower in CBA than in PPA, the decline did not vary between both modes during the test. That is, students’ self-reported effort was higher if the items were easier (compared to students’ abilities). The consequences of these results concerning the further development of CBA tests and large-scale assessments in general are discussed.

How do test-takers rate their effort? A comparative analysis of self-report and log file data

The present study investigates students' test-taking effort by integrating and comparing traditional self-report questionnaire data and students' test-taking behavior, based on log data analyses. Previous studies have shown that different methods often lead to different results. A computer-based measure of complex problem-solving in uncertain situations was used to minimize the influence of factual knowledge on test performance. K-means cluster analysis was used to build groups of students differing in test-taking effort, resulting in 3 distinct groups. The correlation between students' test-taking effort and test performance proved to be weaker based on the self-reported questionnaire data than on their actual test-taking behavior. Both the self-report questionnaire and the log data showed a decrease in test-taking effort during the test. The number of clicks played the largest role in predicting performance. Results suggest that (1) self-report questionnaire data are not consistent with students' actual test-taking behavior and (2) it's not necessary to make the maximum effort to obtain valid test results, but a certain level of effort is needed.

Test-taking motivation and performance: Do self-report and time-based measures of effort reflect the same aspects of test-taking motivation?

Several studies found a low correlation between two measures of test-taking effort: self-reported effort (SRE) and response time effort (RTE). This study examined test-taking motivation in low-stake cognitive ability testing (n = 1614) by applying expectancy-value theory as the framework. We investigated the complex relationship between test performance and test-taking motivation aspects (expectancy, importance, interest, test anxiety, time cost, and test-taking effort). Furthermore, we used both SRE and RTE, allowing us to examine whether the two measures of effort relied on the same underlying mechanism of test-taking motivation. Our finding showed that SRE and RTE simultaneously explained more than half of test performance variance, with the predictive power of RTE being higher. RTE and SRE were correlated lower than expected (r = 0.28). SRE is best predicted by expectancy, while RTE is best predicted by test anxiety. In practice, if motivation-filtering procedures are needed, it is better to use RTE.

Examining the Question-Behavior Effect in Low-Stakes Testing Contexts: A Cheap Strategy to Increase Examinee Effort

ABSTRACT Questioning individuals about future behavior influences the subsequent performance of that behavior. This question-behavior effect may be enhanced when questions include positive self-identity prompts. To evaluate the effectiveness of this intervention in a testing context, we randomly assigned examinees to one of three conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to the test, answering five questions regarding intended effort that referenced positive self-identity prior to the test, or a control condition. We then administered two multiple-choice tests and collected two measures of test-taking effort: self-reported effort and response time effort. Answering questions about effort prior to completing the tests resulted in higher self-reported effort and perceived test importance. Moreover, in the question conditions, significantly fewer examinees were identified to be filtered from the dataset due to low effort. The easy strategy of asking examinees to report their intended effort appears to increase test-taking effort.

Conversation-based assessment: A novel approach to boosting test-taking effort in digital formative assessment

Conversation-based assessment: A novel approach to boosting test-taking effort in digital formative assessment

Development and Internal Validity of the Student Opinion Scale: A Measure of Test-Taking Motivation

In low-stakes testing contexts, there are minimal personal consequences associated with examinee performance. Examples include assessments administered for research, program evaluation, test development, and international comparisons (e.g., Programme for International Student Assessment [PISA]). Because test-taking motivation can suffer in low-stakes conditions, the Student Opinion Scale (SOS) is commonly administered to measure test-taking effort and how personally important the examinee feels it is to do well on the test. Although popular, studies of the scale’s internal validity yield conflicting findings. The present study critically evaluates the creation of the SOS and considers its factor structure across six samples of college students differing in their college experience level and version of the SOS administered. Because findings only support the internal validity of the effort subscale, further study and development of the importance subscale is recommended.

Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating.

Score equating is an essential tool in improving the fairness of test score interpretations when employing multiple test forms. To ensure that the equating functions used to connect scores from one form to another are valid, they must be invariant across different populations of examinees. Given that equating is used in many low-stakes testing programs, examinees' test-taking effort should be considered carefully when evaluating population invariance in equating, particularly as the occurrence of rapid guessing (RG) has been found to differ across subgroups. To this end, the current study investigated whether differential RG rates between subgroups can lead to incorrect inferences concerning population invariance in test equating. A simulation was built to generate data for two examinee subgroups (one more motivated than the other) administered two alternative forms of multiple-choice items. The rate of RG and ability characteristics of rapid guessers were manipulated. Results showed that as RG responses increased, false positive and false negative inferences of equating invariance were respectively observed at the lower and upper ends of the observed score scale. This result was supported by an empirical analysis of an international assessment. These findings suggest that RG should be investigated and documented prior to test equating, especially in low-stakes assessment contexts. A failure to do so may lead to incorrect inferences concerning fairness in equating.

An investigation of item, examinee, and country correlates of rapid guessing in PISA

The objective of the present study was to investigate item-, examinee-, and country-level correlates of rapid guessing (RG) in the context of the 2018 PISA science assessment. Analyzing data from 267,148 examinees across 71 countries showed that over 50% of examinees engaged in RG on an average proportion of one in 10 items. Descriptive differences were noted between countries on the mean number of RG responses per examinee with discrepancies as large as 500%. Country-level differences in the odds of engaging in RG were associated with mean performance and regional membership. Furthermore, based on a two-level cross-classified hierarchical linear model, both item- and examinee-level correlates were found to moderate the likelihood of RG. Specifically, the inclusion of items with multimedia content was associated with a decrease in RG. A number of demographic and attitudinal examinee-level variables were also significant moderators, including sex, linguistic background, SES, and self-rated reading comprehension, motivation mastery, and fear of failure. The findings from this study imply that select subgroup comparisons within and across nations may be biased by differential test-taking effort. To mitigate RG in international assessments, future test developers may look to leverage technology-enhanced items.

Between- versus Within-Examinee Variability in Test-Taking Effort and Test Emotions during a Low-Stakes Test

ABSTRACT When tests are low stakes for examinees, meaning there are little to no personal consequences associated with test results, some examinees put little effort into their performance. To understand the causes and consequences of diminished effort, researchers correlate test-taking effort with other variables, such as test-taking emotions and test performance. Most studies correlate examinees’ overall level of test-taking effort with other variables, with fewer studies considering variables related to changing effort levels during testing. To understand if fluctuations in effort during testing relate to fluctuations in emotions, we collected effort and emotions (anger, boredom, emotionality, enjoyment, pride, worry) data from 768 university students three times during a low-stakes institutional accountability test. Examinees greatly varied in their average levels of effort and average levels of emotions during testing; relatively less variability was observed in these variables during testing. Average levels of emotions were predictive of effort, but fluctuations in emotions during testing were not. Our findings indicate that researchers should consider the proportion of intraindividual and interindividual variability in effort when considering predictors of test-taking effort.

Student test-taking effort in low-stakes assessments: evidence from the English version of the PISA 2015 science test

BackgroundThe idea of using low-stakes assessment results is often mentioned when designing educational system reforms. However, when tests have no consequences for the students, test takers may not make enough effort when completing the test, and their lack of engagement may negatively affect the validity of the conclusions of the studies that use such tests. This article presents analyses of student test-taking engagement in a low-stakes international large-scale assessment, the Programme for International Student Assessment (PISA), and (i) quantifies test-taking effort, (ii) predicts effort by means of expectancy-value proxies and (iii) investigates the relationship between effort and science proficiency.MethodsStudents’ response times on the science items of PISA 2015 were used to derive an index of test-taking effort. Data from six English-speaking countries that administered the computer-based version of the test, were selected. The response time for each item was modelled by means of a two-class finite mixture model, and students’ probabilities of being classified as effortful were combined to derive a global index of effort.ResultsOur findings showed that students’ effort decreased towards the end of the testing sessions. The variance of examinees’ test-taking effort was not substantially explained by the expectancy-value variables. Test-taking effort had a strong relationship with science achievement, with the correlation increasing to more than 0.5 towards the end of the test. Moreover, an important part of the relationship between test-taking effort and achievement is not related to other student characteristics, such as gender, socio-economic and cultural status, attitude towards school or attitude towards science.ConclusionsThis study shows that students put different amounts of effort into test taking, especially towards the end of the assessment, and suggests a possible underestimation of related student achievement that may affect the interpretation of test scores.

Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process

The prevalence and serious consequences of noneffortful responses from unmotivated examinees are well-known in educational measurement. In this study, we propose to apply an iterative purification process based on a response time residual method with fixed item parameter estimates to detect noneffortful responses. The proposed method is compared with the traditional residual method and noniterative method with fixed item parameters in two simulation studies in terms of noneffort detection accuracy and parameter recovery. The results show that when severity of noneffort is high, the proposed method leads to a much higher true positive rate with a small increase of false discovery rate. In addition, parameter estimation is significantly improved by the strategies of fixing item parameters and iteratively cleansing. These results suggest that the proposed method is a potential solution to reduce the impact of data contamination due to severe low test-taking effort and to obtain more accurate parameter estimates. An empirical study is also conducted to show the differences in the detection rate and parameter estimates among different approaches.

Improving Test-Taking Effort in Low-Stakes Group-Based Educational Testing: A Meta-Analysis of Interventions

ABSTRACT Four decades of research have shown that students’ low test-taking effort is a serious threat to the validity of score-based inferences from low-stakes, group-based educational assessments. This meta-analysis sought to identify effective interventions for improving students’ test-taking effort in such contexts. Included studies (a) used a treatment-control group design; (b) administered a low-stakes group-based educational assessment; (c) employed an intervention to improve test-taking motivation; and (d) evaluated test-taking effort and/or test performance as outcomes. The analysis included 53 studies (N = 59,096) that produced 60 and 105 effect sizes of test-taking effort and test performance, respectively. On average, interventions were found to improve test-taking effort and test performance by 0.13 standard deviations (SD) each. The largest gains in test-taking effort were observed when providing external incentives followed by increasing test relevance, while no significant differences were found between these two intervention types in improving test performance. Furthermore, negligible impact was detected on both dependent variables for interventions that modified assessment design or promised feedback. Recommendations for future research and practice are discussed.

Is Differential Noneffortful Responding Associated With Type I Error in Measurement Invariance Testing?

Low test-taking effort as a validity threat is common when examinees perceive an assessment context to have minimal personal value. Prior research has shown that in such contexts, subgroups may differ in their effort, which raises two concerns when making subgroup mean comparisons. First, it is unclear how differential effort could influence evaluations of scale property equivalence. Second, if attaining full scalar invariance, the degree to which differential effort can bias subgroup mean comparisons is unknown. To address these issues, a simulation study was conducted to examine the influence of differential noneffortful responding (NER) on evaluations of measurement invariance and latent mean comparisons. Results showed that as differential rates of NER grew, increased Type I errors of measurement invariance were observed only at the metric invariance level, while no negative effects were apparent for configural or scalar invariance. When full scalar invariance was correctly attained, differential NER led to bias of mean score comparisons as large as 0.18 standard deviations with a differential NER rate of 7%. These findings suggest that test users should evaluate and document potential differential NER prior to both conducting measurement quality analyses and reporting disaggregated subgroup mean performance.

Model meets reality: Validating a new behavioral measure for test-taking effort

ABSTRACT Identifying and considering test-taking effort is of utmost importance for drawing valid inferences on examinee competency in low-stakes tests. Different approaches exist for doing so. The speed-accuracy+engagement model aims at identifying non-effortful test-taking behavior in terms of nonresponse and rapid guessing based on responses and response times. The model allows for identifying rapid-guessing behavior on the item-by-examinee level whilst jointly modeling the processes underlying rapid guessing and effortful responding. To assess whether the model indeed provides a valid measure of test-taking effort, we investigate (1) convergent validity with previously developed behavioral as well as self-report measures on guessing behavior and effort, (2) fit within the nomological network of test-taking motivation derived from expectancy-value theory, and (3) ability to detect differences between groups that can be assumed to differ in test-taking effort. Results suggest that the model captures central aspects of non-effortful test-taking behavior. While it does not cover the whole spectrum of non-effortful test-taking behavior, it provides a measure for some aspects of it, in a manner that is less subjective than self-reports. The article concludes with a discussion of implications for the development of behavioral measures of non-effortful test-taking behavior.