Test-taking Behavior Research Articles

Group differences in test-taking behaviour: an example from a high-stakes testing program

This study investigated whether different groups of test-takers vary in their reported test-taking behaviour in a high-stakes test situation. A between-group design (N = 1129) was used to examine whether high and low achievers, as well as females and males, differ in their use of test-taking strategies, and in level of reported test anxiety and motivation. The results showed differences between high and low achievers on a number of test-taking strategies, where high achievers reported using successful strategies to a higher extent. There were also gender differences: females, for example, reported using random guessing to a higher extent than males. Further, low achievers, especially females, reported significantly higher levels of test anxiety than high achievers, and high achievers reported slightly higher levels of motivation when compared to low achievers. To conclude, test-taking behaviour might bring additional variance to test scores, whether or not this is irrelevant variance is discussed.

Mean and distributional impact of single-sex high schools on students’ cognitive achievement, major choice, and test-taking behavior: Evidence from a random assignment policy in Seoul, Korea

Single-sex schooling has been considered in many countries as a way to promote student achievement. This paper estimates the mean and distributional impact of single-sex high schools on students’ cognitive achievement, major choice, and test-taking behavior—by exploiting the random assignment policy adopted in Seoul, Korea. Based on administrative data for a period of seven years, I find that, on average, the positive effects of single-sex schooling on test scores are small, especially when the parental and teacher sorting are accounted for. Although the magnitude of the estimated effects is small, I find that the effect is relatively larger for students in quantiles 0.5–0.8 of the distribution of test scores. The impact is trivial, on the other hand, for students located at the very bottom and the very top quantiles. Moreover, I do not find any differences, both practically and statistically, in major choice and test-taking behavior.

Testing Students with Special Educational Needs in Large-Scale Assessments - Psychometric Properties of Test Scores and Associations with Test Taking Behavior.

Assessing competencies of students with special educational needs in learning (SEN-L) poses a challenge for large-scale assessments (LSAs). For students with SEN-L, the available competence tests may fail to yield test scores of high psychometric quality, which are—at the same time—measurement invariant to test scores of general education students. We investigated whether we can identify a subgroup of students with SEN-L, for which measurement invariant competence measures of adequate psychometric quality may be obtained with tests available in LSAs. We furthermore investigated whether differences in test-taking behavior may explain dissatisfying psychometric properties and measurement non-invariance of test scores within LSAs. We relied on person fit indices and mixture distribution models to identify students with SEN-L for whom test scores with satisfactory psychometric properties and measurement invariance may be obtained. We also captured differences in test-taking behavior related to guessing and missing responses. As a result we identified a subgroup of students with SEN-L for whom competence scores of adequate psychometric quality that are measurement invariant to those of general education students were obtained. Concerning test taking behavior, there was a small number of students who unsystematically picked response options. Removing these students from the sample slightly improved item fit. Furthermore, two different patterns of missing responses were identified that explain to some extent problems in the assessments of students with SEN-L.

Factors Associated with High School Exit Exam Outcomes among Homeless High School Students

Little is known about academic performance among homeless high school students, although correlates of academic performance are well documented among their peers in the elementary and middle school grades. This study explores the relationship between student-level demographic and academic performance indicators (for example, grade point average [GPA], attendance, standardized test score performance) and high school exit exam performance (that is, whether students took and passed high school exit exams, respectively) in a sample of 10th- to 12th-grade students identified as homeless by their district’s McKinney–Vento program (N = 494 students in 25 high schools). Factors related to taking the exit exams included grade level, cumulative GPA, and previous test-taking behavior. Among students who had taken the exit exams, standardized test scores, GPA, and English fluency related to exit exam outcomes, suggesting the potential for academic skills remediation among this subgroup. The article concludes with a discussion of the implications for future research, practice, and policy.

Does previewing answer choice options improve performance on reading tests?

Previewing answer-choice options before finishing reading the text is a widely employed test-taking behavior. In the present study we examined whether previewing is related to item response accuracy and response time, using data from Chinese learners of varying English proficiency levels and English native speakers. We examined eye movement patterns of participants who completed online multiple-choice sentence completion tasks, and how previewing was related to reading performance and whether the relation varied as a function of English proficiency level. The results showed that, relative to no previewing, previewing was associated with a significantly lower probability of answering an item correctly but not with significantly longer response time. Importantly, these relations varied across English proficiency levels such that participants with higher proficiency performed better without previewing, but there was no difference for lower-intermediate learners of English. These findings suggest that previewing does not facilitate performance on a sentence comprehension task, but instead interferes with the comprehension process, particularly for individuals with relatively high language proficiency.

A mixture hierarchical model for response times and response accuracy

In real testing, examinees may manifest different types of test-taking behaviours. In this paper we focus on two types that appear to be among the more frequently occurring behaviours – solution behaviour and rapid guessing behaviour. Rapid guessing usually happens in high-stakes tests when there is insufficient time, and in low-stakes tests when there is lack of effort. These two qualitatively different test-taking behaviours, if ignored, will lead to violation of the local independence assumption and, as a result, yield biased item/person parameter estimation. We propose a mixture hierarchical model to account for differences among item responses and response time patterns arising from these two behaviours. The model is also able to identify the specific behaviour an examinee engages in when answering an item. A Monte Carlo expectation maximization algorithm is proposed for model calibration. A simulation study shows that the new model yields more accurate item and person parameter estimates than a non-mixture model when the data indeed come from two types of behaviour. The model also fits real, high-stakes test data better than a non-mixture model, and therefore the new model can better identify the underlying test-taking behaviour an examinee engages in on a certain item.

Test taking behaviour and test results

It has been found that in high-stakes testing situations the scores are considerably higher than in low-stakes situations, which implies that besides abilities, motivation plays a role in the results. We investigated the test-taking behaviour of first year students under the condition of absent external reward for testing results. Sample consisted of 326 first year students from Estonian institutions of higher education. The testing procedure was conducted via internet. Preliminary findings: The total test scores correlated with testing time moderately. The strongest correlation between test results and testing time was observed in mathematics subtest (r = .54). Self-reported motivation to do the test measured prior to testing had low predictive validity as for total test score and as for test-taking time. Test-taking motivation reported after completing the test was more closely associated with the test result (R2 = .18) and the time devoted to the test (R2 = .14).

Item-response theory modeling of IQ gains (the Flynn effect) on crystallized intelligence: Rodgers' hypothesis yes, Brand's hypothesis perhaps

Potential explanations for generational intelligence test score gains continue to be subject to intense debate and scrutiny in the scientific community. However, the explanatory value of some of the proposed causes remains difficult to determine, since only little empirical evidence is available. To clarify the role of two scarcely investigated theories accounting for the Flynn effect, this study set out to examine the role of changing test-taking behavior (Brand's hypothesis) and of a narrowing of the IQ ability distribution (Rodgers' hypothesis). Archival records of crystallized intelligence test performance over a time-span of 17years of a large number of psychiatric inpatients and outpatients in Austria were investigated (N=5445; 1978–94). This sample was particularly suitable to investigate our hypotheses since participants were under no pressure to perform which makes observed changes in test taking behavior attributable to personal style and ability rather than differential performance in pressure situations. Analytical approaches of both classical test theory and item response theory (IRT) yielded gains of 1.0 to 2.4 IQ points per decade. Test-taking behavior indicative of guessing and decreasing population IQ variability appeared to contribute both to IQ test score gains. IRT-based analyses showed that gains were largely preserved when controlling for highest educational qualification, while the test instrument showed measurement invariance between cohorts. However, IRT-based results also suggested that changes in test-taking behavior might not necessarily reflect increased guessing, but item drift instead. In all, this evidence emphasizes better performance of individuals of the lower tail of the IQ ability distribution in more recent years as one important contributing factor for generational IQ test score gains.

American College Students' Attitudes Toward Institutional Accountability Testing: Developing Measures

In the United States, government mandates for educational accountability have prompted an increase in testing in K-12 and higher education settings, resulting in a generation of millennial students who have undergone repeated assessment over the course of their educational careers. The cumulative effects of testing may range from increasingly antagonistic attitudes towards assessment to undesirable test-taking behavior, thereby compromising validity of inferences made from test scores. Even though validity of accountability testing is at the forefront of the debate on educational reforms, the role of students' attitudes as determinants of test scores is not well understood. The current study focused on the construction of two new measures of students' attitudes toward institutional accountability testing in K-12 (SAIAT-K-12) and in higher education (SAIAT-HE). The dimensionality of these measures, as well as their utility for examining the link between students' attitudes and test performance, are discussed.

Changes in test-taking patterns over time

The current study aims to investigate the relationship between right, wrong and missing answers to cognitive test items (test-taking patterns) in the context of the Flynn Effect (FE). We compare two cohorts of Estonian students (1933/36, n=890; 2006, n=913) using an Estonian adaptation of the National Intelligence Tests and document three simultaneous trends: fewer missing answers (−1 Cohen's d averaged over subtests), and a rise in the number of right and wrong answers to the subtests (average ds of .86 and .30, respectively). In the Arithmetical Reasoning and Vocabulary subtests, adjustments for false-positive answers (the number of right minus the number of wrong answers) reduced the size of the Flynn Effect by half. These subtests were supposed to be high g-loading subtests. Our conclusion is that rapid guessing has risen over time and influenced tests scores more strongly over the years. The FE is partly explained by changes in test-taking behavior over time.

Self-Efficacy Appraisals and Test-Taking Behavior of Students from Culturally Diverse Populations

This study examined the relationships between self-efficacy calibration and test-taking behavior as measured by the percentage of skipped items, unreached items, and incorrect responses on a mathematics test. Jewish and Arab 8th graders in four achievement levels, as defined by quartiles of the test score distribution, were compared with respect to their mathematics self-efficacy and their test-taking behavior. The results indicated that in every achievement level Arab students as compared to their Jewish counterparts tended to report higher levels of self-efficacy and to attempt more items on the test, which resulted in a higher rate of incorrect responses. The results support previous research findings pointing to the detrimental effect that overestimation of SE has on performance. The results were discussed with reference to the metacognitive processes involved in estimation of self-efficacy and in test performance. The identified gaps between the two ethnic groups were discussed in light of the learning culture that characterizes each of them. The contribution of the Israeli context, whereby students from two culturally diverse groups study according to the same mathematics curriculum but in separate schools, to understanding factors underlying culture-related group differences in mathematics test performance was underscored.

Computer Versus Paper Testing in Precollege Economics

Interest is growing at the precollege level in computer testing (CT) instead of paper-and-pencil testing (PT) for subjects in the school curriculum, including economics. Before economic educators adopt CT, a better understanding of its likely effects on test-taking behavior and performance compared with PT is needed. Using two volunteer student samples of CT and PT test scores collected as part of the field testing and national norming of the Test of Economic Knowledge (Walstad, Rebeck, and Butters 2010), the present authors investigated how CT and PT affect student test responses. The authors found that eighth- and ninth-grade students perform better with CT than PT, that CT has the potential to limit item guessing, and that CT may reduce item bias from the order of item placement on a test.

Test-taking behaviors in a neurocognitive assessment: Associations with school-age outcomes in a Finnish longitudinal follow-up.

Test-taking behaviors (i.e., task focus, maintenance of attention, and cooperation) affect children's cognitive test performance, and, thus, it is critical to take test-taking behavior into account when drawing conclusions and making recommendations. Prior studies have evaluated test-taking behaviors at the end of the assessment; the present study focused on the fluctuation of cooperation and attention during a neuropsychological assessment. We examined the attention and cooperation of 5-year-old children in a test-taking situation; the associations between these aspects of their test-taking behavior and the children's concurrent neurocognitive test performance, IQ, and parent-rated behavior; and the associations with their IQ, behavioral outcomes, and academic achievement at 8 years of age. The data (for 76 boys and 63 girls) were drawn from the Jyväskylä Longitudinal Study of Dyslexia (Lyytinen et al., 2001, 2004). All the children were Caucasian and spoke Finnish as their native language. As a whole, the 5-year-old children showed high cooperation and attention, but a slight decrement in test-taking behavior toward the end of the session was rather common. Three subgroups of children with different levels of cooperation and attention were identified. Children in the subgroup with nonoptimal attention and cooperation showed decreasing neurocognitive test performance toward the end of the assessment session. They also showed more inattentive behavior 3 years later. The findings imply that the examiner's observations of waning attention and cooperation during the assessment session are highly relevant, as these provide stable and clinically meaningful information about the child's behavioral tendencies.

Improvement of developmental outcome between 24 and 36 months corrected age in very preterm infants

To study early developmental course in preschool-aged very preterm infants and its association with perinatal risk factors and test-taking behaviour. Children born <30 weeks gestation and/or <1000g in the Academic Medical Center of Amsterdam were assessed at 24 and 36 months corrected age with the Dutch Bayley Scales of Infant Development-II (BSID-II-NL) and neurological examination. Linear regression analyses for developmental change were performed with perinatal risk factors. One hundred and forty-six children, mean GA 28 weeks and mean birth weight 1043 g, participated. Mental and psychomotor scores improved significantly with 6 and 7 points, respectively, from 24 to 36 months (p < 0.01). Mild to severe problems on at least one domain occurred less often at 36 (32%) compared to 24 months (63%) (p < 0.01), using corrected scores. Mental improvement was associated with being born very small for gestational age or <28 weeks; psychomotor improvement was associated with not being treated with indomethacin. Difficult test behaviour occurred mostly at 24 months and was associated with non-optimal development at 36 months. Improved developmental outcome and test behaviour were found at 36 compared to 24 months in a cohort of very preterm children. Long-term outcome studies and retesting of behaviourally difficult children are recommended.

Children's strategic regulation, metacognitive monitoring, and control processes during test taking

From the perspective of self-regulated learning, the interplay between learners' individual characteristics and the context of testing have been emphasized for assessing learning outcomes. The present study examined metacognitive processes in children's test-taking behaviour and explored their impacts on performance. Further, it was investigated whether differences in retrieval processes (operationalized through item difficulty) contribute to performance in strategic regulation skills. A total of 107 participants (8-/9- and 11-/12-year-olds) solved a cloze test including answerable (easy, medium, and difficult questions) and unanswerable questions about an earlier presented educational film, gave confidence judgments to every answer, and were then allowed to withdraw answers if they wished. Two different scoring schemes for test performance were compared to a control group. Analyses revealed relatively adequate monitoring processes when metacognitively distinguishing easy, difficult, and unanswerable items and correct and incorrect answers. At the same time, there was developmental progression in the ability to accurately monitor uncertainty. As to control processes, all children proved to be able to adjust their test-taking behaviour to the benefit of test accuracy by withdrawing mainly incorrect answers. Controlling was more efficient for easy than for difficult and unanswerable items. The study offers evidence for the impact of metacognitive skills in children's learning outcomes and documents strategic behaviour during test taking, as well as developmental progression in the involved skills. Further, findings underline the importance of memory retrieval for subsequent metacognitive processes.

The Word Associates Format: Validation evidence

Although the Word Associates Format (WAF) is becoming more frequently used as a depth-of-knowledge measure, relatively little validation has been carried out on it. This report of two validation studies tackles various important WAF issues yet to be satisfactorily resolved. Study 1 conducted introspective interviews regarding students’ WAF test-taking behavior along with interviews on featured target words to determine how accurately the most common scoring system for the WAF reflects the examinees’ actual knowledge of the words. Analysis is provided concerning WAF accuracy and item answering strategies and patterns. Study 2 repeated the interview procedures from Study 1 with several modifications, including the addition of a receptive dimension in the word knowledge interview. The various WAF-scoring methods were compared, and the format types (6 and 8 option), distractor types, and distribution of answers were examined in depth. Both studies indicate that the WAF reflects true lexical knowledge fairly well at the extremes of the scoring scale while scores in the middle do not lead to any reliable interpretation. Furthermore, there is the likelihood that the WAF may both underestimate and overestimate vocabulary knowledge. Suggestions regarding item construction and use of the WAF are given to improve its accuracy and reliability.

Undergraduates' Intentions to Take a Second Language Proficiency Test: A Comparison of Predictions from the Theory of Planned Behavior and Social Cognitive Theory

English competency has become essential for obtaining a better job or succeeding in higher education in Taiwan. Thus, passing the General English Proficiency Test is important for college students in Taiwan. The current study applied Ajzen's theory of planned behavior and the notions of outcome expectancy and self-efficacy from Bandura's social cognitive theory to investigate college students' intentions to take the General English Proficiency Test. The formal sample consisted of 425 undergraduates (217 women, 208 men; M age = 19.5 yr., SD = 1.3). The theory of planned behavior showed greater predictive ability (R2 = 33%) of intention than the social cognitive theory (R2 = 7%) in regression analysis and made a unique contribution to prediction of actual test-taking behavior one year later in logistic regression. Within-model analyses indicated that subjective norm in theory of planned behavior and outcome expectancy in social cognitive theory are crucial factors in predicting intention. Implications for enhancing undergraduates' intentions to take the English proficiency test are discussed.

Testing Person Fit in Cognitive Diagnosis

In cognitive diagnosis, the test-taking behavior of some examinees may be idiosyncratic so that their test scores may not reflect their true cognitive abilities as much as that of more typical examinees. Statistical tests are developed to recognize the following: (a) nonmasters of the required attributes who correctly answer the item (spuriously high scores) and (b) masters of the attributes who fail to correctly answer the item (spuriously low scores). For a person, nonzero probability of aberrant behavior is tested as the alternative hypothesis, against normal behavior as the null hypothesis. The two generalized likelihood ratio test statistics used, with the null hypothesis parameter on the boundary of the parameter space in each, have asymptotic distributions of a 50:50 mixture of a chi-square distribution with one degree of freedom and a degenerate distribution that is a constant of 0 under the null hypothesis. Simulation results, primarily based on the DINA model (deterministic inputs, noisy ‘‘AND’’ gate), are used to investigate the following: (a) how accurately the statistical tests identify normal/aberrant behaviors, (b) how the power of the tests depends on the length of the cognitive exam and the degree of the inclination toward aberrance, and (c) how sensitive the tests are to inaccurate estimation of model parameters.

Influence of behaviour and risk factors on motor performance in preterm infants at age 2 to 3 years

The aim of this cross-sectional study was to determine the influence of test-taking behaviour and risk factors for delayed motor performance in 437 preterm infants (244 males, 193 females; < or = 32 weeks of gestation) at the corrected age of 2 to 3 years (mean 29mo [SD 3.3]). Other mean (SD) sample demographics were: postmenstrual age 29(+5) weeks (1(+5)), range 25(+0)-32(+0); birthweight 1213.7g (331.7), range 468-2350; and days in the neonatal intensive care unit 21.1 (21.3), range 1-165. Children (n=23) with a severe disability were excluded. We assessed motor performance and behaviour during testing with the Motor Scale and the Behaviour Rating Scale (BRS) of the Bayley Scales of Infant Development, 2nd edition (BSID-II). Risk factors were tested against delayed motor performance as the dependent variable in binary logistic regression analysis. Median score on the Motor Scale in terms of the BSID-II Psychomotor Developmental Index (PDI) was 86. 'Delayed' motor performance was observed in 46.5% of the children tested, and behaviour was 'not-optimal' in 31.4%. The Motor Scale and BRS scores were significantly correlated (r(s)=0.62, p<0.01). Risk factors for delayed motor performance were: neonatal convulsions (odds ratio [OR] 4.5; 95% confidence interval [CI] 1.6-12.9), low maternal educational level (OR 3.3; 95% CI 1.7-6.5), male sex (OR 2.8; 95% CI 1.8-4.3), and chronic lung disease (OR 2.1; 95% CI 1.1- 4.1). We conclude that preterm infants are at high risk of delayed motor performance and non-optimal test-taking behaviour.

The Relationship Between Experience With Standardized Patient Examinations and Subsequent Standardized Patient Examination Performance

Standardized patient (SP) exams are becoming an important part of testing clinical skills. The dynamics of this test format warrant exploration. This study is designed to measure the relationship between experience with SPs and performance on a SP examination. A general linear model is used to determine if a relationship exists and the pattern of that relationship as experience increases. There is a linear association between increased experience and better performance in technical domains. In interpersonal skill domains, there is a curvilinear association between experience and performance with improved performance with moderate levels of experience but not in the most experienced group. Different levels of experience with SPs are associated with differences in performance. The varied pattern of associations suggests a possible practice effect or test-taking behavior. Additional studies are needed to assess this potential effect on SP exam validity.