Test-taking Behavior Research Articles

Providing a brief corrective statement does not improve test performance in patients invalidating testing: A multisite, single-blind randomized controlled trial

Objective: Performance below the actual abilities of the examinee can be measured using performance validity tests (PVTs). PVT failure negatively impacts the quality of the neuropsychological assessment. In our study, we addressed this issue by providing a brief corrective statement regarding invalidity to improve test-taking behavior. Methods: This study is a multisite single-blind randomized controlled trial in a consecutive sample of clinically referred adult patients (N = 196) in a general hospital setting. Patients who failed a PVT (n = 71) were randomly allocated to a corrective statement approach (CS; n = 39), in which a brief verbal corrective statement was given by the technician, or received no corrective statement upon indications of invalid performance (NO-CS; n = 32). Both groups (CS and NO-CS) were provided with the same subsequently repeated and newly administered tests. Results: There were no group (CS vs. NO-CS) differences on both the repeated and single-administered PVTs and standard cognitive tests. Furthermore, invalid performing participants benefited significantly less from the repeated test administration compared to the valid performing group. Conclusions: Our study found that a brief corrective within-session statement, to address PVT failure and improve test-taking behavior, did not improve consequent test performance. These results suggest limited value of a brief verbal corrective statement to influence performance below best of capabilities. It highlights the need for more research to identify more effective approaches that can enhance patients test-taking behavior. Ultimately, such efforts are critical in ensuring accurate diagnosis and effective treatment recommendations for patients.

Online Reading Comprehension Assessment of Primary School Students: Analysis of Testing Behavior Indicators

Online Reading Comprehension Assessment of Primary School Students: Analysis of Testing Behavior Indicators

A Pilot Study for Measuring Complex Problem-Solving in Jordan: Feasibility, Construct Validity, and Behavior Pattern Analyses

In the 21st century, complex problem-solving (CPS) serves as a key indicator of educational achievement. However, the elements of successful CPS have not yet been fully explored. This study investigates the role of strategic exploration and different problem-solving and test-taking behaviors in CPS success, using logfile data to visualize and quantify students’ problem-solving behavior on 10 CPS problems with different characteristics and levels of difficulty. Additionally, in the present study, we go beyond the limits of most studies that focus on students’ problem-solving behavior pattern analyses in European cultures and education systems to examine Arabic students’ CPS behavior. The results show that computer-based assessments of CPS are feasible and valid in Jordanian higher education. The findings also confirm the structural validity of CPS, indicating that the processes of knowledge acquisition (KAC) and knowledge application (KAP) can be distinguished and separated in the problem-solving process. Large differences were identified in students’ test-taking behavior in terms of the efficacy of their exploration strategy. We identified four latent classes based on the students’ exploration strategy behavior. The study thus leads to a better understanding of how students solve problems and behave during the problem-solving process in uncertain situations.

Examining successful and unsuccessful time management through process data: two novel indicators of test-taking behaviors

Test-taking behavior is a potential source of construct irrelevant variance for test scores in international large-scale assessments where test-taking effort, motivation, and behaviors in general tend to be confounded with test scores. In an attempt to disentangle this relationship and gain further insight into examinees’ test-taking processes, researchers can now utilize process and timing data to obtain a more comprehensive view of test-taking behaviors, such as test-taking effort. The purpose of this study is to propose and evaluate two novel response-based, standardized indicators of test-taking behaviors that utilize a combination of examinee response and process (timing) data to better understand and describe test-taking effort in ILSAs. These indices were empirically estimated with USA data from two booklets from e-TIMSS 2019 in mathematics for grade 4. In addition, their predictive validity was examined with respect to achievement estimates. Their network of associations with other relevant variables such as motivation, interest in the subject, as well as across subjects were also examined to test their intra-individual stability in e-TIMSS.

Trends in Participation in Rubella Antibody Testing among Working-Aged Men during Annual Health Check-Ups from 2019 to 2022 in Japan.

In 2019, the Japanese government established a scheme for rubella antibody testing of men born between 1962 and 1978 during workplace health check-ups. However, the use of vouchers for rubella antibody testing was limited. Health check-up data analyses are needed to determine the reason why rubella antibody testing is not widely used. In this study, we aimed to describe changes in rubella antibody test-taking behavior during health check-ups during the first 3 years of the rubella catch-up campaign in Japan. In 2019, 2020, and 2021 (2020 in some areas) vouchers were sent to men born during the fiscal years 1972-1978, 1966-1971, and 1962-1965, respectively. We calculated the prevalence in men born between 1962 and 1978 who underwent rubella antibody testing during mandatory health check-ups under the Industrial Health and Safety Act. Rubella antibody testing uptake was relatively high (approximately 15%) in all three age groups soon after the distribution of the vouchers and then declined to below 2% during the second and third years. Further population-based approaches with continuous public engagement are required in workplaces to effectively promote and expand the rubella vaccination program in Japan.

Understanding transitions in complex problem-solving: Why we succeed and where we fail

Problem-solving is an essential transversal and learning skill in the 21st century. To better support it, we need a deeper understanding of its processes, i.e., problem exploration, problem representation and knowledge use. Understanding the mechanism behind the transitions between these processes plays an important role in supporting its development. With a sample of N = 1828 first-year university students, this study is the first to measure how students transition between the different problem-solving processes and how this influences their overall problem-solving performance as well as how this is related to their other test-taking behaviours. Results indicate that mastering the first transition – correctly understanding and depicting the problem structure – plays a crucial role in problem-solving. Students who failed to master it regularly failed in the next transition. Problem complexity strongly influenced transitions during the entire problem-solving process. Based on students’ behavioural patterns, we distinguished four qualitatively different latent transition classes: expert transitioners, advanced transitioners, beginner transitioners and non-transitioners. The number of interactions proved to be a more effective profile characteristic, especially on high-complexity problems, than time spent on the problem-solving process. The results of the current study provide important insights into how students’ transition between the different problem-solving processes and how their test-taking behaviour indicates their transition skills.

Change in Engagement During Test Events: An Argument for Weighted Scoring?

ABSTRACT Recent research has provided evidence that performance change during a student’s test event can indicate the presence of test-taking disengagement. Meaningful performance change implies that some portions of the test event reflect assumed maximum performance better than others and, because disengagement tends to diminish performance, lower-performing portions are less likely to reflect maximum performance than higher-performing portions. This empirical study explored the use of differential weighting of item responses during scoring, with weighting schemes representing either declining or increasing performance. Results indicated that weighted scoring could substantially decrease the score distortion due to disengagement factors and thereby improve test score validity. The study findings support the use of scoring procedures that manage disengagement by adapting to student test-taking behavior.

Multimodal Data Fusion to Detect Preknowledge Test-Taking Behavior Using Machine Learning.

In various fields, including college admission, medical board certifications, and military recruitment, high-stakes decisions are frequently made based on scores obtained from large-scale assessments. These decisions necessitate precise and reliable scores that enable valid inferences to be drawn about test-takers. However, the ability of such tests to provide reliable, accurate inference on a test-taker's performance could be jeopardized by aberrant test-taking practices, for instance, practicing real items prior to the test. As a result, it is crucial for administrators of such assessments to develop strategies that detect potential aberrant test-takers after data collection. The aim of this study is to explore the implementation of machine learning methods in combination with multimodal data fusion strategies that integrate bio-information technology, such as eye-tracking, and psychometric measures, including response times and item responses, to detect aberrant test-taking behaviors in technology-assisted remote testing settings.

Gender differences in item nonresponse in the PISA 2018 student questionnaire

Gender differences in item nonresponse are well-documented in high-stakes achievement tests, where female students are shown to omit more items than male students. These gender differences in item nonresponse are often linked to differential risk-taking strategies, with females being risk-averse and unwilling to guess on an item, even if it could gain them credits. In low-stakes settings, similar trends should not apply, as the students carry no consequence for their performance. Instead, test-taking motivation is argued to be the pivoting factor, with female students seen as more motivated and omitting fewer items than male students. In contrast to the high- and low-stakes achievement tests, less is known about gender differences in item nonresponse in student background questionnaires. Using cross-classified mixed effects models, we examined gender differences in item nonresponse on the Programme for International Student Assessment (PISA) 2018 student questionnaire across 80 countries and 71 scales. On average, the odds of male students omitting a questionnaire item were double the odds of female students, consistent with the expected trend in the low-stakes setting. However, we show that gender differences in item nonresponse are not merely a function of the stakes involved for individual students but a more complex phenomenon that is context-dependent and not necessarily stable across countries, scales’ formats, and contents. We argue that examining differences in item nonresponse patterns could serve as a source of additional information about the students’ test-taking behaviour and the quality of the questionnaire.

Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times

Change-point analysis (CPA) is a method for detecting abrupt changes in parameter(s) underlying a sequence of random variables. It has been applied to detect examinees’ aberrant test-taking behavior by identifying abrupt test performance change. Previous studies utilized maximum likelihood estimations of ability parameters, focusing on detecting one change point for each examinee. This article proposes a Bayesian CPA procedure using response times (RTs) to detect abrupt changes in examinee speed, which may be related to aberrant responding behaviors. The lognormal RT model is used to derive a procedure for detecting aberrant RT patterns. The method takes the numbers and locations of the change points as parameters in the model to detect multiple change points or multiple aberrant behaviors. Given the change points, the corresponding speed of each segment in the test can be estimated, which enables more accurate inferences about aberrant behaviors. Simulation study results indicate that the proposed procedure can effectively detect simulated aberrant behaviors and estimate change points accurately. The method is applied to data from a high-stakes computerized adaptive test, where its applicability is demonstrated.

How do test-takers rate their effort? A comparative analysis of self-report and log file data

The present study investigates students' test-taking effort by integrating and comparing traditional self-report questionnaire data and students' test-taking behavior, based on log data analyses. Previous studies have shown that different methods often lead to different results. A computer-based measure of complex problem-solving in uncertain situations was used to minimize the influence of factual knowledge on test performance. K-means cluster analysis was used to build groups of students differing in test-taking effort, resulting in 3 distinct groups. The correlation between students' test-taking effort and test performance proved to be weaker based on the self-reported questionnaire data than on their actual test-taking behavior. Both the self-report questionnaire and the log data showed a decrease in test-taking effort during the test. The number of clicks played the largest role in predicting performance. Results suggest that (1) self-report questionnaire data are not consistent with students' actual test-taking behavior and (2) it's not necessary to make the maximum effort to obtain valid test results, but a certain level of effort is needed.

Some thoughts on analytical choices in the scaling model for test scores in international large-scale assessment studies

International large-scale assessments (LSAs), such as the Programme for International Student Assessment (PISA), provide essential information about the distribution of student proficiencies across a wide range of countries. The repeated assessments of the distributions of these cognitive domains offer policymakers important information for evaluating educational reforms and received considerable attention from the media. Furthermore, the analytical strategies employed in LSAs often define methodological standards for applied researchers in the field. Hence, it is vital to critically reflect on the conceptual foundations of analytical choices in LSA studies. This article discusses the methodological challenges in selecting and specifying the scaling model used to obtain proficiency estimates from the individual student responses in LSA studies. We distinguish design-based inference from model-based inference. It is argued that for the official reporting of LSA results, design-based inference should be preferred because it allows for a clear definition of the target of inference (e.g., country mean achievement) and is less sensitive to specific modeling assumptions. More specifically, we discuss five analytical choices in the specification of the scaling model: (1) specification of the functional form of item response functions, (2) the treatment of local dependencies and multidimensionality, (3) the consideration of test-taking behavior for estimating student ability, and the role of country differential items functioning (DIF) for (4) cross-country comparisons and (5) trend estimation. This article’s primary goal is to stimulate discussion about recently implemented changes and suggested refinements of the scaling models in LSA studies.

Measuring depth of academic vocabulary knowledge

In second language (L2) vocabulary studies there is continuing interest in tests of depth of vocabulary knowledge, measuring various aspects of word knowledge other than just the form–meaning link that is the focus of breadth (or size) tests. This study aimed to explore new formats that could be used as diagnostic tools for assessing depth of academic vocabulary knowledge. The participants were 222 first-year students taking a programme of English for academic purposes (EAP) at a university in Vietnam. The depth measure was a newly developed test of a sample of words from Gardner and Davies’ Academic Vocabulary List, including sub-tests on receptive knowledge of synonyms, collocations and word parts. The participants also took the Updated Vocabulary Levels Test as a breadth measure, as well as a translation test. Results showed that most of the students had a relatively limited knowledge of general English vocabulary. There was a moderate correlation between the breadth and depth tests. The students had the best knowledge of synonyms, followed by word parts and then collocations. The collocation and word parts sub-tests included a Not Sure option to discourage blind guessing, and analyses of responses to this option offered useful insights into the students’ test-taking behaviour.

Not-reached Items: An Issue of Time and of test-taking Disengagement? the Case of PISA 2015 Reading Data

ABSTRACT Many low-stakes assessments, such as international large-scale surveys, are administered during time-limited testing sessions and some test-takers are not able to endorse the last items of the test, resulting in not-reached (NR) items. However, because the test has no consequence for the respondents, these NR items can also stem from quitting the test. This article, by means of mixture modeling, investigates heterogeneity in the onset of NR items in reading in PISA 2015. Test-taking behavior, assessed by the response times on the first items of the test, and the risk of NR item onset are modeled simultaneously in a 3-class model that distinguishes rapid, slow and typical respondents. Results suggest that NR items can come from a lack of time or from disengaged behaviors and that the relationship between the number of NR items and ability estimate can be affected by these non-effortful NR responses.

How we explore, interpret, and solve complex problems: A cross-national study of problem-solving processes

Complex problem solving (CPS) is considered an important educational outcome in the 21st century. Despite its importance, we have only little only knowledge of its measurability, development, or comparability in some countries, in particular in those with a short history of computer-based assessment. The results of the current study provide insights into the validity of CPS measurements and shed light on the different behavioral patterns and test-taking behavior in two convenience samples with different sample characteristics of Jordanian (N = 431) and Hungarian (N = 1844) students as they solve complex problems. CPS proved to be measurement-invariant in Jordan and Hungary among university students. Analyzing log data, we identified differences in students' test-taking behavior in terms of the effectiveness of their exploration strategy, time-on-task, and number of trials. Based on the students’ exploration strategy behavior, we identified four latent classes in both samples. The tested process indicators proved to be non-invariant over the different latent profiles; that is, there are differences in the role of the number of manipulations executed, time-on-task, and type of strategy used in actual problem-solving achievement between students that fall within different profiles. This study contributes to our understanding of how students from different educational contexts behave while solving complex problems.

Scientific, Legal, and Ethical Concerns About AI-Based Personnel Selection Tools: A Call to Action

Organizations are increasingly turning toward personnel selection tools that rely on artificial intelligence (AI) technologies and machine learning algorithms that, together, intend to predict the future success of employees better than traditional tools. These new forms of assessment include online games, video-based interviews, and big data pulled from many sources, including test responses, test-taking behavior, applications, resumes, and social media. Speedy processing, lower costs, convenient access, and applicant engagement are often and rightfully cited as the practical advantages for using these selection tools. At the same time, however, these tools raise serious concerns about their effectiveness in terms their conceptual relevance to the job, their basis in a job analysis to ensure job relevancy, their measurement characteristics (reliability and stability), their validity in predicting employee-relevant outcomes, their evidence and normative information being updated appropriately, and the associated ethical concerns around what information is being represented to employers and told to job candidates. This paper explores these concerns, concluding with an urgent call to industrial and organizational psychologists to extend existing professional standards for employment testing to these new AI and machine learning based forms of testing, including standards and requirements for their documentation.

Latent print quality in blind proficiency testing: Using quality metrics to examine laboratory performance

Calls for blind proficiency testing in forensic science disciplines intensified following the 2009 National Academy of Sciences report and were echoed in the 2016 report by the President’s Council of Advisors on Science and Technology. Both practitioners and scholars have noted that “open” proficiency tests, in which analysts know they are being tested, allow for test-taking behavior that is not representative of behavior in routine casework. This study reports the outcomes of one laboratory’s blind quality control (BQC) program. Specifically, we describe results from approximately 2.5 years of blind cases in the latent print section (N = 376 latent prints submitted as part of 144 cases). We also used a widely available quality metrics software (LQMetrics) to explore relationships between objective print quality and case outcomes. Results revealed that nearly all BQC prints (92.0%) were of sufficient quality to enter into AFIS. When prints had a source present in AFIS, 41.7% of print searches resulted in a candidate list containing the true source. Examiners committed no false positive errors but other types of errors were more common. Average print quality was in the midpoint of the range (53.4 on a 0-to-100 scale), though prints were evenly distributed across the Good, Bad, and Ugly categories. Quality metrics were significantly associated with sufficiency determinations, examiner conclusions, and examiner accuracy. Implications for blind testing and the use of quality metrics in routine casework as well as proficiency testing are discussed.

Reframing rankings in educational assessments.

Ability and test-taking behavior should be disentangled and jointly reported to improve interpretation and fairness

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting.

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Test Taking Strategies

This study aims at exploring the debatable issue in a high-stakes testing situation which is desirable for any test taker. It has been found that, beside content knowledge, test-taking behavior, such as risk-taking strategies, motivation, and test anxiety, is important for test performance. This study assumes that teaching test-taking strategies will result in increasing students' overall scores in their tests. It is also assumed that having such strategies will reduce the level of test stress and improve students' overall attitudes toward tests.
 Studies in this area often compare groups of test takers. They have for example found that high achievers tend to report using more effective test-taking strategies when compared to low achievers (Stenlund et al. 2017), that males are more prone to taking risks when answering test items (see, e.g., Baldiga 2014), and that females and low achievers seem to experience higher levels of test anxiety than males and high achievers (Stenlund et al. 2017). Findings like these can help understand and to some extent possibly explain performance differences that are often observed between manifest groups in achievement tests.