Item Response Times Research Articles

Measurement Invariance for Multilingual Learners Using Item Response and Response Time in PISA 2018

AbstractThis study examines potential assessment bias based on students' primary language status in PISA 2018. Specifically, multilingual (MLs) and nonmultilingual (non‐MLs) students in the United States are compared with regard to their response time as well as scored responses across three cognitive domains (reading, mathematics, and science). Differential item functioning (DIF) analysis reveals that 7–14% of items exhibit DIF‐related problems in scored responses between the two groups, aligning with PISA technical report results. While MLs generally spend more time on the test than non‐MLs across cognitive levels, differential response time (DRT) functioning identifies significant time differences in 7–10% of items for students with similar cognitive levels. It was noticeable that items with DIF and DRT issues show limited overlap, suggesting diverse reasons for student struggles in the assessment. A deeper examination of item characteristics is recommended for test developers and teachers to gain a better understanding of these nuances.

Decompositions of Information Criteria in Joint Model Under Probit Link for Response

ABSTRACTThe item response and response time are naturally encountered in variety of computerized testing, which are jointly analysed to achieve efficient model fitting. Several novel Bayesian assessment criteria, which quantify the enhancement in the fit of one dataset given the other, are proposed by utilizing decompositions of the deviance information criterion (DIC) and the logarithm of the pseudo‐marginal likelihood (LPML) under a logistic link. However, the calculation of these criteria under a probit link, a commonly employed alternative, remains unexplored. Moreover, decomposing DIC and LPML under probit link presents greater challenges than in the logistic model due to the intricate additional dimension integral. This paper addresses this gap by demonstrating the application of Bayesian model assessment criteria with a normal ogive model in the context of item response data within a joint modelling framework. Additionally, we also illustrate how to implement Bayesian sampling via a powerful R package ‘nimble’. Extensive simulation studies are conducted to evaluate the empirical performance of those criteria. Further, a detailed analysis of the Programme for International Student Assessment (PISA) data is carried out to illustrate the methodology.

An empirical evaluation of probing questions investigating question comprehensibility in web surveys

ABSTRACT It is essential for the quality of survey data that questions are easy to understand. We developed four variants of comprehensibility probes measuring the comprehensibility of a survey question as a whole or of individual aspects (vocabulary, sentence complexity, content). We compared the probes on four experimental questions in terms of response distributions and mean comprehensibility scores to assess their sensitivity to capture comprehensibility problems at different levels of granularity, as well as nonresponse and response times as indicators of respondents’ willingness and required effort to answer the probes. We found no differences in the response distributions and mean comprehensibility scores, while item nonresponse and response times differed significantly between the probe versions.

Using Response Times in Answer Similarity Analysis

Recent decades have seen a tremendous growth in the development of collusion detection methods, many of which rest on the assumption that examinees who engage in collusion will display unusually similar scores/responses. In this article, we expand the definition of answer similarity to include not only the item scores/responses but also the item response times (RTs). Using detailed simulations and an experimental data set, we show that (a) both the new and existing similarity statistics are able to control the Type I error rate in most of the studied conditions and (b) the new statistics are much more powerful, on average, than the existing statistics at detecting several types of simulated collusion.

Relating violations of measurement invariance to group differences in response times.

Measurement invariance is an assumption underlying the regression of a latent variable on a background variable. It requires the measurement model parameters of the latent variable to be equal across the levels of the background variable. Item-specific violations of this assumption are referred to as differential item functioning and are ideally substantively explainable to warrant theoretically valid and meaningful results. Past research has focused on developing statistical approaches to explain differential item functioning effects in terms of item- or person-specific covariates. In this study, we propose a modeling approach that can be used to test if differences in item response times can be used to statistically explain differential item functioning. To this end, we operationalize a latent response process factor and test if item-specific group differences on this factor can account for the observed differences in item scores. We investigate the properties of the model in a simulation study, and we apply the model to a real data set. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Does value-based prioritization at working memory enhance long-term memory?

Research has demonstrated that individuals can direct their attention to valuable information in both working memory and long-term memory tasks with observable effects on performance. However, it is currently unclear whether prioritising an item for a working memory task automatically translates into a boost at long-term memory. This was examined in two experiments using relatively short (250 ms per item; Experiment 1) and longer (500 ms per item; Experiment 2) encoding times. Participants first completed a visual working memory task, in which they were presented with series of photographs of everyday objects. Following a brief delay (1,000 ms), they completed a four-alternative forced-choice test. Prior to encoding, participants were informed of the point values associated with each item. In some trials, the first item in the sequence was worth more points than the rest. In other trials, all items were equally valuable. After a filled delay, participants completed a surprise long-term memory task. At working memory, a value effect was reliably observed on recognition accuracy, along with some evidence of faster response times for high-value items. However, there was little consistent evidence of this effect automatically persisting into long-term memory. Thus, the benefits of attentional prioritization in working memory do not always translate into longer-term performance. More broadly, this provides further evidence that manipulations that enhance working memory performance do not necessarily enhance long-term memory.

Conditional Dependence across Slow and Fast Item Responses: With a Latent Space Item Response Modeling Approach.

There recently have been many studies examining conditional dependence between response accuracy and response times in cognitive tests. While most previous research has focused on revealing a general pattern of conditional dependence for all respondents and items, it is plausible that the pattern may vary across respondents and items. In this paper, we attend to its potential heterogeneity and examine the item and person specificities involved in the conditional dependence between item responses and response times. To this end, we use a latent space item response theory (LSIRT) approach with an interaction map that visualizes conditional dependence in response data in the form of item-respondent interactions. We incorporate response time information into the interaction map by applying LSIRT models to slow and fast item responses. Through empirical illustrations with three cognitive test datasets, we confirm the presence and patterns of conditional dependence between item responses and response times, a result consistent with previous studies. Our results further illustrate the heterogeneity in the conditional dependence across respondents, which provides insights into understanding individuals' underlying item-solving processes in cognitive tests. Some practical implications of the results and the use of interaction maps in cognitive tests are discussed.

Emotional arousal lingers in time to bind discrete episodes in memory

ABSTRACT Temporal stability and change in neutral contexts can transform continuous experiences into distinct and memorable events. However, less is known about how shifting emotional states influence these memory processes, despite ample evidence that emotion impacts non-temporal aspects of memory. Here, we examined if emotional stimuli influence temporal memory for recent event sequences. Participants encoded lists of neutral images while listening to auditory tones. At regular intervals within each list, participants heard emotional positive, negative, or neutral sounds, which served as “emotional event boundaries” that divided each sequence into discrete events. Temporal order memory was tested for neutral item pairs that either spanned an emotional sound or were encountered within the same auditory event. Encountering a highly arousing event boundary led to faster response times for items encoded within the next event. Critically, we found that highly arousing sounds had different effects on binding ongoing versus ensuing sequential representations in memory. Specifically, highly arousing sounds were significantly more likely to enhance temporal order memory for ensuing information compared to information that spanned those boundaries, especially for boundaries with negative valence. These findings suggest that within aversive emotional contexts, fluctuations in arousal help shape the temporal organisation of events in memory.

Computer-Based Listening Test with Full Video, Visual-Limited Video, and Audio: A Comparative Analysis Based on Difficulty, Discrimination Power, and Response Time

ABSTRACT This study aimed to compare item characteristics and response time between stimulus conditions in computer-delivered listening tests. Listening materials had three variants: regular videos, frame-by-frame videos, and only audios without visuals. Participants were 228 Japanese high school students who were requested to complete one of nine listening test forms. Results showed that item difficulty did not significantly differ between conditions, but item discrimination was much weaker in regular video versions than in the other two conditions. Response time was not different in the three conditions in any quantile. This suggests that listening items full of visual information have different aspects of the general (traditional) listening construct, despite the possibility of construct-irrelevant factors being affected.

What Can We Learn from a Semiparametric Factor Analysis of Item Responses and Response Time? An Illustration with the PISA 2015 Data.

It is widely believed that a joint factor analysis of item responses and response time (RT) may yield more precise ability scores that are conventionally predicted from responses only. For this purpose, a simple-structure factor model is often preferred as it only requires specifying an additional measurement model for item-level RT while leaving the original item response theory (IRT) model for responses intact. The added speed factor indicated by item-level RT correlates with the ability factor in the IRT model, allowing RT data to carry additional information about respondents' ability. However, parametric simple-structure factor models are often restrictive and fit poorly to empirical data, which prompts under-confidence in the suitablity of a simple factor structure. In the present paper, we analyze the 2015 Programme for International Student Assessment mathematics data using a semiparametric simple-structure model. We conclude that a simple factor structure attains a decent fit after further parametric assumptions in the measurement model are sufficiently relaxed. Furthermore, our semiparametric model implies that the association between latent ability and speed/slowness is strong in the population, but the form of association is nonlinear. It follows that scoring based on the fitted model can substantially improve the precision of ability scores.

A Multistrategy Cognitive Diagnosis Model Incorporating Item Response Times Based on Strategy Selection Theories

Response times (RTs) facilitate the quantification of underlying cognitive processes in problem-solving behavior. To provide more comprehensive diagnostic feedback on strategy selection and attribute profiles with multistrategy cognitive diagnosis model (CDM) and utilize additional information for item RTs, this study develops a multistrategy cognitive diagnosis modeling framework combined with RTs. The proposed model integrates individual response accuracy and RT into a unified framework to define strategy selection and make it closer to the individual’s strategy selection process. Simulation studies demonstrated that the proposed model had reasonable parameter recovery and attribute classification accuracy and outperformed the existing multistrategy CDMs and single-strategy CDMs in terms of performance. Empirical results further illustrated the practical application and the advantages of the proposed model.

Using item scores and response times in person-fit assessment.

The use of joint models for item scores and response times is becoming increasingly popular in educational and psychological testing. In this paper, we propose two new person-fit statistics for such models in order to detect aberrant behaviour. The first statistic is computed by combining two existing person-fit statistics: one for the item scores, and one for the item response times. The second statistic is computed directly using the likelihood function of the joint model. Using detailed simulations, we show that the empirical null distributions of the new statistics are very close to the theoretical null distributions, and that the new statistics tend to be more powerful than several existing statistics for item scores and/or response times. A real data example is also provided using data from a licensure examination.

Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries

Unfavorable test-taking behaviors, such as speededness and disengagement, have long been a validity concern for large-scale low-stakes assessments. Understanding the presence and extent of such behaviors is important for ensuring the validity of inferences based on test scores. This study examined test-taking behaviors using item response time (RT), a process data-derived variable from the TIMSS 2019 database. Analyses compared the United States to three other countries (England, Singapore, and the United Arab Emirates) that administered the digital version of TIMSS (eTIMSS) 2019 in English at grade 8. Test-taking behaviors were identified within each country and compared within and across countries. Specifically, to identify distinct types of test-taking behaviors, mixture modeling was employed on RT and item scores from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment. The results indicated that each country had several latent classes of students with different pacing trajectories and performance. The test-taking behaviors of these latent classes were labeled as Steady; Disengaged or Very disengaged; Speeded or Very speeded; and Efficient and high-performing. Most of the students in each country had a Steady pace (medium to high sum score; steady RT throughout the test): 71% in England, 74% in both Singapore and the United Arab Emirates, and 84% in the United States. Disengaged or Very disengaged students (low sum score; short RT) were identified in each country but were more prevalent in England and the United Arab Emirates (above 20% in both) than in the United States and Singapore (both below 10%). The study also revealed small percentages of Speeded or Very speeded students (low to medium sum score; long RT at first but very short RT toward the end) in England, the United Arab Emirates, and the United States (1%, 5%, and 6%, respectively) but not in Singapore. A unique class of Efficient and high-performing students (high sum score; short RT) was identified only in Singapore (24%). This study demonstrated that mixture modeling is a useful technique for identifying distinct test-taking behaviors and highlighted the presence and extent of unfavorable test-taking behaviors within each selected country using data from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment.

Should Intelligence Tests Be Speeded or Unspeeded? A Brief Review of the Effects of Time Pressure on Response Processes and an Experimental Study with Raven's Matrices.

Intelligence tests are often performed under time constraints for practical reasons, but the effects of time pressure on reasoning performance are poorly understood. The first part of this work provides a brief review of major expected effects of time pressure, which includes forcing participants to skip items, convoking a mental speed factor, constraining response times, qualitatively altering cognitive processing, affecting anxiety and motivation, and interacting with individual differences. The second part presents data collected with Raven's matrices under three conditions of speededness to provide further insight into the complex effects of time pressure, with three major findings. First, even mild time pressure (with enough time available for all participants to complete the task at a leisurely pace) induced speeding throughout the whole task, starting with the very first item, and participants sped up more than was actually required. Second, time pressure came with lower confidence and poorer strategy use and a substantial decrease of accuracy (d = 0.35), even when controlling for response time at the item level-indicating a detrimental effect on cognitive processing beyond speeding. Third, time pressure disproportionately reduced response times for difficult items and participants with high ability, working memory capacity, or need for cognition, although this did not differentially affect ability estimates. Overall, both the review and empirical sections show that the effects of time pressure go well beyond forcing participants to speed or skip the last few items and make even mild time constraints inadvisable when attempting to measure maximal performance, especially for high-performing samples.

Reliability and Validity of Noncognitive Ecological Momentary Assessment Survey Response Times as an Indicator of Cognitive Processing Speed in People's Natural Environment: Intensive Longitudinal Study.

Various populations with chronic conditions are at risk for decreased cognitive performance, making assessment of their cognition important. Formal mobile cognitive assessments measure cognitive performance with greater ecological validity than traditional laboratory-based testing but add to participant task demands. Given that responding to a survey is considered a cognitively demanding task itself, information that is passively collected as a by-product of ecological momentary assessment (EMA) may be a means through which people's cognitive performance in their natural environment can be estimated when formal ambulatory cognitive assessment is not feasible. We specifically examined whether the item response times (RTs) to EMA questions (eg, mood) can serve as approximations of cognitive processing speed. This study aims to investigate whether the RTs from noncognitive EMA surveys can serve as approximate indicators of between-person (BP) differences and momentary within-person (WP) variability in cognitive processing speed. Data from a 2-week EMA study investigating the relationships among glucose, emotion, and functioning in adults with type 1 diabetes were analyzed. Validated mobile cognitive tests assessing processing speed (Symbol Search task) and sustained attention (Go-No Go task) were administered together with noncognitive EMA surveys 5 to 6 times per day via smartphones. Multilevel modeling was used to examine the reliability of EMA RTs, their convergent validity with the Symbol Search task, and their divergent validity with the Go-No Go task. Other tests of the validity of EMA RTs included the examination of their associations with age, depression, fatigue, and the time of day. Overall, in BP analyses, evidence was found supporting the reliability and convergent validity of EMA question RTs from even a single repeatedly administered EMA item as a measure of average processing speed. BP correlations between the Symbol Search task and EMA RTs ranged from 0.43 to 0.58 (P<.001). EMA RTs had significant BP associations with age (P<.001), as expected, but not with depression (P=.20) or average fatigue (P=.18). In WP analyses, the RTs to 16 slider items and all 22 EMA items (including the 16 slider items) had acceptable (>0.70) WP reliability. After correcting for unreliability in multilevel models, EMA RTs from most combinations of items showed moderate WP correlations with the Symbol Search task (ranged from 0.29 to 0.58; P<.001) and demonstrated theoretically expected relationships with momentary fatigue and the time of day. The associations between EMA RTs and the Symbol Search task were greater than those between EMA RTs and the Go-No Go task at both the BP and WP levels, providing evidence of divergent validity. Assessing the RTs to EMA items (eg, mood) may be a method of approximating people's average levels of and momentary fluctuations in processing speed without adding tasks beyond the survey questions.

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators.

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables-text reread, item revisit, and answer change-were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

Online Monitoring of Test‐Taking Behavior Based on Item Responses and Response Times

AbstractThe study presents multivariate sequential monitoring procedures for examining test‐taking behaviors online. The procedures monitor examinee's responses and response times and signal aberrancy as soon as significant change is identifieddetected in the test‐taking behavior. The study in particular proposes three schemes to track different indicators of a test‐taking mode—the observable manifest variables, latent trait variables, and measurement likelihood. For each procedure, sequential sampling strategies are presented to implement online monitoring. Numerical experimentation based on simulated data suggests that the proposed procedures demonstrate adequate performance. The procedures identified examinees with aberrant behaviors with high detection power and timeliness, while maintaining error rates reasonably small. Experimental application to real data also suggested that the procedures have practical relevance to real assessments. Based on the observations from the experiential analysis, the study discusses implications and guidelines for practical use.

Understanding examinees’ item responses through cognitive modeling of response accuracy and response times

Understanding the cognitive processes, skills and strategies that examinees use in testing is important for construct validity and score interpretability. Although response processes evidence has long been included as an important aspect of validity (i.e., Standards for Educational and Psychological Tests, 1999), relevant studies are often lacking, especially in large scale educational and psychological testing. An important method for studying response processes involves explanatory mathematical modeling of item responses and item response times from variables that represent sources of cognitive complexity. For many item types, examinees may differ in strategies applied to responding to items. Mixture class item response theory models can identify latent classes of examinees with different processes, skills and strategies based on their pattern of item responses. This study will illustrate the use of response times in conjunction with explanatory item response theory models and mixture models, to provide information relevant to test validity and, hence, to score interpretations.

Process and Product in Computer-Based Assessments

Abstract: There is no consensus among assessment researchers about many of the central problems of response process data, including what is it and what is it comprised of. The Standards for Educational and Psychological Testing ( American Educational Research Association et al., 2014 ) locate process data within their five sources of validity evidence. However, we rarely see a conceptualization of response processes; rather, the focus is on the techniques and methods of assembling response process indices or statistical models. The method often overrides clear definitions, and, as a field, we may therefore conflate method and methodology – much like we have conflated validity and validation ( Zumbo, 2007 ). In this paper, we aim to clear the conceptual ground to explore the scope of a holistic framework for the validation of process and product. We review prominent conceptualizations of response processes and their sources and explore some fundamental questions: Should we make a theoretical and practical distinction between response processes and response data? To what extent do the uses of process data reflect the principles of deliberate, educational, and psychological measurement? To answer these questions, we consider the case of item response times and the potential for variation associated with disability and neurodiversity.

Typicality in the brain during semantic and episodic memory decisions

Concept typicality is a key semantic dimension supporting the categorical organization of items based on their features, such that typical items share more features with other members of their category than atypical items, which are more distinctive. Typicality effects manifest in better accuracy and faster response times during categorization tasks, but higher performance for atypical items in episodic memory tasks, due to their distinctiveness. At a neural level, typicality has been linked to the anterior temporal lobe (ATL) and the inferior frontal gyrus (IFG) in semantic decision tasks, but patterns of brain activity during episodic memory tasks remain to be understood. We investigated the neural correlates of typicality in semantic and episodic memory to determine the brain regions associated with semantic typicality and uncover effects arising when items are reinstated during retrieval. In an fMRI study, 26 healthy young subjects first performed a category verification task on words representing typical and atypical concepts (encoding), and then completed a recognition memory task (retrieval). In line with previous literature, we observed higher accuracy and faster response times for typical items in the category verification task, while atypical items were better recognized in the episodic memory task. During category verification, univariate analyses revealed a greater involvement of the angular gyrus for typical items and the inferior frontal gyrus for atypical items. During the correct recognition of old items, regions belonging to the core recollection network were activated. We then compared the similarity of the representations from encoding to retrieval (ERS) using Representation Similarity Analyses. Results showed that typical items were reinstated more than atypical ones in several regions including the left precuneus and left anterior temporal lobe (ATL). This suggests that the correct retrieval of typical items requires finer-grained processing, evidenced by greater item-specific reinstatement, which is needed to resolve their confusability with other members of the category due to their higher feature similarity. Our findings confirm the centrality of the ATL in the processing of typicality while extending it to memory retrieval.