Item Response Times Research Articles

Using Item Response Times in Online Questionnaires to Detect Mild Cognitive Impairment.

With the increase in web-based data collection, response times (RTs) for survey items have become a readily available byproduct in most online studies. We examined whether RTs in online questionnaires can prospectively discriminate between cognitively normal respondents and those with cognitive impairment, no dementia (CIND). Participants were 943 members of a nationally representative internet panel, aged 50 and older. We analyzed RTs that were passively recorded as paradata for 37 surveys (1,053 items) administered online over 6.5 years. A multilevel location-scale model derived 3 RT parameters for each survey: (1) a respondent's average RT and 2 components of intraindividual RT variability addressing (2) systematic RT adjustments and (3) unsystematic RT fluctuations. CIND status was determined at the end of the 6.5-year period. All 3 RT parameters were significantly associated with CIND, with a combined predictive accuracy of area under the receiver-operating characteristic curve = 0.74. Slower average RTs, smaller systematic RT adjustments, and greater unsystematic RT fluctuations prospectively predicted a greater likelihood of CIND over periods of up to 6.5, 4.5, and 1.5 years, respectively. RTs for survey items are a potential early indicator of CIND, which may enhance analyses of predictors, correlates, and consequences of cognitive impairment in online survey research.

Detecting Item Preknowledge Using Revisits With Speed and Accuracy

Examinees with item preknowledge tend to obtain inflated test scores that undermine test score validity. With the availability of process data collected in computer-based assessments, the research on detecting item preknowledge has progressed on using both item scores and response times. Item revisit patterns of examinees can also be utilized as an additional source of information. This study proposes a new statistic for detecting item preknowledge when compromised items are known by utilizing the hierarchical speed–accuracy revisits model. By simultaneously evaluating abnormal changes in the latent abilities, speeds, and revisit propensities of examinees, the procedure was found to provide greater statistical power and stronger substantive evidence that an examinee had indeed benefited from item preknowledge.

Joint cognitive diagnostic modeling for probabilistic attributes incorporating item responses and response times

Joint cognitive diagnostic modeling for probabilistic attributes incorporating item responses and response times

Multiple‐Group Joint Modeling of Item Responses, Response Times, and Action Counts with the Conway‐Maxwell‐Poisson Distribution

AbstractMultiple group modeling is one of the methods to address the measurement noninvariance issue. Traditional studies on multiple group modeling have mainly focused on item responses. In computer‐based assessments, joint modeling of response times and action counts with item responses helps estimate the latent speed and action levels in addition to latent ability. These two new data sources can also be used to further address the measurement noninvariance issue. One challenge, however, is to correctly model action counts which can be underdispersed, overdispersed, or equidispersed in real data sets. To address this, we adopted the Conway‐Maxwell‐Poisson distribution that accounts for different types of dispersion in action counts and incorporated it in the multiple group joint modeling of item responses, response times, and action counts. Bayesian Markov Chain Monte Carlo method was used for model parameter estimation. To illustrate an application of the proposed model, an empirical data analysis was conducted using the Programme for International Student Assessment (PISA) 2015 collaborative problem‐solving items where potential measurement noninvariance issue existed between gender groups. Results indicated that Conway‐Maxwell‐Poisson model yielded better model fit than alternative count data models such as negative binomial and Poisson models. In addition, response times and action counts provided further information on performance differences between groups.

Diagnostic accuracy of the Stricker Learning Span and Mayo Test Drive Composite for amnestic Mild Cognitive Impairment.

AbstractBackgroundRemote assessment tools offer significant promise for aiding early detection of cognitive impairment. Mayo Test Drive (MTD): Test Development through Rapid Iteration, Validation and Expansion, is a web‐based platform for remote self‐administered assessment that includes a computer adaptive word list memory test (Stricker Learning Span; SLS) and a measure of processing speed (Symbols Test). We examined the diagnostic accuracy of the SLS and a MTD composite (SLS max span, SLS trials 1–5 total correct, SLS delay correct, [Symbols correct item response time*‐1]) for amnestic mild cognitive impairment (aMCI). We also explored diagnostic accuracy for a broader group that included individuals with possible MCI (pMCI; see Figure 1).MethodsParticipants were recruited from the Mayo Clinic Study of Aging for this ancillary remote study. 226 were cognitively unimpaired (CU; concordant CU diagnosis by 3 independent raters). Fifty‐six participants had possible MCI (at least 1 of 3 raters indicated MCI) and 16 had a consensus diagnosis of aMCI. Primary outcome variables were SLS sum of trials, AVLT sum of trials, Symbols correct items response time, and MTD composite. Mean difference analyses used linear model ANOVAs (alpha = .05). Receiver operating characteristic (ROC) curves were applied; we derived optimal cutoff scores based on the Youden index method.ResultsBoth aMCI and possible MCI groups showed significantly lower performance than the CU group on SLS (Hedge’s g aMCI = ‐1.72, pMCI = ‐1.04), Symbols (Hedge’s g aMCI = 1.38, pMCI = 0.64), and MTD composite (Hedge’s g aMCI = ‐1.89, pMCI = ‐1.09); see Tables 1 and 2 (all p‘s<.01 even when additionally covarying age, education, sex). Total area under the curve was high for differentiating of CU vs. aMCI (MTD Composite = 0.91, SLS = 0.91) and acceptable for CU vs. pMCI (MTD Composite = 0.77, SLS = 0.77); see Table 2. Consistent with our flexible platform, a variety of device types were used to complete remote testing (32% smartphone, 12% tablet, 56% PC).ConclusionsMTD and the SLS show high diagnostic accuracy for aMCI. MTD is a flexible, brief, and easy‐to‐use remote cognitive assessment tool with great potential for scalable use in future studies seeking to maximize inclusion of individuals with aMCI in clinical trials or for cognitive screening in clinical settings.

Detecting Preknowledge Cheating via Innovative Measures: A Mixture Hierarchical Model for Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts.

Preknowledge cheating jeopardizes the validity of inferences based on test results. Many methods have been developed to detect preknowledge cheating by jointly analyzing item responses and response times. Gaze fixations, an essential eye-tracker measure, can be utilized to help detect aberrant testing behavior with improved accuracy beyond using product and process data types in isolation. As such, this study proposes a mixture hierarchical model that integrates item responses, response times, and visual fixation counts collected from an eye-tracker (a) to detect aberrant test takers who have different levels of preknowledge and (b) to account for nuances in behavioral patterns between normally-behaved and aberrant examinees. A Bayesian approach to estimating model parameters is carried out via an MCMC algorithm. Finally, the proposed model is applied to experimental data to illustrate how the model can be used to identify test takers having preknowledge on the test items.

Modeling Rapid Guessing Behaviors in Computer-Based Testlet Items.

In traditional test models, test items are independent, and test-takers slowly and thoughtfully respond to each test item. However, some test items have a common stimulus (dependent test items in a testlet), and sometimes test-takers lack motivation, knowledge, or time (speededness), so they perform rapid guessing (RG). Ignoring the dependence in responses to testlet items can negatively bias standard errors of measurement, and ignoring RG by fitting a simpler item response theory (IRT) model can bias the results. Because computer-based testing captures response times on testlet responses, we propose a mixture testlet IRT model with item responses and response time to model RG behaviors in computer-based testlet items. Two simulation studies with Markov chain Monte Carlo estimation using the JAGS program showed (a) good recovery of the item and person parameters in this new model and (b) the harmful consequences of ignoring RG (biased parameter estimates: overestimated item difficulties, underestimated time intensities, underestimated respondent latent speed parameters, and overestimated precision of respondent latent estimates). The application of IRT models with and without RG to data from a computer-based language test showed parameter differences resembling those in the simulations.

Exploration of the Stacking Ensemble Machine Learning Algorithm for Cheating Detection in Large-Scale Assessment.

Cheating detection in large-scale assessment received considerable attention in the extant literature. However, none of the previous studies in this line of research investigated the stacking ensemble machine learning algorithm for cheating detection. Furthermore, no study addressed the issue of class imbalance using resampling. This study explored the application of the stacking ensemble machine learning algorithm to analyze the item response, response time, and augmented data of test-takers to detect cheating behaviors. The performance of the stacking method was compared with that of two other ensemble methods (bagging and boosting) as well as six base non-ensemble machine learning algorithms. Issues related to class imbalance and input features were addressed. The study results indicated that stacking, resampling, and feature sets including augmented summary data generally performed better than its counterparts in cheating detection. Compared with other competing machine learning algorithms investigated in this study, the meta-model from stacking using discriminant analysis based on the top two base models-Gradient Boosting and Random Forest-generally performed the best when item responses and the augmented summary statistics were used as the input features with an under-sampling ratio of 10:1 among all the study conditions.

Bridging Models of Biometric and Psychometric Assessment: A Three-Way Joint Modeling Approach of Item Responses, Response Times, and Gaze Fixation Counts.

Recently, joint models of item response data and response times have been proposed to better assess and understand test takers' learning processes. This article demonstrates how biometric information such as gaze fixation counts obtained from an eye-tracking machine can be integrated into the measurement model. The proposed joint modeling framework accommodates the relations among a test taker's latent ability, working speed and test engagement level via a person-side variance-covariance structure, while simultaneously permitting the modeling of item difficulty, time-intensity, and the engagement intensity through an item-side variance-covariance structure. A Bayesian estimation scheme is used to fit the proposed model to data. Posterior predictive model checking based on three discrepancy measures corresponding to various model components are introduced to assess model-data fit. Findings from a Monte Carlo simulation and results from analyzing experimental data demonstrate the utility of the model.

Joint Modeling of Response Accuracy and Time in Between-Item Multidimensional Tests Based on Bi-Factor Model.

With the popularity of computer-based testing (CBT), it is easier to collect item response times (RTs) in psychological and educational assessments. RTs can provide an important source of information for respondents and tests. To make full use of RTs, the researchers have invested substantial effort in developing statistical models of RTs. Most of the proposed models posit a unidimensional latent speed to account for RTs in tests. In psychological and educational tests, many tests are multidimensional, either deliberately or inadvertently. There may be general effects in between-item multidimensional tests. However, currently there exists no RT model that considers the general effects to analyze between-item multidimensional test RT data. Also, there is no joint hierarchical model that integrates RT and response accuracy (RA) for evaluating the general effects of between-item multidimensional tests. Therefore, a bi-factor joint hierarchical model using between-item multidimensional test is proposed in this study. The simulation indicated that the Hamiltonian Monte Carlo (HMC) algorithm works well in parameter recovery. Meanwhile, the information criteria showed that the bi-factor hierarchical model (BFHM) is the best fit model. This means that it is necessary to take into consideration the general effects (general latent trait) and the multidimensionality of the RT in between-item multidimensional tests.

Can we measure students’ task engagement using item response time in a low stake test? Evidence from PISA 2015

이 연구는 컴퓨터기반 학업성취도평가에서 산출되는 프로세스데이터 중 응답시간 정보를 활용하여 학생들의 과제참여도, 구체적으로 자기 능력보다 어려운 과제를 해결하기 위해 인지적 활동을 지속하는 정도를 측정하는 방법을 탐색하기 위한 목적으로 수행되었다. PISA 2015 자료 중 한국 학생 5,568명의 능력추정치와 문항난이도 추정치, 그리고 응답시간 자료를 활용하였으며, 주요 연구결과는 다음과 같다. 첫째, 문항응답시간에 대한 개인 간 분산은 학생의 성취도와 다른 동기적 특성을 고려한 후에도 유의하였다. 둘째, 문항응답시간을 활용하여 구성한 과제참여도 지수는 효능감, 흥미, 성취동기와 약한 정적 상관을, 성취도와 중간 정도의 정적 상관을 보였다. 셋째, 문항응답시간을 활용하여 구성한 과제참여도 지수는 성취도를 예측하는 데 다른 동기적 특성들과 상호 보완적인 역할을 하는 것으로 나타났다. 마지막으로, 응답시간을 활용하여 학생의 비인지적 특성을 진단하는 것의 유용성과 제한점, 향후 연구과제가 논의되었다.

Examining Students' Item Response Times in eTIMSS According to their Proficiency Levels, Self-Confidence and Item Characteristics

The aim of this study was to examine whether the time spent answering science and mathematics items by Turkish students participating in the Trends in International Mathematics and Science Study (TIMSS) at the 8th grade level showed a significant difference according to their proficiency levels, self-confidence, and the item characteristics. This study was correlational research to explore the relationship between the variables discussed. A total of 577 students who participated in the TIMSS 2019 study at the 8th grade level in Turkey and answered the common 24 (11 mathematics and 13 science) items in Booklets 1 and 2 constituted the study participants. In the data analysis, the Kruskal Wallis-H test, Mann-Whitney U test, and Latent Class Analysis were used. As a result, it was determined that the type of item and cognitive level had a significant relation to the item response times of students. The students were found to spend more time on open-ended items than multiple-choice items. On the other hand, the time spent on items in the applying level was significantly higher than the knowledge level. However, there was no significant difference between the time spent answering items in the applying level and reasoning level. It was observed that if the students' confidence level in science was high, the rate of correct answers was high, and they answered the items in a short amount of time. Students who were somewhat self-confident in mathematics were more successful in difficult mathematics items and spent less time answering the items.

Ice Is Hot and Water Is Dry

Abstract: To monitor students’ progress and adapt instruction to students’ needs, teachers increasingly use repeated assessments of equivalent tests. The present study investigates whether equivalent reading tests can be successfully developed via rule-based item design. Based on theoretical considerations, we identified 3-item features for reading comprehension at the word, sentence, and text levels, respectively, which should influence the difficulty and time intensity of reading processes. Using optimal design algorithms, a design matrix was calculated, and four equivalent test forms of the German reading test series for second graders (quop-L2) were developed. A total of N = 7,751 students completed the tests. We estimated item difficulty and time intensity parameters as well as person ability and speed parameters using bivariate item response theory (IRT) models, and we investigated the influence of item features on item parameters. Results indicate that all item properties significantly affected either item difficulty or response time. Moreover, as indicated by the IRT-based test information functions and analyses of variance, the four different test forms showed similar levels of difficulty and time-intensity at the word, sentence, and text levels (all η2< .002). Results were successfully cross-validated using a sample of N = 5,654 students.

Modeling Not-Reached Items in Timed Tests: A Response Time Censoring Approach.

Time limits are imposed on many computer-based assessments, and it is common to observe examinees who run out of time, resulting in missingness due to not-reached items. The present study proposes an approach to account for the missing mechanisms of not-reached items via response time censoring. The censoring mechanism is directly incorporated into the observed likelihood of item responses and response times. A marginal maximum likelihood estimator is proposed, and its asymptotic properties are established. The proposed method was evaluated and compared to several alternative approaches that ignore the censoring through simulation studies. An empirical study based on the PISA 2018 Science Test was further conducted.

Measuring Opportunity Cost in Statistics Using Evaluative Space Grid Items: Results from a Pilot Study

Challenges to measuring students’ attitudes toward statistics remain despite decades of focused research. Measuring the expectancy-value theory (EVT) Cost construct has been especially challenging owing in part to the historical lack of research about it. To measure the EVT Cost construct better, this study asked university students to respond to items using both a Likert-type response and an Evaluative Space Grid (ESG)- type response. ESG items enable bivariate responses in a single item and permit distinguishing among two different types of neutral attitudes: indifferent and ambivalent. This pilot study evaluates the appropriateness of ESG-type items for measuring the EVT Cost construct by analyzing student response patterns to ESG-type items and comparing them with Likert-type items. Validity evidence is documented using descriptive statistics and graphs, correlations among items, and a trinomial hypothesis test. Internal consistency reliability indices are also reported. Friedman’s Test is used to compare the average response times for items of different types. Results indicate that students can meaningfully respond to ESG-type items in ways that are similar to their Likert-type responses, that students respond to ESG-type items quicker with more practice, and that distinguishing among indifferent and ambivalent attitudes seems appropriate for the EVT Cost construct. These findings suggest that ESG-type items may provide new insights not possible with Likert-type items but also that more research should be conducted to better understand their advantages and disadvantages within statistics education.

Linear Ballistic Accumulator Item Response Theory Model for Multidimensional Multiple-Alternative Forced-Choice Measurement of Personality

There has been a growing interest in psychological measurements that use the multiple-alternative forced-choice (MAFC) response format for its resistance to response biases. Although several models have been proposed for the data obtained from such measurements, none have succeeded in incorporating the response time information. Given that currently, many psychological measurements are performed via computers, it would be beneficial to develop a joint model involving an MAFC item response and response time. The present study proposes the first model that combines a cognitive process model that underlies the observed response time and the forced-choice item response model. Specifically, the proposed model is based on the linear ballistic accumulator model of response time, which is substantially extended by reformulating its parameters so as to incorporate the MAFC item responses. The model parameters are estimated by the Markov chain Monte Carlo (MCMC) algorithm. A simulation study confirmed that the proposed approach could appropriately recover the parameters. Two empirical applications are reported to demonstrate the use of the proposed model and compare it with existing models. The results showed that the proposed model could be a useful tool for jointly modeling the MAFC item responses and response times.

Improving the Delivery of Function-Directed Care During Acute Hospitalizations: Methods to Develop and Validate the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT)

Objective:To (1) develop a patient-reported, multidomain functional assessment tool focused on medically ill patients in acute care settings; (2) characterize the measure's psychometric performance; and (3) establish clinically actionable score strata that link to easily implemented mobility preservation plans. Design:This article describes the approach that our team pursued to develop and characterize this tool, the Functional Assessment in Acute Care Multidimensional Computer Adaptive Test (FAMCAT). Development involved a multistep process that included (1) expanding and refining existing item banks to optimize their salience for hospitalized patients; (2) administering candidate items to a calibration cohort; (3) estimating multidimensional item response theory models; (4) calibrating the item banks; (5) evaluating potential multidimensional computerized adaptive testing (MCAT) enhancements; (6) parameterizing the MCAT; (7) administering it to patients in a validation cohort; and (8) estimating its predictive and psychometric characteristics. Setting:A large (2000-bed) Midwestern Medical Center. Participants:The overall sample included 4495 adults (2341 in a calibration cohort, 2154 in a validation cohort) who were admitted either to medical services with at least 1 chronic condition or to surgical/medical services if they required readmission after a hospitalization for surgery (N=4495). Intervention:Not applicable. Main Outcome Measures:Not applicable. Results:The FAMCAT is an instrument designed to permit the efficient, precise, low-burden, multidomain functional assessment of hospitalized patients. We tried to optimize the FAMCAT's efficiency and precision, as well as its ability to perform multiple assessments during a hospital stay, by applying cutting edge methods such as the adaptive measure of change (AMC), differential item functioning computerized adaptive testing, and integration of collateral test-taking information, particularly item response times. Evaluation of these candidate methods suggested that all may enhance MCAT performance, but none were integrated into initial MCAT parameterization. Conclusions:The FAMCAT has the potential to address a longstanding need for structured, frequent, and accurate functional assessment among patients hospitalized with medical diagnoses and complications of surgery.

Bayesian Mixed Effects Model and Data Visualization for Understanding Item Response Time and Response Order in Open Online Assessment

Open (open-book) online assessment has become a great tool in higher education, which is frequently used for monitoring learning progress and teaching effectiveness. It has been gaining popularity because it is flexible to use and makes response behavior data available for researchers to study response processes. However, some challenges are encountered in analyzing these data, such as how to handle outlying response time, how to make use of the information from item response order, how item response time, response order and item scores are related, and how to help classroom teachers quickly check whether student responses are aligned with the design of the assessment. The purposes of this study are 3-fold: (1) to provide a solution for handling outlying response times due to the design of open online formative assessments (i.e., ample or unrestricted testing time), (2) to propose a new measure for investigating the item response order, and (3) to discuss two analytical approaches that are useful for studying response behaviors–data visualization and the Bayesian generalized linear mixed effects model (B-GLMM). An application of these two approaches is illustrated using open online quiz data. Our findings obtained from B-GLMM showed that item response order was related to item response time, but not to item scores; and item response time was related to item scores, but its effects were moderated by the cognitive level. Additionally, the findings from both B-GLMM and data visualization were consistent, which assisted instructors to see the alignment of student responses with the assessment design.

Latent-variable Approaches Utilizing Both Item Scores and Response Times To Detect Test Fraud

Abstract There is a growing interest in approaches based on latent-variable models for detecting fraudulent behavior on educational tests. Wollack and Schoenig (2018) noted the presence of five types of statistical/psychometric approaches to detect the three broad types of test fraud that occur in educational tests. This paper includes a brief review of the five types of statistical/psychometric approaches mentioned by Wollack and Schoenig (2018). This paper then includes a more detailed review of the recent approaches for detecting test fraud using both item scores and response times—all of these approaches are based on latent-variable models. A real data example demonstrates the use of two of the approaches.

On the Cover: Is my Test Speeded?

“Is my test speeded?” is a question that test developers should ask when developing a power test, a test that measures ability rather than speed. A test is not speeded if most test takers have sufficient time to answer all test questions in the time allotted. One way to check the speededness of a test is to compute the completion rate – that is, the proportion of test takers who answered the test questions at the end of the test. When the time spent on test questions is available, the second way to evaluate the speededness is to compare the time spent on the questions at the end to the time spent on the questions in the beginning. The graphic shows the average response times for items according to their position in the test sequence and their difficulty (i.e., proportion of correct responses). The visual here answers the following research questions: 1) “Is there a general downward trend in mean item response times toward the end of the test, controlling for item difficulty?” and 2) “Do items become more difficult at the end of the test?” The plot shows data from multiple 60-item modules on a high-stakes certification exam. This graph was generated using the R package ggplot2 (Wickham, 2009), and code is available upon request. The solid black line is the grand mean response time by item position across all modules. First, as indicated by the solid black line, it does not appear that item response times are generally decreasing toward the end of the test, indicating that examinees are not “speeding up” as they reach the time limit. Second, the colored lines, indicating the trends in average response time by position as a function of item difficulty, show that more difficult items take more time to complete than easier items, but this relationship does not depend on item position. That is, items do not become more difficult if placed at the end of the test. Overall, the graphic suggests this exam is not speeded. Interested readers may contact Bozhidar M. Bashkov ([email protected]) for questions on this graph or the code to create it. I would recommend readers try varying the number of categories of item difficulty when creating the graph. The comparison may look sharper with a small number of categories – for example, three.