IRT Models Research Articles

Remote cognitive evaluation in aging adults: A new software implementation during COVID‐19

AbstractBackgroundRemote cognitive testing is a convenient and efficient method to evaluate older adults especially because of the sanitary restrictions due to COVID 19. Our aim was to test the psychometric properties of newly developed versions of two visuospatial attention subtests, Hidden Patterns and Identical Pictures, administered by remote cognitive evaluation platform developed by Neurobehavioral Systems, Inc. We conducted testing and analysis in a sample of healthy aging adults.MethodsParticipants were recruited in East‐Bay, California; 184 aging adults completed the computerized hidden patterns test, and 160 older adults answered the identical picture test. The mean age was 70.11 years old with a range of 60 to 85 years old. Most of the participants were female (59.60%), and dementia symptoms were not reported. Participants were evaluated twice in one year at home, the software created by Neurobehavioral Systems, Inc allowed a synchronous interaction with the remote evaluator, which helped to monitor and answer any question before, during or after the evaluation.ResultDifficulty and discrimination analysis (classical test theory) was performed in R (version 4.1.0),. We estimated a Bayesian 1PL Item Response Model (IRT) for each test along with DIF testing by gender. The IRT models were calculated with the package rstan (Stan Development Team, 2020)ConclusionThe IRT models showed group differences at the items’ difficulty. The estimated reliability showed to be adequate (α > .75), and test – retest correlations were medium to high (r > .50). Correlations with age ranged from r = ‐.33 to r = .25.

An Attempt to Measure and Model Women’s Attitudes to Saving for Retirement

Abstract Research background Because of the convergence of demographic, political, and economic trends, retirement has become a significant stage of human life, which currently requires thoughtful and careful planning. Therefore, social pension schemes – a type of system under which the benefits are being provided – and women’s individual saving decisions should give consideration to the fact that it is necessary to accumulate additional funds which should be sufficient for several decades after retirement. Purpose Social pension schemes are widely discussed in all European countries, and in this context we can pose the following question: What is it exactly that makes women save for retirement? This paper has two objectives: to identify the factors that encourage women to save and to measure these factors and develop a model which shows correlations. Research methodology A questionnaire survey was used for collecting the primary data. Subsequently, the analysis is based on 4 indexes created for the purposes of this paper. Then, the indexes were estimated on the basis of the unidimensional two-parameter IRT model, rho and CR coefficients, and multiple-group logistic regression analysis. Novelty The conducted calculations lead to the conclusion that regardless of age, several factors are statistically significant for the whole analysed group, e.g. a place of residence. This is the first paper that shows determinants important for women when it comes to saving for retirement.

SAS PROC IRT and the R mirt Package: A Comparison of Model Parameter Estimation for Multidimensional IRT Models

This study investigates the performance of estimation methods for multidimensional IRT models with dichotomous and polytomous data in two well-known IRT programs: SAS PROC IRT and the mirt package in R. A simulation study was used to compare performance on a simple structure Rasch model, complex structure 2PL model, and bifactor graded response model. Under RMSE and bias criteria regarding item parameter recovery, PROC IRT and mirt showed nearly identical performance in the simple structure condition. When a complex structure was used, mirt performed better in terms of the recovery of intercept parameters, while the recovery of slope parameters depended on the program and the sample sizes: PROC IRT tended to be better with small samples (N=500) according to RMSE, and mirt was better for larger samples (N=1000 and 2500) according to RMSE and bias for the slope parameter recovery. When a bifactor structure was used, mirt was preferred in all cases; differences lessened as sample size increased.

The analysis of TIMSS 2015 data with confirmatory mixture item response theory: A multidimensional approach

In this study, we illustrated an application of the confirmatory mixture IRT model for multidimensional tests. We aimed to examine the differences in student performance by domains with a confirmatory mixture IRT modeling approach. A three-dimensional and three-class model was analyzed by assuming content domains as dimensions and cognitive domains as item groups. We estimated the item performance differences among the students through structural parameters. There were 463 students from Turkey and 880 students from Canada who participated in the TIMSS 2015 4th-grade mathematics assessment. Results for Turkey indicated, students in Class 2 had better performance in knowing and reasoning compared to those in Classes 1 and 3. Students in Class 2 and Class 3 were similar in applying math concepts compared to students in Class 1. For the Canadian sample, students in Class 2 had better performance in knowing, applying, and reasoning compared to those in Class 1 and 3. Also, Class 3 students were better at applying domain than Class 1. Also, mean values were obtained for all content domains in the two countries. Confirmatory mixture IRT modeling approaches appear to differentiate students’ mathematics competencies.

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior.

Rapid guessing (RG) is a form of non-effortful responding that is characterized by short response latencies. This construct-irrelevant behavior has been shown in previous research to bias inferences concerning measurement properties and scores. To mitigate these deleterious effects, a number of response time threshold scoring procedures have been proposed, which recode RG responses (e.g., treat them as incorrect or missing, or impute probable values) and then estimate parameters for the recoded dataset using a unidimensional or multidimensional IRT model. To date, there have been limited attempts to compare these methods under the possibility that RG may be misclassified in practice. To address this shortcoming, the present simulation study compared item and ability parameter recovery for four scoring procedures by manipulating sample size, the linear relationship between RG propensity and ability, the percentage of RG responses, and the type and rate of RG misclassifications. Results demonstrated two general trends. First, across all conditions, treating RG responses as incorrect produced the largest degree of combined systematic and random error (larger than ignoring RG). Second, the remaining scoring approaches generally provided equal accuracy in parameter recovery when RG was perfectly identified; however, the multidimensional IRT approach was susceptible to increased error as misclassification rates grew. Overall, the findings suggest that recoding RG as missing and employing a unidimensional IRT model is a promising approach.

A Stan tutorial on Bayesian IRTree models: Conventional models and explanatory extension.

IRTree models have been receiving increasing attention. However, to date, there are limited sources that provide a systematic introduction to Bayesian modeling techniques using modern probabilistic programming frameworks for the implementation of IRTree models. To facilitate the research and application of IRTree models, this paper introduces how to perform two families of Bayesian IRTree models (i.e., response tree models and latent tree models) in Stan and how to extend them in an explanatory way. Some suggestions on executing Stan codes and checking convergence are also provided. An empirical study based on the Oxford Achieving Resilience during COVID-19 data was conducted as an example to further illustrate how to apply Bayesian IRTree models to address research questions. Finally, strengths and future directions are discussed.

Modeling local item dependence in C-tests with the loglinear Rasch model

C-tests are gap-filling tests mainly used as rough and economical measures of second-language proficiency for placement and research purposes. A C-test usually consists of several short independent passages where the second half of every other word is deleted. Owing to their interdependent structure, C-test items violate the local independence assumption of IRT models. This poses some problems for IRT analysis of C-tests. A few strategies and psychometric models have been suggested and employed in the literature to circumvent the problem. In this research, a new psychometric model, namely, the loglinear Rasch model, is used for C-tests and the results are compared with the dichotomous Rasch model where local item dependence is ignored. Findings showed that the loglinear Rasch model fits significantly better than the dichotomous Rasch model. Examination of the locally dependent items did not reveal anything as regards their contents. However, it did reveal that 50% of the dependent items were adjacent items. Implications of the study for modeling local dependence in C-tests using different approaches are discussed.

General intelligence in middle school students from different Russian regions: Results of PISA-like tests

This study is aimed at investigating the contribution of the general intelligence factor if six PISA domains (reading, mathematical, scientific, financial literacies, global competence, and creative thinking) are combined in one measurement instrument. For achieving our goal, items based on the PISA frameworks are developed, students in grades 5–8 from three different Russian regions are assessed, and three IRT models (unidimensional, multidimensional, and bifactor) are applied to process the data. In addition, the correlations from the multidimensional model are estimated to examine the degree of cognitive specificity and mixture modeling is implemented to investigate ability differentiation across grades. Statistical analysis reveals that the bifactor model comprising one general and six specific factors, has a better fit in each grade. Based on this model, we compute the variance explained by the general factor, with the estimates varying between 60% and 70%. In general, the pure variance explained by specific factors does not exceed 10%. The correlations are above 0.40 in each grade and the averaged associations tend to increase from 6th to 8th grade, although they are smaller in years 6 and 7 compared to year 5. The general ability differentiation effect is observed in grades 6 to 8 and is not present in grade 5. Specific ability differentiation is more pronounced in reading literacy, especially in grade 5 to 7. The results obtained are discussed from the perspective of the ability and developmental differentiation/dedifferentiation problem.

A Tree-Based Approach to Identifying Response Styles with Anchoring Vignettes

ABSTRACT Survey score interpretations are often plagued by sources of construct-irrelevant variation, such as response styles. In this study, we propose the use of an IRTree Model to account for response styles by making use of self-report items and anchoring vignettes. Specifically, we investigate how the IRTree approach with anchoring vignettes compares to traditional approaches that either do not include anchoring vignettes or do not account for response styles. We analyze secondary data using four different models: 1) total score; 2) graded response model; 3) IRTree without the consideration of anchoring vignettes, and 4) IRTree considering anchoring vignettes. We found significant differences in trait estimates from models that account for response styles compared to those that do not. Additionally, we found differences in trait estimates between the IRTree Models when considering anchoring vignettes and when not. Model comparisons suggest that trait differences are due to adjusting for acquiescence response style.

R-package LNIRT for joint modeling of response accuracy and times.

In computer-based testing it has become standard to collect response accuracy (RA) and response times (RTs) for each test item. IRT models are used to measure a latent variable (e.g., ability, intelligence) using the RA observations. The information in the RTs can help to improve routine operations in (educational) testing, and provide information about speed of working. In modern applications, the joint models are needed to integrate RT information in a test analysis. The R-package LNIRT supports fitting joint models through a user-friendly setup which only requires specifying RA, RT data, and the total number of Gibbs sampling iterations. More detailed specifications of the analysis are optional. The main results can be reported through the summary functions, but output can also be analysed with Markov chain Monte Carlo (MCMC) output tools (i.e., coda, mcmcse). The main functionality of the LNIRT package is illustrated with two real data applications.

The impact of the first wave of COVID-19 on students’ attainment, analysed by IRT modelling method

Universities around the world were closed for several months to slow down the spread of the COVID-19 pandemic. During this crisis, a tremendous amount of effort was made to use online education to support the teaching and learning process. The COVID-19 pandemic gave us a profound insight into how online education can radically affect students and how students adapt to new challenges. The question is how switching to online education affected dropout? This study shows the results of a research project clarifying the impact of the transition to online courses on dropouts. The data analysed are from a large public university in Europe where online education was introduced in March 2020. This study compares the academic progress of students newly enroled in 2018 and 2019 using IRT modelling. The results show that (1) this period did not contribute significantly to the increase in dropout, and we managed to retain our students.(2) Subjects became more achievable during online education, and students with less ability were also able to pass their exams. (3) Students who participated in online education reported lower average grade points than those who participated in on-campus education. Consequently, on-campus students could win better scholarships because of better grades than students who participated in online education. Analysing students’ results could help (1) resolve management issues regarding scholarship problems and (2) administrators develop programmes to increase retention in online education.

Developing A Computerized Adaptive Test Form of the Occupational Field Interest Inventory

In this research, the aim was to apply the Occupational Field Interest Inventory (OFII), which was developed in paper-pencil format, as a Computerized Adaptive Test (CAT). For this purpose, the paper and pencil form of the OFII was applied to 1425 high school students and post-hoc simulations were carried out with the obtained data. According to results obtained from the simulations, it was decided that the most ideal criteria for the CAT application were GPCM as the IRT model, .40 standard error value as the test termination rule, and MFI as the item selection method. The OFII ended with an average of 59 items, and the correlations between scores obtained from the paper-pencil form and thetas (θ) estimated by simulation ranged between .91-.97. According to post-hoc simulation results, the CAT application was applied to 150 students. It was observed that the correlations between the scores of students from the online application of the paper-pencil form and θ levels estimated by the CAT form varied between .73 and .91.

A Mixed Sequential IRT Model for Mixed-Format Items.

To provide more insight into an individual's response process and cognitive process, this study proposed three mixed sequential item response models (MS-IRMs) for mixed-format items consisting of a mixture of a multiple-choice item and an open-ended item that emphasize a sequential response process and are scored sequentially. Relative to existing polytomous models such as the graded response model (GRM), generalized partial credit model (GPCM), or traditional sequential Rasch model (SRM), the proposed models employ an appropriate processing function for each task to improve conventional polytomous models. Simulation studies were carried out to investigate the performance of the proposed models, and the results indicated that all proposed models outperformed the SRM, GRM, and GPCM in terms of parameter recovery and model fit. An application illustration of the MS-IRMs in comparison with traditional models was demonstrated by using real data from TIMSS 2007.

A Monte Carlo study of IRTree models' ability to recover item parameters.

Item response tree (IRTree) models are theorized to extract response styles from self-report data by utilizing multidimensional item response theory (IRT) models based on theoretical decision processes. Despite the growing popularity of the IRTree framework, there has been little research that has systematically examined the ability of its most popular models to recover item parameters across sample size and test length. This Monte Carlo simulation study explored the ability of IRTree models to recover item parameters based on data created from the midpoint primary process model. Results indicate the IRTree model can adequately recover item parameters early in the decision process model, specifically the midpoint node. However, as the model progresses through the decision hierarchy, item parameters have increased associated error variance. The authors ultimately recommend caution when employing the IRTree framework.

Measurement Does Not Take Place in a Legal Vacuum

Measurement Does Not Take Place in a Legal Vacuum

Correcting for Extreme Response Style: Model Choice Matters.

Extreme response style (ERS), the tendency of participants to select extreme item categories regardless of the item content, has frequently been found to decrease the validity of Likert-type questionnaire results. For this reason, various item response theory (IRT) models have been proposed to model ERS and correct for it. Comparisons of these models are however rare in the literature, especially in the context of cross-cultural comparisons, where ERS is even more relevant due to cultural differences between groups. To remedy this issue, the current article examines two frequently used IRT models that can be estimated using standard software: a multidimensional nominal response model (MNRM) and a IRTree model. Studying conceptual differences between these models reveals that they differ substantially in their conceptualization of ERS. These differences result in different category probabilities between the models. To evaluate the impact of these differences in a multigroup context, a simulation study is conducted. Our results show that when the groups differ in their average ERS, the IRTree model and MNRM can drastically differ in their conclusions about the size and presence of differences in the substantive trait between these groups. An empirical example is given and implications for the future use of both models and the conceptualization of ERS are discussed.

Item selection methods in multidimensional computerized adaptive testing for forced-choice items using Thurstonian IRT model.

Multidimensional computerized adaptive testing for forced-choice items (MFC-CAT) combines the benefits of multidimensional forced-choice (MFC) items and computerized adaptive testing (CAT) in that it eliminates response biases and reduces administration time. Previous studies that explored designs of MFC-CAT only discussed item selection methods based on the Fisher information (FI), which is known to perform unstably at early stages of CAT. This study proposes a set of new item selection methods based on the KL information for MFC-CAT (namely MFC-KI, MFC-KB, and MFC-KLP) based on the Thurstonian IRT (TIRT) model. Three simulation studies, including one based on real data, were conducted to compare the performance of the proposed KL-based item selection methods against the existing FI-based methods in three- and five-dimensional MFC-CAT scenarios with various test lengths and inter-trait correlations. Results demonstrate that the proposed KL-based item selection methods are feasible for MFC-CAT and generate acceptable trait estimation accuracy and uniformity of item pool usage. Among the three proposed methods, MFC-KB and MFC-KLP outperformed the existing FI-based item selection methods and resulted in the most accurate trait estimation and relatively even utilization of the item pool.

Dynamic Response Strategies: Accounting for Response Process Heterogeneity in IRTree Decision Nodes

It is essential to control self-reported trait measurements for response style effects to ensure a valid interpretation of estimates. Traditional psychometric models facilitating such control consider item responses as the result of two kinds of response processes—based on the substantive trait, or based on response styles—and they assume that both of these processes have a constant influence across the items of a questionnaire. However, this homogeneity over items is not always given, for instance, if the respondents’ motivation declines throughout the questionnaire so that heuristic responding driven by response styles may gradually take over from cognitively effortful trait-based responding. The present study proposes two dynamic IRTree models, which account for systematic continuous changes and additional random fluctuations of response strategies, by defining item position-dependent trait and response style effects. Simulation analyses demonstrate that the proposed models accurately capture dynamic trajectories of response processes, as well as reliably detect the absence of dynamics, that is, identify constant response strategies. The continuous version of the dynamic model formalizes the underlying response strategies in a parsimonious way and is highly suitable as a cognitive model for investigating response strategy changes over items. The extended model with random fluctuations of strategies can adapt more closely to the item-specific effects of different response processes and thus is a well-fitting model with high flexibility. By using an empirical data set, the benefits of the proposed dynamic approaches over traditional IRTree models are illustrated under realistic conditions.

DIGITALIZATION AT THE SERVICE OF PSYCHODIAGNOSTICS: MODERN OPPORTUNITIES IN INTELLIGENCE TESTING

The assessment of intellectual development in children and adolescents is an essential component of psychodiagnostics, facilitating an understanding of the nature of learning diffi culties. Intelligence tests have a history that spans more than a century, and even contemporary versions of widely used intelligence tests (such as the Wechsler or Stanford-Binet tests) are limited by the infl uence of their traditional paper-based formats. Nevertheless, the advent of computational capabilities and the digitalization of tests have led psychometrics to experience substantial development, opening new possibilities for assessment. The complete application of modern psychometrics holds the promise of revolutionizing the measurement of intelligence. Present-day psychometrics is deeply entwined with computational behavioral sciences, going beyond outdated principles in the development of psychological tests. Methodological approaches that notably enhance the accuracy, reliability, and equity of measurement instruments currently exist. Furthermore, the accumulation of supplementary validity evidence, simplifi - cation of result interpretation, and practical implementation far exceed conventional psychodiagnostic approaches. This presentation will show the potentialities presented by modern psychometrics, including the utilization of innovative task formats such as technology-enhanced items, the integration of universal design principles in test development, the adoption of computerized adaptive testing (CAT) and multistage testing (MST) methodologies, the application of multidimensional IRT models to establish both construct and criterion validity, the shift from discrete norms to continuous scaling, and the incorporation of longitudinal measurements. The combination of these features within a computerized intelligence testing framework will enhance respondents’ motivation and simplify the test administration process. It will facilitate frequent assessments of a child’s progress by presenting varying test versions on each occasion, thereby signifi cantly reducing errors in administration and scoring, while also safeguarding against misuse by non-professionals.

Analyzing the Interaction of Item Position Effect and Student Characteristics within Explanatory IRT Models

The location of the same item in different positions among booklets leads to a biased estimation of item parameters. This undesirable effect on the probability of answering the items correctly is referred as the item position effect. The purpose of this study is to examine the items that are more sensitive to the item position effect and to investigate the student characteristics related to the item position effect. In the study, the items in the PISA 2015 reading domain are used. The study group consists of 2418 students who responded to the items in the reading domain from PISA 2015 Turkey Sample. Explanatory IRT models are used in the analysis of the research. According to the results, 42% of the items are affected by the item position. The most important characteristic related to item position is the SES level of students. In addition, male students are more affected by item position than female students.