IRT Models Research Articles

Investigating Heterogeneity in Response Strategies: A Mixture Multidimensional IRTree Approach.

To improve the validity of self-report measures, researchers should control for response style (RS) effects, which can be achieved with IRTree models. A traditional IRTree model considers a response as a combination of distinct decision-making processes, where the substantive trait affects the decision on response direction, while decisions about choosing the middle category or extreme categories are largely determined by midpoint RS (MRS) and extreme RS (ERS). One limitation of traditional IRTree models is the assumption that all respondents utilize the same set of RS in their response strategies, whereas it can be assumed that the nature and the strength of RS effects can differ between individuals. To address this limitation, we propose a mixture multidimensional IRTree (MM-IRTree) model that detects heterogeneity in response strategies. The MM-IRTree model comprises four latent classes of respondents, each associated with a different set of RS traits in addition to the substantive trait. More specifically, the class-specific response strategies involve (1) only ERS in the "ERS only" class, (2) only MRS in the "MRS only" class, (3) both ERS and MRS in the "2RS" class, and (4) neither ERS nor MRS in the "0RS" class. In a simulation study, we showed that the MM-IRTree model performed well in recovering model parameters and class memberships, whereas the traditional IRTree approach showed poor performance if the population includes a mixture of response strategies. In an application to empirical data, the MM-IRTree model revealed distinct classes with noticeable class sizes, suggesting that respondents indeed utilize different response strategies.

5 Cross-cultural Diagnostic Validity of the Multilingual Naming Test (MINT) in a Sample of Older Adults.

Objective:The current study aimed to evaluate the psychometric properties and diagnostic accuracy of the 32-item version of the Multilingual Naming Test (MINT) in a sample of English and Spanish monolinguals and bilingual older adults from two ethnic groups (EA; European Americans and HA; Hispanic American) with typical and atypical aging. An IRT model was used to identify 24 MINT Items assessed across ethnicity and language testing groups (Spanish and English). We analyzed the discriminant and predictive validity of the 32-item and 24-item scales across diagnostic groups (cognitively normal [CN], mild cognitive impairment [MCI], and dementia [AD]). Diagnostic accuracy was then assessed with both versions applying ROC (Receiver Operating Characteristics) curve reporting using AUC (Area Under the Curve). We expected the MINT to distinguish between the CN and AD groups but not between CN and MCI and the MCI and AD. We conducted IRT analyses to evaluate the cross-language validity of the items from the 32-item MINT in English and Spanish through Rasch Analysis across our two ethnic groups. Finally, we tested the association between MINT scores and MRI volumetric measures of language-related areas in both cerebral hemispheres’ temporal and frontal lobes.Participants and Methods:The sample comprised 281 participants (178 females) enrolled in the 1Florida Alzheimer’s Disease Research Center (ADRC), with 175 participants self-identified as HA (51 tested in English and 124 in Spanish) and 106 EA, all of them monolingual English speakers. The participants were classified into three diagnostic groups: 1. CN (n = 94); 2. MCI (n = 148); and 3. AD (n = 39). Participants are evaluated yearly through a comprehensive neuropsychological battery, including the MINT is a standard CN task that requires patients to retrieve words upon presentation of a line drawing.Results:We obtained a ceiling effect in four items (Butterfly, Glove, Watch, and Candle). Four items were easier in English (Blind, Gauge, Porthole, and Pestle) and four in Spanish (Dustpan, Funnel, Anvil, and Mortar). In the 32-item version of the MINT, EA scored significantly higher than HA, but when removing those eight items, the ethnic difference was attenuated and no longer statistically significant (controlling for education). The ROC curves showed that both versions of the MINT had poor accuracy when identifying CN participants and were acceptable in identifying dementia participants but unacceptable for classifying MCI participants. The 32-item MINT in English and Spanish and the 24-item MINT in Spanish were significantly correlated with the bilateral MTG. However, the 24-item MINT in English was only correlated with this area’s volume in the right hemisphere. The left FG correlated with MINT scores regardless of language and MINT version. We also found some differential correlations depending on the language of administration. The bilateral hippocampi, STG, MTG and FG, and right ITG were significantly correlated only with MINT Spanish scores, while the left ITG was significant only when either version of the MINT was administered in English.Conclusions:Our results highlight the importance of analyzing cross-cultural samples when implementing neuropsychological tests.

Multilevel IRT models to explore heterogeneity in subjective financial knowledge at individual and regional levels: the Italian case

IntroductionModern FinTech tools (e.g., instant payments, blockchain, roboadvisor) represent the new frontier of digital finance. Consequently, the evaluation of the knowledge level of the population about these topics is a crucial concern. In this context, several exogenous factors may influence individual differences in financial literacy. In particular, the territorial characteristics can have an impact on FinTech. In this work, we investigate individual heterogeneity in subjective financial knowledge in Italy, specifically focusing on modern FinTech tools, and exploring the differences at the individual and regional levels.MethodsA sample of 598 Italian individuals from 10 different Italian regions was involved. A multilevel IRT model is performed to evaluate the level of FinTech individual knowledge and the differences according to Italian regions to account for the hierarchical structure of the data.ResultsResults reported a weak regional effect, revealing that heterogeneity in financial knowledge can be mainly attributed to individual characteristics. At the individual level, age, economic condition, knowledge of traditional financial objects and numeracy showed a significant effect. In addition, a scientific field of study and work have an impact on respondents' knowledge level.DiscussionWhat is shown and discussed in this contribution can inspire policymakers' actions to increase financial literacy in the population. In particular, the obtained results imply that policymakers should improve the population's awareness of less popular FinTech tools and foster individuals' literacy about numbers and traditional financial tools, which proved to have a great influence in explaining FinTech knowledge differences.

Influence of respondents’ Differentiation of subjective response on water knowledge stock test scale: Evaluation based on two-parameter-multidimensional IRT model

Insufficient awareness of water issues is a crucial bottleneck restricting the sustainable utilization of water resources. To accurately measure citizens' water knowledge stock and overcome the differences between scales and respondents' characteristic levels on test results, the research focuses on developing and evaluating water knowledge stock test scales. The mechanism for identifying indicators is designed based on the grounded theory, and as a result, the water knowledge stock test indicator system is derived. The data was collected by the form of survey questionnaire developed with the test indicator system. A two-parameter multidimensional item response theoretical model is constructed based on item parameter estimation, data model fitting, and item information function. The survey data and optimization model are used to optimize the water knowledge stock test scale and verify the fitting degree with the characteristics of the respondents. The test information function and standard error function indicate that the scale is most informative for individuals with characteristic levels ranging from −2 to 3, resulting in a highly reliable test effect. The research has established a measurement indicator system, methodology, and presented results that serve as a foundation for measuring the stock of water knowledge.

Nutrition-associated health levels in persons with cancer: item response modelling based on the International Classification of Functioning, Disability and Health.

The nutritional status of individuals with cancer is a crucial determinant of their health and well-being, and addressing nutrition-related functioning conditions is essential for maintaining physical activity levels and participating in daily activities. This study aims to identify an evidence-based International Classification of Functioning, Disability and Health (ICF) scale using item response theory for nutrition conditions in patients with cancer, which can differentiate and assess nutrition-related functioning conditions of cancer survivors. Cross-sectional study. Sir Run Run Hospital, Nanjing Medical University. One hundred cancer patients were enrolled. Via convenience sampling, the study administered a questionnaire consisting of 89 ICF items to participants. The original five-point rating system was binarized (1 = no problem, 0 = problem). Through data shaping, non-parametric IRT analysis and parametric IRT analysis, psychometric properties of nutritional ICF scale were calculated using R software. The study extracted a unidimensional scale with 32 items and constructed 2-parameter logistic model with good fitness, whose root mean square error approximation (RMSEA) = 0.0759, Tucker-Lewis Index (TLI) = 0.9655, and Comparative Fit Index (CFI) = 0.9677. The model demonstrated high reliability, as indicated by a Cronbach's α of 0.95, Guttman λ<inf>2</inf> of 0.95, Molenaar Sijtsma statistic of 0.96 and a latent class reliability coefficient (LCRC) of 0.98. Besides, there was a strong correlation between the total score of 32 ICF items and the score of Patient-Generated Subjective Global Assessment (PG-SGA, P<0.001, r=-0.77) and Mini Nutritional Assessment (MNA, P<0.001, r=0.76), suggesting the 32-item scale had high validity. The study constructed an evidence-based ICF scale for nutrition conditions in patients with cancer with 32-item and 2PLM for evaluating nutrition-associated health level in persons with cancer, including high validity and reliability. The IRT model based on ICF provided a promising assessment tool to discriminate nutrition-associated health level of persons with cancer, and offered an auxiliary method for selecting rehabilitation intervention targets. This has the potential to lead to improved outcomes in cancer treatment and increased quality of life for cancer survivors.

Estimating and Using Block Information in the Thurstonian IRT Model

Multidimensional forced-choice (MFC) tests are increasing in popularity but their construction is complex. The Thurstonian item response model (Thurstonian IRT model) is most often used to score MFC tests that contain dominance items. Currently, in a frequentist framework, information about the latent traits in the Thurstonian IRT model is computed for binary outcomes of pairwise comparisons, but this approach neglects stochastic dependencies. In this manuscript, it is shown how to estimate Fisher information on the block level. A simulation study showed that the observed and expected standard errors based on the block information were similarly accurate. When local dependencies for block sizes >,2 were neglected, the standard errors were underestimated, except with the maximum a posteriori estimator. It is shown how the multidimensional block information can be summarized for test construction. A simulation study and an empirical application showed small differences between the block information summaries depending on the outcome considered. Thus, block information can aid the construction of reliable MFC tests.

Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching

The cross-classified data structure is ubiquitous in education, psychology, and health outcome sciences. In these areas, assessment instruments that are made up of multiple items are frequently used to measure latent constructs. The presence of both the cross-classified structure and multivariate categorical outcomes leads to the so-called item-level data with cross-classified structure. An example of such data structure is the routinely collected student evaluation of teaching (SET) data. Motivated by the lack of research on multilevel IRT modeling with crossed random effects and the need of an approach that can properly handle SET data, this study proposed a cross-classified IRT model, which takes into account both the cross-classified data structure and properties of multiple items in an assessment instrument. A new variant of the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm was introduced to address the computational complexities in estimating the proposed model. A preliminary simulation study was conducted to evaluate the performance of the algorithm for fitting the proposed model to data. The results indicated that model parameters were well recovered. The proposed model was also applied to SET data collected at a large public university to answer empirical research questions. Limitations and future research directions were discussed.

Comparison of the response time-based effort-moderated IRT model and three-parameter logistic model according to computerized adaptive test performances: a simulation study

Depending on the developments in technology and information, paper-pencil tests leave their place for computerized adaptive tests (CATs). CAT is widely used in the field of health, mainly in psychiatry. Many item response theory models have been proposed in the literature regarding the use of response time focusing on item difficulty and personal characteristics by ignoring the multidimensional interactions, therefore these results may cause bias in estimates of individual ability levels. The present simulation study was conducted to compare the performance of CAT applications of the effort-moderated item response theory (EM-IRT) model, which is based on response time, and the three-parameter logistic (3PL) model. While simulating CAT with the EM-IRT model and the 3PL model, the hybrid method was used for ability estimation and maximum Fisher information (MFI) was used for item selection. The CAT process proceeded until the standard error of the estimation was <0.3 and <0.5, or all items in the item bank were used. The number of individuals was specified as 1000, while the number of items was changed to 50, 100, and 250. All six scenarios were repeated 1000 times. With the increase in the number of items and the decrease in the standard error as a stopping criterion, consistent results were obtained with true ability levels in both methods. The CAT with the EM-IRT model estimated true ability level slightly lower than CAT with the 3PL model. The EM-IRT model enables measuring the response time that could yield additional data to the physician about the mental and cognitive condition of the patient. The CAT method can be a promising method of telemedicine in the era of the pandemic.

Адаптивное оценивание в аттестационных процедурах студентов и выпускников

The competency-based approach, which today serves as the basis for the design and implementation of educational programs for the training of professional specialists, provides a certain set of competencies formed by students and graduates (universal, general professional, professional) as learning outcomes. It is possible to assess competencies as latent characteristics only in activity. This latent characteristic, which is the reason why students and graduates are able to perform professional tasks, is the goal of measurement. But in practice, observable assessments of abilities or skills for performing quasi-professional tasks are obtained, according to which conclusions about the level of formation of latent competencies are made. In turn, the implementation of the competency-based approach involves a change in approaches to assess learning outcomes, since it is impossible to talk about obtaining objective (reliable), comparable (valid) and reliable information about achieved learning outcomes and the level of competency formation. This gives rise to the need to refer to a mixed (biparadigm) methodology of educational measurements, on the one hand, and a special mathematical apparatus of the modern test theory (Item Response Theory — IRT), designed to assess the latent parameters of the subjects and the parameters of the tasks of the assessment tools, on the other hand. The proposed approaches to the organization and conduct of certification procedures for students and graduates provide objective, comparable and reasonable results. Unlike traditional formats for conducting the state exam for the final certification of graduates, adaptive assessment is attractive from the point of view of obtaining more accurate estimates of examinees’ parameters (level of preparedness or level of competence formation) due to fewer tasks and creating a “success situation” for each student in the measurement process through machine algorithms selection of such tasks that he will be able to perform. IRT algorithms based, for example, on the maximum likelihood method, make it possible to implement the humanistic ideas of control and evaluation activities. To implement any of the described adaptive approaches in the intermediate or final certification, the following conditions must be met: availability of a bank of calibrated tasks with stable characteristics (difficulties, differentiating ability) or algorithms for their cloning; availability of computer programs or software and instrumental environment (service, platform) that use one or more selected IRT models that help achieve the highest possible measurement accuracy when assessing the level of preparedness or the formation of student competencies; availability of specifications for assessment tools that ensure the meaningful validity of the measurement results. From a didactic point of view, the latter condition is especially important, since in order to obtain objective and comparable results during certification, it is necessary to take into account the content elements of the educational program, the verification of which is planned in the specification of the assessment tools.

An Illustration of an IRTree Model for Disengagement.

Low-stakes test performance commonly reflects examinee ability and effort. Examinees exhibiting low effort may be identified through rapid guessing behavior throughout an assessment. There has been a plethora of methods proposed to adjust scores once rapid guesses have been identified, but these have been plagued by strong assumptions or the removal of examinees. In this study, we illustrate how an IRTree model can be used to adjust examinee ability for rapid guessing behavior. Our approach is flexible as it does not assume independence between rapid guessing behavior and the trait of interest (e.g., ability) nor does it necessitate the removal of examinees who engage in rapid guessing. In addition, our method uniquely allows for the simultaneous modeling of a disengagement latent trait in addition to the trait of interest. The results indicate the model is quite useful for estimating individual differences among examinees in the disengagement latent trait and in providing more precise measurement of examinee ability relative to models ignoring rapid guesses or accommodating it in different ways. A simulation study reveals that our model results in less biased estimates of the trait of interest for individuals with rapid responses, regardless of sample size and rapid response rate in the sample. We conclude with a discussion of extensions of the model and directions for future research.

Gaining a deeper understanding of the meaning of the carelessness parameter in the 4PL IRT model and strategies for estimating it

Three popular models are used to describe the characteristics of the test items and estimate the ability of examinees under the dichotomous IRT model, namely the one-, two-, and three-parameter logistic models. The three-item parameters are discriminating power, difficulty, and pseudo-guessing. In the development of the dichotomous IRT model, carelessness or upper asymptote parameter was proposed, which forms a four-parameter logistic (4PL) model to accommodate a condition where a high-ability examinee gives an incorrect response to a test item when he/she should be able to respond to the test item correctly. However, the carelessness parameter and the 4PL model have not been widely accepted and used due to several factors, and people’s understanding of that parameter and strategies for estimating it is still inadequate. Therefore, this study aims to shed light on ideas underlying the 4PL model, the meaning of the carelessness parameter, and strategies used to estimate that parameter based on the extant literature. The focus of this study was then extended to demonstrating practical examples of estimating item and person parameters using the 4PL model using empirical data on responses of 1,000 students from the Indonesia Open University (Universitas Terbuka) on 21 of 30 multiple-choice items on the Business English test, a paper-and-pencil test. We mainly analyzed empirical data using the ‘mirt’ package in RStudio. We present the analysis results coherently so that IRT users would have a sufficient understanding of the 4PL model and the carelessness parameter, and they can estimate item and person parameters under the 4PL model.

Rejoinder to Commentaries on Lyu, Bolt and Westby's "Exploring the Effects of Item Specific Factors in Sequential and IRTree Models".

We respond to the commentaries on Lyu, Bolt and Westby's "Exploring the effects of item specific factors in sequential and IRTree models." The commentaries raise important points that allow us to clarify our theoretical expectation for item specific factors in many educational and psychological test items. At the same time, we agree with the commentaries in acknowledging challenges associated with providing empirical evidence for their presence and reflect on strategies that might support their estimation. We maintain that the principal concern is the ambiguity item specific factors create in attempting to interpret or use the parameters beyond the first node.

Item-Specific Factors in IRTree Models: When They Matter and When They Don’t

Lyu et al. (Psychometrika, 2023) demonstrated that item-specific factors can cause spurious effects on the structural parameters of IRTree models for multiple nested response processes per item. Here, we discuss some boundary conditions and argue that person selection effects on item parameters are not unique to item-specific factors and that the effects presented by Lyu et al. (Psychometrika, 2023) may not generalize to the family of IRTree models as a whole. We conclude with the recommendation that IRTree model specification should be guided by theoretical considerations, rather than driven by data, in order to avoid misinterpretations of parameter differences.

Exploring the Effects of Item-Specific Factors in Sequential and IRTree Models.

Test items for which the item score reflects a sequential or IRTree modeling outcome are considered. For such items, we argue that item-specific factors, although not empirically measurable, will often be present across stages of the same item. In this paper, we present a conceptual model that incorporates such factors. We use the model to demonstrate how the varying conditional distributions of item-specific factors across stages become absorbed into the stage-specific item discrimination and difficulty parameters, creating ambiguity in the interpretations of item and person parameters beyond the first stage. We discuss implications in relation to various applications considered in the literature, including methodological studies of (1) repeated attempt items; (2) answer change/review, (3) on-demand item hints; (4) item skipping behavior; and (5) Likert scale items. Our own empirical applications, as well as several examples published in the literature, show patterns of violations of item parameter invariance across stages that are highly suggestive of item-specific factors. For applications using sequential or IRTree models as analytical models, or for which the resulting item score might be viewed as outcomes of such a process, we recommend (1) regular inspection of data or analytic results for empirical evidence (or theoretical expectations) of item-specific factors; and (2) sensitivity analyses to evaluate the implications of item-specific factors for the intended inferences or applications.

Person Specific Parameter Heterogeneity in the 2PL IRT Model

Following Kelderman and Molenaar’s demonstration that a factor model with person specific factor loadings is almost indistinguishable from the standard factor model in terms of overall fit, we examined person specific measurement models in Item Response Theory, person specific discrimination and difficulty parameters were created by adding random variation at the item by person level. Using standard fitting algorithms for the 2PL IRT there was modest evidence of person- or item-level misfit using common diagnostic tools. The item difficulties were well-estimated, but the item discriminations were noticeably underestimated. As found by Kelderman and Molenaar, factor scores were estimated with less than expected reliability due to the underlying heterogeneity. The person specific models considered here are basically limiting cases of IRT models with multilevel, mixture, or differential item functioning structure. We conclude with some thoughts regarding real-world sources of heterogeneity that might go unacknowledged in common testing applications.

CUDA-aware MPI implementation of Gibbs sampling for an IRT model

CUDA-aware MPI implementation of Gibbs sampling for an IRT model

Remote cognitive evaluation in aging adults: A new software implementation during COVID‐19

AbstractBackgroundRemote cognitive testing is a convenient and efficient method to evaluate older adults especially because of the sanitary restrictions due to COVID 19. Our aim was to test the psychometric properties of newly developed versions of two visuospatial attention subtests, Hidden Patterns and Identical Pictures, administered by remote cognitive evaluation platform developed by Neurobehavioral Systems, Inc. We conducted testing and analysis in a sample of healthy aging adults.MethodsParticipants were recruited in East‐Bay, California; 184 aging adults completed the computerized hidden patterns test, and 160 older adults answered the identical picture test. The mean age was 70.11 years old with a range of 60 to 85 years old. Most of the participants were female (59.60%), and dementia symptoms were not reported. Participants were evaluated twice in one year at home, the software created by Neurobehavioral Systems, Inc allowed a synchronous interaction with the remote evaluator, which helped to monitor and answer any question before, during or after the evaluation.ResultDifficulty and discrimination analysis (classical test theory) was performed in R (version 4.1.0),. We estimated a Bayesian 1PL Item Response Model (IRT) for each test along with DIF testing by gender. The IRT models were calculated with the package rstan (Stan Development Team, 2020)ConclusionThe IRT models showed group differences at the items’ difficulty. The estimated reliability showed to be adequate (α &gt; .75), and test – retest correlations were medium to high (r &gt; .50). Correlations with age ranged from r = ‐.33 to r = .25.

An Attempt to Measure and Model Women’s Attitudes to Saving for Retirement

Abstract Research background Because of the convergence of demographic, political, and economic trends, retirement has become a significant stage of human life, which currently requires thoughtful and careful planning. Therefore, social pension schemes – a type of system under which the benefits are being provided – and women’s individual saving decisions should give consideration to the fact that it is necessary to accumulate additional funds which should be sufficient for several decades after retirement. Purpose Social pension schemes are widely discussed in all European countries, and in this context we can pose the following question: What is it exactly that makes women save for retirement? This paper has two objectives: to identify the factors that encourage women to save and to measure these factors and develop a model which shows correlations. Research methodology A questionnaire survey was used for collecting the primary data. Subsequently, the analysis is based on 4 indexes created for the purposes of this paper. Then, the indexes were estimated on the basis of the unidimensional two-parameter IRT model, rho and CR coefficients, and multiple-group logistic regression analysis. Novelty The conducted calculations lead to the conclusion that regardless of age, several factors are statistically significant for the whole analysed group, e.g. a place of residence. This is the first paper that shows determinants important for women when it comes to saving for retirement.

SAS PROC IRT and the R mirt Package: A Comparison of Model Parameter Estimation for Multidimensional IRT Models

This study investigates the performance of estimation methods for multidimensional IRT models with dichotomous and polytomous data in two well-known IRT programs: SAS PROC IRT and the mirt package in R. A simulation study was used to compare performance on a simple structure Rasch model, complex structure 2PL model, and bifactor graded response model. Under RMSE and bias criteria regarding item parameter recovery, PROC IRT and mirt showed nearly identical performance in the simple structure condition. When a complex structure was used, mirt performed better in terms of the recovery of intercept parameters, while the recovery of slope parameters depended on the program and the sample sizes: PROC IRT tended to be better with small samples (N=500) according to RMSE, and mirt was better for larger samples (N=1000 and 2500) according to RMSE and bias for the slope parameter recovery. When a bifactor structure was used, mirt was preferred in all cases; differences lessened as sample size increased.

The analysis of TIMSS 2015 data with confirmatory mixture item response theory: A multidimensional approach

In this study, we illustrated an application of the confirmatory mixture IRT model for multidimensional tests. We aimed to examine the differences in student performance by domains with a confirmatory mixture IRT modeling approach. A three-dimensional and three-class model was analyzed by assuming content domains as dimensions and cognitive domains as item groups. We estimated the item performance differences among the students through structural parameters. There were 463 students from Turkey and 880 students from Canada who participated in the TIMSS 2015 4th-grade mathematics assessment. Results for Turkey indicated, students in Class 2 had better performance in knowing and reasoning compared to those in Classes 1 and 3. Students in Class 2 and Class 3 were similar in applying math concepts compared to students in Class 1. For the Canadian sample, students in Class 2 had better performance in knowing, applying, and reasoning compared to those in Class 1 and 3. Also, Class 3 students were better at applying domain than Class 1. Also, mean values were obtained for all content domains in the two countries. Confirmatory mixture IRT modeling approaches appear to differentiate students’ mathematics competencies.