Item Response Models Research Articles

The InterModel Vigorish as a Lens for Understanding (and Quantifying) the Value of Item Response Models for Dichotomously Coded Items.

The deployment of statistical models-such as those used in item response theory-necessitates the use of indices that are informative about the degree to which a given model is appropriate for a specific data context. We introduce the InterModel Vigorish (IMV) as an index that can be used to quantify accuracy for models of dichotomous item responses based on the improvement across two sets of predictions (i.e., predictions from two item response models or predictions from a single such model relative to prediction based on the mean). This index has a range of desirable features: It can be used for the comparison of non-nested models and its values are highly portable and generalizable. We use this fact to compare predictive performance across a variety of simulated data contexts and also demonstrate qualitative differences in behavior between the IMV and other common indices (e.g., the AIC and RMSEA). We also illustrate the utility of the IMV in empirical applications with data from 89 dichotomous item response datasets. These empirical applications help illustrate how the IMV can be used in practice and substantiate our claims regarding various aspects of model performance. These findings indicate that the IMV may be a useful indicator in psychometrics, especially as it allows for easy comparison of predictions across a variety of contexts.

Interpretation of course conceptual structure and student self-efficacy: an integrated strategy of knowledge graphs with item response modeling

BackgroundThere is a scarcity of studies that quantitatively assess the difficulty and importance of knowledge points (KPs) depending on students’ self-efficacy for learning (SEL). This study aims to validate the practical application of psychological measurement tools in physical therapy education by analyzing student SEL and course conceptual structure.MethodsFrom the “Therapeutic Exercise” course curriculum, we extracted 100 KPs and administered a difficulty rating questionnaire to 218 students post-final exam. The pipeline of the non-parametric Item Response Theory (IRT) and parametric IRT modeling was employed to estimate student SEL and describe the hierarchy of KPs in terms of item difficulty. Additionally, Gaussian Graphical Models with Non-Convex Penalties were deployed to create a Knowledge Graph (KG) and identify the main components. A visual analytics approach was then proposed to understand the correlation and difficulty level of KPs.ResultsWe identified 50 KPs to create the Mokken scale, which exhibited high reliability (Cronbach’s alpha = 0.9675) with no gender bias at the overall or at each item level (p > 0.05). The three-parameter logistic model (3PLM) demonstrated good fitness with questionnaire data, whose Root Mean Square Error Approximation was < 0.05. Also, item-model fitness unveiled good fitness, as indicated by each item with non-significant p-values for chi-square tests. The Wright map revealed item difficulty relative to SEL levels. SEL estimated by the 3PLM correlated significantly with the high-ability range of average Grade-Point Average (p < 0.05). The KG backbone structure consisted of 58 KPs, with 29 KPs overlapping with the Mokken scale. Visual analysis of the KG backbone structure revealed that the difficulty level of KPs in the IRT could not replace their position parameters in the KG.ConclusionThe IRT and KG methods utilized in this study offer distinct perspectives for visualizing hierarchical relationships and correlations among the KPs. Based on real-world teaching empirical data, this study helps to provide a research foundation for updating course contents and customizing learning objectives.Trial registrationNot applicable.

A nonparametric Bayesian item response modelling approach for clustering items and individuals simultaneously

Item response theory (IRT) is a popular modelling paradigm for measuring subject latent traits and item properties according to discrete responses in tests or questionnaires. There are very limited discussions on heterogeneity pattern detection for both items and individuals. In this paper, we introduce a nonparametric Bayesian approach for clustering items and individuals simultaneously under the Rasch model. Specifically, our proposed method is based on the mixture of finite mixtures (MFM) model, which obtains the number of clusters and the clustering configurations for both items and individuals simultaneously. The performance of parameter estimation and parameter clustering under the MFM Rasch model is evaluated by simulation studies, and two real examples are applied to further illustrate the MFM Rasch modelling.

Validation of an elicited Imitation test as a measure of Korean language proficiency

This study presents a modified version of the Korean Elicited Imitation (EI) test, designed to resemble natural spoken language, and validates its reliability as a measure of proficiency. The study assesses the correlation between average test scores and Test of Proficiency in Korean (TOPIK) levels, examining score distributions among beginner, intermediate, and advanced learner groups. Using item response theory (IRT), the study explores the influence of four key facets—learners, items, raters, and constructs—on performance evaluation. An explanatory item response modeling (EIRM) analysis identified linguistic factors impacting the EI test’s performance. Notably, the study uncovered a robust positive correlation between EI test scores and TOPIK levels. Significant score disparities were observed between beginner and intermediate, as well as beginner and advanced, learner cohorts. An IRT-based exploration of each facet revealed that item difficulty was comparatively lower in contrast to learners’ commendable performance, and raters exhibited a high degree of scoring consistency. The EIRM analysis underscores the significance of variables such as the number of syllables, vocabulary score, and content word density in influencing test performance.

Are factor scores measurement invariant?

There has been increased interest in practical methods for integrative analysis of data from multiple studies or samples, and using factor scores to represent constructs has become a popular and practical alternative to latent variable models with all individual items. Although researchers are aware that scores representing the same construct should be on a similar metric across samples-namely they should be measurement invariant-for integrative data analysis, the methodological literature is unclear whether factor scores would satisfy such a requirement. In this note, we show that even when researchers successfully calibrate the latent factors to the same metric across samples, factor scores-which are estimates of the latent factors but not the factors themselves-may not be measurement invariant. Specifically, we prove that factor scores computed based on the popular regression method are generally not measurement invariant. Surprisingly, such scores can be noninvariant even when the items are invariant. We also demonstrate that our conclusions generalize to similar shrinkage scores in item response models for discrete items, namely the expected a posteriori scores and the maximum a posteriori scores. Researchers should be cautious in directly using factor scores for cross-sample analyses, even when such scores are obtained from measurement models that account for noninvariance. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Extended Asymptotic Identifiability of Nonparametric Item Response Models.

Nonparametric item response models provide a flexible framework in psychological and educational measurements. Douglas (Psychometrika 66(4):531-540, 2001) established asymptotic identifiability for a class of models with nonparametric response functions for long assessments. Nevertheless, the model class examined in Douglas (2001) excludes several popular parametric item response models. This limitation can hinder the applications in which nonparametric and parametric models are compared, such as evaluating model goodness-of-fit. To address this issue, We consider an extended nonparametric model class that encompasses most parametric models and establish asymptotic identifiability. The results bridge the parametric and nonparametric item response models and provide a solid theoretical foundation for the applications of nonparametric item response models for assessments with many items.

Differential and Functional Response Time Item Analysis: An Application to Understanding Paper versus Digital Reading Processes

AbstractDespite the growing interest in incorporating response time data into item response models, there has been a lack of research investigating how the effect of speed on the probability of a correct response varies across different groups (e.g., experimental conditions) for various items (i.e., differential response time item analysis). Furthermore, previous research has shown a complex relationship between response time and accuracy, necessitating a functional analysis to understand the patterns that manifest from this relationship. In this study, response time data are incorporated into an item response model for two purposes: (a) to examine how individuals' speed within an experimental condition affects their response accuracy on an item, and (b) to detect the differences in individuals' speed between conditions in the presence of within‐condition effects. For these two purposes, by‐variable smooth functions are employed to model differential and functional response time effects by experimental condition for each item. This model is illustrated using an empirical data set to describe the effect of individuals' speed on their reading comprehension ability in two experimental conditions of reading medium (paper vs. digital) by item. A simulation study showed that the recovery of parameters and by‐variable smooth functions of response time was satisfactory, and that the type I error rate and power of the test for the by‐variable smooth function of response time were acceptable in conditions similar to the empirical data set. In addition, the proposed method correctly identified the range of response time where between‐condition differences in the effect of response time on the probability of a correct response were accurate.

Guesses and Slips as Proficiency‐Related Phenomena and Impacts on Parameter Invariance

AbstractTraditional approaches to the modeling of multiple‐choice item response data (e.g., 3PL, 4PL models) emphasize slips and guesses as random events. In this paper, an item response model is presented that characterizes both disjunctively interacting guessing and conjunctively interacting slipping processes as proficiency‐related phenomena. We show how evidence for this perspective is seen in the systematic form of invariance violations for item slip and guess parameters under four‐parameter IRT models when compared across populations of different mean proficiency levels. Specifically, higher proficiency populations tend to show higher guess and lower slip probabilities than lower proficiency populations. The results undermine the use of traditional models for IRT applications that require invariance and would suggest greater attention to alternatives.

Smooth Information Criterion for Regularized Estimation of Item Response Models

Item response theory (IRT) models are frequently used to analyze multivariate categorical data from questionnaires or cognitive test data. In order to reduce the model complexity in item response models, regularized estimation is now widely applied, adding a nondifferentiable penalty function like the LASSO or the SCAD penalty to the log-likelihood function in the optimization function. In most applications, regularized estimation repeatedly estimates the IRT model on a grid of regularization parameters λ. The final model is selected for the parameter that minimizes the Akaike or Bayesian information criterion (AIC or BIC). In recent work, it has been proposed to directly minimize a smooth approximation of the AIC or the BIC for regularized estimation. This approach circumvents the repeated estimation of the IRT model. To this end, the computation time is substantially reduced. The adequacy of the new approach is demonstrated by three simulation studies focusing on regularized estimation for IRT models with differential item functioning, multidimensional IRT models with cross-loadings, and the mixed Rasch/two-parameter logistic IRT model. It was found from the simulation studies that the computationally less demanding direct optimization based on the smooth variants of AIC and BIC had comparable or improved performance compared to the ordinarily employed repeated regularized estimation based on AIC or BIC.

The Impact of Non-Effortful Responding on Item and Person Parameters in Item-Pool Scaling Linking

ABSTRACT Low test-taking effort of examinees has been considered a source of construct-irrelevant variance in item response modeling, leading to serious consequences on parameter estimation. This study aims to investigate how non-effortful response (NER) influences the estimation of item and person parameters in item-pool scale linking (IPSL) and whether differential NER for two groups has a different effect on parameter estimation under the equivalent and non-equivalent group designs. Three simulation studies and an empirical study were conducted to answer these research questions. Results show that NER on items in a certain item set (e.g. common new items, exclusive new items) have a dominant negative effect on parameter estimation of items from the same item set, while NER on items from the item pool would seriously impact the accuracy of ability parameters. Discussions and recommendations for item set orders are provided.

A Recent Development of a Network Approach to Assessment Data: Latent Space Item Response Modeling for Intelligence Studies.

This article aims to provide an overview of the potential advantages and utilities of the recently proposed Latent Space Item Response Model (LSIRM) in the context of intelligence studies. The LSIRM integrates the traditional Rasch IRT model for psychometric data with the latent space model for network data. The model has person-wise latent abilities and item difficulty parameters, capturing the main person and item effects, akin to the Rasch model. However, it additionally assumes that persons and items can be mapped onto the same metric space called a latent space and distances between persons and items represent further decreases in response accuracy uncaptured by the main model parameters. In this way, the model can account for conditional dependence or interactions between persons and items unexplained by the Rasch model. With two empirical datasets, we illustrate that (1) the latent space can provide information on respondents and items that cannot be captured by the Rasch model, (2) the LSIRM can quantify and visualize potential between-person variations in item difficulty, (3) latent dimensions/clusters of persons and items can be detected or extracted based on their latent positions on the map, and (4) personalized feedback can be generated from person-item distances. We conclude with discussions related to the latent space modeling integrated with other psychometric models and potential future directions.

HBMIRT: A SAS macro for estimating uni- and multidimensional 1- and 2-parameter item response models in small (and large!) samples

Item response theory (IRT) has evolved as a standard psychometric approach in recent years, in particular for test construction based on dichotomous (i.e., true/false) items. Unfortunately, large samples are typically needed for item refinement in unidimensional models and even more so in the multidimensional case. However, Bayesian IRT approaches with hierarchical priors have recently been shown to be promising for estimating even complex models in small samples. Still, it may be challenging for applied researchers to set up such IRT models in general purpose or specialized statistical computer programs. Therefore, we developed a user-friendly tool – a SAS macro called HBMIRT – that allows to estimate uni- and multidimensional IRT models with dichotomous items. We explain the capabilities and features of the macro and demonstrate the particular advantages of the implemented hierarchical priors in rather small samples over weakly informative priors and traditional maximum likelihood estimation with the help of a simulation study. The macro can also be used with the online version of SAS OnDemand for Academics that is freely accessible for academic researchers.

Item Response Analysis of a Structured Mixture Item Response Model with mirt Package in R

Structured mixture item response models (StrMixIRMs) are a special type of constrained confirmatory mixture item response theory (IRT) model for detecting latent performance differences in a measurement instrument by characteristic item groups, and classifying respondents according to these differences. In light of limited software options for estimating StrMixIRMs under existing frameworks, this paper proposes reparameterizing it as a confirmatory mixture IRT model using interaction effects between latent classes and item groups. The reparameterization allows for easier implementation of StrMixIRMs with multiple software programs that have mixture modeling capabilities, including open-source ones. This widens the accessibility to these models to a broad range of users and thus can facilitate research and applications of StrMixIRMs. This paper serves two main goals: First, we introduce StrMixIRMs, focusing on the proposed reparameterization based on interaction effects and its various extensions. Second, we illustrate use cases of this novel reparameterization within the mirt 1.41 package in R by employing two empirical datasets. Detailed R code with notes are provided for the applications along with an interpretation of the outputs.

Recognize the Value of the Sum Score, Psychometrics' Greatest Accomplishment.

The sum score on a psychological test is, and should continue to be, a tool central in psychometric practice. This position runs counter to several psychometricians' belief that the sum score represents a pre-scientific conception that must be abandoned from psychometrics in favor of latent variables. First, we reiterate that the sum score stochastically orders the latent variable in a wide variety of much-used item response models. In fact, item response theory provides a mathematically based justification for the ordinal use of the sum score. Second, because discussions about the sum score often involve its reliability and estimation methods as well, we show that, based on very general assumptions, classical test theory provides a family of lower bounds several of which are close to the true reliability under reasonable conditions. Finally, we argue that eventually sum scores derive their value from the degree to which they enable predicting practically relevant events and behaviors. None of our discussion is meant to discredit modern measurement models; they have their own merits unattainable for classical test theory, but the latter model provides impressive contributions to psychometrics based on very few assumptions that seem to have become obscured in the past few decades. Their generality and practical usefulness add to the accomplishments of more recent approaches.

A Comparative Study of Item Response Theory Models for Mixed Discrete-Continuous Responses.

Language proficiency assessments are pivotal in educational and professional decision-making. With the integration of AI-driven technologies, these assessments can more frequently use item types, such as dictation tasks, producing response features with a mixture of discrete and continuous distributions. This study evaluates novel measurement models tailored to these unique response features. Specifically, we evaluated the performance of the zero-and-one-inflated extensions of the Beta, Simplex, and Samejima's Continuous item response models and incorporated collateral information into the estimation using latent regression. Our findings highlight that while all models provided highly correlated results regarding item and person parameters, the Beta item response model showcased superior out-of-sample predictive accuracy. However, a significant challenge was the absence of established benchmarks for evaluating model and item fit for these novel item response models. There is a need for further research to establish benchmarks for evaluating the fit of these innovative models to ensure their reliability and validity in real-world applications.

Conditional Dependence across Slow and Fast Item Responses: With a Latent Space Item Response Modeling Approach.

There recently have been many studies examining conditional dependence between response accuracy and response times in cognitive tests. While most previous research has focused on revealing a general pattern of conditional dependence for all respondents and items, it is plausible that the pattern may vary across respondents and items. In this paper, we attend to its potential heterogeneity and examine the item and person specificities involved in the conditional dependence between item responses and response times. To this end, we use a latent space item response theory (LSIRT) approach with an interaction map that visualizes conditional dependence in response data in the form of item-respondent interactions. We incorporate response time information into the interaction map by applying LSIRT models to slow and fast item responses. Through empirical illustrations with three cognitive test datasets, we confirm the presence and patterns of conditional dependence between item responses and response times, a result consistent with previous studies. Our results further illustrate the heterogeneity in the conditional dependence across respondents, which provides insights into understanding individuals' underlying item-solving processes in cognitive tests. Some practical implications of the results and the use of interaction maps in cognitive tests are discussed.

Comparison of Two Methods for Determining Item Characteristic Functions and Latent Variable Time-Course for Pharmacometric Item Response Models

There are examples in the literature demonstrating different approaches to defining the item characteristic functions (ICF) and characterizing the latent variable time-course within a pharmacometrics item response theory (IRT) framework. One such method estimates both the ICF and latent variable time-course simultaneously, and another method establishes the ICF first then models the latent variable directly. To date, a direct comparison of the “simultaneous” and “sequential” methodologies described in this work has not yet been systematically investigated. Item parameters from a graded response IRT model developed from Parkinson’s Progression Marker Initiative (PPMI) study data were used as simulation parameters. Each method was evaluated under the following conditions: (i) with and without drug effect and (ii) slow progression rate with smaller sample size and rapid progression rate with larger sample size. Overall, the methods performed similarly, with low bias and good precision for key parameters and hypothesis testing for drug effect. The ICF parameters were well determined when the model was correctly specified, with an increase in precision in the scenario with rapid progression. In terms of drug effect, both methods had large estimation bias for the slow progression rate; however, this bias can be considered small relative to overall progression rate. Both methods demonstrated type 1 error control and similar discrimination between model with and without drug effect. The simultaneous method was slightly more precise than the sequential method while the sequential method was more robust towards longitudinal model misspecification and offers practical advantages in model building.

Rotation Local Solutions in Multidimensional Item Response Theory Models

We conducted an extensive Monte Carlo study of factor-rotation local solutions (LS) in multidimensional, two-parameter logistic (M2PL) item response models. In this study, we simulated more than 19,200 data sets that were drawn from 96 model conditions and performed more than 7.6 million rotations to examine the influence of (a) slope parameter sizes, (b) number of indicators per factor (trait), (c) probabilities of cross-loadings, (d) factor correlation sizes, (e) model approximation error, and (f) sample sizes on the local solution rates of the oblimin and (oblique) geomin rotation algorithms. To accommodate these design variables, we extended the standard M2PL model to include correlated major factors and uncorrelated minor factors (to represent model error). Our results showed that both rotation methods converged to LS under some conditions with geomin producing the highest local solution rates across many models. Our results also showed that, for identical item response patterns, rotation LS can produce different latent trait estimates with different levels of measurement precision (as indexed by the conditional standard error of measurement). Follow-up analyses revealed that when rotation algorithms converged to multiple solutions, quantitative indices of structural fit, such as numerical measures of simple structure, will often misidentify the rotation that is closest in mean-squared error to the factor pattern (or item-slope pattern) of the data-generating model.

Measuring a Critical Component of Contraceptive Decision Making: The Contraceptive Concerns and Beliefs Scale

IntroductionConcerns about safety and side effects from contraceptives are widespread and related to reluctance to use them. Measuring these concerns is an essential component of understanding contraceptive decision-making and guiding contraceptive and interpregnancy clinical care.MethodsWe used qualitative research and item response theory to develop and test a psychometric instrument to measure contraceptive concerns and beliefs. We developed 55 candidate scale items and tested them among 572 adolescents and adults across nine California healthcare facilities in 2019–2020. We derived a 6-item scale and assessed differences by age and social determinants of health with multivariable regression.ResultsIn qualitative data, participants voiced both concerns and positive beliefs about contraception. Quantitative survey respondents were aged 21 years on average, and 24% were parous. Over half (54%) worried contraception has dangerous side effects, and 39% worried it is unnatural. The mean Contraceptive Concerns score, increasing with higher concerns, was 1.85 (SD: 1.00, range 0–4, α = 0.81). Items fit a partial credit item response model and met prespecified criteria for internal structure validity. Contraceptive use declined with increasing Concerns score (adjusted prevalence ratio [aPR] = 0.81 [0.72–0.92]). Scores were elevated among Black (mean: 2.06; aβ = 0.34 [0.09, 0.59]) and Multiracial or other race (2.11; aβ = 0.34 [0.02, 0.66]) respondents vs. White (1.66), but not Latinx respondents (1.81; aβ = 0.11 [− 0.11, 0.33]). Scores were also elevated among participants with lower maternal education (high school/Associate’s 1.89 versus college 1.60; aβ = 0.28 [0.04, 0.53]).DiscussionThe psychometrically robust Concerns instrument can be used in research to measure autonomous contraceptive decision-making and to design person-centered care.

Simplified functioning assessment for low back pain: ICF-based item response theory modelling.

Despite the growing interest of the item response theory (IRT) in assessment of person abilities and functioning difficulties in screening tools, there is scarcity of research using IRT on ICF-based tools for persons with low back pain (LBP). To generate and validate a parsimonious core set of ICF (PCSI) for LBP based on the IRT modelling. A cross-sectional study. Nine hospitals in Jiangsu Province, China. We recruited patients with LBP. All participants completed the 78 items of the comprehensive ICF Core Set for LBP. The five-point scoring system was converted to dichotomous data with 1 as functioning/independent and 0 as impairment/dependent. Psychometric properties of the data were examined using Mokken Scale analysis and parametric item response modelling. This study recruited 306 participants (185 females and 121 males) with LBP. The overall median age of the study participants was 50.28 (95% CI 23.34; 82.05) years. We constructed a three-parameter logistic model with 28 ICF categories (8 of body function, 18 of activities and participation, and two of body structures). The internal consistency was good with Cronbach's alpha = 0.927 and latent class reliability coefficient (LCRC) = 0.955. The model was validated by significant correlations (P<0.001) of its estimated person abilities with the Oswestry Disability Index (ODI, r=-0.41), the Roland-Morris Disability Questionnaire (RMDQ, r=-0.57), the Physical Component Summary (PCS, r=0.63), and the Mental Component Summary (MCS, r = 0.46) of 12-Item Short Form Survey (SF-12). The person abilities and item difficulties were integrated into a Wright map that offered a background for making individualized clinical decisions. The PCSI of LBP with 28 categories has good construct validity and internal consistency, and is a convenient instrument for assessing functioning among persons with LBP. The IRT model provided theoretical and algorithmic support for deriving a simplified model for functioning assessment hence serving a basis for formulating rehabilitation plans in clinical practice and research. A parsimonious core set of ICF (PCSI) for LBP based on the IRT modelling provides a background for making individualized clinical decisions based on item difficulties.