Multidimensional Item Response Theory Models Research Articles

Co-occurring dominance and ideal point processes: A general IRTree framework for multidimensional item responding

AbstractResponding to rating scale items is a multidimensional process, since not only the substantive trait being measured but also additional personal characteristics can affect the respondents’ category choices. A flexible model class for analyzing such multidimensional responses are IRTree models, in which rating responses are decomposed into a sequence of sub-decisions. Different response processes can be involved in item responding both sequentially across those sub-decisions and as co-occurring processes within sub-decisions. In the previous literature, modeling co-occurring processes has been exclusively limited to dominance models, where higher trait levels are associated with higher expected scores. However, some response processes may rather follow an ideal point rationale, where the expected score depends on the proximity of a person’s trait level to the item’s location. Therefore, we propose a new multidimensional IRT model of co-occurring dominance and ideal point processes (DI-MIRT model) as a flexible framework for parameterizing IRTree sub-decisions with multiple dominance processes, multiple ideal point processes, and combinations of both. The DI-MIRT parameterization opens up new application areas for the IRTree model class and allows the specification of a wide range of theoretical assumptions regarding the cognitive processing of item responding. A simulation study shows that IRTree models with DI-MIRT parameterization provide excellent parameter recovery and accurately reflect co-occurring dominance and ideal point processes. In addition, a clear advantage over traditional IRTree models with purely sequential processes is demonstrated. Two application examples from the field of response style analysis highlight the benefits of the general IRTree framework under real-world conditions.

Smooth Information Criterion for Regularized Estimation of Item Response Models

Item response theory (IRT) models are frequently used to analyze multivariate categorical data from questionnaires or cognitive test data. In order to reduce the model complexity in item response models, regularized estimation is now widely applied, adding a nondifferentiable penalty function like the LASSO or the SCAD penalty to the log-likelihood function in the optimization function. In most applications, regularized estimation repeatedly estimates the IRT model on a grid of regularization parameters λ. The final model is selected for the parameter that minimizes the Akaike or Bayesian information criterion (AIC or BIC). In recent work, it has been proposed to directly minimize a smooth approximation of the AIC or the BIC for regularized estimation. This approach circumvents the repeated estimation of the IRT model. To this end, the computation time is substantially reduced. The adequacy of the new approach is demonstrated by three simulation studies focusing on regularized estimation for IRT models with differential item functioning, multidimensional IRT models with cross-loadings, and the mixed Rasch/two-parameter logistic IRT model. It was found from the simulation studies that the computationally less demanding direct optimization based on the smooth variants of AIC and BIC had comparable or improved performance compared to the ordinarily employed repeated regularized estimation based on AIC or BIC.

A Method of Empirical Q-Matrix Validation for Multidimensional Item Response Theory

ABSTRACT A Q-matrix is a binary matrix that defines the relationship between items and latent variables and is widely used in diagnostic classification models (DCMs), and can also be adopted in multidimensional item response theory (MIRT) models. The construction process of the Q-matrix is typically carried out by experts in the subject area of the items and statistical procedures can be used to verify its suitability. In DCMs, different approaches have been proposed for validating the Q-matrix through iterative algorithms. This article presents a method of empirical Q-matrix validation for MIRT models. A simulation study and an application to real data of morphology skills of elementary students are conducted to examine the viability of the method. Relevant issues regarding the implementation of the method and the results obtained are discussed.

HBMIRT: A SAS macro for estimating uni- and multidimensional 1- and 2-parameter item response models in small (and large!) samples

Item response theory (IRT) has evolved as a standard psychometric approach in recent years, in particular for test construction based on dichotomous (i.e., true/false) items. Unfortunately, large samples are typically needed for item refinement in unidimensional models and even more so in the multidimensional case. However, Bayesian IRT approaches with hierarchical priors have recently been shown to be promising for estimating even complex models in small samples. Still, it may be challenging for applied researchers to set up such IRT models in general purpose or specialized statistical computer programs. Therefore, we developed a user-friendly tool – a SAS macro called HBMIRT – that allows to estimate uni- and multidimensional IRT models with dichotomous items. We explain the capabilities and features of the macro and demonstrate the particular advantages of the implemented hierarchical priors in rather small samples over weakly informative priors and traditional maximum likelihood estimation with the help of a simulation study. The macro can also be used with the online version of SAS OnDemand for Academics that is freely accessible for academic researchers.

How Scoring Approaches Impact Estimates of Growth in the Presence of Survey Item Ceiling Effects.

Survey scores are often the basis for understanding how individuals grow psychologically and socio-emotionally. A known problem with many surveys is that the items are all "easy"-that is, individuals tend to use only the top one or two response categories on the Likert scale. Such an issue could be especially problematic, and lead to ceiling effects, when the same survey is administered repeatedly over time. In this study, we conduct simulation and empirical studies to (a) quantify the impact of these ceiling effects on growth estimates when using typical scoring approaches like sum scores and unidimensional item response theory (IRT) models and (b) examine whether approaches to survey design and scoring, including employing various longitudinal multidimensional IRT (MIRT) models, can mitigate any bias in growth estimates. We show that bias is substantial when using typical scoring approaches and that, while lengthening the survey helps somewhat, using a longitudinal MIRT model with plausible values scoring all but alleviates the issue. Results have implications for scoring surveys in growth studies going forward, as well as understanding how Likert item ceiling effects may be contributing to replication failures.

Variational Estimation for Multidimensional Generalized Partial Credit Model.

Multidimensional item response theory (MIRT) models have generated increasing interest in the psychometrics literature. Efficient approaches for estimating MIRT models with dichotomous responses have been developed, but constructing an equally efficient and robust algorithm for polytomous models has received limited attention. To address this gap, this paper presents a novel Gaussian variational estimation algorithm for the multidimensional generalized partial credit model. The proposed algorithm demonstrates both fast and accurate performance, as illustrated through a series of simulation studies and two real data analyses.

Examining Differential Item Functioning from a Multidimensional IRT Perspective.

Differential item functioning (DIF) is a standard analysis for every testing company. Research has demonstrated that DIF can result when test items measure different ability composites, and the groups being examined for DIF exhibit distinct underlying ability distributions on those composite abilities. In this article, we examine DIF from a two-dimensional multidimensional item response theory (MIRT) perspective. We begin by delving into the compensatory MIRT model, illustrating and how items and the composites they measure can be graphically represented. Additionally, we discuss how estimated item parameters can vary based on the underlying latent ability distributions of the examinees. Analytical research highlighting the consequences of ignoring dimensionally and applying unidimensional IRT models, where the two-dimensional latent space is mapped onto a unidimensional, is reviewed. Next, we investigate three different approaches to understanding DIF from a MIRT standpoint: 1. Analytically Uniform and Nonuniform DIF: When two groups of interest have different two-dimensional ability distributions, a unidimensional model is estimated. 2. Accounting for complete latent ability space: We emphasize the importance of considering the entire latent ability space when using DIF conditional approaches, which leads to the mitigation of DIF effects. 3. Scenario-Based DIF: Even when underlying two-dimensional distributions are identical for two groups, differing problem-solving approaches can still lead to DIF. Modern software programs facilitate routine DIF procedures for comparing response data from two identified groups of interest. The real challenge is to identify why DIF could occur with flagged items. Thus, as a closing challenge, we present four items (Appendix A) from a standardized test and invite readers to identify which group was favored by a DIF analysis.

Variation of subclinical psychosis across 16 sites in Europe and Brazil: findings from the multi-national EU-GEI study.

Incidence of first-episode psychosis (FEP) varies substantially across geographic regions. Phenotypes of subclinical psychosis (SP), such as psychotic-like experiences (PLEs) and schizotypy, present several similarities with psychosis. We aimed to examine whether SP measures varied across different sites and whether this variation was comparable with FEP incidence within the same areas. We further examined contribution of environmental and genetic factors to SP. We used data from 1497 controls recruited in 16 different sites across 6 countries. Factor scores for several psychopathological dimensions of schizotypy and PLEs were obtained using multidimensional item response theory models. Variation of these scores was assessed using multi-level regression analysis to estimate individual and between-sites variance adjusting for age, sex, education, migrant, employment and relational status, childhood adversity, and cannabis use. In the final model we added local FEP incidence as a second-level variable. Association with genetic liability was examined separately. Schizotypy showed a large between-sites variation with up to 15% of variance attributable to site-level characteristics. Adding local FEP incidence to the model considerably reduced the between-sites unexplained schizotypy variance. PLEs did not show as much variation. Overall, SP was associated with younger age, migrant, unmarried, unemployed and less educated individuals, cannabis use, and childhood adversity. Both phenotypes were associated with genetic liability to schizophrenia. Schizotypy showed substantial between-sites variation, being more represented in areas where FEP incidence is higher. This supports the hypothesis that shared contextual factors shape the between-sites variation of psychosis across the spectrum.

Development and initial validation of a creative self-efficacy scale for undergraduates: categorical confirmatory factor analysis and multidimensional item response theory

Creative self-efficacy (CSE) has recently received much attention due to its association with student learning and creativity. To that end, a CSE scale was developed for undergraduates and sources of validity evidence based on scale content, response processes, and internal structure were collected. Score reliability, using categorical omega based on the categorical confirmatory factor analysis model and marginal reliability for response pattern scores based on item response theory (IRT), were estimated. After various revision iterations of the initial 28-item pool by 10 subject matter experts and 18 undergraduates, some items were revised, four items were dropped, and ultimately 24 items were field tested for measuring two hypothesized dimensions of CSE among 602 undergraduates. Categorical confirmatory factor analysis results indicated that the two-dimensional model had better fit. Similarly, between the two competing multidimensional IRT models, the two-dimensional graded response model had the best fit. Categorical omega coefficients and marginal reliability for response pattern scores were, respectively, 0.88 and 0.81 for the two underlying dimensions.

How Modern Psychometrics Can Change Intelligence Testing of Children and Adolescents in Russia

The assessment of intellectual development in children and adolescents is an essential component of psychodiagnostics, facilitating an understanding of the nature of learning difficulties. Intelligence tests have a history that spans more than a century, and even contemporary versions of widely used intelligence tests (such as the Wechsler or Stanford-Binet tests) are limited by the influence of their traditional paper-based formats. Nevertheless, the advent of computational capabilities and the digitalization of tests have led psychometrics to experience substantial development, opening new possibilities for assessment. The complete application of modern psychometrics holds the promise of revolutionizing the measurement of intelligence. This paper presents the current situation of measuring intelligence in children and adolescents in the world and in Russia. It describes the potentialities presented by modern psychometrics, including the utilization of innovative task formats such as technology-enhanced items, the integration of universal design principles in test development, the adoption of computerized adaptive testing (CAT) and multistage testing (MST) methodologies, the application of multidimensional IRT models to establish both construct and criterion validity, the shift from discrete norms to continuous scaling, and the incorporation of longitudinal measurements. The combination of these features within a computerized intelligence testing framework will enhance respondents' motivation and simplify the test administration process. It will facilitate frequent assessments of a child's progress by presenting varying test versions on each occasion, thereby significantly reducing errors in administration and scoring, while also safeguarding against misuse by non-professionals.

Estimation of a Simple Structure in a Multidimensional IRT Model Using Structure Regularization.

We develop a method for estimating a simple matrix for a multidimensional item response theory model. Our proposed method allows each test item to correspond to a single latent trait, making the results easier to interpret. It also enables clustering of test items based on their corresponding latent traits. The basic idea of our approach is to use the prenet (product-based elastic net) penalty, as proposed in factor analysis. For optimization, we show that combining stochastic EM algorithms, proximal gradient methods, and coordinate descent methods efficiently yields solutions. Furthermore, our numerical experiments demonstrate its effectiveness, especially in cases where the number of test subjects is small, compared to methods using the existing L1 penalty.

Psychometric evaluation of the United States-adapted perceived perioperative competence scale-revised: A national survey

BackgroundNursing competence is integral to ensuring patient safety, especially in high-risk environments such as the operating room. Instruments which facilitate self-assessment of specialty specific nursing competence allow nurses to gain important insights into their practice to facilitate continuous growth in their professional practice. Currently, there are no psychometrically tested tools to assess perioperative competence applicable to the United States context. ObjectiveTest the psychometric properties of the Perceived Perioperative Competence Scale-Revised in the United States context. MethodsA cross-sectional survey design was used. Perioperative nurses were recruited via four professional associations and the survey was administered online. Construct validity of the six-dimensional Perceived Perioperative Competence Scale-Revised was tested using a multidimensional item response theory model known as the graded response model. Measurement invariance was assessed relative to years of perioperative experience. Internal consistency was estimated using McDonald's Omega and Cronbach's alpha reliability coefficients. ResultsResponses from a total of 1,581 participants were analyzed in the psychometric analysis. The six-dimensional graded response model of the Perceived Perioperative Competence Scale-Revised displayed satisfactory model fit for the sample (Chi-square(df) = 5,699.09(725); root mean square error of approximation = 0.066, 90% confidence interval: 0.064, 0.067; comparative fit index = 0.955; Tucker-Lewis index = 0.952; standardized root mean squared residual = 0.045). Scalar invariance was established when assessing the psychometric equivalence of the scale across years of perioperative experience (<10 years, ≥ 10 years) (Chi-square(df) = 5,785.29(1,573); root mean square error of approximation = 0.058, 90% confidence interval: 0.057, 0.060; comparative fit index = 0.959; Tucker–Lewis index = 0.959). Reliability across the six subscales ranged from alpha = 0.87 - 0.94 and Omega = 0.93 - 0.97. ConclusionsResults suggest that the Perceived Perioperative Competence Scale-Revised is suitable to use with perioperative nurses practicing in clinical settings in the United States. Measurement invariance testing indicates the scale is measuring the same construct and is being interpreted in a conceptually similar manner across groups based on years of perioperative experience.

The validity and IRT psychometric analysis of Chinese version of Difficult Doctor-Patient Relationship Questionnaire (DDPRQ-10)

ObjectiveThe doctor-patient relationship (DPR) plays a crucial role in the Chinese healthcare system, functioning to improve medical quality and reduce medical costs. This study examined the psychometric properties of the Chinese version of the Difficult Doctor-Patient Relationship Questionnaire (DDPRQ-10) among general hospital inpatients in China.MethodsThe research recruited 38 resident doctors responsible for 120 participants, and factor analyses were used to assess the construct validity of the scale. Convergent validity was evaluated by examining the correlation between DDPRQ-10 and depressive symptoms, burnout, and self-efficacy, using the Patient Health Questionnaire Depression Scale-9 item (PHQ-9), and the Maslach Burnout Inventory (MBI). Both multidimensional item response theory (MIRT) and unidimensional item response theory (IRT) frameworks were used to estimate the parameters of each item.ResultsThe Chinese version of DDPRQ-10 showed satisfactory internal consistency (Cronbach's alpha = 0.931), and fitted in a modified two-factor model of positive feelings and negative feelings (χ2/df = 1.494, GFI = 0.925, RMSEA = 0.071, SRMR = 0.008, CFI = 0.985, NFI = 0.958, NNFI = 0.980, TLI = 0.980, IFI = 0.986). Significant correlations with PHQ-9 with DDPRQ-10 and both subscales were revealed (r = 0.293 ~ 0.333, p < .001), while DDPRQ-10 score also significantly correlated with doctors’ MBI score (r = -0.467, p < .001). The MIRT model of full scale and IRT models of both subscales showed high discrimination of all items (a = 2.30 ~ 10.18), and the test information within the range of low-quality relationship was relatively high.ConclusionThe Chinese version of DDPRQ-10 displayed satisfactory reliability and validity and thus was appropriate for measuring the DPR in Chinese medical settings.

A generalized expectation model selection algorithm for latent variable selection in multidimensional item response theory models

A generalized expectation model selection algorithm for latent variable selection in multidimensional item response theory models

Multidimensional item response theory models for testlet-based doubly bounded data.

A testlet-based visual analogue scale (VAS) is a doubly bounded scaling approach (e.g., from 0% to 100% or from 0 to 1) composed of multiple adjectives, nouns, or sentences (statements/items) within testlets for measuring individuals' attitudes, opinions, or career interests. While testlet-based VASs have many advantages over Likert scales, such as reducing response style effects, the development of proper statistical models for analyzing testlet-based VAS data lags behind. This paper proposes a novel beta copula model and a competing logit-normal model based on the item response theory framework, assessed by Bayesian parameter estimation, model comparison, and goodness-of-fit statistics. An empirical career interest dataset based on a testlet-based VAS design was analyzed using the proposed models. Simulation studies were conducted to assess the two models' parameter recovery. The results show that the beta copula model had superior fit in the empirical data analysis, and also exhibited good parameter recovery in the simulation studies, suggesting that it is a promising statistical approach to testlet-based doubly bounded responses.

A Note on Improving Variational Estimation for Multidimensional Item Response Theory.

Survey instruments and assessments are frequently used in many domains of social science. When the constructs that these assessments try to measure become multifaceted, multidimensional item response theory (MIRT) provides a unified framework and convenient statistical tool for item analysis, calibration, and scoring. However, the computational challenge of estimating MIRT models prohibits its wide use because many of the extant methods can hardly provide results in a realistic time frame when the number of dimensions, sample size, and test length are large. Instead, variational estimation methods, such as Gaussian variational expectation-maximization (GVEM) algorithm, have been recently proposed to solve the estimation challenge by providing a fast and accurate solution. However, results have shown that variational estimation methods may produce some bias on discrimination parameters during confirmatory model estimation, and this note proposes an importance-weighted version of GVEM (i.e., IW-GVEM) to correct for such bias under MIRT models. We also use the adaptive moment estimation method to update the learning rate for gradient descent automatically. Our simulations show that IW-GVEM can effectively correct bias with modest increase of computation time, compared with GVEM. The proposed method may also shed light on improving the variational estimation for other psychometrics models.

SAS PROC IRT and the R mirt Package: A Comparison of Model Parameter Estimation for Multidimensional IRT Models

This study investigates the performance of estimation methods for multidimensional IRT models with dichotomous and polytomous data in two well-known IRT programs: SAS PROC IRT and the mirt package in R. A simulation study was used to compare performance on a simple structure Rasch model, complex structure 2PL model, and bifactor graded response model. Under RMSE and bias criteria regarding item parameter recovery, PROC IRT and mirt showed nearly identical performance in the simple structure condition. When a complex structure was used, mirt performed better in terms of the recovery of intercept parameters, while the recovery of slope parameters depended on the program and the sample sizes: PROC IRT tended to be better with small samples (N=500) according to RMSE, and mirt was better for larger samples (N=1000 and 2500) according to RMSE and bias for the slope parameter recovery. When a bifactor structure was used, mirt was preferred in all cases; differences lessened as sample size increased.

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior.

Rapid guessing (RG) is a form of non-effortful responding that is characterized by short response latencies. This construct-irrelevant behavior has been shown in previous research to bias inferences concerning measurement properties and scores. To mitigate these deleterious effects, a number of response time threshold scoring procedures have been proposed, which recode RG responses (e.g., treat them as incorrect or missing, or impute probable values) and then estimate parameters for the recoded dataset using a unidimensional or multidimensional IRT model. To date, there have been limited attempts to compare these methods under the possibility that RG may be misclassified in practice. To address this shortcoming, the present simulation study compared item and ability parameter recovery for four scoring procedures by manipulating sample size, the linear relationship between RG propensity and ability, the percentage of RG responses, and the type and rate of RG misclassifications. Results demonstrated two general trends. First, across all conditions, treating RG responses as incorrect produced the largest degree of combined systematic and random error (larger than ignoring RG). Second, the remaining scoring approaches generally provided equal accuracy in parameter recovery when RG was perfectly identified; however, the multidimensional IRT approach was susceptible to increased error as misclassification rates grew. Overall, the findings suggest that recoding RG as missing and employing a unidimensional IRT model is a promising approach.

Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation.

Multidimensional item response theory (MIRT) is a statistical test theory that precisely estimates multiple latent skills of learners from the responses in a test. Both compensatory and non-compensatory models have been proposed for MIRT: the former assumes that each skill can complement other skills, whereas the latter assumes they cannot. This non-compensatory assumption is convincing in many tests that measure multiple skills; therefore, applying non-compensatory models to such data is crucial for achieving unbiased and accurate estimation. In contrast to tests, latent skills will change over time in daily learning. To monitor the growth of skills, dynamical extensions of MIRT models have been investigated. However, most of them assumed compensatory models, and a model that can reproduce continuous latent states of skills under the non-compensatory assumption has not been proposed thus far. To enable accurate skill tracing under the non-compensatory assumption, we propose a dynamical extension of non-compensatory MIRT models by combining a linear dynamical system and a non-compensatory model. This results in a complicated posterior of skills, which we approximate with a Gaussian distribution by minimizing the Kullback-Leibler divergence between the approximated posterior and the true posterior. The learning algorithm for the model parameters is derived through Monte Carlo expectation maximization. Simulation studies verify that the proposed method is able to reproduce latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors. Furthermore, experiments on an actual data set demonstrate that our dynamical non-compensatory model can infer practical skill tracing and clarify differences in skill tracing between non-compensatory and compensatory models.

A Monte Carlo study of IRTree models' ability to recover item parameters.

Item response tree (IRTree) models are theorized to extract response styles from self-report data by utilizing multidimensional item response theory (IRT) models based on theoretical decision processes. Despite the growing popularity of the IRTree framework, there has been little research that has systematically examined the ability of its most popular models to recover item parameters across sample size and test length. This Monte Carlo simulation study explored the ability of IRTree models to recover item parameters based on data created from the midpoint primary process model. Results indicate the IRTree model can adequately recover item parameters early in the decision process model, specifically the midpoint node. However, as the model progresses through the decision hierarchy, item parameters have increased associated error variance. The authors ultimately recommend caution when employing the IRTree framework.