Multidimensional Item Response Theory Models Research Articles

Person explanatory multidimensional item response theory with the instrument package in R.

We present the new R package instrument to perform Bayesian estimation of person explanatory multidimensional item response theory. The package implements an exploratory multidimensional item response theory model and a higher-order multidimensional item response theory model, a type of confirmatory multidimensional item response theory. Explanation of person parameters is accomplished by fixed and random effect linear regression models. Estimation is carried out using Hamiltonian Monte Carlo in Stan. In this article, we provide a detailed description of the models; we use the instrument package to demonstrate fitting explanatory item response models with fixed and random effects (i.e., mixed modeling) of person parameters in R; and, we perform a simulation study to evaluate the performance of our implementation of the models.

Bayesian Adaptive Lasso for Detecting Item-Trait Relationship and Differential Item Functioning in Multidimensional Item Response Theory Models.

In multidimensional tests, the identification of latent traits measured by each item is crucial. In addition to item-trait relationship, differential item functioning (DIF) is routinely evaluated to ensure valid comparison among different groups. The two problems are investigated separately in the literature. This paper uses a unified framework for detecting item-trait relationship and DIF in multidimensional item response theory (MIRT) models. By incorporating DIF effects in MIRT models, these problems can be considered as variable selection for latent/observed variables and their interactions. A Bayesian adaptive Lasso procedure is developed for variable selection, in which item-trait relationship and DIF effects can be obtained simultaneously. Simulation studies show the performance of our method for parameter estimation, the recovery of item-trait relationship and the detection of DIF effects. An application is presented using data from the Eysenck Personality Questionnaire.

Multidimensionality of children’s foundational skills: an IRT-based validation of the NEPSY factorial structure

ABSTRACT Objective Measurement tools for children’s foundational skills provide important insights towards the developmental mechanisms of these domains. This study examines the structure of sensorimotor, visuospatial processing (VP), executive function (EF) skills and their interrelationship. The NEPSY is a widely-used neuropsychological instrument, but while validated at the domain-level through classical test theory approaches, no studies have conducted in-depth item-level analyses on its structural and psychometric properties. Method This study examined whether the designed structure of NEPSY is empirically supported by fitting unidimensional and multidimensional Item Response Theory models to item response data of early elementary students in the United States. A confirmatory factor analysis was also conducted to examine the factorial structure of the data. Results Results provide empirical support to the factorial structure of NEPSY, both at the domain level and within the VP sub-domain. The results show that VP and EF are separate constructs and subtests within the VP domain can further be modelled as interrelated but distinct factors – non-motor and motor-based processing. Conclusion Our specific results support the theoretical stance that VP consists of multiple component skills that are distinct because they require different neurological processes. This multidimensional NEPSY model shares some structural commonalities with the Cattell–Horn–Carroll model.

The General Psychopathology 'p' Factor in Adolescence: Multi-Informant Assessment and Computerized Adaptive Testing.

Accumulating evidence supports the presence of a general psychopathology dimension, the p factor ('p'). Despite growing interest in the p factor, questions remain about how p is assessed. Although multi-informant assessment of psychopathology is commonplace in clinical research and practice with children and adolescents, almost no research has taken a multi-informant approach to studying youth p or has examined the degree of concordance between parent and youth reports. Further, estimating p requires assessment of a large number of symptoms, resulting in high reporter burden that may not be feasible in many clinical and research settings. In the present study, we used bifactor multidimensional item response theory models to estimate parent- and adolescent-reported p in a large community sample of youth (11-17 years) and parents (N = 5,060 dyads). We examined agreement between parent and youth p scores and associations with assessor-rated youth global functioning. We also applied computerized adaptive testing (CAT) simulations to parent and youth reports to determine whether adaptive testing substantially alters agreement on p or associations with youth global functioning. Parent-youth agreement on p was moderate (r =.44) and both reports were negatively associated with youth global functioning. Notably, 7 out of 10 of the highest loading items were common across reporters. CAT reduced the average number of items administered by 57%. Agreement between CAT-derived p scores was similar to the full form (r =.40) and CAT scores were negatively correlated with youth functioning. These novel results highlight the promise and potential clinical utility of a multi-informant p factor approach.

A Comparison of Mixed and Partial Membership Diagnostic Classification Models with Multidimensional Item Response Models

Diagnostic classification models (DCM) are latent structure models with discrete multivariate latent variables. Recently, extensions of DCMs to mixed membership have been proposed. In this article, ordinary DCMs, mixed and partial membership models, and multidimensional item response theory (IRT) models are compared through analytical derivations, three example datasets, and a simulation study. It is concluded that partial membership DCMs are similar, if not structurally equivalent, to sufficiently complex multidimensional IRT models.

Co-occurring dominance and ideal point processes: A general IRTree framework for multidimensional item responding

Responding to rating scale items is a multidimensional process, since not only the substantive trait being measured but also additional personal characteristics can affect the respondents’ category choices. A flexible model class for analyzing such multidimensional responses are IRTree models, in which rating responses are decomposed into a sequence of sub-decisions. Different response processes can be involved in item responding both sequentially across those sub-decisions and as co-occurring processes within sub-decisions. In the previous literature, modeling co-occurring processes has been exclusively limited to dominance models, where higher trait levels are associated with higher expected scores. However, some response processes may rather follow an ideal point rationale, where the expected score depends on the proximity of a person’s trait level to the item’s location. Therefore, we propose a new multidimensional IRT model of co-occurring dominance and ideal point processes (DI-MIRT model) as a flexible framework for parameterizing IRTree sub-decisions with multiple dominance processes, multiple ideal point processes, and combinations of both. The DI-MIRT parameterization opens up new application areas for the IRTree model class and allows the specification of a wide range of theoretical assumptions regarding the cognitive processing of item responding. A simulation study shows that IRTree models with DI-MIRT parameterization provide excellent parameter recovery and accurately reflect co-occurring dominance and ideal point processes. In addition, a clear advantage over traditional IRTree models with purely sequential processes is demonstrated. Two application examples from the field of response style analysis highlight the benefits of the general IRTree framework under real-world conditions.

Smooth Information Criterion for Regularized Estimation of Item Response Models

Item response theory (IRT) models are frequently used to analyze multivariate categorical data from questionnaires or cognitive test data. In order to reduce the model complexity in item response models, regularized estimation is now widely applied, adding a nondifferentiable penalty function like the LASSO or the SCAD penalty to the log-likelihood function in the optimization function. In most applications, regularized estimation repeatedly estimates the IRT model on a grid of regularization parameters λ. The final model is selected for the parameter that minimizes the Akaike or Bayesian information criterion (AIC or BIC). In recent work, it has been proposed to directly minimize a smooth approximation of the AIC or the BIC for regularized estimation. This approach circumvents the repeated estimation of the IRT model. To this end, the computation time is substantially reduced. The adequacy of the new approach is demonstrated by three simulation studies focusing on regularized estimation for IRT models with differential item functioning, multidimensional IRT models with cross-loadings, and the mixed Rasch/two-parameter logistic IRT model. It was found from the simulation studies that the computationally less demanding direct optimization based on the smooth variants of AIC and BIC had comparable or improved performance compared to the ordinarily employed repeated regularized estimation based on AIC or BIC.

A Method of Empirical Q-Matrix Validation for Multidimensional Item Response Theory

ABSTRACT A Q-matrix is a binary matrix that defines the relationship between items and latent variables and is widely used in diagnostic classification models (DCMs), and can also be adopted in multidimensional item response theory (MIRT) models. The construction process of the Q-matrix is typically carried out by experts in the subject area of the items and statistical procedures can be used to verify its suitability. In DCMs, different approaches have been proposed for validating the Q-matrix through iterative algorithms. This article presents a method of empirical Q-matrix validation for MIRT models. A simulation study and an application to real data of morphology skills of elementary students are conducted to examine the viability of the method. Relevant issues regarding the implementation of the method and the results obtained are discussed.

HBMIRT: A SAS macro for estimating uni- and multidimensional 1- and 2-parameter item response models in small (and large!) samples

Item response theory (IRT) has evolved as a standard psychometric approach in recent years, in particular for test construction based on dichotomous (i.e., true/false) items. Unfortunately, large samples are typically needed for item refinement in unidimensional models and even more so in the multidimensional case. However, Bayesian IRT approaches with hierarchical priors have recently been shown to be promising for estimating even complex models in small samples. Still, it may be challenging for applied researchers to set up such IRT models in general purpose or specialized statistical computer programs. Therefore, we developed a user-friendly tool – a SAS macro called HBMIRT – that allows to estimate uni- and multidimensional IRT models with dichotomous items. We explain the capabilities and features of the macro and demonstrate the particular advantages of the implemented hierarchical priors in rather small samples over weakly informative priors and traditional maximum likelihood estimation with the help of a simulation study. The macro can also be used with the online version of SAS OnDemand for Academics that is freely accessible for academic researchers.

How Scoring Approaches Impact Estimates of Growth in the Presence of Survey Item Ceiling Effects.

Survey scores are often the basis for understanding how individuals grow psychologically and socio-emotionally. A known problem with many surveys is that the items are all "easy"-that is, individuals tend to use only the top one or two response categories on the Likert scale. Such an issue could be especially problematic, and lead to ceiling effects, when the same survey is administered repeatedly over time. In this study, we conduct simulation and empirical studies to (a) quantify the impact of these ceiling effects on growth estimates when using typical scoring approaches like sum scores and unidimensional item response theory (IRT) models and (b) examine whether approaches to survey design and scoring, including employing various longitudinal multidimensional IRT (MIRT) models, can mitigate any bias in growth estimates. We show that bias is substantial when using typical scoring approaches and that, while lengthening the survey helps somewhat, using a longitudinal MIRT model with plausible values scoring all but alleviates the issue. Results have implications for scoring surveys in growth studies going forward, as well as understanding how Likert item ceiling effects may be contributing to replication failures.

Variational Estimation for Multidimensional Generalized Partial Credit Model.

Multidimensional item response theory (MIRT) models have generated increasing interest in the psychometrics literature. Efficient approaches for estimating MIRT models with dichotomous responses have been developed, but constructing an equally efficient and robust algorithm for polytomous models has received limited attention. To address this gap, this paper presents a novel Gaussian variational estimation algorithm for the multidimensional generalized partial credit model. The proposed algorithm demonstrates both fast and accurate performance, as illustrated through a series of simulation studies and two real data analyses.

Examining Differential Item Functioning from a Multidimensional IRT Perspective.

Differential item functioning (DIF) is a standard analysis for every testing company. Research has demonstrated that DIF can result when test items measure different ability composites, and the groups being examined for DIF exhibit distinct underlying ability distributions on those composite abilities. In this article, we examine DIF from a two-dimensional multidimensional item response theory (MIRT) perspective. We begin by delving into the compensatory MIRT model, illustrating and how items and the composites they measure can be graphically represented. Additionally, we discuss how estimated item parameters can vary based on the underlying latent ability distributions of the examinees. Analytical research highlighting the consequences of ignoring dimensionally and applying unidimensional IRT models, where the two-dimensional latent space is mapped onto a unidimensional, is reviewed. Next, we investigate three different approaches to understanding DIF from a MIRT standpoint: 1. Analytically Uniform and Nonuniform DIF: When two groups of interest have different two-dimensional ability distributions, a unidimensional model is estimated. 2. Accounting for complete latent ability space: We emphasize the importance of considering the entire latent ability space when using DIF conditional approaches, which leads to the mitigation of DIF effects. 3. Scenario-Based DIF: Even when underlying two-dimensional distributions are identical for two groups, differing problem-solving approaches can still lead to DIF. Modern software programs facilitate routine DIF procedures for comparing response data from two identified groups of interest. The real challenge is to identify why DIF could occur with flagged items. Thus, as a closing challenge, we present four items (Appendix A) from a standardized test and invite readers to identify which group was favored by a DIF analysis.

Variation of subclinical psychosis across 16 sites in Europe and Brazil: findings from the multi-national EU-GEI study.

Incidence of first-episode psychosis (FEP) varies substantially across geographic regions. Phenotypes of subclinical psychosis (SP), such as psychotic-like experiences (PLEs) and schizotypy, present several similarities with psychosis. We aimed to examine whether SP measures varied across different sites and whether this variation was comparable with FEP incidence within the same areas. We further examined contribution of environmental and genetic factors to SP. We used data from 1497 controls recruited in 16 different sites across 6 countries. Factor scores for several psychopathological dimensions of schizotypy and PLEs were obtained using multidimensional item response theory models. Variation of these scores was assessed using multi-level regression analysis to estimate individual and between-sites variance adjusting for age, sex, education, migrant, employment and relational status, childhood adversity, and cannabis use. In the final model we added local FEP incidence as a second-level variable. Association with genetic liability was examined separately. Schizotypy showed a large between-sites variation with up to 15% of variance attributable to site-level characteristics. Adding local FEP incidence to the model considerably reduced the between-sites unexplained schizotypy variance. PLEs did not show as much variation. Overall, SP was associated with younger age, migrant, unmarried, unemployed and less educated individuals, cannabis use, and childhood adversity. Both phenotypes were associated with genetic liability to schizophrenia. Schizotypy showed substantial between-sites variation, being more represented in areas where FEP incidence is higher. This supports the hypothesis that shared contextual factors shape the between-sites variation of psychosis across the spectrum.

Development and initial validation of a creative self-efficacy scale for undergraduates: categorical confirmatory factor analysis and multidimensional item response theory

Creative self-efficacy (CSE) has recently received much attention due to its association with student learning and creativity. To that end, a CSE scale was developed for undergraduates and sources of validity evidence based on scale content, response processes, and internal structure were collected. Score reliability, using categorical omega based on the categorical confirmatory factor analysis model and marginal reliability for response pattern scores based on item response theory (IRT), were estimated. After various revision iterations of the initial 28-item pool by 10 subject matter experts and 18 undergraduates, some items were revised, four items were dropped, and ultimately 24 items were field tested for measuring two hypothesized dimensions of CSE among 602 undergraduates. Categorical confirmatory factor analysis results indicated that the two-dimensional model had better fit. Similarly, between the two competing multidimensional IRT models, the two-dimensional graded response model had the best fit. Categorical omega coefficients and marginal reliability for response pattern scores were, respectively, 0.88 and 0.81 for the two underlying dimensions.

How Modern Psychometrics Can Change Intelligence Testing of Children and Adolescents in Russia

The assessment of intellectual development in children and adolescents is an essential component of psychodiagnostics, facilitating an understanding of the nature of learning difficulties. Intelligence tests have a history that spans more than a century, and even contemporary versions of widely used intelligence tests (such as the Wechsler or Stanford-Binet tests) are limited by the influence of their traditional paper-based formats. Nevertheless, the advent of computational capabilities and the digitalization of tests have led psychometrics to experience substantial development, opening new possibilities for assessment. The complete application of modern psychometrics holds the promise of revolutionizing the measurement of intelligence. This paper presents the current situation of measuring intelligence in children and adolescents in the world and in Russia. It describes the potentialities presented by modern psychometrics, including the utilization of innovative task formats such as technology-enhanced items, the integration of universal design principles in test development, the adoption of computerized adaptive testing (CAT) and multistage testing (MST) methodologies, the application of multidimensional IRT models to establish both construct and criterion validity, the shift from discrete norms to continuous scaling, and the incorporation of longitudinal measurements. The combination of these features within a computerized intelligence testing framework will enhance respondents' motivation and simplify the test administration process. It will facilitate frequent assessments of a child's progress by presenting varying test versions on each occasion, thereby significantly reducing errors in administration and scoring, while also safeguarding against misuse by non-professionals.

Estimation of a Simple Structure in a Multidimensional IRT Model Using Structure Regularization.

We develop a method for estimating a simple matrix for a multidimensional item response theory model. Our proposed method allows each test item to correspond to a single latent trait, making the results easier to interpret. It also enables clustering of test items based on their corresponding latent traits. The basic idea of our approach is to use the prenet (product-based elastic net) penalty, as proposed in factor analysis. For optimization, we show that combining stochastic EM algorithms, proximal gradient methods, and coordinate descent methods efficiently yields solutions. Furthermore, our numerical experiments demonstrate its effectiveness, especially in cases where the number of test subjects is small, compared to methods using the existing L1 penalty.

Effects of the Quantity and Magnitude of Cross-Loading and Model Specification on MIRT Item Parameter Recovery.

In real-world situations, multidimensional data may appear on large-scale tests or psychological surveys. The purpose of this study was to investigate the effects of the quantity and magnitude of cross-loadings and model specification on item parameter recovery in multidimensional Item Response Theory (MIRT) models, especially when the model was misspecified as a simple structure, ignoring the quantity and magnitude of cross-loading. A simulation study that replicated this scenario was designed to manipulate the variables that could potentially influence the precision of item parameter estimation in the MIRT models. Item parameters were estimated using marginal maximum likelihood, utilizing the expectation-maximization algorithms. A compensatory two-parameter logistic-MIRT model with two dimensions and dichotomous item-responses was used to simulate and calibrate the data for each combination of conditions across 500 replications. The results of this study indicated that ignoring the quantity and magnitude of cross-loading and model specification resulted in inaccurate and biased item discrimination parameter estimates. As the quantity and magnitude of cross-loading increased, the root mean square of error and bias estimates of item discrimination worsened.

Psychometric evaluation of the United States-adapted perceived perioperative competence scale-revised: A national survey

BackgroundNursing competence is integral to ensuring patient safety, especially in high-risk environments such as the operating room. Instruments which facilitate self-assessment of specialty specific nursing competence allow nurses to gain important insights into their practice to facilitate continuous growth in their professional practice. Currently, there are no psychometrically tested tools to assess perioperative competence applicable to the United States context. ObjectiveTest the psychometric properties of the Perceived Perioperative Competence Scale-Revised in the United States context. MethodsA cross-sectional survey design was used. Perioperative nurses were recruited via four professional associations and the survey was administered online. Construct validity of the six-dimensional Perceived Perioperative Competence Scale-Revised was tested using a multidimensional item response theory model known as the graded response model. Measurement invariance was assessed relative to years of perioperative experience. Internal consistency was estimated using McDonald's Omega and Cronbach's alpha reliability coefficients. ResultsResponses from a total of 1,581 participants were analyzed in the psychometric analysis. The six-dimensional graded response model of the Perceived Perioperative Competence Scale-Revised displayed satisfactory model fit for the sample (Chi-square(df) = 5,699.09(725); root mean square error of approximation = 0.066, 90% confidence interval: 0.064, 0.067; comparative fit index = 0.955; Tucker-Lewis index = 0.952; standardized root mean squared residual = 0.045). Scalar invariance was established when assessing the psychometric equivalence of the scale across years of perioperative experience (<10 years, ≥ 10 years) (Chi-square(df) = 5,785.29(1,573); root mean square error of approximation = 0.058, 90% confidence interval: 0.057, 0.060; comparative fit index = 0.959; Tucker–Lewis index = 0.959). Reliability across the six subscales ranged from alpha = 0.87 - 0.94 and Omega = 0.93 - 0.97. ConclusionsResults suggest that the Perceived Perioperative Competence Scale-Revised is suitable to use with perioperative nurses practicing in clinical settings in the United States. Measurement invariance testing indicates the scale is measuring the same construct and is being interpreted in a conceptually similar manner across groups based on years of perioperative experience.

The validity and IRT psychometric analysis of Chinese version of Difficult Doctor-Patient Relationship Questionnaire (DDPRQ-10)

ObjectiveThe doctor-patient relationship (DPR) plays a crucial role in the Chinese healthcare system, functioning to improve medical quality and reduce medical costs. This study examined the psychometric properties of the Chinese version of the Difficult Doctor-Patient Relationship Questionnaire (DDPRQ-10) among general hospital inpatients in China.MethodsThe research recruited 38 resident doctors responsible for 120 participants, and factor analyses were used to assess the construct validity of the scale. Convergent validity was evaluated by examining the correlation between DDPRQ-10 and depressive symptoms, burnout, and self-efficacy, using the Patient Health Questionnaire Depression Scale-9 item (PHQ-9), and the Maslach Burnout Inventory (MBI). Both multidimensional item response theory (MIRT) and unidimensional item response theory (IRT) frameworks were used to estimate the parameters of each item.ResultsThe Chinese version of DDPRQ-10 showed satisfactory internal consistency (Cronbach's alpha = 0.931), and fitted in a modified two-factor model of positive feelings and negative feelings (χ2/df = 1.494, GFI = 0.925, RMSEA = 0.071, SRMR = 0.008, CFI = 0.985, NFI = 0.958, NNFI = 0.980, TLI = 0.980, IFI = 0.986). Significant correlations with PHQ-9 with DDPRQ-10 and both subscales were revealed (r = 0.293 ~ 0.333, p < .001), while DDPRQ-10 score also significantly correlated with doctors’ MBI score (r = -0.467, p < .001). The MIRT model of full scale and IRT models of both subscales showed high discrimination of all items (a = 2.30 ~ 10.18), and the test information within the range of low-quality relationship was relatively high.ConclusionThe Chinese version of DDPRQ-10 displayed satisfactory reliability and validity and thus was appropriate for measuring the DPR in Chinese medical settings.

A generalized expectation model selection algorithm for latent variable selection in multidimensional item response theory models

A generalized expectation model selection algorithm for latent variable selection in multidimensional item response theory models