Multidimensional Item Response Models Research Articles

A Comparison of Mixed and Partial Membership Diagnostic Classification Models with Multidimensional Item Response Models

Diagnostic classification models (DCM) are latent structure models with discrete multivariate latent variables. Recently, extensions of DCMs to mixed membership have been proposed. In this article, ordinary DCMs, mixed and partial membership models, and multidimensional item response theory (IRT) models are compared through analytical derivations, three example datasets, and a simulation study. It is concluded that partial membership DCMs are similar, if not structurally equivalent, to sufficiently complex multidimensional IRT models.

Metacognitive Skills Assessment in Research-Proposal Writing (MSARPW) in the Indonesian University Context: Scale Development and Validation Using Multidimensional Item Response Models

This study aimed to develop the Metacognitive Skills Assessment in a Research Proposal Writing Context (MSARPW), a multidimensional measure, based on student perspectives, for assessing metacognitive skills in thesis writing, and assess the instrument’s psychometric properties using item factor analysis (IFA) and multidimensional item response models. The 40-item MSARPW was administered to 602 Indonesian university students (Mage = 25.254 SDage = 6.854). The IFA showed that the two-dimensional factor structure of MSARPW was satisfactory; however, only 24 of the 40 items were found to fit the model. Multidimensional graded response models (MGRM) were applied to the subsequent 24-item MSARPW, which showed that one item (Item 4) did not satisfy the criteria. The estimated reliabilities of each subscale showed that the 23-item MSARPW has good internal consistency (0.891 and 0.902). To conclude, the 23-item MSARPW appears to be a valid and reliable tool for assessing metacognitive skills in the context of research-proposal writing among Indonesian university students.

A New Approach to Desirable Responding: Multidimensional Item Response Model of Overclaiming Data.

A variety of approaches have been presented for assessing desirable responding in self-report measures. Among them, the overclaiming technique asks respondents to rate their familiarity with a large set of real and nonexistent items (foils). The application of signal detection formulas to the endorsement rates of real items and foils yields indices of (a) knowledge accuracy and (b) knowledge bias. This overclaiming technique reflects both cognitive ability and personality. Here, we develop an alternative measurement model based on multidimensional item response theory (MIRT). We report three studies demonstrating this new model's capacity to analyze overclaiming data. First, a simulation study illustrates that MIRT and signal detection theory yield comparable indices of accuracy and bias-although MIRT provides important additional information. Two empirical examples-one based on mathematical terms and one based on Chinese idioms-are then elaborated. Together, they demonstrate the utility of this new approach for group comparisons and item selection. The implications of this research are illustrated and discussed.

Improving test efficiency for a grid multidimensional computerized classification test by the application of a conditional latent-trait distribution to a sequential probability ratio test.

The measurement efficiency of a multidimensional computerized adaptive testing (MCAT) can be improved by taking the correlations between the dimensions into account during the item selection and latent-trait estimation procedures (Segall, 1996; Wang & Chen, 2004). Although a multidimensional computerized classification test (MCCT), which was based on a multidimensional itemresponse model, was previously found more efficient than its unidimensional counterpart, the difference was negligible (Seitz & Frey, 2013); the researchers had adopted a sequential probability ratio test (SPRT) as the termination criterion in this MCCT study. To make a classification decision on each dimension, which is called a grid classification (Wang et al., 2019), only items that loaded on that dimension were used to calculate the likelihood ratio, which squandered the available information of the correlations between the dimensions. The current study utilizes such useful information to improve the measurement efficiency of the MCCT by applying a conditional distribution of the latent-trait estimates and then including all the administered items to calculate the likelihood ratio in the SPRT. The performance of this newly proposed method was evaluated through a series of simulation studies. The results showed that the proposed method can sizably improve the measurement efficiency of an MCCT by saving 1% to 32% of the test length in comparison with the SPRT when the two test dimensions are at least moderately correlated. The findings and further applications of this study are discussed.

Between-Item Multidimensional IRT: How Far Can the Estimation Methods Go?

Multidimensional item response models are known to be difficult to estimate, with a variety of estimation and modeling strategies being proposed to handle the difficulties. While some previous studies have considered the performance of these estimation methods, they typically include only one or two methods, or a small number of factors. In this paper, we report on a large simulation study of between-item multidimensional IRT estimation methods, considering five different methods, a variety of sample sizes, and up to eight factors. This study provides a comprehensive picture of the methods’ relative performance, as well as each individual method’s strengths and weaknesses. The study results lead us to make recommendations for applied research, related to which estimation methods should be used under various scenarios.

Grit: The long and short of it.

Grit is defined as passion and perseverance for achieving long-term goals and consists of two proposed subcomponents: consistency of interests and perseverance of effort. It has become a much-discussed construct even though research on its underlying factor structure has produced inconclusive results. Furthermore, grit as measured by its most frequently used measure, the Grit-S, does not clearly define “long term” or include the word passion. In addition, only a few studies have looked at factor structure variation and predictive validity of grit in different age and cultural groups. We examined, using state of the art multidimensional item response models and structural equation models, the factor structure of both the Grit-S and a newly developed scale (LT-Grit scale) that specifies what is meant by long term. Participants included 1,250 U.S. high school students, 600 U.S. college students, and 500 Korean college students. We found varying factor structures for the Grit-S and a stable one-factor structure for LT-Grit, across age and culture. Perseverance of effort as measured by the Grit-S was the strongest predictor of grades in the three samples. LT-Grit predicted grades only in the U.S. high school and Korean college samples. Thus, there is no evidence that a consistent grit factor or factors exist across age and culture. (PsycInfo Database Record (c) 2021 APA, all rights reserved)

A study for application of item change sensitivity to longitudinal academic achievement tests

학생 능력의 변화를 측정하는 검사를 개발할 때, 문항이 변화/성장을 얼마나 잘 감지하는지 확인하는 것은 검사 목적의 타당성을 높이기 위해 중요하다. 본 연구는 종단 데이터에 적용된 다차원 문항반응모형의 모수가 변화에 민감한 정도를 나타낼 수 있음을 소개하고, 모수 추정의 정확성을 살펴보았다. 구체적으로, 1년의 시간 간격을 두고 시행된 능력검사 데이터를 이용하여 변별도와 변화에 민감한 정도를 나타내는 모수가 별개의 특성임을 확인하였다. 또한 세 가지 조건(요인 간 상관, 기울기 모수 간 차이, 피험자 수)을 고려한 모의 데이터를 생성하여, 두 문항 특성이 안정적으로 분리되어 추정되는지 확인하였다. 연구 결과, 문항 모수는 대체로 안정적으로 추정되었으나, 요인 간 상관이 부적일 때와 사례수가 적을 때는 추정성이 떨어졌다. 두 문항 모수 간의 차이가 클수록 문항 모수의 추정성은 높아지나, 잠재특성 모수의 추정성은 낮아지는 경향이 있었다.When developing test items that measure changes in student abilities over time, it is important to increase the validity of the test objectives to determine how well the items detect changes/growth. In this study, three conditions of the multidimensional item response model (correlation between factors, difference between slope parameters, and the number of subjects) are considered for simulation study by referring to the characteristics of the actual longitudinal ability test data used to grasp how much students have grown at a time interval of one year. First, we introduced that the item parameter of the multi-dimensional item response model can provide information about the degree of sensitivity to ability change, and examined its function through the accuracy of parameter estimation. As a result of the study, the latent trait parameter and the item parameter were generally estimated to be stable, but the accuracy was poor when the correlation between the two latent factors was negatively related and the number of cases was small. The greater the difference between the two slope parameters, the higher the accuracy of the item parameter, but the lower the accuracy of the latent trait parameter.

Latent Growth Modeling with Categorical Response Data: A Methodological Investigation of Model Parameterization, Estimation, and Missing Data

ABSTRACT Measuring change in an educational or psychological construct over time is often achieved by repeatedly administering the same items to the same examinees over time and fitting a second-order latent growth curve model. However, latent growth modeling with full information maximum likelihood (FIML) estimation becomes computationally challenging when the observed response data are categorical. This study first discusses some possible options that researchers can take regarding model specification and estimation (e.g., limited-information and various FIML estimators) to circumvent the challenge. To explore the utility of a stochastic Newton-Raphson type of algorithm (i.e., Metropolis Hastings-Robbins Monro; MH-RM) implemented primarily for multidimensional item response model, a re-parameterized latent growth model is also introduced. The viability of each option is examined via Monte-Carlo simulations. Insights on the pros and cons of these options and the conditions under which they are applicable are provided for researchers.

Scale Alignment in the Between-Item Multidimensional Partial Credit Model.

In between-item multidimensional item response models, it is often desirable to compare individual latent trait estimates across dimensions. These comparisons are only justified if the model dimensions are scaled relative to each other. Traditionally, this scaling is done using approaches such as standardization-fixing the latent mean and standard deviation to 0 and 1 for all dimensions. However, approaches such as standardization do not guarantee that Rasch model properties hold across dimensions. Specifically, for between-item multidimensional Rasch family models, the unique ordering of items holds within dimensions, but not across dimensions. Previously, Feuerstahler and Wilson described the concept of scale alignment, which aims to enforce the unique ordering of items across dimensions by linearly transforming item parameters within dimensions. In this article, we extend the concept of scale alignment to the between-item multidimensional partial credit model and to models fit using incomplete data. We illustrate this method in the context of the Kindergarten Individual Development Survey (KIDS), a multidimensional survey of kindergarten readiness used in the state of Illinois. We also present simulation results that demonstrate the effectiveness of scale alignment in the context of polytomous item response models and missing data.

A note on a computationally efficient implementation of the EM algorithm in item response models

This note sketches two computational shortcuts for estimating unidimensional item response models and multidimensional item response models with between-item dimensionality utilizing an expectation-maximization (EM) algorithm that relies on numerical integration with fixed quadrature points. It is shown that the number of operations required in the E-step can be reduced in situations of many cases and many items by appropriate shortcuts. Consequently, software implementations of a modified E-step in the EM algorithm could benefit from gains in computation time.

Bayesian multilevel multidimensional item response modeling approach for multiple latent variables in a hierarchical structure

In this paper, we propose a multilevel multidimensional item response model for studying the relations among multiple abilities and covariates in a hierarchical data structure. As an example, this study is well suited to examining the scenario in which a test measures multidimensional latent traits (e.g., reading ability, cognitive ability, and computing ability) and in which students are nested within classes or schools. The new model can recover the correlations among multidimensional abilities, along with the correlation between person- and school-level covariates and abilities. A fully Gibbs sampling algorithm within the Markov chain Monte Carlo (MCMC) framework is proposed for parameter estimation. A unique form of the deviance information criterion (DIC) is used as a model comparison index. Two simulation studies show that the estimation method is suitable in recovering all model parameters.

Bayesian Hierarchical Multidimensional Item Response Modeling of Small Sample, Sparse Data for Personalized Developmental Surveillance.

Developmental surveillance tools are used to closely monitor the early development of infants and young children. This study provides a novel implementation of a multidimensional item response model, using Bayesian hierarchical priors, to construct developmental profiles for a small sample of children (N = 115) with sparse data collected through an online developmental surveillance tool. The surveillance tool records 348 developmental milestones measured from birth to three years of age, within six functional domains: auditory, hands, movement, speech, tactile, and vision. The profiles were constructed in three steps: (1) the multidimensional item response model, embedded in the Bayesian hierarchical framework, was implemented in order to measure both the latent abilities of the children and attributes of the milestones, while retaining the correlation structure among the latent developmental domains; (2) subsequent hierarchical clustering of the multidimensional ability estimates enabled identification of subgroups of children; and (3) information from the posterior distributions of the item response model parameters and the results of the clustering were used to construct a personalized profile of development for each child. These individual profiles support early identification of, and personalized early interventions for, children with developmental delay.

Reconsidering the construct validity of self-reported chronic stress: A multidimensional item response theory approach.

Emerging evidence shows that the construct validity of commonly used chronic stress measures is questionable. Item response theory was used to evaluate the psychometric properties of the Perceived Stress Questionnaire (PSQ), the Perceived Stress Scale (PSS), and the Screening Scale of Chronic Stress (SSCS). Using multidimensional item response modeling of data from two population-based samples (N₁ = 5,379, N₂ = 504), we identified and conceptually replicated a five-dimensional model combining items of chronic stress, neuroticism (i.e., Big Five Inventory Neuroticism scale; BFI-N), and depressiveness (i.e., Patient Health Questionnaire depressiveness scale; PHQ-9). The findings suggested a major overlap of variance between items of the applied instruments forming a pronounced common latent scale, which appeared to be mainly based on the perceived extent of freely available energy (i.e., vitality vs. fatigue). Accordingly, the overlap between items of the PSQ, PSS, and SSCS was larger with PHQ-9 items than BFI-N items. Specifically, items of chronic stress seem to discriminate subclinical manifestations of fatigue, a core symptom of depressiveness. Subsequently estimated person parameters were used to predict hair cortisol concentrations (HCC), a physiological indicator of chronic stress. Controlling for potential confounders, only the common latent scale showed a weak association with HCC, whereas residual variance components were not predictive of HCC. This finding further supports the notion that instruments like PSQ, PSS, and SSCS show an impaired construct validity. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

The Impact of Item Feature and Response Preference in a Mixed-Format Design

A combination of positively and negatively worded items (termed a mixed-format design) has been widely adopted in personality and attitude assessments. While advocates claim that the inclusion of positively and negatively worded items will encourage respondents to process the items more carefully and avoid response preference, others have reported that negatively worded (NW) items may induce a nuisance factor and contaminate scale scores. The present study examined the extent of the impact of the NW-item feature and further investigated whether a mixed-format design could effectively control acquiescence and the preference for extreme response options using two datasets (Attitude toward Peace Walls, and International Personality Item Pool). A proposed multidimensional item response model was implemented to simultaneously estimate the impact of item feature and response preference. The results suggested that NW items induced an impact on item responses and that affirmative preference was negligible, regardless of the proportion of NW items in a scale. However, participants’ extremity preference was large in both balanced and imbalanced mixed-format designs. It concludes that the impact of the NW-item feature is not negligible in a mixed-format scale, which exhibits good control of acquiescence but not extremity preference.

Rotate and Project: Measurement of the Intended Concept with Unidimensional Item Response Theory from Multidimensional Ordinal Items

What to do when item response data are multidimensional but a unidimensional model is preferred in terms of statistical simplicity and ease of interpretability? The projection method for the compensatory logistic multidimensional item response model for dichotomous data leads to a two parameter logistic model with local item dependence. Despite the local item dependence, the model is unidimensional for many practical purposes. Here, Ip’s projection method is generalized to the case of the graded response model for polytomous variables, extending the applicability of the method to Likert-type response formats. A secondary aim of the paper is the study of rotation techniques intended for use prior to projection. In contrast to rotations aiming at a simple structure of factor loadings, the proposed techniques increase the variance explained before or after projection, facilitate the interpretation of the projected dimension by variants of target rotations or a mix of both. The method is illustrated with an application to the Highly Sensitive Person Scale and R code is provided.

Measuring digital literacy across three age cohorts: Exploring test dimensionality and performance differences

Digital literacy (DL) is an important capacity for students’ learning in a rapidly changing world. However, tension exists between the theoretical conceptualizations of DL as a multidimensional construct and empirical studies reporting unidimensional DL scores. Also, little is known about how DL may vary among different age cohorts, and whether and at which age do performance gaps emerge with respect to gender. The focus of this research is to develop a test appropriate for measuring DL performance at different ages and a comprehensive DL assessment framework has been adopted for this purpose. Using data from three age cohorts of students (one from primary schools and two from secondary schools), the dimensionality of DL and performance differences are examined. Comparisons of unidimensional and multidimensional item response models suggest the measured DL to be a unidimensional construct. The results also show that secondary school students obtained higher levels of DL compared to primary school students. A gender gap in DL is found among secondary school students. There is also a need for further research to understand through longitudinal studies the emergence of the gender gap in DL performance.

A Comparison of Multidimensional IRT Linking Methods

본 연구의 목적은 복수의 하위구인으로 이루어진 실제 검사 구조를 고려하여 모의실험을 통해 다차원 문항반응모형을 이용한 검사의 연계화 방법들의 정확성을 비교하는 것이다. 이를 위해 Oshima, Davey와 Lee(2000), Li와 Lissitz(2000), Min(2003)의 세 가지 분리추정과 동시추정방법을 이용하여 공통척도로의 변환을 수행하였다. 추정방법들의 비교를 위해 피험자 집단의 크기, 공통문항의 비율, 능력차원의 상관 조건 별로 결과를 도출하였다. 연구결과 전반적으로 동시추정방법이 분리추정방법들에 비해 양호한 연계화 결과를 보였다. 단, 능력 차원들의 상관이 낮으며 집단의 크기가 충분히 크고 공통문항의 비율이 높은 경우에는 분리추정을 이용한 연계화 결과의 정확성도 높게 나타났다. 이를 통해 다차원 검사들의 연계화의 일반적인 상황에서 동시추정방법의 사용을 제안하며, 분리추정방법을 이용하여 공통척도로 변환 시에는 충분한 피험자 수의 확보와 공통문항이 비율을 높일 필요가 있음을 제안한다.The purpose of this study is to compare the test linking methods using a multidimensional item response model through simulation technique by considering the actual test structure composed of several constructs. The transformation to the common scale is performed using three separate calibration methods: Oshima, Davey and Lee (2000), Li and Lissitz (2000), Min (2003) and concurrent calibration method. For comparison of calibration methods, results are derived from the number of examinee, the dimensional correlations and the ratio of common items. As a result of the study, the concurrent calibration method performs well in linking than three separate calibration methods. However, when the correlation of ability dimensions is low, the number of examinee is large enough, and the ratio of common items is high, the separate calibration methods also provides accurate linking results. In conclusions, concurrent calibration method has an advantage of multidimensional tests linking. And it is necessary to secure a sufficient number of examinee, increase the ratio of common items when transforming to common scale with separate calibration methods.

S118. TRANSDIAGNOSTIC SYMPTOM DIMENSIONS OF PSYCHOSIS AND THE PREDICTIVE ROLE OF PREMORBID ADJUSTMENT AND COGNITIVE CHARACTERISTICS IN THE MULTINATIONAL EU-GEI STUDY

BackgroundA symptom dimension approach may best examine the heterogeneous expression of psychosis. However, whether and how premorbid predisposition and cognitive factors explain phenotypes variation is still debated. This study aimed to test the predictive value of combined cognition and premorbid adjustment on transdiagnostic symptom dimensions in a large sample of people suffering from the first episode of psychosis (FEP).MethodsFEP patients were part of the EUGEI study. Psychopathology was rated using the OPerational CRITeria system. Multidimensional item response modelling estimate a bifactor model of psychosis by Mplus, composed of a general factor and five specific symptom dimensions. WAIS-brief version measured IQ, and Premorbid Adjustment Scale estimated premorbid social (SF) and academic adjustment (AF). We set a multivariate analysis of the covariance having symptom-dimensions as the output and SF, AF, and IQ as the main predictors, adjusted by age, gender, country, and ethnicity. Since lifetime frequency of cannabis use related to both premorbid and cognitive characteristics, and symptom dimensions, it was added as an independent predictor, collected by the Cannabis Experience Questionnaire. Sensitivity analyses were run with IQ subtests.ResultsThe sample included 785 FEP patients [61.2% (N=481) males; Mean age=33.8 (12.3)]. IQ [Pillai=0.019, F(6, 765)=2.52; p=0.020] and SF [Pillai=0.032, F(6, 765)=4.26; p<0.001] had a discriminant effect on the bifactor model as a whole, whereas AF did not [Pillai=0.007, F(6, 765)=0.93; p=0.468]. Cannabis use had a marginally effect [Pillai=0.027, F(12, 1532)=1.76; p=0.049].Positive symptoms were slightly predicted by a lower IQ (B=-0.005, 95% C.I. -0.01, 0.0, p=0.038) and sensitivity analysis revealed the role of a worse processing speed (B=-0.04, 95% C.I. -0.07, -0.01, p=0.003) and working memory (B=-0.02, 95% C.I. -0.05, -0.04, p=0.013) abilities in this relationship. Negative symptoms were higher in subjects with a worse SF (B=-0.12, 95% C.I. -0.18, -0.06, p<0.001) and a lower IQ (B=-0.005, 95% C.I. -0.01, -0.001, p=0.014), due to a lower perceptual reasoning (B=-0.02, 95% C.I. 0.04, 0.003, p=0.023). On the opposite, manic symptoms were more present if patients had better SF (B=0.07, 95% C.I. 0.01, 0.14, p=0.023) and IQ (B=0.005, 95% C.I. 0.0, 0.009, p=0.030), particularly, a higher perceptual reasoning (B=0.02, 95% C.I. 0.04, 0.008, p=0.014). Lower SF predicted depressive symptoms (B=-0.09, 95% C.I. -0.15, -0.03, p=0.032). The model was not predictive of general and disorganization symptoms. Cannabis use had an effect on positive symptoms (F(2, 770)=4.9, p=0.011); that means, patients who smoked cannabis everyday had higher positive symptoms than occasional (M_difference=0.27, 95% C.I. 0.01, 0.52, p=0.034) and never-users (M_difference=0.25, 95% C.I. 0.02, 0.48, p=0.021).DiscussionThe opposed relationship between premorbid sociability and an IQ, particularly its “hold” perceptual-reasoning sub-test – linked to an early coherent self-construction – with manic and negative symptoms, suggests divergent neurodevelopmental trajectories among patients endorsing these different phenotypes. On the other hand, the specific predictive value for positive symptoms of lower processing speed and working memory – well-recognised cognitive markers in psychosis – implies an encoding problem. Worse premorbid sociability but not low general cognition predicted depressive symptoms, thus suggesting a difference between mood downregulation and flat affect, often unrecognised. The cannabis-effect on positive symptoms was expected; this analysis suggests its possible interaction with cognitive characteristics, which deserves further attention.

Psychometric properties of TAS, TAI, FAT test anxiety scales 6 in Chinese university students: a Bifactor IRT study

In this study, the psychometric properties of three commonly used rating scales of test anxiety were examined, including the test anxiety inventory (TAI), the test anxiety scale (TAS) and the Friedman-Bendas Test Anxiety Scale (FAT). Under the framework of item response theory (IRT), the Bifactor multi-dimensional item response model was employed to compare the psychometric properties of the three scales. Results showed that the Bifactor structures were suitable for the three scales, which were then used in the subsequent Bifactor multidimensional item response theory analysis. Although the three commonly used TA scales were likely to measure the same underlying construct—test anxiety, they had very different psychometric properties. The findings of the Bifactor Multi-IRT provided suggestions for determining which scale to use in a given study design: the TAI and the FAT evaluated information at greatly overlapping ranges; however, the TAI, performing a litter better at the same levels of severity of TA, may be a good choice when we recruit those with various levels of TA severity to ensure a high precision. What’s more, FAT may be a good choice for measuring those with moderate TA severity. Meanwhile, the TAS provided more information at the lower level of TA symptomatology, which was to say, TAS was more suitable for epidemiological TA studies and for measuring those with lower TA severity.

Improving Measurement Precision in Experimental Psychopathology Using Item Response Theory.

Although item response models have grown in popularity in many areas of educational and psychological assessment, there are relatively few applications of these models in experimental psychopathology. In this article, we explore the use of item response models in the context of a computerized cognitive task designed to assess visual working memory capacity in people with psychosis as well as healthy adults. We begin our discussion by describing how item response theory can be used to evaluate and improve unidimensional cognitive assessment tasks in various examinee populations. We then suggest how computerized adaptive testing can be used to improve the efficiency of cognitive task administration. Finally, we explore how these ideas might be extended to multidimensional item response models that better represent the complex response processes underlying task performance in psychopathological populations.