Multiple-group Item Response Theory Research Articles

DIF Detection for Multiple Groups: Comparing Three‐Level GLMMs and Multiple‐Group IRT Models

AbstractFor assessment scales applied to different groups (e.g., students from different states; patients in different countries), multigroup differential item functioning (MG‐DIF) needs to be evaluated in order to ensure that respondents with the same trait level but from different groups have equal response probabilities on a particular item. The current study compares two approaches for DIF detection: a multiple‐group item response theory (MG‐IRT) model and a generalized linear mixed model (GLMM). In the MG‐IRT model approach, item parameters are constrained to be equal across groups and DIF is evaluated for each item in each group. In the GLMM, groups are treated as random, and item difficulties are modeled as correlated random effects with a joint multivariate normal distribution. Its nested structure allows the estimation of item difficulty variances and covariances at the group level. We use an excerpt from the PISA 2015 reading domain as an exemplary empirical investigation, and conduct a simulation study to compare the performance of the two approaches. Results from the empirical investigation show that the detection of countries with DIF is similar in both approaches. Results from the simulation study confirm this finding and indicate slight advantages of the MG‐IRT model approach.

THE MINI-MENTAL STATE EXAMINATION IN A CHINESE POPULATION: RELIABILITY, VALIDITY, AND MEASUREMENT INVARIANCE

Abstract The Mini-Mental State Examination (MMSE) is a widely used screening tool for cognitive impairment and dementia in many cultures and regions. However, the properties of the MMSE in the Chinese context have not been evaluated with large population-representative samples. It is also unclear whether item properties of the MMSE differ across age and education subgroups. We study the psychometric properties of the MMSE using data from 8974 respondents aged 60 years or older in the 2018 Chinese Health and Retirement Longitudinal Study (CHARLS). Using item response theory (IRT) for the ordinal data, we confirm a three-dimensional factor structure for the MMSE in CHARLS in accordance with the underlying theoretical framework and establish acceptable model-data fit (SRMSR = 0.036). Total scores and subscale scores of the MMSE correlated positively with each other and correlated negatively with ADL (r = -0.19), IADL (-0.30), and CESD scores (-0.25). The reliability of the total MMSE scores was estimated at 0.78, indicating moderate measurement precision in the study population. Multiple group IRT analysis found that item characteristics and item functioning of 5 items differed by age group and 14 items differed by educational level. This analysis provided evidence on violations to measurement invariance of MMSE items, pointing to a need to account for item-level differences in measurement properties primarily with respect to education level when using the MMSE in CHARLS and, more generally, in Chinese populations. In addition, revised cognitive measures are advised to be used in CHARLS to improve the precision of measurement.

The Mode Effect of Web-Based Surveying on the 2018 U.S. Health and Retirement Study Measure of Cognitive Functioning.

Measuring cognition in an aging populabtion is a public health priority. A move towards survey measurement via the web (as opposed to phone or in-person) is cost-effective but challenging as it may induce bias in cognitive measures. We examine this possibility using an experiment embedded in the 2018 wave of data collection for the U.S. Health and Retirement Study (HRS). We utilize techniques from multiple group item response theory to assess the effect of survey mode on performance on the HRS cognitive measure. We also study the problem of attrition by attempting to predict dropout and via approaches meant to minimize bias in subsequent inferences due to attrition. We find evidence of an increase in scores for HRS respondents who are randomly assigned to the web-based mode of data collection in 2018. Web-based respondents score higher in 2018 than experimentally matched phone-based respondents, and they show much larger gains relative to 2016 performance and subsequently larger declines in 2020. The differential in favor of web-based responding is observed across all items, but is most pronounced for the Serial 7 task and numeracy items. Due to the relative ease of the web-based mode, we suggest a cutscore of 12 being used to indicate CIND (cognitively impaired but not demented) status when using the web-based version rather than 11. The difference in mode may be nonignorable for many uses of the HRS cognitive measure. In particular, it may require reconsideration of some cutscore-based approaches to identify impairment.

Modifying the social responsiveness scale for adaptive administration.

The social responsiveness scale (SRS) is frequently used to quantify the autism-related phenotype and is gaining use in health outcomes research. However, it has a high respondent burden (65 items) for large-scale studies. Further, most evaluations of it have focused on the school-age form, not the preschool form. More validity evidence of shortened forms is necessary in the general population to support the broader health outcomes context of use. We evaluated the psychometrics of the SRS in 7030 individuals from multiple predominantly neurotypical samples in order to shorten it based on non-autistic sample metrics. Analyses included item factor analysis, differential item functioning (DIF), and multiple-group item response theory (IRT) to place the SRS items on a comparable scale, which was then simulated via computer adaptive testing (CAT) administration. The SRS was broadly unidimensional with few methodological residual dependencies. On average, males had more autistic characteristics than females, and preschoolers had fewer characteristics than school-age children. The final IRT calibration included 45 items equated across forms, and each form had 11 with significant wording discrepancies and 9 items with near-identical wording that exhibited form-related DIF. The CAT simulation suggested a median of 14 items was sufficient to reach a reliable score, demonstrating its feasibility across the range of impairments. IRT allows practitioners the ability to get highly reliable scores with fewer items than the full-length SRS. This supports the future application of the SRS in a computer adaptive testing mode in both neurotypical and ASD samples.

Score-based measurement invariance checks for Bayesian maximum-a-posteriori estimates in item response theory.

A family of score-based tests has been proposed in recent years for assessing the invariance of model parameters in several models of item response theory (IRT). These tests were originally developed in a maximum likelihood framework. This study discusses analogous tests for Bayesian maximum-a-posteriori estimates and multiple-group IRT models. We propose two families of statistical tests, which are based on an approximation using a pooled variance method, or on a simulation approach based on asymptotic results. The resulting tests were evaluated by a simulation study, which investigated their sensitivity against differential item functioning with respect to a categorical or continuous person covariate in the two- and three-parametric logistic models. Whereas the method based on pooled variance was found to be useful in practice with maximum likelihood as well as maximum-a-posteriori estimates, the simulation-based approach was found to require large sample sizes to lead to satisfactory results.

Reliability coefficients for multiple group item response theory models.

Reliability of scores from psychological or educational assessments provides important information regarding the precision of measurement. The reliability of scores is however population dependent and may vary across groups. In item response theory, this population dependence can be attributed to differential item functioning or to differences in the latent distributions between groups and needs to be accounted for when estimating the reliability of scores for different groups. Here, we introduce group‐specific and overall reliability coefficients for sum scores and maximum likelihood ability estimates defined by a multiple group item response theory model. We derive confidence intervals using asymptotic theory and evaluate the empirical properties of estimators and the confidence intervals in a simulation study. The results show that the estimators are largely unbiased and that the confidence intervals are accurate with moderately large sample sizes. We exemplify the approach with the Montreal Cognitive Assessment (MoCA) in two groups defined by education level and give recommendations for applied work.

Scale length does matter: Recommendations for measurement invariance testing with categorical factor analysis and item response theory approaches

In social sciences, the study of group differences concerning latent constructs is ubiquitous. These constructs are generally measured by means of scales composed of ordinal items. In order to compare these constructs across groups, one crucial requirement is that they are measured equivalently or, in technical jargon, that measurement invariance (MI) holds across the groups. This study compared the performance of scale- and item-level approaches based on multiple group categorical confirmatory factor analysis (MG-CCFA) and multiple group item response theory (MG-IRT) in testing MI with ordinal data. In general, the results of the simulation studies showed that MG-CCFA-based approaches outperformed MG-IRT-based approaches when testing MI at the scale level, whereas, at the item level, the best performing approach depends on the tested parameter (i.e., loadings or thresholds). That is, when testing loadings equivalence, the likelihood ratio test provided the best trade-off between true-positive rate and false-positive rate, whereas, when testing thresholds equivalence, the χ2 test outperformed the other testing strategies. In addition, the performance of MG-CCFA’s fit measures, such as RMSEA and CFI, seemed to depend largely on the length of the scale, especially when MI was tested at the item level. General caution is recommended when using these measures, especially when MI is tested for each item individually.

Monster, P.I.: Validation Evidence for an Assessment of Adolescent Language That Assesses Vocabulary Knowledge, Morphological Knowledge, and Syntactical Awareness

Assessment of language skills for upper elementary and middle schoolers is important due to the strong link between language and reading comprehension. Yet, currently few practical, reliable, valid, and instructionally informative assessments of language exist. This study provides validation evidence for Monster, P.I., which is a gamified, standardized, computer-adaptive assessment (CAT) of language for fifth to eighth grade students. Creating Monster, P.I. involved an assessment of the dimensionality of morphology and vocabulary and an assessment of syntax. Results using multiple-group item response theory (IRT) with 3,214 fifth through eighth graders indicated morphology and vocabulary were best assessed via bifactor models and syntax unidimensionally. Therefore, Monster, P.I. provides scores on three component areas of language (multidimensional morphology and vocabulary and unidimensional syntax) with the goal of informing instruction. Validity results also suggest that Monster, P.I. scores show moderate correlations with each other and with standardized reading vocabulary and reading comprehension assessments. Furthermore, hierarchical regression results suggest an important link between Monster, P.I. and standardized reading comprehension, explaining between 56% and 75% of the variance. Such results indicate that Monster, P.I. can provide meaningful understandings of language performance which can guide instruction that can impact reading comprehension performance.

Differential item functioning analysis of the CUDIT and relations with alcohol and tobacco use among men across five ethnic groups: The HELIUS study.

Cannabis use has been rising despite recognition of the negative consequences associated with heavy use. The severity of these consequences has been shown to differ across racial/ethnic groups, even when controlling for consumption levels. The present study conducted an item response theory (IRT) analysis of the Cannabis Use Disorders Identification Test (CUDIT) to better understand the patterns of problematic cannabis use and their relation with other substance use across ethnic groups in the Healthy Life in an Urban Setting (HELIUS) study. CUDIT responses from 1,960 cannabis-using African Surinamese, South-Asian Surinamese, Dutch, Moroccan, and Turkish ethnic origin participants were used to test for differential item functioning (DIF) within an IRT framework. After restricting the sample to men because of low frequency of use among women, several instances of uniform DIF were identified. Multiple-group IRT analysis yielded a harmonized cannabis use phenotype that was used to estimate ethnic group differences in problematic cannabis use and its relation to alcohol and tobacco co-use. These analyses suggested that cannabis users from certain ethnic minority groups experienced higher rates of problematic use than the majority group despite lower rates of cannabis use. Further, cannabis and tobacco use were positively related across groups, whereas only ethnic minority groups showed a positive relation between cannabis and alcohol use. These results demonstrate the importance of accounting for DIF when examining group differences in problematic cannabis use, and support prior evidence suggesting that certain ethnic minority groups may be more likely to experience problematic cannabis use and alcohol co-use relative to the majority group. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Applying Item Response Theory Analysis to the Montreal Cognitive Assessment in a Low-Education Older Population.

The traditional application of the Montreal Cognitive Assessment uses total scores in defining cognitive impairment levels, without considering variations in item properties across populations. Item response theory (IRT) analysis provides a potential solution to minimize the effect of important confounding factors such as education. This research applies IRT to investigate the characteristics of Montreal Cognitive Assessment items in a randomly selected, culturally homogeneous sample of 1,873 older persons with diverse educational backgrounds. Any formal education was used as a grouping variable to estimate multiple-group IRT models. Results showed that item characteristics differed between people with and without formal education. Item functioning of the Cube, Clock Number, and Clock Hand items was superior in people without formal education. This analysis provided evidence that item properties vary with education, calling for more sophisticated modelling based on IRT to incorporate the effect of education.

A multiple group item response theory model with centered skew-normal latent trait distributions under a Bayesian framework

Very often, in psychometric research, as in educational assessment, it is necessary to analyze item response from clustered respondents. The multiple group item response theory (IRT) model proposed by Bock and Zimowski [12] provides a useful framework for analyzing such type of data. In this model, the selected groups of respondents are of specific interest such that group-specific population distributions need to be defined. The usual assumption for parameter estimation in this model, which is that the latent traits are random variables following different symmetric normal distributions, has been questioned in many works found in the IRT literature. Furthermore, when this assumption does not hold, misleading inference can result. In this paper, we consider that the latent traits for each group follow different skew-normal distributions, under the centered parameterization. We named it skew multiple group IRT model. This modeling extends the works of Azevedo et al. [4], Bazán et al. [11] and Bock and Zimowski [12] (concerning the latent trait distribution). Our approach ensures that the model is identifiable. We propose and compare, concerning convergence issues, two Monte Carlo Markov Chain (MCMC) algorithms for parameter estimation. A simulation study was performed in order to evaluate parameter recovery for the proposed model and the selected algorithm concerning convergence issues. Results reveal that the proposed algorithm recovers properly all model parameters. Furthermore, we analyzed a real data set which presents asymmetry concerning the latent traits distribution. The results obtained by using our approach confirmed the presence of negative asymmetry for some latent trait distributions.

DSM-IV alcohol abuse and dependence criteria characteristics for recent onset adolescent drinkers

BackgroundLittle is known about the psychometric properties of alcohol abuse and dependence criteria among recent-onset adolescent drinkers, particularly for those who consume alcohol infrequently. This study evaluated how well DSM-IV alcohol dependence criteria measure an alcohol use disorder (AUD) construct for recent onset adolescent drinkers at different levels of drinking frequency. MethodData were drawn from the National Survey on Drug Use and Health, a nationally representative sample of 9356 recent-onset adolescent drinkers, aged 12–21, who began drinking within the past year. Multiple group item response theory analysis was conducted to assess the 11 DSM-IV alcohol abuse and dependence criteria. ResultsCriteria most likely to be endorsed at lower AUD severity included “withdrawal,” “problems at home, school or work” and “tolerance.” The criteria “drinking larger amounts/longer period of time,” “unsuccessful efforts to cut down” and “continuing to drink despite related health problems” were more likely to be endorsed at higher AUD severity. Two criteria, “tolerance” and “time spent getting, using or recovering from alcohol” showed differential item functioning between drinking frequency groups (<7 vs. ≥7days in past month), with lower discrimination and severity for more frequent drinkers. DSM-IV criteria were most precise for intermediate levels of AUD severity. ConclusionsAll but two DSM-IV criteria had consistent psychometric properties across drinking frequency groups. Symptoms were most precise for a narrow, intermediate range of AUD severity. Those assessing AUD in recent onset adolescent drinkers might consider additional symptoms to capture the full AUD continuum.

An item response theory analysis of nicotine dependence symptoms in recent onset adolescent smokers

Introduction Given absence of a “gold standard” for measuring self-reported nicotine dependence, particularly among less experienced smokers, there is a need to evaluate existing measures to determine how well symptoms measure the underlying nicotine dependence construct and whether symptoms function differently for less experienced smokers. Study aims were to determine (1) likelihood of endorsement of individual symptoms at different levels of a nicotine dependence construct and the ability of symptoms to discriminate between different levels of this construct and (2) whether these symptom properties varied between nondaily and daily smokers. Methods We used multiple group item response theory analysis to evaluate nicotine dependence symptoms from the nicotine dependence syndrome scale based on a nationally representative sample of 8081 recent onset adolescent smokers from the national surveys on drug use and health. Results After controlling for age, gender, smoking quantity and length of smoking exposure, symptoms assessing tolerance were invariant across nondaily and daily smokers, and discriminated well between levels of the nicotine dependence construct. However, the majority of symptoms functioned differently for nondaily and daily smokers. These symptoms did not discriminate as well between levels of the nicotine dependence construct and were more likely to be endorsed at lower levels of this construct for daily smokers. Discussion A measure that encompasses a range of symptoms tapping different aspects of smoking may be ideally suited for nondaily adolescent smokers, while an ideal measure of nicotine dependence for daily smokers might also include more core diagnostic features of nicotine dependence such as withdrawal and tolerance.

DSM-IV nicotine dependence symptom characteristics for recent-onset smokers.

This study evaluated how well DSM-IV nicotine dependence symptoms measure an underlying dependence construct for recent-onset daily and nondaily smokers. Based on a nationally representative sample of 2,758 recent-onset adolescent smokers from the National Surveys on Drug Use and Health, we used multiple group item response theory analysis to assess 7 symptoms representing DSM-IV diagnostic features of nicotine dependence. After controlling for age, gender, current smoking quantity, and length of smoking exposure, all 7 DSM-IV symptoms were invariant across nondaily and daily smokers and discriminated well among levels of the nicotine dependence construct. Symptoms most likely to be endorsed at lower levels of the dependence construct included spending more time getting, using, or getting over the effects of smoking and wanting or trying to stop or cut down. Symptoms most likely to be endorsed only at higher levels of the construct included giving up important activities and emotional/psychological and health problems related to smoking. DSM-IV symptoms were most precise for moderately high levels of the dependence construct and less precise for lower levels for both nondaily and daily smokers. DSM-IV nicotine dependence symptoms appear to have desirable psychometric properties for measuring a nicotine dependence construct among recent-onset adolescent smokers at both daily and nondaily levels, providing justification for the use of these symptoms in a measure that aims to evaluate the full continuum of nicotine dependence severity in this population.

A New Concurrent Calibration Method for Nonequivalent Group Design under Nonrandom Assignment

This study proposes a new item parameter linking method for the common-item nonequivalent groups design in item response theory (IRT). Previous studies assumed that examinees are randomly assigned to either test form. However, examinees can frequently select their own test forms and tests often differ according to examinees’ abilities. In such cases, concurrent calibration or multiple group IRT modeling without modeling test form selection behavior can yield severely biased results. We proposed a model wherein test form selection behavior depends on test scores and used a Monte Carlo expectation maximization (MCEM) algorithm. This method provided adequate estimates of testing parameters.