Polytomous Item Response Theory Models Research Articles

Measurement of bullying in the Progress in International Reading Literacy Study 2021 cohort: Evidence for model misspecification

We evaluated the use of the Rasch model in the scoring and scaling of the context questionnaire scales from the Progress in International Reading Literacy Study 2021 cohort. Focusing on the bullying scale of the Progress in International Reading Literacy Study, we contrasted polytomous item response theory models using free (two-parameter logistic modeling) and fixed (Rasch modeling) discrimination parameters. Our analyses included data from 354,074 students from 63 countries or entities. Across all countries and entities, the Rasch-based model significantly distorted levels of bullying. The rank ordering of countries was consistent between models in 32 of the cases (50.8%), while in other cases there were changes in rank of up to five places. This study provides a robust basis for evidence-based decisions on whether a country is lagging or exceeding the global norm in aberrant behaviors such as bullying.

The effect of real word stimuli versus non-word stimuli on oral diadochokinetic rates across the life span: An item discrimination analysis.

Diadochokinetic (DDK) rate tasks are extensively used in the evaluation of speech disorders; however, it is unclear how the different types of speech stimuli affect DDK rate performance. To investigate the effect of age, gender and type of stimuli (non-words versus real words) on the DDK rates in individuals across the lifespan and to provide normative data for Greek. Also to examine the discrimination ability of the speech DDK stimuli administered (non-words and real words) based on a dual DDK assessment protocol using a polytomous item response theory (IRT) model. The participants were 1747 monolingual Greek speakers (376 children, aged 4-17 years; and 1371 adults, aged 18-90+ years). All participants had normal hearing acuity which allowed them to understand and follow instructions. Participants with a medical condition or a language disorder which would affect DDK rate performance were excluded from the study. The time-by-count method was used, and all participants had to repeat as accurately and fast as possible: (1) four disyllabic non-words (/'gaba/, /'taka/, /'kata/, /'baga/), (2) four disyllabic real words (/'kapa/, /'tapa/, /ka'la/, /'paka/) and (3) two trisyllabic non-words (/'pataka/, /'badaga/). All responses were recorded and the speech samples that did not include at least 5 s of correct repetitions were excluded from the analysis. Age affected significantly DDK rates with performance increasing gradually until approximately the age of 40 and then gradually decreasing. Gender had no effect. Overall, there was a significant advantage of disyllabic real word stimuli over disyllabic non-word stimuli and of trisyllabic non-word stimuli over disyllabic non-word stimuli on DDK rates performance. IRT analysis suggested that the data fit the polytomous model reasonably well and all DDK stimuli (real words and non-words) showed a strong relationship (loadings > 0.50) with the latent trait. The current study complements prior research which supports that age and type of stimuli significantly affect DDK rates performance. It is the first study, that testifies to the benefit of real-word stimuli over non-word stimuli on DDK rates across the lifespan in a large representative sample. The implementation of IRT analysis provides empirical evidence about the discrimination ability of the DDK stimuli administered and confirms the reliability of this dual DDK assessment protocol. These findings are valuable for clinicians who work with motor speech disorders. What is already known on this subject Age, type of stimuli (real words versus non-words) and language significantly affect DDK rates performance. Current research strongly suggests the administration of language-specific norms since language-dependent features seem to have a noteworthy effect on the DDK rates, but scarce evidence exists about the discriminatory ability of the DDK speech stimuli commonly administered. What this study adds to the existing knowledge Conflicting findings have been reported about the effect of different types of DDK speech stimuli (real words and non-words) but no study to date has evaluated their discriminatory abilities. The current study is the first to implement a polytomous IRT model to examine this issue. This is also the first study to attempt an investigation of the effect of types of stimuli (real words versus non-words) on a large representative sample across the lifespan (4-90+ years) and to provide normative data for Greek. What are the practical and clinical implications of this work? The present study offers concrete evidence about the advantage of real-word stimuli over non-word stimuli in Greek, as well as normative data for the Greek-speaking populations. Moreover, the IRT analysis testifies to the discriminatory ability of real-word and non-word stimuli affirming the reliability of the present dual DDK assessment protocol as a psychometrically sound measure of DDK ability. The above has significant value for clinicians who work with individuals with motor speech disorders as the protocol can help them with the diagnosis and differential diagnosis of motor speech disorders.

Psychometric properties of the Psy-flex scale: A validation study in a community sample in Korea

This study examined the psychometric properties of the recently developed six-item Psy-Flex among community samples in Korea. Using a cross-sectional design, data were collected from 1059 participants. Three bilingual experts translated the scale to ensure content validity. Factor analysis was employed to confirm the factor structure, and a polytomous item response theory model was used to examine the individual items and entire scale. The theory-based single-factor structure was confirmed using a Korean community sample, and measurement invariances were found across sex and age groups. In addition, the scale showed a moderate relationship with established measures of interest. Furthermore, item and categorical functioning were investigated using the polytomous Item Response Theory (IRT) model (i.e., the Generalized Partial Credit Model [GPCM]), identifying well-functioning (e.g., items 4, 5, and 6) and suboptimal (e.g., item 2) items. Additionally, the results suggest that the participants might not be able to meaningfully differentiate the original five-response categories. To the best of our knowledge, this is the first attempt to validate the Psy-Flex in Korea. We believe that the findings are of considerable value in facilitating our understanding of the scale and, more broadly, the construct of psychological flexibility.

Developing New Scales of Personal Food Security

BackgroundFood insecurity is standardly measured at the household level or for groups of household members. However, food hardships may differ for individuals within households. Summary measures of people’s individual experiences of food insecurity are needed. ObjectiveThis study aims to develop and analyze psychometrically sound multi-item scales of people’s individual experiences of food insecurity. It further aims to examine whether and how the distributions of personal food insecurity differ across age groups, from household food insecurity, and with people’s observed characteristics. DesignThe study analyzes questionnaire data on personal food security outcomes, household food security outcomes, and other characteristics. Participants and settingThe 29,040 participants in the study responded to the 2005-2006 through 2009-2010 cycles of the National Health and Nutrition Examination Survey in the United States. Main outcomesThe main outcomes are five-item scales of personal food insecurity for children younger than age 12 years, young adolescents aged 12 to 15 years, and people aged 16 years and older. Statistical analysis performedThe study develops the personal food insecurity scales through factor analyses and polytomous Item Response Theory models and analyzes characteristics that are related to the scale outcomes through multivariate regressions. ResultsThe article develops personal food insecurity scales that are related to but distinct from the standard household scales, with different scales being needed to capture the experiences of its three age groups. Children younger than age 12 years have much lower risks of personal food insecurity than other age groups, whereas young adolescents have higher risks than other groups. Three percent of children had an indication of personal food insecurity, whereas 10% of adolescents and 7% of adults had indications. Among adults, women and people between ages 31 and 65 years have higher risks of personal food insecurity than men and people of other ages, but they have similar risks for household food insecurity. ConclusionsPersonal food insecurity is a distinct component of well-being that can be summarized through scale measures. Evidence that characteristics, such as sex and age, are related to personal food insecurity but not household food insecurity indicates that food experiences can differ within households and that some people may be systematically disadvantaged.

Wald χ2 Test for Differential Item Functioning Detection with Polytomous Items in Multilevel Data

Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord’s Wald [Formula: see text] test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.

Detecting Rating Scale Malfunctioning With the Partial Credit Model and Generalized Partial Credit Model.

Rating scale analysis techniques provide researchers with practical tools for examining the degree to which ordinal rating scales (e.g., Likert-type scales or performance assessment rating scales) function in psychometrically useful ways. When rating scales function as expected, researchers can interpret ratings in the intended direction (i.e., lower ratings mean "less" of a construct than higher ratings), distinguish between categories in the scale (i.e., each category reflects a unique level of the construct), and compare ratings across elements of the measurement instrument, such as individual items. Although researchers have used these techniques in a variety of contexts, studies are limited that systematically explore their sensitivity to problematic rating scale characteristics (i.e., "rating scale malfunctioning"). I used a real data analysis and a simulation study to systematically explore the sensitivity of rating scale analysis techniques based on two popular polytomous item response theory (IRT) models: the partial credit model (PCM) and the generalized partial credit model (GPCM). Overall, results indicated that both models provide valuable information about rating scale threshold ordering and precision that can help researchers understand how their rating scales are functioning and identify areas for further investigation or revision. However, there were some differences between models in their sensitivity to rating scale malfunctioning in certain conditions. Implications for research and practice are discussed.

Performance of Polytomous IRT Models With Rating Scale Data: An Investigation Over Sample Size, Instrument Length, and Missing Data

The implementation of polytomous item response theory (IRT) models such as the graded response model (GRM) and the generalized partial credit model (GPCM) to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used. The performance of such models has not been fully investigated and compared across conditions with common survey-specific characteristics such as short test length, small sample size, and data missingness. The purpose of the current simulation study is to inform the literature and guide the implementation of GRM and GPCM under these conditions. For item parameter estimations, results suggest a sample size of at least 300 and/or an instrument length of at least five items for both models. The performance of GPCM is stable across instrument lengths while that of GRM improves notably as the instrument length increases. For person parameters, GRM reveals more accurate estimates when the proportion of missing data is small, whereas GPCM is favored in the presence of a large amount of missingness. Further, it is not recommended to compare GRM and GPCM based on test information. Relative model fit indices (AIC, BIC, LL) might not be powerful when the sample size is less than 300 and the length is less than 5. Synthesis of the patterns of the results, as well as recommendations for the implementation of polytomous IRT models, are presented and discussed.

Bayesian estimation of multidimensional polytomous item response theory models with Q-matrices using Stan

The Q-matrix is commonly used in diagnostic classification models and has recently been incorporated into the multidimensional item response theory (MIRT) models to add information about the relationship between items and dimensions of the latent trait. The reformulation of the MIRT models with Q-matrix (MIRT-Q) has presented to improve the precision of the parameters of these models and to provide a simple and intuitive method for users to define the item-trait relationship. This paper aims to explore the incorporation of the Q-matrix in the formulation of MIRT models for polytomous item responses. Specifically, we introduce the incorporation of the Q-matrix into two of the polytomous MIRT models most known and used: the multidimensional graded response (MGR) model, hereinafter called MGR-Q, and the multidimensional generalized partial credit (MGPC) model, hereinafter called MGPC-Q. We provide readers the code of the MGR-Q and MGPC-Q models in Stan, a Bayesian estimation software, and we conduct a simulation study in order to evaluate the parameter recovery of the estimation method. To illustrate the use of both models in practice, we fit them to an operational dataset from 2400 individuals on 13 items and demonstrate the estimation of MGR-Q and MGPC-Q using the Stan program.

Assessing Cognitive Abilities Using the WAIS-IV: An Item Response Theory Approach.

Background: The Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) has been adapted to 28 different cultures and there has been considerable interest in examining its structure through exploratory and confirmatory factor analysis. This study investigates item and scale properties of the Egyptian WAIS-IV using item response theory (IRT) models. Methods: The sample consisted of 250 adults from Egypt. The item-level and subtest statistical properties of the Egyptian WAIS-IV were established using a combination of four dichotomous IRT models and four polytomous IRT models. In addition, factor analysis was performed to investigate the dimensionality of each subtest. Results: Factor analysis indicated the unidimensionality of each subtest. Among IRT models, the two-parameter logistic model provided a good fit for dichotomous subtests, while the graded response model fitted the polytomous data. Most items of the Egyptian WAIS-IV showed high discrimination, and the scale was adequately informative across the levels of latent traits (i.e., cognitive variables). However, each subtest included at least some items with limited ability to distinguish between individuals with differing levels of the cognitive variable being measured. Furthermore, most subtests have items that do not follow the difficulty rank they are ascribed in the WAIS-IV manual. Conclusions: Overall, the results suggest that the Egyptian WAIS-IV offers a highly valid assessment of intellectual abilities, despite the need for some improvements.

Item response theory as a feature selection and interpretation tool in the context of machine learning.

Optimizing the number and utility of features to use in a classification analysis has been the subject of many research studies. Most current models use end-classifications as part of the feature reduction process, leading to circularity in the methodology. The approach demonstrated in the present research uses item response theory (IRT) to select features independent of the end-classification results without the biased accuracies that this circularity engenders. Dichotomous and polytomous IRT models were used to analyze 30 histological breast cancer features from 569 patients using the Wisconsin Diagnostic Breast Cancer data set. Based on their characteristics, three features were selected for use in a machine learning classifier. For comparison purposes, two machine learning-based feature selection protocols were run-recursive feature elimination (RFE) and ridge regression-and the three features selected from these analyses were also used in the subsequent learning classifier. Classification results demonstrated that all three selection processes performed comparably. The non-biased nature of the IRT protocol and information provided about the specific characteristics of the features as to why they are of use in classification help to shed light on understanding which attributes of features make them suitable for use in a machine learning context.

Do Mothers and Fathers Assess their Children's Behavioral Problems in the Same Way as do their Children? An IRT Investigation on the Strengths and Difficulties Questionnaire

AbstractThis study investigated the equivalence of different types of informants, such as children (or early adolescents) and parents, in evaluating child externalizing and internalizing problems. We applied a polytomous item response theory (IRT) model for the Strengths and Difficulties Questionnaire (SDQ). We obtained responses to three subscales—Conduct Problems, Hyperactivity/Inattention, and Emotional Symptoms—from 541 elementary school students aged 10–12 years, fathers for 233 students, mothers for 275 students, and the homeroom teachers for 524 students. Expected values on the individual item calculated by the discrimination and threshold parameters were compared among students, fathers, and mothers as an investigation of differential item functioning (DIF) or differential informant functioning. Assessing either externalizing or internalizing problems were mostly equivalent between fathers and mothers, and most items for externalizing problems functioned equally between students and parents, whereas items for internalizing problems showed DIF between them. IRT also yielded that the intervals of response categories varied across items, particularly for the conduct problems items “fight” and “steal,” and positively worded items showed an extremely low threshold.

Polytomous item explanatory IRT models with random item effects: Concepts and an application

This paper proposes three polytomous item explanatory models with random item errors in Item Response Theory (IRT), by extending the Linear Logistic Test Model with item error (LLTM + ε) approach to polytomous data. The proposed models, also regarded as polytomous random item effects models, can take the uncertainty in explanation and/or the random nature of item parameters into account for polytomous items. To develop the models, the concepts and types of polytomous random item effects are investigated and then added into the existing polytomous item explanatory models. For estimation of the proposed models with crossed random effects for polytomous data, a Bayesian inference method is adopted for data analysis. An empirical example demonstrates practical implications and applications of the proposed models to the Verbal Aggression data. The empirical findings show that the proposed models with random item errors perform better than the existing models without random item errors in terms of the goodness-of-fit and reconstructing the step difficulties and also demonstrate methodological and practical differences of the proposed models in interpreting the item property effects in each of the item location explanatory Many-Facet Rasch Model and the step difficulty explanatory Linear Partial Credit Model approaches.

Statistical Foundations for Computerized Adaptive Testing with Response Revision.

The compatibility of computerized adaptive testing (CAT) with response revision has been a topic of debate in psychometrics for many years. The problem is to provide test takers opportunities to change their answers during the test, while discouraging deceptive strategies from their side and preserving the statistical efficiency of the traditional CAT. The estimating approach proposed in Wang et al. (Stat Sin 27(4):1987-2010, 2017), based on the nominal response model, allows test takers to provide more than one answer to each item during the test, which they all contribute to the interim and final ability estimation. This approach is here reformulated, extended to incorporate a larger class of polytomous and dichotomous item response theory models, and investigated with simulation studies under different test-taking strategies.

Bayes Factors for Evaluating Latent Monotonicity in Polytomous Item Response Theory Models

The assumption of latent monotonicity is made by all common parametric and nonparametric polytomous item response theory models and is crucial for establishing an ordinal level of measurement of the item score. Three forms of latent monotonicity can be distinguished: monotonicity of the cumulative probabilities, of the continuation ratios, and of the adjacent-category ratios. Observable consequences of these different forms of latent monotonicity are derived, and Bayes factor methods for testing these consequences are proposed. These methods allow for the quantification of the evidence both in favor and against the tested property. Both item-level and category-level Bayes factors are considered, and their performance is evaluated using a simulation study. The methods are applied to an empirical example consisting of a 10-item Likert scale to investigate whether a polytomous item scoring rule results in item scores that are of ordinal level measurement.

The Effect of Alternative Scoring Procedures on the Measurement Properties of a Self-Administered Depression Scale

Abstract. To investigate the effect of response alternatives/scoring procedures on the measurement properties of the Center for Epidemiologic Studies Depression Scale (CES-D) which has the four response alternatives, a polytomous item response theory (IRT) model was applied to the responses of 2,061 workers and university students (1,640 males, 421 females). Test information functions derived from the polytomous IRT analyses on the CES-D data with various scoring procedures indicated that: (1) the CES-D with its standard (0-1-2-3) scoring procedure should be useful for screening to detect subjects with “at high-risk” of depression if the θ point showing the highest information corresponds to the cut-off point, because of its extremely higher information; (2) the CES-D with the 0-1-1-2 scoring procedure could cover wider range of depressive severity, suggesting that this scoring procedure might be useful in cases where more exhaustive discrimination in symptomatology is of interest; and (3) the revised version of CES-D with replacing original positive items into negatively revised items outperformed the original version. These findings have never been demonstrated by the classical test theory analyses, and thus the utility of this kind of psychometric testing should be warranted to further investigation for the standard measures of psychological assessment.

A note on monotonicity of item response functions for ordered polytomous item response theory models.

A monotone relationship between a true score (τ) and a latent trait level (θ) has been a key assumption for many psychometric applications. The monotonicity property in dichotomous response models is evident as a result of a transformation via a test characteristic curve. Monotonicity in polytomous models, in contrast, is not immediately obvious because item response functions are determined by a set of response category curves, which are conceivably non-monotonic in θ. The purpose of the present note is to demonstrate strict monotonicity in ordered polytomous item response models. Five models that are widely used in operational assessments are considered for proof: the generalized partial credit model (Muraki, 1992, Applied Psychological Measurement, 16, 159), the nominal model (Bock, 1972, Psychometrika, 37, 29), the partial credit model (Masters, 1982, Psychometrika, 47, 147), the rating scale model (Andrich, 1978, Psychometrika, 43, 561), and the graded response model (Samejima, 1972, A general model for free-response data (Psychometric Monograph no. 18). Psychometric Society, Richmond). The study asserts that the item response functions in these models strictly increase in θ and thus there exists strict monotonicity between τ and θ under certain specified conditions. This conclusion validates the practice of customarily using τ in place of θ in applied settings and provides theoretical grounds for one-to-one transformations between the two scales.

Asymptotic Variance of Linking Coefficient Estimators for Polytomous IRT Models.

In item response theory (IRT), when two groups from different populations take two separate tests, there is a need to link the two ability scales so that the item parameters of the tests are comparable across the groups. To link the two scales, information from common items are utilized to estimate linking coefficients which place the item parameters on the same scale. For polytomous IRT models, the Haebara and Stocking-Lord methods for estimating the linking coefficients have commonly been recommended. However, estimates of the variance for these methods are not available in the literature. In this article, the asymptotic variance of linking coefficients for polytomous IRT models with the Haebara and Stocking-Lord methods are derived. The results are presented in a general form and specific results are given for the generalized partial credit model. Simulations which investigate the accuracy of the derivations under various settings of model complexity and sample size are provided, showing that the derivations are accurate under the conditions considered and that the Haebara and Stocking-Lord methods have superior performance to several moment methods with performance close to that of concurrent calibration.

Large Sample Confidence Intervals for Item Response Theory Reliability Coefficients.

In applications of item response theory (IRT), an estimate of the reliability of the ability estimates or sum scores is often reported. However, analytical expressions for the standard errors of the estimators of the reliability coefficients are not available in the literature and therefore the variability associated with the estimated reliability is typically not reported. In this study, the asymptotic variances of the IRT marginal and test reliability coefficient estimators are derived for dichotomous and polytomous IRT models assuming an underlying asymptotically normally distributed item parameter estimator. The results are used to construct confidence intervals for the reliability coefficients. Simulations are presented which show that the confidence intervals for the test reliability coefficient have good coverage properties in finite samples under a variety of settings with the generalized partial credit model and the three-parameter logistic model. Meanwhile, it is shown that the estimator of the marginal reliability coefficient has finite sample bias resulting in confidence intervals that do not attain the nominal level for small sample sizes but that the bias tends to zero as the sample size increases.

A Polytomous Item Response Theory Model for Measuring Near-Miss Appraisal as a Psychological Trait

Near-miss experiences have been identified as a contributing factor in responses to risk of disaster events. Researchers have found that specific characteristics of a near-miss event could lead individuals to interpret the risk as either “vulnerable” or “resilient.” Moreover, these interpretations can lead to quite different decisions regarding future protective behavior. We developed the Near-Miss Appraisal Scale (NMAS) to assess an individual’s tendency to interpret near misses as vulnerable (or resilient). We developed an initial item pool of 21 items and recruited a sample of 298 respondents through Amazon Mechanical Turk. The final version of the NMAS is based on 10 of these items, following psychometric analysis for dimensionality, scale reliability, and item functioning. We establish discriminant validity of the NMAS by correlating the NMAS with scales of locus of control and risk taking, and predictive validity by using the NMAS to predict individual responses to a near-miss disaster scenario. The current study demonstrates that responses to near misses are not only determined by the nature of the event itself, but also related to decision makers' near-miss appraisal tendencies.

How Well Does the CWEQ II Measure Structural Empowerment? Findings from Applying Item Response Theory

The main purpose of this paper is to examine the psychometric properties of the original five-point CWEQ II using Item Response Theory (IRT) methods, followed by an examination of the revised three-point CWEQ II. (1) Background: The psychometric properties of the CWEQ II have not been previously assessed using more robust techniques such as IRT. (2) Methods: This is a secondary analysis of baseline data from 1067 staff nurses whose leaders had attended a leadership development program. Data were analyzed using a polytomous IRT model. (3) Results: The two versions of CWEQ II fit the SE data equally as each had only one poor-fitting item. For the five-point CWEQ II, discriminant ability was poor for a majority of the items; one item demonstrated a disordinal step difficulty parameter; and item reliability was supported for a relatively wider range of SE levels. The discriminant ability and reliability of items for the three-point CWEQ II was better than those of the five-point CWEQ II, but for a narrower range of SE levels; and the disordinal step difficulty parameter was resolved. (4) Conclusion: The appropriate use of each version of the scale depends on the conditions of the work setting targeted.