IRT Models Research Articles

Residual-Based Person Fit Statistics over Test Sections

Most tests are composed of multiple sections (each section has group of items) such as different item formats, different content category, competencies, different difficulty levels, test dimensions, testlets, and interpretive exercise items. Students could show unexpected and unacceptable responses across these sections. Studying person fit over item level cannot detect aberrant response over test sections. The study proposes a residual-based person fit statistic over test sections with a dichotomous IRT model. The paper demonstrates the new section-level person fit statistic and investigates its distributional properties and power of detecting aberrance in person responses with comparison to Wright's between person fit statistic. The proposed section-level person fit statistic shows superior distributional properties with both true and real ability and item parameters. Moreover, the performance of the proposed person fit statistic is also examined with real data.

Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items.

One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population, ignoring their possible effects on the scores may compromise the fairness and the validity of the assessments. Also, response-style-induced errors of measurement can affect the reliability estimates and overestimate convergent validity by correlating higher with other Likert-scale-based measures. Conversely, it can attenuate the predictive power over non-Likert-based indicators, given that the scores contain more errors. This study compares the validity of the Big Five personality scores obtained: (1) ignoring the SDR and ACQ in graded-scale items (GSQ), (2) accounting for SDR and ACQ with a compensatory IRT model, and (3) using forced-choice blocks with a multi-unidimensional pairwise preference model (MUPP) variant for dominance items. The overall results suggest that ignoring SDR and ACQ offered the worst validity evidence, with a higher correlation between personality and SDR scores. The two remaining strategies have their own advantages and disadvantages. The results from the empirical reliability and the convergent validity analysis indicate that when modeling social desirability with graded-scale items, the SDR factor apparently captures part of the variance of the Agreeableness factor. On the other hand, the correlation between the corrected GSQ-based Openness to Experience scores, and the University Access Examination grades was higher than the one with the uncorrected GSQ-based scores, and considerably higher than that using the estimates from the forced-choice data. Conversely, the criterion-related validity of the Forced Choice Questionnaire (FCQ) scores was similar to the results found in meta-analytic studies, correlating higher with Conscientiousness. Nonetheless, the FCQ-scores had considerably lower reliabilities and would demand administering more blocks. Finally, the results are discussed, and some notes are provided for the treatment of SDR and ACQ in future studies.

Mixture Rasch Model with Main and Interaction Effects of Covariates on Latent Class Membership

Covariates have been used in mixture IRT models to help explain why examinees are classed into different latent classes. Previous research has considered manifest variables as covariates in a mixture Rasch analysis for prediction of group membership. Latent covariates, however, are more likely to have higher correlations with the latent class variable. This study investigated effects of including latent variables as covariates in a mixture Rasch model, in presence of and in absence of interactions between the covariates. Results indicated the latent and manifest covariates influenced latent class membership but did not have much influence on class ability means or class proportions. The influence was relatively higher for latent covariates compared to manifest covariates. The effects of the covariates on class membership and on item parameters were class specific. Substantial effects of covariates on item parameters yielded smaller standard errors for item parameter estimates. A significant interaction term also had an effect on the coefficients for predicting and explaining latent class membership.

Applications of Mixture IRT Models: A Literature Review

ABSTRACTMixture item response theory (MixIRT) models combine IRT models with latent class model and assume that there exist latent subpopulations in the data. Identification of latent subpopulations via MixIRT models produces more detailed information. Detailed information about the response processing of examinees provides a better understanding of the construct being measured by the test. These features have enabled the MixIRT models to be used in many studies. Applications of MixIRT models have included item analysis, differential item functioning, multilevel analyses, detection of test speededness, identification of different personality styles and solution strategies. We reviewed each of these applications along with other MixIRT model applications in this study. Results showed reporting practices and trends in MixIRT applications. We provided possible explanations for these results and suggested how to advance the application and usage of MixIRT models.

Are We Underestimating Food Insecurity? Partial Identification with a Bayesian 4-Parameter IRT Model

This paper addresses measurement error in food security in the USA. In particular, it uses a Bayesian 4-parameter IRT model to look at the likelihood of over- or under-reporting of the conditions that comprise the food security module (FSM), the data collection administered in many US surveys to assess and monitor food insecurity. While this model’s parameters are only partially identified, we learn about the likely values of these parameters by using a Bayesian framework. My results suggest significant under-reporting of more severe food security items, particularly those in the child module. I find no evidence of over-reporting of food hardships. I show that, under conservative assumptions, this model predicts food insecurity prevalence between 1 and 3 percentage points higher than current estimates, or roughly 4 to 15 percent of prevalence, for the years 2007–2015. Results suggest much larger increases—on the order of 50 percent of prevalence—for very low food security among households that were screened into the food security module.

Prediction of LC-MS/MS Properties of Peptides from Sequence by Deep Learning

Deep learning models for prediction of three key LC-MS/MS properties from peptide sequences were developed. The LC-MS/MS properties or behaviors are indexed retention times (iRT), MS1 or survey scan charge state distributions, and sequence ion intensities of HCD spectra. A common core deep supervised learning architecture, bidirectional long-short term memory (LSTM) recurrent neural networks was used to construct the three prediction models. Two featurization schemes were proposed and demonstrated to allow for efficient encoding of modifications. The iRT and charge state distribution models were trained with on order of 105 data points each. An HCD sequence ion prediction model was trained with 2 × 106 experimental spectra. The iRT prediction model and HCD sequence ion prediction model provide improved accuracies over the start-of-the-art models available in literature. The MS1 charge state distribution prediction model offers excellent performance. The prediction models can be used to enhance peptide identification and quantification in data-dependent acquisition and data-independent acquisition (DIA) experiments as well as to assist MRM (multiple reaction monitoring) and PRM (parallel reaction monitoring) experiment design.

비대칭문항반응모형의 문항복잡성 타당화 가능성 연구: PISA 2012 프로세스 데이터의 응답 시간․횟수 변인을 중심으로

비대칭문항반응모형의 문항복잡성 타당화 가능성 연구: PISA 2012 프로세스 데이터의 응답 시간․횟수 변인을 중심으로

Comparative Analysis of Classical Test Theory and Item Response Theory using Chemistry Test Data

Assessment of learning involves deciding whether or not the content and objectives of education are down pat by administering quality tests. This study assesses the standard of Chemistry action take a look at and compares the item statistics generated mistreatment CTT and IRT strategies. A descriptive survey was adopted involving a sample of N=530 students. The specialised XCALIBRE 4 and ITEMAN 4 softwares were used to conduct the item analysis. Results indicate that, both the two methods commonly identified 13(32.5%) items as “problematic” and 27(67.5%) were “good”. Similarly, a significantly higher correlation exists between item statistics derived from the CTT and IRT models, [(r=-0.985,) and (r=0.801) p<0.05] for item difficulty and discrimination respectively; the study concludes that the Chemistry Achievement test used do not pass through the processes of standardisation. Secondly, CTT and IRT frameworks appeared to be effective and reliable in assessing test items as the two frameworks provide similar and comparable results. The study recommends that the teacher made Chemistry tests used in measuring students’ achievement should be made to pass through all the processes of standardisation. Meanwhile, CTT and IRT approaches of item analysis ought to be integrated within the aspects of item development and analysis because of their superiority within the investigation of reliability and minimising measurement errors.

Comparative Analysis of Classical Test Theory and Item Response Theory using Chemistry Test Data

Assessment of learning involves deciding whether or not the content and objectives of education are down pat by administering quality tests. This study assesses the standard of Chemistry action take a look at and compares the item statistics generated mistreatment CTT and IRT strategies. A descriptive survey was adopted involving a sample of N=530 students. The specialised XCALIBRE 4 and ITEMAN 4 softwares were used to conduct the item analysis. Results indicate that, both the two methods commonly identified 13(32.5%) items as “problematic” and 27(67.5%) were “good”. Similarly, a significantly higher correlation exists between item statistics derived from the CTT and IRT models, [(r=-0.985,) and (r=0.801) p<0.05] for item difficulty and discrimination respectively; the study concludes that the Chemistry Achievement test used do not pass through the processes of standardisation. Secondly, CTT and IRT frameworks appeared to be effective and reliable in assessing test items as the two frameworks provide similar and comparable results. The study recommends that the teacher made Chemistry tests used in measuring students’ achievement should be made to pass through all the processes of standardisation. Meanwhile, CTT and IRT approaches of item analysis ought to be integrated within the aspects of item development and analysis because of their superiority within the investigation of reliability and minimising measurement errors

Contextual Responses to Affirmative and/or Reversed-Worded Items

This paper presents a systematic investigation of how affirmative and polar-opposite items presented either jointly or separately affect yea-saying tendencies. We measure these yea-saying tendencies with item response models that estimate a respondent's tendency to give a "yea"-response that may be unrelated to the target trait. In a re-analysis of the Zhang et al. (PLoS ONE, 11:1-15, 2016) data, we find that yea-saying tendencies depend on whether items are presented as part of a scale that contains affirmative and/or polar-opposite items. Yea-saying tendencies are stronger for affirmative than for polar-opposite items. Moreover, presenting polar-opposite items together with affirmative items creates lower yea-saying tendencies for polar-opposite items than when presented in isolation. IRT models that do not account for these yea-saying effects arrive at a two-dimensional representation of the target trait. These findings demonstrate that the contextual information provided by an item scale can serve as a determinant of differential item functioning.

A Bayesian Random Block Item Response Theory Model for Forced-Choice Formats.

Based on the framework of testlet models, the current study suggests the Bayesian random block item response theory (BRB IRT) model to fit forced-choice formats where an item block is composed of three or more items. To account for local dependence among items within a block, the BRB IRT model incorporated a random block effect into the response function and used a Markov Chain Monte Carlo procedure for simultaneous estimation of item and trait parameters. The simulation results demonstrated that the BRB IRT model performed well for the estimation of item and trait parameters and for screening those with relatively low scores on target traits. As found in the literature, the composition of item blocks was crucial for model performance; negatively keyed items were required for item blocks. The empirical application showed the performance of the BRB IRT model was equivalent to that of the Thurstonian IRT model. The potential advantage of the BRB IRT model as a base for more complex measurement models was also demonstrated by incorporating gender as a covariate into the BRB IRT model to explain response probabilities. Recommendations for the adoption of forced-choice formats were provided along with the discussion about using negatively keyed items.

Measurement properties of the project-level Women's Empowerment in Agriculture Index.

Women's empowerment is a process that includes increases in intrinsic agency (power within); instrumental agency (power to); and collective agency (power with). We used baseline data from two studies-Targeting and Realigning Agriculture for Improved Nutrition (TRAIN) in Bangladesh and Building Resilience in Burkina Faso (BRB)-to assess the measurement properties of survey questions operationalizing selected dimensions of intrinsic, instrumental, and collective agency in the project-level Women's Empowerment in Agricultural Index (pro-WEAI). We applied unidimensional item-response models to question (item) sets to assess their measurement properties, and when possible, their cross-context measurement equivalence-a requirement of measures designed for cross-group comparisons. For intrinsic agency in the right to bodily integrity, measured with five attitudinal questions about intimate partner violence (IPV) against women, model assumptions of unidimensionality and local independence were met. Four items showed good model fit and measurement equivalence across TRAIN and BRB. For item sets designed to capture autonomy in income, intrinsic agency in livelihoods activities, and instrumental agency in: livelihoods activities, the sale or use of outputs, the use of income, and borrowing from financial services, model assumptions were not met, model fit was poor, and items generally were weakly related to the latent (unobserved) agency construct. For intrinsic and instrumental agency in livelihoods activities and for instrumental agency in the sale or use of outputs and in the use of income, items sets had similar precision along the latent-agency continuum, suggesting that similar item sets could be dropped without a loss of precision. IRT models for collective agency were not estimable because of low reported presence and membership in community groups. This analysis demonstrates the use of IRT methods to assess the measurement properties of item sets in pro-WEAI, and empowerment scales generally. Findings suggest that a shorter version of pro-WEAI can be developed that will improve its measurement properties. We recommend revisions to the pro-WEAI questionnaire and call for new measures of women's collective agency.

Illustration of Multilevel Explanatory IRT Model DIF Testing with the Creative Thinking Scale

ABSTRACTCreativity has been well studied in the past several decades, and numerous measures have been developed to assess creativity. However, validity evidence associated with each measure is often mixed. In particular, the social consequence aspect of validity has received little attention. This is partly due to the difficulty of testing for differential item functioning (DIF) within the traditional classical test theory framework, which still remains the most popular approach to assessing creativity. Hence, this study provides an example of examining differential item functioning using multilevel explanatory item response theory models. The Creative Thinking Scale was tested for DIF in a sample of 1043 10th–12th graders. Results revealed significant uniform and non‐uniform DIF for some items. Differentially functioning items are able to produce measurement bias and should be either deleted or modeled. The detailed implications for researchers and practitioners are discussed.

Combining mixture distribution and multidimensional IRTree models for the measurement of extreme response styles.

Personality constructs, attitudes and other non-cognitive variables are often measured using rating or Likert-type scales, which does not come without problems. Especially in low-stakes assessments, respondents may produce biased responses due to response styles (RS) that reduce the validity and comparability of the measurement. Detecting and correcting RS is not always straightforward because not all respondents show RS and the ones who do may not do so to the same extent or in the same direction. The present study proposes the combination of a multidimensional IRTree model with a mixture distribution item response theory model and illustrates the application of the approach using data from the Programme for the International Assessment of Adult Competencies (PIAAC). This joint approach allows for the differentiation between different latent classes of respondents who show different RS behaviours and respondents who show RS versus respondents who give (largely) unbiased responses. We illustrate the application of the approach by examining extreme RS and show how the resulting latent classes can be further examined using external variables and process data from computer-based assessments to develop a better understanding of response behaviour and RS.

Evaluation on types of invariance in studying extreme response bias with an IRTree approach.

In recent years, item response tree (IRTree) approaches have received increasing attention in the response style literature for their ability to partial out response style latent variables as well as associated item parameters. When an IRTree approach is adopted to measure extreme response styles, directional and content invariance could be assumed at the latent variable and item parameter levels. In this study, we propose to evaluate the empirical validity of these invariance assumptions by employing a general IRTree model with relaxed invariance assumptions. This would allow us to examine extreme response biases, beyond extreme response styles. With three empirical applications of the proposed evaluation, we find that relaxing some of the invariance assumptions improves the model fit, which suggests that not all assumed invariances are empirically supported. Specifically, at the latent variable level, we find reasonable evidence for directional invariance but mixed evidence for content invariance, although we also find that estimated correlations between content-specific extreme response latent variables are high, hinting at the potential presence of a general extreme response tendency. At the item parameter level, we find no directional or content invariance for thresholds and no content invariance for slopes. We discuss how the variant item parameter estimates obtained from a general IRTree model can offer useful insight to help us understand response bias related to extreme responding measured within the IRTree framework.

A Comparison of IRT Model Combinations for Assessing Fit in a Mixed Format Elementary School Science Test

Open-ended and multiple choice questions are commonly placed on the same tests; however, there is a discussion on the effects of using different item types on the test and item statistics. This study aims to compare model and item fit statistics in a mixed format test where multiple choice and constructed response items are used together. In this 25-item fourth-grade science test administered to 2351 students in 35 schools in Turkey, items are calibrated separately and concurrently utilizing different IRT models. An important aspect of this study is that the effect of the calibration method on model and item fit is investigated on real data. Firstly, while the 1-, 2-, and 3-Parameter Logistic models are utilized to calibrate the binary coded items, the Graded Response Model and the Generalized Partial Credit Model are used to calibrate the open-ended ones. Then, combinations of dichotomous and polytomous models are employed concurrently. The results based on model comparisons revealed that the combination of the 3PL and the Graded Response Model produced the best fit statistics.

Multivariate Higher-Order IRT Model and MCMC Algorithm for Linking Individual Participant Data From Multiple Studies.

Many clinical and psychological constructs are conceptualized to have multivariate higher-order constructs that give rise to multidimensional lower-order traits. Although recent measurement models and computing algorithms can accommodate item response data with a higher-order structure, there are few measurement models and computing techniques that can be employed in the context of complex research synthesis, such as meta-analysis of individual participant data or integrative data analysis. The current study was aimed at modeling complex item responses that can arise when underlying domain-specific, lower-order traits are hierarchically related to multiple higher-order traits for individual participant data from multiple studies. We formulated a multi-group, multivariate higher-order item response theory (HO-IRT) model from a Bayesian perspective and developed a new Markov chain Monte Carlo (MCMC) algorithm to simultaneously estimate the (a) structural parameters of the first- and second-order latent traits across multiple groups and (b) item parameters of the model. Results from a simulation study support the feasibility of the MCMC algorithm. From the analysis of real data, we found that a bivariate HO-IRT model with different correlation/covariance structures for different studies fit the data best, compared to a univariate HO-IRT model or other alternate models with unreasonable assumptions (i.e., the same means and covariances across studies). Although more work is needed to further develop the method and to disseminate it, the multi-group multivariate HO-IRT model holds promise to derive a common metric for individual participant data from multiple studies in research synthesis studies for robust inference and for new discoveries.

Correction for Item Response Theory Latent Trait Measurement Error in Linear Mixed Effects Models

When latent variables are used as outcomes in regression analysis, a common approach that is used to solve the ignored measurement error issue is to take a multilevel perspective on item response modeling (IRT). Although recent computational advancement allows efficient and accurate estimation of multilevel IRT models, we argue that a two-stage divide-and-conquer strategy still has its unique advantages. Within the two-stage framework, three methods that take into account heteroscedastic measurement errors of the dependent variable in stage II analysis are introduced; they are the closed-form marginal MLE, the expectation maximization algorithm, and the moment estimation method. They are compared to the naïve two-stage estimation and the one-stage MCMC estimation. A simulation study is conducted to compare the five methods in terms of model parameter recovery and their standard error estimation. The pros and cons of each method are also discussed to provide guidelines for practitioners. Finally, a real data example is given to illustrate the applications of various methods using the National Educational Longitudinal Survey data (NELS 88).

A New Perspective on the Multidimensionality of Divergent Thinking Tasks.

In the presented work, a shift of perspective with respect to the dimensionality of divergent thinking (DT) tasks is introduced moving from the question of multidimensionality across DT scores (i.e., fluency, flexibility, or originality) to the question of multidimensionality within one holistic score of DT performance (i.e., snapshot ratings of creative quality). We apply IRTree models to test whether unidimensionality assumptions hold in different task instructions for snapshot scoring of DT tests across Likert-scale points and varying levels of fluency. It was found that evidence for unidimensionality across scale points was stronger with be-creative instructions as compared to be-fluent instructions which suggests better psychometric quality of ratings when be-creative instructions are used. In addition, creative quality latent variables pertaining to low-fluency and high-fluency ideational pools shared around 50% of variance which suggests both strong overlap, and evidence for differentiation. The presented approach allows to further examine the psychometric quality of subjective ratings and to examine new questions with respect to within-item multidimensionality in DT.

ITEM DIMENSIONALITY EXPLORATION BY MEANS OF CONSTRUCT MAP AND CATEGORICAL PRINCIPAL COMPONENTS ANALYSIS

The aim of this research is to study the merits and complementarity of Construct Mapping and Categorical Principal Components Analysis as two approaches that explore the dimensionality of multiple-choice items in achievement tests. Data from the two forms of the Romanian National Assessment Tests on Science were used to explore the dimensionality of items and to identify potentially problematic items that affect the equivalence of the two parallel forms. The findings confirm that the two tests have at best partial equivalence, but while the two methods both agree on test unidimensionality, they flag in part different items as potentially problematic. The results enable researchers and practitioners to make coherent data-driven decision regarding the use of unidimensional vs multidimensional IRT models. Keywords: categorical principal components analysis, construct map, item response theory, unidimensionality.