Explanatory Item Response Theory Models Research Articles

Modelling the Differences in Social and Emotional Skills with Polytomous Explanatory IRT: The Example of Assertiveness Skill

Explanatory item response theory models can simultaneously decompose the covariance between persons and items, as well as analyze items by adding item-related predictors for differences between item difficulties and/or person-related predictors for differences between individuals. In the current study, we calculated the parameter estimations regarding the skill of assertiveness according to the rating scale model and partial credit model, which are descriptive (traditional) item response theory models as well as latent regression partial credit model including only person-level predictors, and then examined the results comparatively. We used the raw score belonging to the skill of assertiveness of Türkiye belonging to the OECD Social and Emotional Skills Study, and we included gender, socioeconomic level, perceived relationships with teachers, bullying at school, sense of belonging at school, global mindedness, and test anxiety as person-level predictors. Current study findings suggests that; (1) the latent regression partial credit model produces a better data fit when compared to the rating scale model and partial credit model, and (2) sense of belonging at school, global mindedness, and socioeconomic level are significant predictors to explain the differences between persons. We discussed the current study findings in terms of the rich body of knowledge provided by explanatory item response theory and presented some suggestions.

Understanding examinees’ item responses through cognitive modeling of response accuracy and response times

Understanding the cognitive processes, skills and strategies that examinees use in testing is important for construct validity and score interpretability. Although response processes evidence has long been included as an important aspect of validity (i.e., Standards for Educational and Psychological Tests, 1999), relevant studies are often lacking, especially in large scale educational and psychological testing. An important method for studying response processes involves explanatory mathematical modeling of item responses and item response times from variables that represent sources of cognitive complexity. For many item types, examinees may differ in strategies applied to responding to items. Mixture class item response theory models can identify latent classes of examinees with different processes, skills and strategies based on their pattern of item responses. This study will illustrate the use of response times in conjunction with explanatory item response theory models and mixture models, to provide information relevant to test validity and, hence, to score interpretations.

Impact of word properties on list learning: An explanatory item analysis.

A variety of factors affect list learning performance and relatively few studies have examined the impact of word selection on these tests. This study examines the effect of both language and memory processing of individual words on list learning. Item-response data from 1,219 participants, Mage = 74.41 (SD = 7.13), Medu = 13.30 (SD = 2.72), in the Harmonized Cognitive Assessment Protocol were used. A Bayesian generalized (non)linear multilevel modeling framework was used to specify the measurement and explanatory item-response theory models. Explanatory effects on items due to learning over trials, serial position of words, and six word properties obtained through the English Lexicon Project were modeled. A two parameter logistic (2PL) model with trial-specific learning effects produced the best measurement fit. Evidence of the serial position effect on word learning was observed. Robust positive effects on word learning were observed for body-object integration while robust negative effects were observed for word frequency, concreteness, and semantic diversity. A weak negative effect of average age of acquisition and a weak positive effect for the number of phonemes in the word were also observed. Results demonstrate that list learning performance depends on factors beyond the repetition of words. Identification of item factors that predict learning could extend to a range of test development problems including translation, form equating, item revision, and item bias. In data harmonization efforts, these methods can also be used to help link tests via shared item features and testing of whether these features are equally explanatory across samples. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Polytomous Item Explanatory Item Response Theory Models.

This study investigates polytomous item explanatory item response theory models under the multivariate generalized linear mixed modeling framework, using the linear logistic test model approach. Building on the original ideas of the many-facet Rasch model and the linear partial credit model, a polytomous Rasch model is extended to the item location explanatory many-facet Rasch model and the step difficulty explanatory linear partial credit model. To demonstrate the practical differences between the two polytomous item explanatory approaches, two empirical studies examine how item properties explain and predict the overall item difficulties or the step difficulties each in the Carbon Cycle assessment data and in the Verbal Aggression data. The results suggest that the two polytomous item explanatory models are methodologically and practically different in terms of (a) the target difficulty parameters of polytomous items, which are explained by item properties; (b) the types of predictors for the item properties incorporated into the design matrix; and (c) the types of item property effects. The potentials and methodological advantages of item explanatory modeling are discussed as well.

DOES THE PATIENT HEALTH QUESTIONNAIRE MEASURE MOOD-RELATED QUALITY OF LIFE IN NURSING FACILITY RESIDENTS?

The Patient Health Questionnaire-9 (PHQ-9) is a depressive symptom questionnaire administered to nursing facility (NF) residents in the Minimum Data Set (MDS). Does the PHQ-9 measure mood-related aspects of quality of life (QoL)? We assessed the PHQ-9’s convergent validity with negative and positive mood items from Minnesota’s QoL survey, which is administered annually to a random sample of residents. We also examined if scores on both instruments were associated with various psychiatric diagnoses on the MDS. Using item response theory (IRT) models, we estimated that depressive symptoms (PHQ-9) had a correlation of 0.546 with negative mood and -0.425 with positive mood. With explanatory IRT modeling, we estimated that diagnoses of anxiety, depression, and bipolar disorder were respectively associated with 0.261, 0.339, and 0.301 (all p < 0.001) standard deviation increases in (SD) depressive symptoms, and with 0.235, 0.261, and 0.306 SD increases in negative mood (all p < 0.001), thus indicating convergent validity. For positive mood, depression and bipolar disorder had associations of similar magnitude as the other two constructs. However, anxiety disorders were not associated with lower positive mood (-0.014 SD, p = 0.636). Thus, the PHQ-9 can measure mood-related aspects of QoL. However, the PHQ-9 appears to be sensitive to relatively serious depression, whereas the Minnesota items are more sensitive to lower levels of negative mood. Also, the PHQ-9 does not measure positive mood directly. Thus, the PHQ-9 is a more limited measure of mood-related QoL than the Minnesota items.

Polytomous item explanatory IRT models with random item effects: Concepts and an application

This paper proposes three polytomous item explanatory models with random item errors in Item Response Theory (IRT), by extending the Linear Logistic Test Model with item error (LLTM + ε) approach to polytomous data. The proposed models, also regarded as polytomous random item effects models, can take the uncertainty in explanation and/or the random nature of item parameters into account for polytomous items. To develop the models, the concepts and types of polytomous random item effects are investigated and then added into the existing polytomous item explanatory models. For estimation of the proposed models with crossed random effects for polytomous data, a Bayesian inference method is adopted for data analysis. An empirical example demonstrates practical implications and applications of the proposed models to the Verbal Aggression data. The empirical findings show that the proposed models with random item errors perform better than the existing models without random item errors in terms of the goodness-of-fit and reconstructing the step difficulties and also demonstrate methodological and practical differences of the proposed models in interpreting the item property effects in each of the item location explanatory Many-Facet Rasch Model and the step difficulty explanatory Linear Partial Credit Model approaches.

Illustration of Multilevel Explanatory IRT Model DIF Testing with the Creative Thinking Scale

ABSTRACTCreativity has been well studied in the past several decades, and numerous measures have been developed to assess creativity. However, validity evidence associated with each measure is often mixed. In particular, the social consequence aspect of validity has received little attention. This is partly due to the difficulty of testing for differential item functioning (DIF) within the traditional classical test theory framework, which still remains the most popular approach to assessing creativity. Hence, this study provides an example of examining differential item functioning using multilevel explanatory item response theory models. The Creative Thinking Scale was tested for DIF in a sample of 1043 10th–12th graders. Results revealed significant uniform and non‐uniform DIF for some items. Differentially functioning items are able to produce measurement bias and should be either deleted or modeled. The detailed implications for researchers and practitioners are discussed.

Is creativity domain specific or domain general? Evidence from multilevel explanatory item response theory models

Creativity, as one of the key 21st century skills, has become increasingly important. Yet despite the huge volume of research on creativity in the past 60 years, a fundamental debate about the nature of creativity still remains unsolved: Is creativity domain specific or domain general? In the present study, multilevel explanatory item response theory models were used to analyze 359 undergraduate and graduate students’ participation and achievements in a wide range of creative activities in six distinct domains. Results suggested that creativity was relatively domain general rather than domain specific. The findings will provide valuable insights for researchers and educators in issues related to creativity assessment and efforts to fulfill students’ creative potential in schools.

Explanatory item response theory models: Theory and application

Explanatory item response theory models: Theory and application

Individual differences in addition strategy choice: A psychometric evaluation.

The strategy choice model (SCM) is a highly influential theory of human problem-solving. One strength of this theory is the allowance for both item and person variance to contribute to problem-solving outcomes, but this central tenet of the model has not been empirically tested. Explanatory item response theory (EIRT) provides an ideal approach to testing this core feature of SCM, as it allows for simultaneous estimation of both item and person effects on problem-solving outcomes. We used EIRT to test and confirm this central tenet of the SCM for adolescents’ (n = 376) solving of addition problems. The approach also allowed us to identify the strategy choices of adolescents who still struggle with basic arithmetic. The synthesis of SCM theory and EIRT modeling has implications for more fully investigating the sources of individual differences in students’ problem solving, and for identifying problem-solving patterns associated with poor academic achievement. (PsycINFO Database Record (c) 2019 APA, all rights reserved)

An explanatory item response theory method for alleviating the cold-start problem in adaptive learning environments.

Electronic learning systems have received increasing attention because they are easily accessible to many students and are capable of personalizing the learning environment in response to students' learning needs. To that end, using fast and flexible algorithms that keep track of the students' ability change in real time is desirable. Recently, the Elo rating system (ERS) has been applied and studied in both research and practical settings (Brinkhuis & Maris, 2009; Klinkenberg, Straatemeier, & van der Maas in Computers & Education, 57, 1813-1824, 2011). However, such adaptive algorithms face the cold-start problem, defined as the problem that the system does not know a new student's ability level at the beginning of the learning stage. The cold-start problem may also occur when a student leaves the e-learning system for a while and returns (i.e., a between-session period). Because external effects could influence the student's ability level during the period, there is again much uncertainty about ability level. To address these practical concerns, in this study we propose alternative approaches to cold-start issues in the context of the e-learning environment. Particularly, we propose making the ERS more efficient by using an explanatory item response theory modeling to estimate students' ability levels on the basis of their background information and past trajectories of learning. A simulation study was conducted under various conditions, and the results showed that the proposed approach substantially reduces ability estimation errors. We illustrate the approach using real data from a popular learning platform.

Longitudinal measurement invariance and explanatory IRT models for adolescents\u2019 oral health-related quality of life

BackgroundLongitudinal invariance is a perquisite for a valid comparison of oral health-related quality of life (OHRQoL) scores over time. Item response theory (IRT) models can assess measurement invariance and allow better estimation of the associations between predictors and latent construct. By extending IRT models, this study aimed to investigate the longitudinal invariance of the two 8-item short forms of the Child Perception Questionnaire (CPQ11–14) regression short form (RSF:8) and item-impact short form (ISF:8) and identify factors associated with adolescents’ OHRQoL and its change.MethodsAll students from S1 and S2 (equivalent to US grades 6 and 7) who were born in April 1997 and May 1997 (at age 12) from 45 randomly selected secondary schools were invited to participate in this study and followed up after 3 years. Data on the CPQ11–14 RSF:8 and CPQ11–14 ISF:8, demographics, oral health behavior and status were collected. Explanatory graded response models were fitted to both short forms of the CPQ11–14 data for assessing longitudinal invariance and factors associated with OHRQoL. The Bayesian estimation method – Monte Carlo Markov Chain (MCMC) with Gibbs sampling was adopted for parameter estimation and the credible intervals were used for inference.ResultsData from 649 children at age 12 at baseline and 415 children at age 15 at follow up were analyzed. For the 12 years old children, healthier oral health behavior, better gum status, families with both parents employed and parents’ education level were found to be associated with better OHRQoL. Four items among the 2 short forms lacked longitudinal invariance. With statistical adjustment of longitudinal invariance, OHRQoL were found improved in general over the 3 years but no predictor was associated with OHRQoL in follow-up. For those with decreased family income, their OHRQoL had worsened over 3 years.ConclusionsIRT explanatory analysis enables a more valid identification of the factors associated with OHRQoL and its changes over time. It provides important information to oral healthcare researchers and policymakers.

Role of Working Memory and Strategy-Use in Feedback Effects on children\u2019s Progression in Analogy Solving:an Explanatory Item Response Theory Account

This study contrasted the effects of tutoring, multiple try and no feedback on children’s progression in analogy solving and examined individual differences herein. Feedback that includes additional hints or explanations leads to the greatest learning gains in adults. However, children process feedback differently from adults and effective feedback likely differs between learners with different characteristics or at different stages in the learning process. In this paper multilevel explanatory item response theory models were used to examine individual differences in feedback effects in children’s performance on a computerized pretest-training-posttest assessment of analogical reasoning. The role of working memory and ability level, based on initial strategy-use, were examined in a sample of 999 5–10 year-old children who received either tutoring feedback, multiple tries or no feedback during the training sessions. The results indicate that tutoring feedback leads to the greatest performance gains; however, this was moderated by working memory and ability level. Children who initially used less advanced strategies benefited more from each type of feedback than children who used advanced strategies at pretest. Higher working memory scores were linked to greater benefit from tutoring feedback or no feedback, whereas learning gains in the multiple try condition were not related to working memory. The findings of this study contribute to the growing literature on how to personalize feedback to the learner’s instructional-needs.

Extended Mixed‐Effects Item Response Models With the MH‐RM Algorithm

A mixed‐effects item response theory (IRT) model is presented as a logical extension of the generalized linear mixed‐effects modeling approach to formulating explanatory IRT models. Fixed and random coefficients in the extended model are estimated using a Metropolis‐Hastings Robbins‐Monro (MH‐RM) stochastic imputation algorithm to accommodate for increased dimensionality due to modeling multiple design‐ and trait‐based random effects. As a consequence of using this algorithm, more flexible explanatory IRT models, such as the multidimensional four‐parameter logistic model, are easily organized and efficiently estimated for unidimensional and multidimensional tests. Rasch versions of the linear latent trait and latent regression model, along with their extensions, are presented and discussed, Monte Carlo simulations are conducted to determine the efficiency of parameter recovery of the MH‐RM algorithm, and an empirical example using the extended mixed‐effects IRT model is presented.

Visual relations children find easy and difficult to process in figural analogies.

Analogical reasoning, the ability to learn about novel phenomena by relating it to structurally similar knowledge, develops with great variability in children. Furthermore, the development of analogical reasoning coincides with greater working memory efficiency and increasing knowledge of the entities and relations present in analogy problems. In figural matrices, a classical form of analogical reasoning assessment, some features, such as color, appear easier for children to encode and infer than others, such as orientation. Yet, few studies have structurally examined differences in the difficulty of visual relations across different age-groups. This cross-sectional study of figural analogical reasoning examined which underlying rules in figural analogies were easier or more difficult for children to correctly process. School children (N = 1422, M = 7.0 years, SD = 21 months, range 4.5–12.5 years) were assessed in analogical reasoning using classical figural matrices and memory measures. The visual relations the children had to induce and apply concerned the features: animal, color, orientation, position, quantity and size. The role of age and memory span on the children's ability to correctly process each type of relation was examined using explanatory item response theory models. The results showed that with increasing age and/or greater memory span all visual relations were processed more accurately. The “what” visual relations animal, color, quantity and size were easiest, whereas the “where” relations orientation and position were most difficult. However, the “where” visual relations became relatively easier with age and increased memory efficiency. The implications are discussed in terms of the development of visual processing in object recognition vs. position and motion encoding in the ventral (“what”) and dorsal (“where”) pathways respectively.

An Explanatory Item Response Theory Approach for a Computer-Based Case Simulation Test

Problem: Practitioners working with multiple-choice tests have long utilized Item Response Theory (IRT) models to evaluate the performance of test items for quality assurance. The use of similar applications for performance tests, however, is often encumbered due to the challenges encountered in working with complicated data sets in which local calibrations alone provide a poor model fit. Purpose: The purpose of this study was to investigate whether the item calibration process for a performance test, computer-based case simulations (CCS), taken from the United States Medical Licensing Examination® (USMLE®) Step 3® examination may be improved through explanatory IRT models. It was hypothesized that explanatory IRT may help improve data modeling for performance assessment tests by allowing important predictors to be added to a conventional IRT model, which are limited to item predictors alone. Methods: The responses of 767 examinees from a six-item CCS test were modeled using the Partial Credit Model (PCM) and four explanatory model extensions, each incorporating one predictor variable of interest. Predictor variables were the examinees’ gender, the order in which examinees encountered an individual item (item sequence), the time it took each examinee to respond to each item (response time), and examinees’ ability score on the multiple-choice part of the examination. Results: Results demonstrate a superior model fit for the explanatory PCM with examinee ability score from the multiple-choice portion of Step 3. Explanatory IRT model extensions might prove useful in complex performance assessment test settings where item calibrations are often problematic due to short tests and small samples. Recommendations: Findings of this study have great value in practice and implications for researchers working with small or complicated response data. Explanatory IRT methodology not only provides a way to improve data modeling for performance assessment tests but also enhances the

A generalized longitudinal mixture IRT model for measuring differential growth in learning environments

This article describes a generalized longitudinal mixture item response theory (IRT) model that allows for detecting latent group differences in item response data obtained from electronic learning (e-learning) environments or other learning environments that result in large numbers of items. The described model can be viewed as a combination of a longitudinal Rasch model, a mixture Rasch model, and a random-item IRT model, and it includes some features of the explanatory IRT modeling framework. The model assumes the possible presence of latent classes in item response patterns, due to initial person-level differences before learning takes place, to latent class-specific learning trajectories, or to a combination of both. Moreover, it allows for differential item functioning over the classes. A Bayesian model estimation procedure is described, and the results of a simulation study are presented that indicate that the parameters are recovered well, particularly for conditions with large item sample sizes. The model is also illustrated with an empirical sample data set from a Web-based e-learning environment.

Direct Likelihood Analysis and Multiple Imputation for Missing Item Scores in Multilevel Cross-Classification Educational Data

Multiple imputation (MI) has become a highly useful technique for handling missing values in many settings. In this article, the authors compare the performance of a MI model based on empirical Bayes techniques to a direct maximum likelihood analysis approach that is known to be robust in the presence of missing observations. Specifically, they focus on handling of missing item scores in multilevel cross-classification item response data structures that may require more complex imputation techniques, and for situations where an imputation model can be more general than the analysis model. Through a simulation study and an empirical example, the authors show that MI is more effective in estimating missing item scores and produces unbiased parameter estimates of explanatory item response theory models formulated as cross-classified mixed models.

Using Explanatory Item Response Theory Modeling to Investigate Context Effects of Differential Item Functioning for Students With Disabilities

To address whether or not modifications in test administration influence item functioning for students with disabilities on a high-stakes statewide problem-solving assessment, a sample of 868 students (with and without disabilities) from 74 Georgia schools were randomly assigned to one of three testing conditions (resource guide, calculator, or standard administration). The authors describe a conceptual framework for investigating measurement invariance, specifically differential item functioning within the context of assessing students with disabilities. Specifically, we illustrate how the hierarchical generalized linear model, an explanatory model that incorporates item response models into hierarchical models in multilevel settings, can be used to explore issues of measurement invariance traditionally addressed using descriptive item response models, that is, the many-facet Rasch model. Results obtained from the two approaches are reported and discussed.

Individual differences in strategy use on division problems: Mental versus written computation.

Individual differences in strategy use (choice and accuracy) were analyzed. A sample of 362 Grade 6 students solved complex division problems under 2 different conditions. In the choice condition students were allowed to use either a mental or a written strategy. In the subsequent no-choice condition, they were required to use a written strategy. Latent class analysis showed that there were 3 subgroups of students with respect to pattern of strategy choices: primarily using a written strategy (more girls than boys); primarily using a mental strategy (more boys than girls); and using a written strategy on more difficult items but a mental strategy on the easier ones (almost no weak mathematical achievers). Strategy accuracies were analyzed with explanatory item response theory modeling. A between-subjects comparison in the choice condition showed that written strategies were usually more accurate than mental strategies, especially for the weak achievers. A within-subject comparison showed that the performance of students who used mental calculation on a particular item in the choice condition improved if they were required to use a written strategy in the no-choice condition.