IRT Models Research Articles

Development and validation of composite scores for language and visuospatial functioning in ADNI

AbstractBackgroundComposite scores may be useful to summarize overall language or visuospatial functioning. We generated composite scores for these domains from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and evaluated their validity.MethodWe used item response theory to derive composite measures for language (ADNI‐Lan) and visuospatial functioning (ADNI‐VS). We performed multiple validation analyses comparing ADNI‐Lan and ADNI‐VS to individual test and global scores among participants with normal cognition (NC), mild cognitive impairment (MCI), and Alzheimer’s disease (AD). We evaluated ability to detect change, strength of association with a priori selected structural brain imaging findings, association with risk of AD conversion for people with MCI, and differences in intercepts and slopes for people with MCI with vs. without the CSF AD biomarker signature.ResultThere were 1,913 people with imaging data and age ³60. Single factor IRT models fit the data well for both composites. ADNI‐Lan outperformed MMSE language items, ADAS‐Cog language items, the Boston Naming Test (BNT), and fluency items in detecting change for MCI and AD. The clock score was modestly better than ADNI‐VS at detecting change for MCI and AD. ADNI Lan was the best at predicting MCI to AD conversion, while the clock score did slightly better than ADNI‐VS. Strength of association of language scores with structural MRI findings are shown in Table 1. ADNI‐Lan and the BNT had stronger associations than other scores for MCI in ADNI1, and also in ADNI2/ADNIGO except for BNT/temporal pole. For AD, ADNI‐Lan had strong associations with each region, similar to other scores except MMSE. Strength of association of visuospatial scores with structural MRI findings are shown in Table 2. ADNI‐VS and the clock had similar strong associations for most regions. People with MCI with the CSF AD biomarker signature had lower language intercepts, captured best by ADNI‐Lan, and more rapid decline, also captured best by ADNI‐Lan. Intercepts were not different for either visuospatial score, and ADNI‐VS and the clock score had similar slope terms.ConclusionADNI‐Lang and ADNI‐VS had good evidence of validity and may be useful composites for language and visuospatial functioning in ADNI.

Item response theory analysis of the Clinical Dementia Rating.

The Clinical Dementia Rating (CDR) is widely used in Alzheimer's disease research studies and has well established reliability and validity. To facilitate the development of an online, electronic CDR (eCDR) for more efficient clinical applications, this study aims to produce a shortened version of the CDR, and to develop the statistical model for automatic scoring. Item response theory (IRT) was used for item evaluation and model development. An automatic scoring algorithm was validated using existing CDR global and domain box scores as the reference standard. Most CDR items discriminate well at mild and very mild levels of cognitive impairment. The bi-factor IRT model fits best and the shortened CDR still demonstrates very high classification accuracy (81%∼92%). The shortened version of the CDR and the automatic scoring algorithm has established a good foundation for developing an eCDR and will ultimately improve the efficiency of cognitive assessment.

Item response theory and its applications in educational measurement Part I: Item response theory and its implementation in R

AbstractItem response theory (IRT) is a class of latent variable models, which are used to develop educational and psychological tests (e.g., standardized tests, personality tests, tests for licensure, and certification). We review the theory and practices of IRT across two articles. In Part 1, we provide a broad range of topics such as foundations of educational measurement, basics of IRT, and applications of IRT using R. We focus particularly on the topics that the mirt package covers. These include unidimensional and multidimensional IRT models for dichotomous and polytomous items with continuous and discrete factors, confirmatory analysis and multigroup analysis in IRT, and estimation algorithms. In Part 2, on the other hand, we focus on more practical aspects of IRT, namely scoring, scaling, and equating.This article is categorized under: Applications of Computational Statistics > Psychometrics Software for Computational Statistics > Software/Statistical Software

Development of a Multidimensional Computerized Adaptive Test based on the Bifactor Model

The purpose of this study was to conduct a real-time multidimensional computerized adaptive test (MCAT) using data from a previous paper-pencil test (PPT) regarding the grammar and vocabulary dimensions of an end-of-term proficiency exam conducted on students in a preparatory class at a university. An item pool was established through four separate 50-item sets applied in four different semesters. The fit between unidimensional, multi-unidimensional and bifactor IRT models was compared during item calibration, with the bifactor model providing the best fit for all data sets. This was followed by a hybrid simulation for 36 conditions obtained using six item selection methods, two ability estimation methods and three termination rules. The statistics and graphs obtained indicate D-rule item selection, maximum a posteriori (MAP) ability estimation and standard error termination rule as the best algorithm for the real-time MCAT application. With the minimum number of items to be administered determined as 10, the real-time application conducted on 99 examinees yielded an average number of items of 13.4. The PPT format proficiency exam consists of 50 items, leading to the conclusion that the examinees participating in the real-time MCAT are administered an average of 74.4% fewer items than the PPT. Additionally, 86 of the examinees answered between 10-13 items. The item pool use rate is 30%. Lastly, the correlation between the PPT scores and general trait scores of 32 examinees was calculated as .77.

Identified and unidentified cases of the fixed-effects three-parameter IRT model

Identified and unidentified cases of the fixed-effects three-parameter IRT model

Parameter Estimation Accuracy of the Effort-Moderated Item Response Theory Model Under Multiple Assumption Violations.

As low-stakes testing contexts increase, low test-taking effort may serve as a serious validity threat. One common solution to this problem is to identify noneffortful responses and treat them as missing during parameter estimation via the effort-moderated item response theory (EM-IRT) model. Although this model has been shown to outperform traditional IRT models (e.g., two-parameter logistic [2PL]) in parameter estimation under simulated conditions, prior research has failed to examine its performance under violations to the model's assumptions. Therefore, the objective of this simulation study was to examine item and mean ability parameter recovery when violating the assumptions that noneffortful responding occurs randomly (Assumption 1) and is unrelated to the underlying ability of examinees (Assumption 2). Results demonstrated that, across conditions, the EM-IRT model provided robust item parameter estimates to violations of Assumption 1. However, bias values greater than 0.20 SDs were observed for the EM-IRT model when violating Assumption 2; nonetheless, these values were still lower than the 2PL model. In terms of mean ability estimates, model results indicated equal performance between the EM-IRT and 2PL models across conditions. Across both models, mean ability estimates were found to be biased by more than 0.25 SDs when violating Assumption 2. However, our accompanying empirical study suggested that this biasing occurred under extreme conditions that may not be present in some operational settings. Overall, these results suggest that the EM-IRT model provides superior item and equal mean ability parameter estimates in the presence of model violations under realistic conditions when compared with the 2PL model.

Students\u2019 perceptions of teaching behaviour in Turkish secondary education: a Mokken Scaling of My Teacher Questionnaire

Teacher behaviour has significant impact on student learning and outcomes and determines the teaching quality in learning environments. The My Teacher Questionnaire (MTQ) has been used to assess students’ perceptions of teaching behaviour in national and international studies with well-cited outcomes. In this cross-sectional survey study, we adjusted and shortened the MTQ for diverse settings in Turkish secondary education, using the nonparametric IRT model, Mokken Scaling (MS). The sample consisted of 12,036 students (grade 9–12, age 15–19 years) involving 446 classes/teachers from 24 general public high schools in Turkey. More than half of the students (n = 6544, 54.40%) were females, while 306 students (2.5%) did not report their gender. The MS polytomous Double Monotonicity Model (DMM) was employed for scaling the individual student data. The ten selected items (MTQ10) showed a strong unidimensional structure (H = 0.61) with good internal reliability (Cronbach’s α = 0.93, Molenaar Sijtsma ρ = 0.93) and construct validity. The final structure was tested on three random data sets and convergent validity of the MTQ10 was examined using student engagement in learning. The scale MTQ10 functioned well across various groups (random samples, genders, grades, subjects). Based on these results, MTQ10 reveals strong psychometric quality for the assessment of students’ perceptions of teaching behaviour with the potential to deepen our understanding of teaching behaviours and teaching quality in Turkey.

Can High Numbers of Traits Solve the Ipsativity Issue of Forced-Choice Questionnaires?

Forced-choice questionnaires can prevent faking and other response biases typically associated with rating scales. However, the derived trait scores are often unreliable and ipsative, making inter- individual comparisons in high-stakes situations impossible. Several studies suggest that these problems vanish if the number of measured traits is high. To determine the necessary number of traits under varying sample sizes, factor loadings, and inter- trait-correlations, simulations were performed for the two most widely used scoring methods, namely the classical (ipsative) approach and Thurstonian IRT models. Results demonstrate that while especially Thurstonian IRT models perform well under ideal conditions, both methods yield insufficient reliabilities in most conditions resembling applied contexts. Moreover, not only the classical estimates but also the Thurstonian IRT estimates remain (partially) ipsative, even when the number of traits is very high (i.e., 30). This result not only questions earlier assumptions regarding the use of classical scores in high dimensional questionnaires, but it also raises doubts about validation studies on Thurstonian IRT models because correlations of (partially) ipsative scores with external criteria cannot be interpreted in a usual way.

Validating Rubric Scoring Processes: An Application of an Item Response Tree Model

ABSTRACT When rating performance assessments, raters may ascribe different scores for the same performance when rubric application does not align with the intended application of the scoring criteria. Given performance assessment score interpretation assumes raters apply rubrics as rubric developers intended, misalignment between raters’ scoring processes and the intended scoring processes may lead to invalid inferences from these scores. In an effort to standardize raters’ scoring processes, an alternative scoring method was used. With this method, rubric developers’ intended scoring processes are made explicit by requiring raters to respond to a series of selected-response statements resembling a decision tree. To determine if raters scored essays as intended using a traditional rubric and the alternative scoring method, an IRT model with a tree-like structure (IRTree) was specified to depict the intended scoring processes and fit to data from each scoring method. Results suggest raters using the alternative method may be better able to rate as intended and thus the alternative method may be a viable alternative to traditional rubric scoring. Implications of the IRTree model are discussed.

An IRT Mixture Model for Rating Scale Confusion Associated with Negatively Worded Items in Measures of Social-Emotional Learning

ABSTRACT We illustrate the application of mixture IRT models to evaluate respondent confusion due to the negative wording of certain items on a social-emotional learning (SEL) assessment. Using actual student self-report ratings on four social-emotional learning scales collected from students in grades 3–12 from CORE Districts in the state of California, we also evaluate the consequences of the potential confusion in biasing student- and school-level scores as well as the estimated correlational relationships between SEL constructs and student-level variables. Models of both full and partial confusion are examined. Our results suggest that (1) rating scale confusion due to negatively worded items does appear to be present; (2) the confusion is most prevalent at lower grade levels (third–fifth); and (3) the occurrence of confusion is positively related to both reading proficiency and ELL status, as anticipated, and consequently biases estimates of SEL correlations with these student-level variables. For these reasons, we suggest future iterations of the SEL measures use only positively oriented items.

An Evaluation of 4PL IRT and DINA Models for Estimating Pseudo-Guessing and Slipping Parameters

In an achievement test, the examinees with the required knowledge and skill on a test item are expected to answer the item correctly while the examinees with a lack of necessary information on the item are expected to give an incorrect answer. However, an examinee can give a correct answer to the multiple-choice test items through guessing or sometimes give an incorrect response to an easy item due to anxiety or carelessness. Either case may cause a bias estimation of examinee abilities and item parameters. 4PL IRT model and the DINA model can be used to mitigate these negative impacts on the parameter estimations. The current simulation study aims to compare the estimated pseudo-guessing and slipping parameters from the 4PL IRT model and the DINA model under several study conditions. The DINA model was used to simulate the datasets in the study. The study results showed that the bias of the estimated slipping and guessing parameters from both 4PL IRT and DINA models were reasonably small in general although the estimated slipping and guessing parameters were more biased when datasets were analyzed through the 4PL IRT model rather than the DINA model (i.e., the average bias for both guessing and slipping parameters = .00 from DINA model, but .08 from 4PL IRT model). Accordingly, both 4PL IRT and DINA models can be considered for analyzing the datasets contaminated with guessing and slipping effects.

Psychometric analysis of forensic examiner behavior

Forensic science often involves the comparison of crime-scene evidence to a known-source sample to determine if the evidence and the reference sample came from the same source. Even as forensic analysis tools become increasingly objective and automated, final source identifications are often left to individual examiners’ interpretation of the evidence. Each source identification relies on judgements about the features and quality of the crime-scene evidence that may vary from one examiner to the next. The current approach to characterizing uncertainty in examiners’ decision-making has largely centered around the calculation of error rates aggregated across examiners and identification tasks, without taking into account these variations in behavior. We propose a new approach using IRT and IRT-like models to account for differences among examiners and additionally account for the varying difficulty among source identification tasks. In particular, we survey some recent advances (Luby 2019a) in the application of Bayesian psychometric models, including simple Rasch models as well as more elaborate decision tree models, to fingerprint examiner behavior.

Comparison of Scale Identification Methods in Mixture IRT Models

The effects of three scale identification constraints in mixture IRT models were studied. A simulation study found no constraint effect on the mixture Rasch and mixture 2PL models, but the item anchoring constraint was the only one that worked well on selecting correct model with the mixture 3PL model.

Power Divergence Family of Statistics for Person Parameters in IRT Models.

We generalize the power divergence (PD) family of statistics to the two-parameter logistic IRT model for the purpose of constructing hypothesis tests and confidence intervals of the person parameter. The well-known score test statistic is a special case of the proposed PD family. We also prove the proposed PD statistics are asymptotically equivalent and converge in distribution to [Formula: see text]. In addition, a moment matching method is introduced to compare statistics and choose the optimal one within the PD family. Simulation results suggest that the coverage rate of the associated confidence interval is well controlled even under small sample sizes for some PD statistics. Compared to some other approaches, the associated confidence intervals exhibit smaller lengths while maintaining adequate coverage rates. The utilities of the proposed method are demonstrated by analyzing a real data set.

O10.4. THE ROLE OF CONTEXTUAL FACTORS AND ASSESSMENT STRATEGIES IN THE ACCURATE SCREENING OF PSYCHOSIS-RISK SYMPTOMS

BackgroundFalsely observing symptoms of psychosis risk in youth is a critical limitation to pluripotent prevention efforts. Factors including race, age, and measurement construction may affect the validity of semi-structured interviews and self-report screening tools designed to identify symptoms. Consideration of different constructs measured within a single screening tool, what a screening tool can offer beyond prediction of psychosis-like symptoms, and how to maximize efficiency may all influence the ability to effectively identify people who would benefit from services.MethodsHelp-seeking adolescents (N=134) ages 12–25 completed various screening tools for psychosis risk, and the Structured Interview for Psychosis-risk Syndromes (SIPS). The influence of race and age on screen scores and SIPS (CHR) symptoms were examined. We also evaluated alternatives to CHR symptom screening beyond conventional CHR screening tools.ResultsHigher Prime Screen scores increased the probability of clinician assessed symptoms among White participants, but not Black participants (interaction: b = -0.50 Wald χ2[1] = 4.63, p = .03, Exp[B] = 0.60), and younger participants were more likely to use the Prime Screen as a dichotomous (yes/no) screener relative to older participants, reducing screener efficacy (b = -0.04, Wald χ2[1] = 3.75, p = .05, Exp[B] = 0.96). Given discrepancies across groups, we attempted to expand screening strategies. First, to better understand the Prime Screen, factor analysis suggested two distinct factors that appear to measure different constructs. Additionally, an IRT model of several CHR screening tools suggested the value of an ultra-brief two-item measure. Finally, analyses of the BASC-2, a more comprehensive measure of behavior that includes, but is not limited to, psychosis risk items, also demonstrated accurate prediction of psychosis symptoms, as well as more pluripotent conceptualizations of symptoms.DiscussionResults suggest that contextual or individual factors influence the accurate identification of individuals exhibiting psychosis-risk symptoms. Tailored approaches to screening based on an awareness of context, identity, setting, and preferences of clients are possible, and customizing assessment efforts accordingly may be useful for the accurate identification of a variety of people who might benefit from services.

Generalized Discrimination Index

<p style="text-align:justify">Kelley’s Discrimination Index (DI) is a simple and robust, classical non-parametric short-cut to estimate the item discrimination power (IDP) in the practical educational settings. Unlike item–total correlation, DI can reach the ultimate values of +1 and ‒1, and it is stable against the outliers. Because of the computational easiness, DI is specifically suitable for the rough estimation where the sophisticated tools for item analysis such as IRT modelling are not available as is usual, for example, in the classroom testing. Unlike most of the other traditional indices for IDP, DI uses only the extreme cases of the ordered dataset in the estimation. One deficiency of DI is that it suits only for dichotomous datasets. This article generalizes DI to allow polytomous dataset and flexible cut-offs for selecting the extreme cases. A new algorithm based on the concept of the characteristic vector of the item is introduced to compute the generalized DI (GDI). A new visual method for item analysis, the cut-off curve, is introduced based on the procedure called exhaustive splitting.</p>

Bivariate Model for Dichotomous Responses and Latent Variables Jointly Assessing Attitude and Attitudinal Stability

In any given survey, individuals are likely to differ in attitudes toward the subject matter. They also may differ in terms of the duration and persistence of attitudes, with some persons’ beliefs being much more stable than others. For the purpose of jointly assessing attitude and temporal attitudinal stability, we propose a latent bivariate item response model. Attitudinal stability is operationalized as a construct called response consistency, which is indicated by the concordance of observed responses between two-time points. A simulation experiment assesses the parameter recovery of the proposed model. A real data analysis example uses data collected from a study on folklore beliefs about diabetes (563 individuals from multiple rural communities in North Carolina). On two different occasions, the individuals in the sample completed a 31-item common-sense model of diabetes inventory, which measures the congruence of their beliefs with a biomedical model. Results from the simulation study showed that the model parameters and factor correlation in the latent bivariate IRT model overall recovered well. Results from the real data analysis demonstrated the saliency of the construct. A weak association between having beliefs congruent with the biomedical model and response consistency across the two administrations was found.

Seeing the Forest and the Trees: Comparison of Two IRTree Models to Investigate the Impact of Full Versus Endpoint-Only Response Option Labeling

In this study, we examined the results and interpretations produced from two different IRTree models-one using paths consisting of only dichotomous decisions, and one using paths consisting of both dichotomous and polytomous decisions. We used data from two versions of an impulsivity measure. In the first version, all the response options had labels; in the second version, only the endpoints were labeled. Based on past research, we hypothesized that the endpoints would be selected more frequently in the endpoint-only labeled condition, and the midpoint response option would be selected more frequently in the fully labeled condition. Results from the two models (dichotomous and polytomous) were similar and indicated that our hypotheses were partially supported-specifically, there was no consistent pattern in terms of which condition saw a higher frequency of midpoint response selection. However, our hypotheses regarding extreme responding in the endpoint-only labeling condition were supported.

A Mixture IRTree Model for Extreme Response Style: Accounting for Response Process Uncertainty.

This paper presents a mixture item response tree (IRTree) model for extreme response style. Unlike traditional applications of single IRTree models, a mixture approach provides a way of representing the mixture of respondents following different underlying response processes (between individuals), as well as the uncertainty present at the individual level (within an individual). Simulation analyses reveal the potential of the mixture approach in identifying subgroups of respondents exhibiting response behavior reflective of different underlying response processes. Application to real data from the Students Like Learning Mathematics (SLM) scale of Trends in International Mathematics and Science Study (TIMSS) 2015 demonstrates the superior comparative fit of the mixture representation, as well as the consequences of applying the mixture on the estimation of content and response style traits. We argue that methodology applied to investigate response styles should attend to the inherent uncertainty of response style influence due to the likely influence of both response styles and the content trait on the selection of extreme response categories.

A Mixture IRTree Model for Performance Decline and Nonignorable Missing Data.

In educational assessments and achievement tests, test developers and administrators commonly assume that test-takers attempt all test items with full effort and leave no blank responses with unplanned missing values. However, aberrant response behavior-such as performance decline, dropping out beyond a certain point, and skipping certain items over the course of the test-is inevitable, especially for low-stakes assessments and speeded tests due to low motivation and time limits, respectively. In this study, test-takers are classified as normal or aberrant using a mixture item response theory (IRT) modeling approach, and aberrant response behavior is described and modeled using item response trees (IRTrees). Simulations are conducted to evaluate the efficiency and quality of the new class of mixture IRTree model using WinBUGS with Bayesian estimation. The results show that the parameter recovery is satisfactory for the proposed mixture IRTree model and that treating missing values as ignorable or incorrect and ignoring possible performance decline results in biased estimation. Finally, the applicability of the new model is illustrated by means of an empirical example based on the Program for International Student Assessment.