Standard Item Response Theory Research Articles

Validity of test score interpretations and cross-cultural comparisons in the First and Second International Science Studies

AbstractIn international large-scale assessments, student performance comparisons across educational systems are frequently done to assess the state and development in different domains. These results often have a large impact on educational policy and on the perceptions of an educational system’s performance. Early assessments, such as the First and Second International Science Studies (FISS and SISS), have been used alongside recent studies to create unique scales for investigating changes in constructs. The implicit assumptions in system comparisons are that the measures are valid, reliable, and comparable. However, these assumptions have not always been investigated thoroughly. This study aims to investigate the validity and cross-system comparability of scores from the FISS and SISS, conducted by the International Association for the Evaluation of Educational Achievement in 1970–1971 and 1983–1984. Findings based on item response theory (IRT) modeling indicate that scores in most educational systems can be viewed as reliable measures of a single science construct, supporting the validity of test score interpretations in these educational systems individually. In a robust assessment of measurement invariance using standard IRT methods, an alignment-based method, and the root mean square difference (RMSD) fit statistic, we demonstrate that measurement invariance is violated across systems. The alignment-based method identified a well-fitting model with complex restrictions but no items exhibited invariance across all systems, a result supported by the RMSD statistics. These results question the appropriateness of score comparisons across systems in FISS and SISS. We discuss the implications of these results and outline consequences for score comparisons across time.

Psychometric properties of the health literacy instrument in Brazil (HLS-EU-BR47)

BackgroundHealth literacy (HL) is a key component of health promotion and sustainability and contributes to well-being. Despite its global relevance, HL is an under-researched topic in South America but is now debuting its exploration in Brazil. To leverage its benefits for South America, the mere translation of validated tools into Portuguese is insufficient. Rather, it is necessary to examine their validity. This study aims to assess the psychometric properties of the European Health Literacy Questionnaire (HLS-EU-BR47) using the Item Response Theory (IRT) in a population-based sample of adults in Brazil.MethodsA cross-sectional online study was conducted across Brazil and included 1028 participants aged 18 years and above (80% women). Cronbach’s alpha, McDonald’s omega, factor analysis, graded responses model, Item Characteristic Curve, HL levels based on this, HL standard calculation, IRT, and regular score correlation were computed.ResultsThe instrument exhibit high reliability (Cronbach’s alpha 0.95). Factor analysis yielded one factor. IRT was appropriate for data analysis because it allowed quality evaluation of items and constructed a scale to quantify HL. The 47 items and latent features of respondents in the same unit of measurement are positioned in the construction of the HLS-EU-BR47 instrument. The percentages of individuals at each HL level, calculated using IRT, were found to be comparable to those obtained through the standard computation, e.g., 3.2% of people reported very low HL versus 10.8% inadequate HL, 56.2% reported low HL versus 39.5% problematic HL, 31.1% had moderate HL versus 30.1% sufficient HL, and 9.5% had high HL versus 19.7% with excellent HL. The mean HL scores were comparable between women and men (33.9 vs. 33.7, P = 0.36).ConclusionThis study provides new evidence of the validity of a widely used HL instrument for the population of South America (in this case, Brazil). This tool can be utilized by citizens, health professionals, and regional/national policymakers to inform the development of initiatives to assess and improve the HL of individuals, groups, and communities. Further studies are needed to confirm and extend the findings and to explore the influence of local cultures and practices in the vast Brazilian territory on HL.

State-Aware Deep Item Response Theory using student facial features.

This paper introduces a novel approach to Item Response Theory (IRT) by incorporating deep learning to analyze student facial expressions to enhance the prediction and understanding of student responses to test items. This research is based on the assertion that students' facial expressions offer crucial insights into their cognitive and affective states during testing, subsequently influencing their item responses. The proposed State-Aware Deep Item Response Theory (SAD-IRT) model introduces a new parameter, the student state parameter, which can be viewed as a relative subjective difficulty parameter. It is latent-regressed from students' facial features while solving test items using state-of-the-art deep learning techniques. In an experiment with 20 students, SAD-IRT boosted prediction performance in students' responses compared to prior models without the student state parameter, including standard IRT and its deep neural network implementation, while maintaining consistent predictions of student ability and item difficulty parameters. The research further illustrates the model's early prediction ability in predicting the student's response result before the student answered. This study holds substantial implications for educational assessment, laying the groundwork for more personalized and effective learning and assessment strategies that consider students' emotional and cognitive states.

A Nonparametric Bayesian Model for Detecting Differential Item Functioning: An Application to Political Representation in the US

AbstractA common approach when studying the quality of representation involves comparing the latent preferences of voters and legislators, commonly obtained by fitting an item response theory (IRT) model to a common set of stimuli. Despite being exposed to the same stimuli, voters and legislators may not share a common understanding of how these stimuli map onto their latent preferences, leading to differential item functioning (DIF) and incomparability of estimates. We explore the presence of DIF and incomparability of latent preferences obtained through IRT models by reanalyzing an influential survey dataset, where survey respondents expressed their preferences on roll call votes that U.S. legislators had previously voted on. To do so, we propose defining a Dirichlet process prior over item response functions in standard IRT models. In contrast to typical multistep approaches to detecting DIF, our strategy allows researchers to fit a single model, automatically identifying incomparable subgroups with different mappings from latent traits onto observed responses. We find that although there is a group of voters whose estimated positions can be safely compared to those of legislators, a sizeable share of surveyed voters understand stimuli in fundamentally different ways. Ignoring these issues can lead to incorrect conclusions about the quality of representation.

The Bayesian Covariance Structure Model for Testlets

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items. The inclusion of testlet parameters significantly increases the number of model parameters, which can lead to computational problems. To avoid these problems, a Bayesian covariance structure model (BCSM) for testlets is proposed, where standard IRT models are extended with a covariance structure model to account for dependences among testlet items. In the BCSM, the dependence among testlet items is modeled without using testlet effects. This approach does not imply any sample size restrictions and is very efficient in terms of the number of parameters needed to describe testlet dependences. The BCSM is compared to the well-known Bayesian random effects model for testlets using a simulation study. Specifically for testlets with a few items, a small number of test takers, or weak associations among testlet items, the BCSM shows more accurate estimation results than the random effects model.

InDisc: An R Package for Assessing Person and Item Discrimination in Typical-Response Measures.

InDisc is an R Package that implements procedures for estimating and fitting unidimensional Item Response Theory (IRT) Dual Models (DMs). DMs are intended for personality and attitude measures and are, essentially, extended standard IRT models with an extra person parameter that models the discriminating power of the individual. The package consists of a main function, which calls subfunctions for fitting binary, graded, and continuous responses. The program, a detailed user's guide, and an empirical example are available at no cost to the interested practitioner.

The Impact of Test and Sample Characteristics on Model Selection and Classification Accuracy in the Multilevel Mixture IRT Model.

The standard item response theory (IRT) model assumption of a single homogenous population may be violated in real data. Mixture extensions of IRT models have been proposed to account for latent heterogeneous populations, but these models are not designed to handle multilevel data structures. Ignoring the multilevel structure is problematic as it results in lower-level units aggregated with higher-level units and yields less accurate results, because of dependencies in the data. Multilevel data structures cause such dependencies between levels but can be modeled in a straightforward way in multilevel mixture IRT models. An important step in the use of multilevel mixture IRT models is the fit of the model to the data. This fit is often determined based on relative fit indices. Previous research on mixture IRT models has shown that performances of these indices and classification accuracy of these models can be affected by several factors including percentage of class-variant items, number of items, magnitude and size of clusters, and mixing proportions of latent classes. As yet, no studies appear to have been reported examining these issues for multilevel extensions of mixture IRT models. The current study aims to investigate the effects of several features of the data on the accuracy of model selection and parameter recovery. Results are reported on a simulation study designed to examine the following features of the data: percentages of class-variant items (30, 60, and 90%), numbers of latent classes in the data (with from 1 to 3 latent classes at level 1 and 1 and 2 latent classes at level 2), numbers of items (10, 30, and 50), numbers of clusters (50 and 100), cluster size (10 and 50), and mixing proportions [equal (0.5 and 0.5) vs. non-equal (0.25 and 0.75)]. Simulation results indicated that multilevel mixture IRT models resulted in less accurate estimates when the number of clusters and the cluster size were small. In addition, mean Root mean square error (RMSE) values increased as the percentage of class-variant items increased and parameters were recovered more accurately under the 30% class-variant item conditions. Mixing proportion type (i.e., equal vs. unequal latent class sizes) and numbers of items (10, 30, and 50), however, did not show any clear pattern. Sample size dependent fit indices BIC, CAIC, and SABIC performed poorly for the smaller level-1 sample size. For the remaining conditions, the SABIC index performed better than other fit indices.

Minimum Distance Estimation of Multidimensional Diffusion-Based Item Response Theory Models

Diffusion-based item response theory models are models for responses and response times on psychological tests, which can be used as measurement models in the same way as standard item response theory models (Tuerlinckx, Molenaar, & van der Maas, 2016). Their range of application, however, is narrowed by the fact that multidimensional versions of the model are not easy to fit. Marginal maximum likelihood estimation (e.g., Molenaar, Tuerlinckx, & van der Maas, 2015a) is computationally intensive and infeasible for multidimensional versions. The weighted least squares estimator of Ranger, Kuhn, and Szardenings (2016) is inefficient. Here, we propose an alternative estimator that is more efficient than the least squares estimator and less demanding than the maximum likelihood estimator. The estimator is based on minimum distance estimation and consists in modeling the sample quantiles and sample covariances. The performance of the estimator is investigated in a simulation study. The simulation study corroborates that the estimator performs well. The application of the estimator is demonstrated with real data.

Bayesian treatment of non-standard problems in test analysis

This paper extends the methods of [10] in an attempt to handle non-standard problems in test analysis. The approach is based on a Bayesian framework where test characteristics are treated as random parameters for which posterior probability assessments are available. The generality of the approach permits straightforward analyses of problems that may be difficult using standard classical test theory and standard item response theory. We first illustrate the methods on aviation test scores where the test outcomes are not dichotomous (i.e. correct and incorrect responses). Instead, the approach is modified to handle questions with answers on a five-point ordinal scale. The second problem addresses the complication of the assessment of instructors in addition to the assessment of test questions and students.

PCN244 UNBIASED TREATMENT EFFICACY DETECTION METHODS WHEN USING PATIENT REPORTED OUTCOMES IN SURVIVAL-DEPENDENT DROP-OUT

Missing patient reported outcome (PRO) data can arise from several processes. In the context of survival designs, the missingness may very well be outcome-dependent. This type of missing data is known as missing not at random (MNAR). In this circumstance, the PRO scores will produce biased treatment efficacy estimates. This bias can lead to the incorrect inference of no difference in treatment efficacy across arms. Outcome-dependent missingness poses serious challenges and requires the use of statistical approaches that properly adjust for this type of missing data. Currently, the state of the art in modeling PROs in a survival analysis context is to simultaneously model observed scores and the drop-out mechanism. This is referred to as a joint process model. However, interest in item response theory (IRT) approaches has grown in recent years. An IRT-based framework has been developed for evaluating PRO-based scores in survival analysis context. A longitudinal IRT model that incorporates the drop-out mechanism has been created. This allows the IRT scores and the drop-out mechanism to be modeled simultaneously. A simulation study based on empirical PRO data is employed here to illustrate the advantages of such an approach. The results of the simulation study show that the IRT approach to jointly model Quality of Life (QoL) scores and drop-out allows for the accurate estimation of separation of the treatment arms. Other approaches, such as a longitudinal IRT model that ignores drop-out, fail to detect separation. An IRT-based method has been developed that can better estimate the treatment effect in a survival analysis context. Notably, this approach can be implemented in standard commercial IRT software.

The Impact of Ignoring Multilevel Data Structure on the Estimation of Dichotomous Item Response Theory Models

Applying single-level statistical models to multilevel data typically produces underestimated standard errors, which may result in misleading conclusions. This study examined the impact of ignoring multilevel data structure on the estimation of item parameters and their standard errors of the Rasch, two-, and three- parameter logistic models in item response theory (IRT) to demonstrate the degree of such underestimation in IRT. Also, the Lord’s chi-square test using the underestimated standard errors was used to test differential item functioning (DIF) to show the impact of such underestimation on the practical applications of IRT. The results of simulation studies showed that, in the most severe case of multilevel data, the standard error estimate from the standard single-level IRT models was about half of the minimal asymptotic standard error, and the type I error rate of the Lord’s chi-square test was inflated up to .35. The results of this study suggest that standard single-level IRT models may seriously mislead our conclusions in the presence of multilevel data, and therefore multilevel IRT models need to be considered as alternatives.

A General Unfolding IRT Model for Multiple Response Styles.

It is commonly known that respondents exhibit different response styles when responding to Likert-type items. For example, some respondents tend to select the extreme categories (e.g., strongly disagree and strongly agree), whereas some tend to select the middle categories (e.g., disagree, neutral, and agree). Furthermore, some respondents tend to disagree with every item (e.g., strongly disagree and disagree), whereas others tend to agree with every item (e.g., agree and strongly agree). In such cases, fitting standard unfolding item response theory (IRT) models that assume no response style will yield a poor fit and biased parameter estimates. Although there have been attempts to develop dominance IRT models to accommodate the various response styles, such models are usually restricted to a specific response style and cannot be used for unfolding data. In this study, a general unfolding IRT model is proposed that can be combined with a softmax function to accommodate various response styles via scoring functions. The parameters of the new model can be estimated using Bayesian Markov chain Monte Carlo algorithms. An empirical data set is used for demonstration purposes, followed by simulation studies to assess the parameter recovery of the new model, as well as the consequences of ignoring the impact of response styles on parameter estimators by fitting standard unfolding IRT models. The results suggest the new model to exhibit good parameter recovery and seriously biased estimates when the response styles are ignored.

IRT Scoring and Test Blueprint Fidelity

This article focuses on the topic of how item response theory (IRT) scoring models reflect the intended content allocation in a set of test specifications or test blueprint. Although either an adaptive or linear assessment can be built to reflect a set of design specifications, the method of scoring is also a critical step. Standard IRT models employ a set of optimal scoring weights, and these weights depend on item parameters in the two-parameter logistic (2PL) and three-parameter logistic (3PL) models. The current article is an investigation of whether the scoring models reflect an intended set of weights defined as the proportion of item falling into each cell of the test blueprint. The 3PL model is of special interest because the optimal scoring weights depend on ability. Thus, the concern arises that for examinees of low ability, the intended weights are implicitly altered.

Parameter Estimation in Rasch Models for Examinee‐Selected Items

AbstractThe examinee‐selected‐item (ESI) design, in which examinees are required to respond to a fixed number of items in a given set of items (e.g., choose one item to respond from a pair of items), always yields incomplete data (i.e., only the selected items are answered and the others have missing data) that are likely nonignorable. Therefore, using standard item response theory models, which assume ignorable missing data, can yield biased parameter estimates so that examinees taking different sets of items to answer cannot be compared. To solve this fundamental problem, in this study the researchers utilized the specific objectivity of Rasch models by adopting the conditional maximum likelihood estimation (CMLE) and pairwise estimation (PE) methods to analyze ESI data, and conducted a series of simulations to demonstrate the advantages of the CMLE and PE methods over traditional estimation methods in recovering item parameters in ESI data. An empirical data set obtained from an experiment on the ESI design was analyzed to illustrate the implications and applications of the proposed approach to ESI data.

Mixture Item Response Models for Inattentive Responding Behavior

Inattentive responses can threaten measurement quality, yet they are common in rating- or Likert-scale data. In this study, we proposed a new mixture item response theory model to distinguish inattentive responses from normal responses so that test validity can be ascertained. Simulation studies demonstrated that the parameters of the new model were recovered fairly well using the Bayesian methods implemented in the freeware WinBUGS, and fitting the new model to data that lacked inattentive responses did not result in severely biased parameter estimates. In contrast, ignoring inattentive responses by fitting standard item response theory models to data containing inattentive responses yielded seriously biased parameter estimates and a failure to distinguish inattentive participants from normal participants; the person-fit statistic lz was also unsatisfactory in identifying inattentive responses. Two empirical examples demonstrate the applications of the new model.

Summed Score Likelihood-Based Indices for Testing Latent Variable Distribution Fit in Item Response Theory.

In standard item response theory (IRT) applications, the latent variable is typically assumed to be normally distributed. If the normality assumption is violated, the item parameter estimates can become biased. Summed score likelihood-based statistics may be useful for testing latent variable distribution fit. We develop Satorra-Bentler type moment adjustments to approximate the test statistics' tail-area probability. A simulation study was conducted to examine the calibration and power of the unadjusted and adjusted statistics in various simulation conditions. Results show that the proposed indices have tail-area probabilities that can be closely approximated by central chi-squared random variables under the null hypothesis. Furthermore, the test statistics are focused. They are powerful for detecting latent variable distributional assumption violations, and not sensitive (correctly) to other forms of model misspecification such as multidimensionality. As a comparison, the goodness-of-fit statistic M 2 has considerably lower power against latent variable nonnormality than the proposed indices. Empirical data from a patient-reported health outcomes study are used as illustration.

Non-ignorable missingness item response theory models for choice effects in examinee-selected items.

Examinee-selected item (ESI) design, in which examinees are required to respond to a fixed number of items in a given set, always yields incomplete data (i.e., when only the selected items are answered, data are missing for the others) that are likely non-ignorable in likelihood inference. Standard item response theory (IRT) models become infeasible when ESI data are missing not at random (MNAR). To solve this problem, the authors propose a two-dimensional IRT model that posits one unidimensional IRT model for observed data and another for nominal selection patterns. The two latent variables are assumed to follow a bivariate normal distribution. In this study, the mirt freeware package was adopted to estimate parameters. The authors conduct an experiment to demonstrate that ESI data are often non-ignorable and to determine how to apply the new model to the data collected. Two follow-up simulation studies are conducted to assess the parameter recovery of the new model and the consequences for parameter estimation of ignoring MNAR data. The results of the two simulation studies indicate good parameter recovery of the new model and poor parameter recovery when non-ignorable missing data were mistakenly treated as ignorable.

A Bayesian Robust IRT Outlier-Detection Model.

In psychometric practice, the parameter estimates of a standard item-response theory (IRT) model can become biased when item-response data, of persons' individual responses to test items, contain outliers relative to the model. Also, the manual removal of outliers can be a time-consuming and difficult task. Besides, removing outliers leads to data information loss in parameter estimation. To address these concerns, a Bayesian IRT model that includes person and latent item-response outlier parameters, in addition to person ability and item parameters, is proposed and illustrated, and is defined by item characteristic curves (ICCs) that are each specified by a robust, Student's t-distribution function. The outlier parameters and the robust ICCs enable the model to automatically identify item-response outliers, and to make estimates of the person ability and item parameters more robust to outliers. Hence, under this IRT model, it is unnecessary to remove outliers from the data analysis. Our IRT model is illustrated through the analysis of two data sets, involving dichotomous- and polytomous-response items, respectively.

IRT Modeling in the Presence of Zero-Inflation With Application to Psychiatric Disorder Severity.

Item response theory (IRT) has been increasingly utilized in psychiatry for the purpose of describing the relationship among items in psychiatric disorder symptom batteries hypothesized to be indicators of an underlying latent continuous trait representing the severity of the psychiatric disorder. It is common to find zero-inflated (ZI) data such that a large proportion of the sample has none of the symptoms. It has been argued that standard IRT models of psychiatric disorder symptoms may be problematic due to the unipolar nature of many clinical traits. In the current article, the authors propose to address this by using a mixture model to approximate the unknown latent trait distribution in the IRT model while allowing for the presence of a non-pathological subgroup. The basic idea is that instead of assuming normality for the underlying trait, the latent trait will be allowed to follow a mixture of normals including a degenerate component that is fixed to represent a non-pathological group for whom the psychiatric symptoms simply are not relevant and hence are all expected to be zero. The authors demonstrate how the ZI mixture IRT method can be implemented in Mplus and present a simulation study comparing its performance with a standard IRT model assuming normality under different scenarios representative of psychiatric disorder symptom batteries. The model incorrectly assuming normality is shown to have biased discrimination and severity estimates. An application further illustrates the method using data from an alcohol use disorder criteria battery.

Using Testlet Response Theory to Examine Local Dependence in C-Tests

C-tests are gap-filling tests widely used to assess general language proficiency for purposes of placement, screening, or provision of feedback to language learners. C-tests consist of several short texts in which parts of words are missing. We addressed the issue of local dependence in C-tests using an explicit modeling approach based on testlet response theory. Data were provided by a total sample of 4,708 participants working on eight C-test texts with 20 gaps each. The resulting parameter estimates were compared to those obtained from (a) a polytomous item response theory (IRT) model and (b) a standard IRT model that ignored the dependence structure. Testlet effects proved to be very small and correspondence between parameters obtained from the different modeling approaches was high. When local dependence was ignored reliability of the C-test was slightly overestimated. Implications for the analysis of testlets in general, and C-tests in particular are discussed.