Three-parameter Logistic Item Response Theory Research Articles

On the Choice of the Item Response Model for Scaling PISA Data: Model Selection Based on Information Criteria and Quantifying Model Uncertainty.

In educational large-scale assessment studies such as PISA, item response theory (IRT) models are used to summarize students’ performance on cognitive test items across countries. In this article, the impact of the choice of the IRT model on the distribution parameters of countries (i.e., mean, standard deviation, percentiles) is investigated. Eleven different IRT models are compared using information criteria. Moreover, model uncertainty is quantified by estimating model error, which can be compared with the sampling error associated with the sampling of students. The PISA 2009 dataset for the cognitive domains mathematics, reading, and science is used as an example of the choice of the IRT model. It turned out that the three-parameter logistic IRT model with residual heterogeneity and a three-parameter IRT model with a quadratic effect of the ability provided the best model fit. Furthermore, model uncertainty was relatively small compared to sampling error regarding country means in most cases but was substantial for country standard deviations and percentiles. Consequently, it can be argued that model error should be included in the statistical inference of educational large-scale assessment studies.

Estimating standard errors of IRT true score equating coefficients using imputed item parameters

Reporting standard errors of equating has been advocated as a standard practice when conducting test equating. The two most widely applied procedures for standard errors of equating including the bootstrap method and the delta method are either computationally intensive or confined to the derivations of complicated formulas. In the current study, a hypothetical example was used to illustrate how the multiple imputation method could be taken as an alternative procedure for obtaining the standard errors for the item response theory (IRT) true score equating coefficients in the context of the common-item nonequivalent groups equating design under the three-parameter logistic IRT model. This method makes use of multiple sets of imputed item parameter values. By using the simulated and real data, the performance of the multiple imputation method was examined and compared with that of the bootstrap and delta methods. The results indicated that the multiple imputation method performed as effectively as the bootstrap method and the delta method when using the characteristic curve methods. The multiple imputation method produced very similar results to the delta method when the moment methods were used.

Testing the developmental theory of sex differences in intelligence using latent modeling: Evidence from the TEA Ability Battery (BAT-7)

Here we test the developmental theory of sex differences in intelligence using latent modeling. The theory has been recently updated (Lynn, 2017). Based on men's greater average brain volumes and premature maturation in women, the theory predicts an intelligence advantage for women (or null sex differences) in early adolescence, but an advantage for men in late adolescence. The findings reported by Lynn are consistent with the theory, but basic measurement issues were not met. Are we taping the same construct regardless of sex and age and can we provide sound conclusions? We considered 10,335 individuals (4992 boys and 5343 girls) within an age range of 12–18 years. They completed the subtests included in the TEA Ability Battery (BAT-7). Theta scores for each subtest were estimated using a three-parameter logistic Item Response Theory model (3PL IRT). Results fitted properly to invariance criteria across age and sex. Latent estimates for the general factor of intelligence (g) were used for testing the theory. The observed findings were in agreement with the developmental theory: there were null sex differences at 12 years of age, but there was an average difference favoring boys equivalent to 5 IQ points at 18 years of age.

Using the Stan Program for Bayesian Item Response Theory.

Stan is a new Bayesian statistical software program that implements the powerful and efficient Hamiltonian Monte Carlo (HMC) algorithm. To date there is not a source that systematically provides Stan code for various item response theory (IRT) models. This article provides Stan code for three representative IRT models, including the three-parameter logistic IRT model, the graded response model, and the nominal response model. We demonstrate how IRT model comparison can be conducted with Stan and how the provided Stan code for simple IRT models can be easily extended to their multidimensional and multilevel cases.

Comment on 3PL IRT Adjustment for Guessing

Guessing behavior is an issue discussed widely with regard to multiple choice tests. Its primary effect is on number-correct scores for examinees at lower levels of proficiency. This is a systematic error or bias, which increases observed test scores. Guessing also can inflate random error variance. Correction or adjustment for guessing formulas has been applied to address some of these issues. The purpose of this research comment is to draw attention to the adjustment for guessing implicit in the three-parameter logistic item response theory model. Potential equity issues also arise with respect to this adjustment.

Alternative Matching Scores to Control Type I Error of the Mantel–Haenszel Procedure for DIF in Dichotomously Scored Items Conforming to 3PL IRT and Nonparametric 4PBCB Models

When the matching score is either less than perfectly reliable or not a sufficient statistic for determining latent proficiency in data conforming to item response theory (IRT) models, Type I error (TIE) inflation may occur for the Mantel—Haenszel (MH) procedure or any differential item functioning (DIF) procedure that matches on summed-item score, but primarily on short tests. Alternative matching scores were developed based on sufficient statistics, reliability, and explicit corrections for measurement error. Manipulated factors were tests (20, 24, 26 items), reference/focal sample sizes (1,000/1,000, 800/200), proficiency distributions (identical, means differed, variances differed, means and variances differed), and simulation technique (three-parameter logistic IRT model and four-parameter beta compound-binomial model with nonparametric nonmonotonic item-true score step functions). Outcomes were as follows: TIE of MH chi-square test at the .05 nominal level; and the bias, standard error, and root mean square error of the MH delta-DIF statistic under null-DIF conditions. Of eight categorized alternative matching scores, four scores controlled TIE as well as or better than traditional summed-item score in almost all items for all conditions: (a) estimated latent proficiency from a 3PL IRT model, (b) the sum of weighted item scores where the weight was the item— total score biserial correlation coefficient excluding the item from total score, (c) the sum of weighted item scores where the weight was the item loading on the single common factor from factor analysis of tetrachoric correlation coefficients, and (d) Kelley’s linear regressed true score estimate.

Estimating Consistency and Accuracy Indices for Multiple Classifications

This article describes procedures for estimating various indices of classification consistency and accuracy for multiple category classifications using data from a single test administration. The estimates of the classification consistency and accuracy indices are compared under three different psychometric models: the two-parameter beta binomial, four-parameter beta binomial, and three-parameter logistic IRT (item response theory) models. Using real data sets, the estimation procedures are illustrated, and the characteristics of the estimated classification indices are examined. This article also examines the behavior of the estimated classification indices as a function of the latent variable. All three components of the models (i.e., the estimated true score distributions, fitted observed score distributions, and estimated conditional error variances) appear to have considerable influence on the magnitudes of the estimated classification indices. Choosing a model in practice should be based on various considerations including the degree of model fit to the data, suitability of the model assumptions, and the computational feasibility.

Discussion

Here we test the developmental theory of sex differences in intelligence using latent modeling. The theory has been recently updated (Lynn, 2017). Based on men's greater average brain volumes and premature maturation in women, the theory predicts an intelligence advantage for women (or null sex differences) in early adolescence, but an advantage for men in late adolescence. The findings reported by Lynn are consistent with the theory, but basic measurement issues were not met. Are we taping the same construct regardless of sex and age and can we provide sound conclusions? We considered 10,335 individuals (4992 boys and 5343 girls) within an age range of 12–18 years. They completed the subtests included in the TEA Ability Battery (BAT-7). Theta scores for each subtest were estimated using a three-parameter logistic Item Response Theory model (3PL IRT). Results fitted properly to invariance criteria across age and sex. Latent estimates for the general factor of intelligence (g) were used for testing the theory. The observed findings were in agreement with the developmental theory: there were null sex differences at 12 years of age, but there was an average difference favoring boys equivalent to 5 IQ points at 18 years of age.

Optimal Item Discrimination and Maximum Information for Logistic IRT Models

Items with the highest discrimination parameter values in a logistic item response theory model do not necessarily give maximum information. This paper derives discrimination parameter values, as functions of the guessing parameter and distances between person parameters and item difficulty, that yield maximum information for the three-parameter logistic item response theory model. An upper bound for information as a function of these parameters is also derived. An algorithm is suggested for the maximum information item selection criterion for adaptive testing and is compared with a full bank search algorithm.

Some New Item Selection Criteria for Adaptive Testing

In this study some alternative item selection criteria for adaptive testing are proposed. These criteria take into account the uncertainty of the ability estimates. A general weighted information criterion is suggested of which the usual maximum information criterion and the suggested alternative criteria are special cases. A simulation study was conducted to compare the different criteria. The results showed that the likelihood weighted mean information criterion was a good alternative to the maximum information criterion. Another good alternative was a maximum information criterion with the maximum likelihood estimate of ability replaced by the Bayesian EAP estimate. An appendix discusses the interval information criterion for the two- and three-parameter logistic item response theory model.

A Simulation Study of Methods for Assessing Differential Item Functioning in Computerized Adaptive Tests

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

Irt Versus Conventional Equating Methods: A Comparative Study of Scale Stability

Scale drift for the verbal and mathematical portions of the Scholastic Aptitude Test (SAT) was investigated using linear, equipercentile and item response theory (IRT) equating methods. The linear methods investigated were the Tucker, Levine Equally Reliable and Levine Unequally Reliable models. Three IRT calibration designs were employed. These designs are referred to as (1) concurrent, (2) fixed b’s method, and (3) characteristic curve transformation method. The results of the various equating methods were compared both graphically and analytically. These results indicated that for reasonably parallel tests, linear equating methods perform adequately. However, when tests differ somewhat in content and length, methods based on the three-parameter logistic IRT model lead to greater stability of equating results. Of the conventional equating methods investigated, the Levine Equally Reliable model appears to be the most robust for the type of equating situation used in this study. The IRT method that provided the most stable equating results overall was the concurrent calibration method.