Item Parameter Values Research Articles

Robustness of Adaptive Measurement of Change to Item Parameter Estimation Error.

Adaptive measurement of change (AMC) is a psychometric method for measuring intra-individual change on one or more latent traits across testing occasions. Three hypothesis tests-a Z test, likelihood ratio test, and score ratio index-have demonstrated desirable statistical properties in this context, including low false positive rates and high true positive rates. However, the extant AMC research has assumed that the item parameter values in the simulated item banks were devoid of estimation error. This assumption is unrealistic for applied testing settings, where item parameters are estimated from a calibration sample before test administration. Using Monte Carlo simulation, this study evaluated the robustness of the common AMC hypothesis tests to the presence of item parameter estimation error when measuring omnibus change across four testing occasions. Results indicated that item parameter estimation error had at most a small effect on false positive rates and latent trait change recovery, and these effects were largely explained by the computerized adaptive testing item bank information functions. Differences in AMC performance as a function of item parameter estimation error and choice of hypothesis test were generally limited to simulees with particularly low or high latent trait values, where the item bank provided relatively lower information. These simulations highlight how AMC can accurately measure intra-individual change in the presence of item parameter estimation error when paired with an informative item bank. Limitations and future directions for AMC research are discussed.

Estimating standard errors of IRT true score equating coefficients using imputed item parameters

Reporting standard errors of equating has been advocated as a standard practice when conducting test equating. The two most widely applied procedures for standard errors of equating including the bootstrap method and the delta method are either computationally intensive or confined to the derivations of complicated formulas. In the current study, a hypothetical example was used to illustrate how the multiple imputation method could be taken as an alternative procedure for obtaining the standard errors for the item response theory (IRT) true score equating coefficients in the context of the common-item nonequivalent groups equating design under the three-parameter logistic IRT model. This method makes use of multiple sets of imputed item parameter values. By using the simulated and real data, the performance of the multiple imputation method was examined and compared with that of the bootstrap and delta methods. The results indicated that the multiple imputation method performed as effectively as the bootstrap method and the delta method when using the characteristic curve methods. The multiple imputation method produced very similar results to the delta method when the moment methods were used.

Comparison of Cognitive Diagnosis Models under Changing Conditions: DINA, RDINA, HODINA and HORDINA

The application of CDMs to fraction subtraction data revealed problems on the classification of examinees, latent class sizes, and the use of higher-order models. Additionally, selecting the most appropriate model assumes critical importance if there are several appropriate models available for the data. In the present study, DINA–RDINA and HODINA–HORDINA models were compared under changing conditions (i.e., number of attributes, g and s item parameter values, and number of items) with simulated and real data. The results show that for conditions where the g–s parameter values and the number of attributes were low (0.1 and 3, respectively), the reparameterized models generated values that were virtually identical to those obtained using DINA models. However, when the g–s parameter values and the number of attributes were increased (0.5 and 5, respectively), the parameter estimations obtained from the models, latent class estimates, AIC, and BIC show differences through the values from the models.

Translation of Big-Five Personality Questionnaire into Turkish and comparing it with Five Factor Personality Inventory Short Form

Objective: There is not great number of Turkish personality tests developed or adapted based on five factor model. For this reason, Big Five Personality Questionnaire (B5KT-50-Tr) was translated into Turkish. Consequently, B5KT-50-Tr and Five Factor Personality Inventory Short Form (5FPI-SF) were applied to the same group and constructs and factors of these two test were compared by various procedures. In this way we aimed to reveal the similarities and differences of two tests. Methods: The study was carried out in two phases and with two separate studies. In the first phase of the study B5KT-50-Tr was translated into Turkish. A total of 1167 people, 589 women and 578 men ranging in age from 18 to 70 participated in this study. Out of these participants 98 people took part in the test- retest study. In the second phase of the study both tests were administered and the total number of participants were 1463, 786 women and 677 men, ranging in age from 18 to 80. Results: In the translation study B5KT-50-Tr explained 35.58% of the total variance by five factors and CFA results showed model fit. Internal consistencies of the factors of the test were calculated between 0.65 and 0.79, and test-retest coefficients were between 0.55 and 0.80. As for the comparison study, B5KT-50-Tr explained 36.10% of the total variance and 5FPI-SF explained 32.11% of the total variance. According to the results of CFA both tests showed model fit at a close level to each other. Internal consistencies of the factors of B5KT-50-Tr were between 0.68 and 0.79, and for 5FPI-SF were between 0.68 and 0.85. The correlation coefficients of the factors of the two test with each other were found between 0.51 and 0.66. Although there was no difference between IRT item parameter values between two tests, mean factor scores of all factors were found different. Conclusion: Translation study show that B5KT-50-Tr has met the basic psychometric requirements. As for the comparison study, it has been observed that the tests are largely similar in terms of the features compared. The obtained results are discussed in terms of the details.

A-Stratified Computerized Adaptive Testing in the Presence of Calibration Error.

a-Stratified computerized adaptive testing with b-blocking (AST), as an alternative to the widely used maximum Fisher information (MFI) item selection method, can effectively balance item pool usage while providing accurate latent trait estimates in computerized adaptive testing (CAT). However, previous comparisons of these methods have treated item parameter estimates as if they are the true population parameter values. Consequently, capitalization on chance may occur. In this article, we examined the performance of the AST method under more realistic conditions where item parameter estimates instead of true parameter values are used in the CAT. Its performance was compared against that of the MFI method when the latter is used in conjunction with Sympson-Hetter or randomesque exposure control. Results indicate that the MFI method, even when combined with exposure control, is susceptible to capitalization on chance. This is particularly true when the calibration sample size is small. On the other hand, AST is more robust to capitalization on chance. Consistent with previous investigations using true item parameter values, AST yields much more balanced item pool usage, with a small loss in the precision of latent trait estimates. The loss is negligible when the test is as long as 40 items.

Optimal ABC inventory classification using interval programming

Inventory classification is one of the most important activities in inventory management, whereby inventories are classified into three or more classes. Several inventory classifications have been proposed in the literature, almost all of which have two main shortcomings in common. That is, the previous methods mainly rely on an expert opinion to derive the importance of the classification criteria which results in subjective classification, and they need precise item parameters before implementing the classification. While the problem has been predominantly considered as a multi-criteria, we examine the problem from a different perspective, proposing a novel optimisation model for ABC inventory classification in the form of an interval programming problem. The proposed interval programming model has two important features compared to the existing methods: it provides optimal results instead of an expert-based classification and it does not require precise values of item parameters, which are not almost always available before classification. Finally, by illustrating the proposed classification model in the form of numerical example, conclusion and suggestions for future works are presented.

Theory-Based Parameterization of Semiotics for Measuring Pre-literacy Development

A probabilistic model was applied to problem of measuring pre-literacy in young children. First, semiotic philosophy and contemporary cognition research were conceptually integrated to establish theoretical foundations for rating 14 characteristics of children's drawings and narratives (N = 120). Then ratings were transformed with a Rasch model, which estimated linear item parameter values that accounted for 79 percent of rater variance. Principle Components Analysis of item residual matrix confirmed variance remaining after item calibration was largely unsystematic. Validation analyses found positive correlations between semiotic measures and preschool literacy outcomes. Practical implications of a semiotics dimension for preschool practice were discussed.

The Influence of Item Calibration Error on Variable-Length Computerized Adaptive Testing

Variable-length computerized adaptive testing (VL-CAT) allows both items and test length to be “tailored” to examinees, thereby achieving the measurement goal (e.g., scoring precision or classification) with as few items as possible. Several popular test termination rules depend on the standard error of the ability estimate, which in turn depends on the item parameter values. However, items are chosen on the basis of their parameter estimates, and capitalization on chance may occur. In this article, the authors investigated the effects of capitalization on chance on test length and classification accuracy in several VL-CAT simulations. The results confirm that capitalization on chance occurs in VL-CAT and has complex effects on test length, ability estimation, and classification accuracy. These results have important implications for the design and implementation of VL-CATs.

The Impact of Multidirectional Item Parameter Drift on IRT Scaling Coefficients and Proficiency Estimates

Item parameter drift (IPD) occurs when item parameter values change from their original value over time. IPD may pose a serious threat to the fairness and validity of test score interpretations, especially when the goal of the assessment is to measure growth or improvement. In this study, we examined the effect of multidirectional IPD (i.e., some items become harder while other items become easier) on the linking procedure and rescaled proficiency estimates. The impact of different combinations of linking items with various multidirectional IPD on the test equating procedure was investigated for three scaling methods (mean-mean, mean-sigma, and TCC method) via a series of simulation studies. It was observed that multidirectional IPD had a substantive effect on examinees' scores and achievement level classifications under some of the studied conditions. Choice of linking method had a direct effect on the results, as did the pattern of IPD.

LINKING FOR THE GENERAL DIAGNOSTIC MODEL

ABSTRACTThree strategies for linking two consecutive assessments are investigated and compared by analyzing reading data for the National Assessment of Educational Progress (NAEP) using the general diagnostic model. These strategies are compared in terms of marginal and joint expectations of skills, joint probabilities of skill patterns, and item parameter estimates. The results indicate that fixing item parameter values at their previously calibrated values is sufficient to establish a comparable scale for the subsequent year.

Usability of Extended-matching Type Items in the Korean Medical Licensing Examinations (2002, 2003)

Purpose: In 2002, extended-matching type (R-type) items were introduced to the Korean Medical Licensing Examination. To evaluate the usability of R-type items, the results of the Korean Medical Licensing Examination in 2002 and 2003 were analyzed based on item types and knowledge levels. Methods: Item parameters, such as difficulty and discrimination indexes, were calculated using the classical test theory. The item parameters were compared across three item types and three knowledge levels. Results: The values of R-type item parameters were higher than those of A- or K-type items. There was no significant difference in item parameters according to knowledge level, including recall, interpretation, and problem solving. The reliability of R-type items exceeded 0.99. With the R-type, an increasing number in correct answers was associated with a decreasing difficulty index. Conclusion: The introduction of R-type items is favorable from the perspective of item parameters. However, an increase in the number of correct answers in pick 'n' type questions results in the items being more difficult to solve.

Theoretical and Empirical Comparisons between Two Models for Continuous Item Response

This article analyzes the relations between two continuous response models intended for typical response items: the linear congeneric model and Samejima's continuous response model (CRM). Using a factor analytical (FA) approach based on the assumption of underlying response variables, I describe how a particular case of the CRM can be considered as a nonlinear counterpart of Spearman's FA model. The mathematical relations between the: item-trait regressions, item parameter values, and conditional and marginal distributions of both models are obtained. The results allow (a) the item parameter values of the linear model to be obtained from CRM item parameter values, and (b) the conditions in which the congeneric model will be a good approximation to the CRM to be predicted. The relations described are illustrated using an empirical example and assessed by means of a simulation study.

Detection of Known Items in Adaptive Testing with a Statistical Quality Control Method

Due to previous exposure of items in adaptive testing, items may become known to a substantial portion of examinees. A disclosed item is bound to show drift in the item parameter values. In this paper it is suggested to use a statistical quality control method for the detection of known items. The method is worked out in detail for the 1-PL and 3-PL models. Adaptive test data are used to re-estimate the item parameters, and these estimates are used in a test of parameter drift. The method is illustrated in a number of simulation studies, including a power study.

Item parameter estimation via minimum logit chi‐square

The minimum logit chi‐square procedure was used in the item parameter estimation stage of the Birnbaum joint estimation paradigm. Simulated item response data for 36 data sets were analysed via this approach. Overall, the results yielded by the minimum chi‐square procedure compared very favourably with the results yielded by the LOGIST computer program for these data sets. As would be expected, test length, sample size and match of the test difficulty to the ability of the examinees all had an impact upon the level of agreement of the estimates with the underlying item parameter values. The empirical results suggest that in this context the minimum logit‐square procedure yields acceptable item parameter estimates.

A Closer Look at Latent Trait Parameter Invariance

The relationship between item parameter values obtained from independent Birnbaum (1968) model calibrations of the same item set from two different samples is examined.

Within and Among Population Item Fit with the Simple Logistic Model

The Rasch simple logistic model is used to examine the relative item parameter values of a personality inventory with respect to populations cross classified by sex and country. If the model holds across populations these values should be statistically equivalent. However, while the items conform to the model within each of the populations, half the items show significant discrepancies among the populations. These discrepancies are interpreted both statistically and substantively in terms of main effects and interactions.

Developments in Latent Trait Theory: Models, Technical Issues, and Applications

There are many shortcomings of standard testing and measurement technology.' For one, the values of standard item parameters (item difficulty and item discrimination) are not invariant across groups of examinees that differ in ability. This means that standard item statistics are only useful in test construction for examinee populations very similar to the sample of examinees in which the item statistics were obtained. Another shortcoming is that comparisons of examinees on an ability measured by a set of test items comprising a test are limited to situations where examinees are administered the same (or parallel) test items. Finally, standard testing technology provides no basis for determining what a particular examinee might do when confronted with a

A Simple Technique for Estimating Latent Trait Mental Test Parameters

A very simple and economical method for estimating initial parameter values for the normal ogive or logistic latent trait mental test model is outlined. The accuracy of the method in comparison with maximum likelihood estimation is investigated through the use of Monte-Carlo data. The study yields a number of observations concerning how sample size and true item parameter values of the data influence both estimation methods.