Examinee Parameters Research Articles

Cognitive Diagnosis Testlet Model for Multiple-Choice Items

The testlet design is very popular in educational and psychological assessments. This article proposes a new cognitive diagnosis model, the multiple-choice cognitive diagnostic testlet (MC-CDT) model for tests using testlets consisting of MC items. The MC-CDT model uses the original examinees’ responses to MC items instead of dichotomously scored data (i.e., correct or incorrect) to retain information of different distractors and thus enhance the MC items’ diagnostic power. The Markov chain Monte Carlo algorithm was adopted to calibrate the model using the WinBUGS software. Then, a thorough simulation study was conducted to evaluate the estimation accuracy for both item and examinee parameters in the MC-CDT model under various conditions. The results showed that the proposed MC-CDT model outperformed the traditional MC cognitive diagnostic model. Specifically, the MC-CDT model fits the testlet data better than the traditional model, while also fitting the data without testlets well. The findings of this empirical study show that the MC-CDT model fits real data better than the traditional model and that it can also provide testlet information.

Pengembangan Model Waktu Respon Untuk Memahami Proses Kognitif Peserta Tes

The popular measurement theory is Item Responds Theory (IRT). IRT measures the parameter of item test and examineer based on the examinee responds, true or false answer. In computer based test, beside responds of the examinee, there is another variable that should be notice, that is the time for item responds. Responds time shows how long examinee can administer item test. The information about responds time in test can be the valuable information source for test administrator , for instance when we analyze the speeded test, calibrate test items, detect about the cheating, and design a test. The responds time can also be used to measure the item parameters of a test (item difficulties and item slowness) and examinee parameters of a test (ability and speedness). Relating with those purpose, the responds time be modelled based on the basic distribution. Researcher choose the lognormal distribution as time responds distribution, because of the easy intrepretation. The responds time is dependent variable, and independent variable are item difficulty, item slowness, ability dan speedness. The method used to estimate parameters is Bayesian, Markov Chain Monte Carlo.

The Potential for Interpretational Confounding in Cognitive Diagnosis Models.

Binary examinee mastery/nonmastery classifications in cognitive diagnosis models may often be an approximation to proficiencies that are better regarded as continuous. Such misspecification can lead to inconsistencies in the operational definition of "mastery" when binary skills models are assumed. In this paper we demonstrate the potential for an interpretational confounding of the latent skills when truly continuous skills are treated as binary. Using the DINA model as an example, we show how such forms of confounding can be observed through item and/or examinee parameter change when (1) different collections of items (such as representing different test forms) previously calibrated separately are subsequently calibrated together; and (2) when structural restrictions are placed on the relationships among skill attributes (such as the assumption of strictly nonnegative growth over time), among other possibilities. We examine these occurrences in both simulation and real data studies. It is suggested that researchers should regularly attend to the potential for interpretational confounding by studying differences in attribute mastery proportions and/or changes in item parameter (e.g., slip and guess) estimates attributable to skill continuity when the same samples of examinees are administered different test forms, or the same test forms are involved in different calibrations.

Sample Size and Test Length for Item Parameter Estimate and Exam Parameter Estimate

The Item Response Theory (IRT) model contains one or more parameters in the model. These parameters are unknown, so it is necessary to predict them. This paper aims (1) to determine the sample size (N) on the stability of the item parameter (2) to determine the length (n) test on the stability of the estimate parameter examinee (3) to determine the effect of the model on the stability of the item and the parameter to examine (4) to find out Effect of sample size and test length on item stability and examinee parameter estimates (5) Effect of sample size, test length, and model on item stability and examinee parameter estimates. This paper is a simulation study in which the latent trait (q) sample simulation is derived from a standard normal population of ~ N (0.1), with a specific Sample Size (N) and test length (n) with the 1PL, 2PL and 3PL models using Wingen. Item analysis was carried out using the classical theory test approach and modern test theory. Item Response Theory and data were analyzed through software R with the ltm package. The results showed that the larger the sample size (N), the more stable the estimated parameter. For the length test, which is the greater the test length (n), the more stable the estimated parameter (q).

Farklı Yetenek Dağılımlarında Madde Güçlük Dağılımı, Test Uzunluğu ve Örneklem Büyüklüğünün İncelenmesi

This is a post-hoc simulation study which investigates the effect of different item difficulty distributions, sample sizes, and test lengths on measurement precision while estimating the examinee parameters in right and left-skewed distributions. First of all, the examinee parameters were obtained from 20-item real test results for the right-skewed and left-skewed sample groups of 500, 1000, 2500, 5000, and 10000. In the second phase of the study, four different tests were formed according to the b parameter values: normal, uniform, left skewed and right skewed distributions. A total of 80 conditions were formed within the scope of this research by selecting 20-item and 30-item condition as the test length variable. In determining the measurement precision, the RMSE and AAD values were calculated. The results were evaluated in terms of the item difficulty distributions, sample sizes, and test lengths. As a result, in right-skewed examinee distribution, the highest measurement precision was obtained at the normal b distribution and the lowest measurement precision was obtained at the right skewed b distribution. A higher measurement precision was obtained in the 30-item test, however, it was observed that the change in the sample size didn’t affect the measurement precision significantly in right-skewed examinee distribution. In the left skewed distribution, the highest measurement precision was obtained at the normal b distribution and the lowest measurement precision was obtained at the left-skewed b distribution. Also it was observed that the change in the sample size and test length didn’t affect the measurement precision significantly in the left-skewed distribution.

A Mixture Cure-Rate Model for Responses and Response Times in Time-Limit Tests.

Many large-scale standardized tests are intended to measure skills related to ability rather than the rate at which examinees can work. Time limits imposed on these tests make it difficult to distinguish between the effect of low proficiency and the effect of lack of time. This paper proposes a mixture cure-rate model approach to address this issue. Maximum likelihood estimation is proposed for parameter and variance estimation for three cases: when examinee parameters are to be estimated given precalibrated item parameters, when item parameters are to be calibrated given known examinee parameters, and when item parameters are to be estimated without assuming known examinee parameters. Large-sample properties are established for the cases under suitable regularity conditions. Simulation studies suggest that the proposed approach is appropriate for inferences concerning model parameters. In addition, not distinguishing between the effect of low proficiency and the effect of lack of time is shown to have considerable consequences for parameter estimation. A real data example is presented to demonstrate the new model. Choice of survival models for the latent power times is also discussed.

Using Neural Network Analysis to Define Methods of DINA Model Estimation for Small Sample Sizes

The DINA model is a commonly used model for obtaining diagnostic information. Like many other Diagnostic Classification Models (DCMs), it can require a large sample size to obtain reliable item and examinee parameter estimation. Neural Network (NN) analysis is a classification method that uses a training dataset for calibration. As a result, if this training dataset is determined theoretically, as was the case in Gierl’s attribute hierarchical method (AHM), the NN analysis does not have any sample size requirements. However, a NN approach does not provide traditional item parameters of a DCM or allow for item responses to influence test calibration. In this paper, the NN approach will be implemented for the DINA model estimation to explore its effectiveness as a classification method beyond its use in AHM. The accuracy of the NN approach across different sample sizes, item quality and Q-matrix complexity is described in the DINA model context. Then, a Markov Chain Monte Carlo (MCMC) estimation algorithm and Joint Maximum Likelihood Estimation is used to extend the NN approach so that item parameters associated with the DINA model are obtained while allowing examinee responses to influence the test calibration. The results derived by the NN, the combination of MCMC and NN (NN MCMC) and the combination of JMLE and NN are compared with that of the well-established Hierarchical MCMC procedure and JMLE with a uniform prior on the attribute profile to illustrate their strength and weakness.

THE FUSION MODEL FOR SKILLS DIAGNOSIS: BLENDING THEORY WITH PRACTICALITY

ABSTRACTThis paper presents the development of the fusion model skills diagnosis system (fusion model system), which can help integrate standardized testing into the learning process with both skills‐level examinee parameters for modeling examinee skill mastery and skills‐level item parameters, giving information about the diagnostic power of the test. The development of the fusion model system involves advancements in modeling, parameter estimation, model‐fitting methods, and model‐fit evaluation procedures, which are described in detail in the paper. To document the accuracy of the estimation procedure and the effectiveness of the model‐fitting and model‐fit evaluation procedures, this paper also presents a series of simulation studies. Special attention is given to evaluating the robustness of the fusion model system to violations of various modeling assumptions. The results demonstrate that the fusion model system is a promising tool for skills diagnosis that merits further research and development.

Understanding Parameter Invariance in Unidimensional IRT Models

One theoretical feature that makes item response theory (IRT) models those of choice for many psychometric data analysts is parameter invariance, the equality of item and examinee parameters from different examinee populations or measurement conditions. In this article, using the well-known fact that item and examinee parameters are identical only up to a set of linear transformations specific to the functional form of a given IRT model, violations of these transformations for unidimensional IRT models are investigated using analytical, numerical, and visual tools. Because item parameter drift (IPD) constitutes a lack of invariance (LOI) at the individual item level or item set level, the magnitudes and effects of IPD on examinee response probabilities and true scores are algebraically derived and connected to empirical results from a recent simulation study. Thus, this article facilitates a deeper understanding of the exact statistical formulation of parameter invariance as a fundamental property of latent variable measurement and explicates some practical consequences of LOI for decision making.

A Note on How to Quantify and Report Whether Irt Parameter Invariance Holds: When Pearson Correlations are Not Enough

Based on seminal work by Lord and Hambleton, Swaminathan, and Rogers, this article is an analytical, graphical, and conceptual reminder that item response theory (IRT) parameter invariance only holds for perfect model fit in multiple populations or across multiple conditions and is thus an ideal state. In practice, one attempts to quantify the degree to which a lack of invariance is likely to be present through repeated calibrations of item and examinee parameters. Motivated by two recent studies on item parameter invariance, this article shows how a seemingly intuitive measure such as Pearson’s Product-Moment Correlation Coefficient (PPMCC) is insufficient for that purpose, as it is not sensitive to restrictive linear relationships such as identities, which are required for parameter invariance to hold. It thus misses, for example, additive group-level effects, which are observed in practice with translated instruments or with large-scale assessments such as TIMSS.

An Item Response Theory Analysis of Response Stability in Personality Measurement

An item response theory model of response stability is developed, based on the local independence principle. The model predicts response changes under repeated administrations of the same instrument using item and examinee parameter estimates as predictors. Real data were used to assess how the model functioned. Results indicated that the model predictions were approximately fulfilled. Limitations of the model and the empirical study are discussed.

Likert Scaling Using Continuous, Censored, and Graded Response Models: Effects on Criterion-Related Validity

This study examined how three item response models performed when they were applied to data collected from a conventionally developed Likert-type personality scale. Each model examined isbased on a different response assumption: the multiple linear factor analysis model (continuous responses), the TOBIT factor analysis model (censored responses), and the multidimensional graded response model. The item and examinee parameters of the models were estimated using different discrepancy functions and current software implementations. Comparisons were made in terms of the goodness-of-fit of the model, parameter estimates, and criterion-related validity. Results showed that the models’ response assumptions were reasonably tenable and that the solutions obtained in all cases were similar. More complex models did not lead to substantial improvement in predictive validity.

Monte Carlo Studies in Item Response Theory

Monte carlo studies are being used in item response theory (IRT) to provide information about how validly these methods can be applied to realistic datasets (e.g., small numbers of examinees and multidimensional data). This paper describes the conditions under which monte carlo studies are appropriate in IRT-based re search, the kinds of problems these techniques have been applied to, available computer programs for gen erating item responses and estimating item and exam inee parameters, and the importance of conceptualizing these studies as statistical sampling experiments that should be subject to the same principles of experimen tal design and data analysis that pertain to empirical studies. The number of replications that should be used in these studies is also addressed.

The Rasch Poisson Counts Model for Incomplete Data: An Application of the EM Algorithm

Rasch's Poisson counts model is a latent trait model for the situation in which K tests are administered to N examinees and the test score is a count [e.g., the repeated occurrence of some event, such as the number of items completed or the number of items answered (in)correctly]. The Rasch Poisson counts model assumes that the test scores are Poisson distributed random variables. In the approach presented here, the Poisson parameter is assumed to be a product of a fixed test difficulty and a gamma-distributed random examinee latent trait parameter. From these assumptions, marginal maximum likelihood estimators can be derived for the test difficulties and the parameters of the prior gamma distribution. For the examinee parameters, there are a number of options. The model can be applied in a situa tion in which observations result from an incomplete design. When examinees are assigned to different sub sets of tests using background information, this informa tion must be taken into account when using marginal maximum likelihood estimation. If the focus is on test calibration and there is no interest in the characteristics of the latent traits in relation to the background informa tion, conditional maximum likelihood methods may be preferred because they are easier to implement and are justified for incomplete data for test parameter estima tion.

MODELING ITEM RESPONSES WHEN DIFFERENT SUBJECTS EMPLOY DIFFERENT SOLUTION STRATEGIES1

ABSTRACTA model is presented for item responses when different examinees employ different strategies to arrive at their answers, and when only those answers, not choice of strategy or subtask results, can be observed. Using substantive theory to differentiate the likelihoods of response vectors under a fixed set of solution strategies, we model responses in terms of item parameters associated with each strategy, proportions of the population employing each, and the distributions of examinee parameters within each. Posterior distributions can then be obtained for each examinee, giving the probabilities that they employed each of the strategies and their proficiency under each. The ideas are illustrated with a conceptual example about response strategies for spatial rotation items, and a numerical example resolving a population of examinees into subpopulations of valid responders and random guessers.