Cognitive Diagnostic Computerized Adaptive Testing Research Articles

On initial item selection in cognitive diagnostic computerized adaptive testing

There has recently been much interest in computerized adaptive testing (CAT) for cognitive diagnosis. While there exist various item selection criteria and different asymptotically optimal designs, these are mostly constructed based on the asymptotic theory assuming the test length goes to infinity. In practice, with limited test lengths, the desired asymptotic optimality may not always apply, and there are few studies in the literature concerning the optimal design of finite items. Related questions, such as how many items we need in order to be able to identify the attribute pattern of an examinee and what types of initial items provide the optimal classification results, are still open. This paper aims to answer these questions by providing non-asymptotic theory of the optimal selection of initial items in cognitive diagnostic CAT. In particular, for the optimal design, we provide necessary and sufficient conditions for the Q-matrix structure of the initial items. The theoretical development is suitable for a general family of cognitive diagnostic models. The results not only provide a guideline for the design of optimal item selection procedures, but also may be applied to guide item bank construction.

Exploration of Item Selection in Dual-Purpose Cognitive Diagnostic Computerized Adaptive Testing: Based on the RRUM.

Cognitive diagnostic computerized adaptive testing (CD-CAT) can be divided into two broad categories: (a) single-purpose tests, which are based on the subject's knowledge state (KS) alone, and (b) dual-purpose tests, which are based on both the subject's KS and traditional ability level ( ). This article seeks to identify the most efficient item selection method for the latter type of CD-CAT corresponding to various conditions and various evaluation criteria, respectively, based on the reduced reparameterized unified model (RRUM) and the two-parameter logistic model of item response theory (IRT-2PLM). The Shannon entropy (SHE) and Fisher information methods were combined to produce a new synthetic item selection index, that is, the "dapperness with information (DWI)" index, which concurrently considers both KS and within one step. The new method was compared with four other methods. The results showed that, in most conditions, the new method exhibited the best performance in terms of KS estimation and the second-best performance in terms of estimation. Item utilization uniformity and computing time are also considered for all the competing methods.

High-Efficiency Response Distribution-Based Item Selection Algorithms for Short-Length Cognitive Diagnostic Computerized Adaptive Testing.

Cognitive diagnostic computerized adaptive testing (CD-CAT) purports to obtain useful diagnostic information with great efficiency brought by CAT technology. Most of the existing CD-CAT item selection algorithms are evaluated when test length is fixed and relatively long, but some applications of CD-CAT, such as in interim assessment, require to obtain the cognitive pattern with a short test. The mutual information (MI) algorithm proposed by Wang is the first endeavor to accommodate this need. To reduce the computational burden, Wang provided a simplified scheme, but at the price of scale/sign change in the original index. As a result, it is very difficult to combine it with some popular constraint management methods. The current study proposes two high-efficiency algorithms, posterior-weighted cognitive diagnostic model (CDM) discrimination index (PWCDI) and posterior-weighted attribute-level CDM discrimination index (PWACDI), by modifying the CDM discrimination index. They can be considered as an extension of the Kullback-Leibler (KL) and posterior-weighted KL (PWKL) methods. A pre-calculation strategy has also been developed to address the computational issue. Simulation studies indicate that the newly developed methods can produce results comparable with or better than the MI and PWKL in both short and long tests. The other major advantage is that the computational issue has been addressed more elegantly than MI. PWCDI and PWACDI can run as fast as PWKL. More importantly, they do not suffer from the problem of scale/sign change as MI and, thus, can be used with constraint management methods together in a straightforward manner.

The Development of CD-CAT with Polytomous Attributes

Cognitive diagnostic computerized adaptive testing(CD-CAT) purports to combine the strengths of both CAT and cognitive diagnosis. The attributes in CD-CAT have been supposed to be dichotomous, "0" represents non-master for examinee and non-measurement for item while "1" represents master for examinee and measurement for item. But recently, polytomous attribute framework has been proposed by some studies(Karelitz, 2004; de la Torre, et al., 2010; Chen, et al., 2013). Based on the conventional computerized adaptive testing for cognitive diagnosis with dichotomous attributes, this study developed a new CD-CAT with polytomous attributes, called p CD-CAT, which is adaptive to polytomous or ordered category attributes framework. During the procedure, two key parts were involved in the p CD-CAT. One was parameters estimation. Here the cognitive diagnosis model with polytomous attributes was employed to estimate the individual's polytomous knowledge states with maximum likelihood estimator(MLE) algorithm. Another was the item selection strategy. In this paper, the PAKL and PAPWKL methods based on polytomous attributes were developed and implied to select items adaptively from item pool. Three Monte Carlo simulation studies with different experimental conditions were conducted here, which mainly focused on the performance of the proposed p CD-CAT by this paper. These experimental conditions were the fixed test length CAT(15, 20 and 25 items respectively), the variable test length CAT(the post probability of knowledge state is 0.75,0.80 and 0.85 respectively) and the compare between p CD-CAT and the conventional dichotomous CD-CAT, respectively. There studies showed: The classification accuracy, test security and test efficiency under p CD-CAT were all acceptable and reasonable. The PA-KL item selection strategy had low classification, test security and test efficiency, which indicated that PA-KL was unfit to the p CD-CAT. However PA-PWKL and PA-HKL item selection strategies had high classification accuracy, test security and test efficiency. In addition, while using conventional CD-CAT to fit the p CD-CAT with polytomous attributes, the PMA(attribute pattern match ration) was less than 30% and the test security indexes(e.g. item exposure and test overlap) were poorer than the p CD-CAT. All above results indicated that the conventional CD-CAT should not be employed to fit the polytomous attributes, while the method proposed by this paper is a good choose. All in all, the p CD-CAT overcame the shortcomings stemmed from dichotomous CD-CAT, thus they might expect a good prospect and application. And it provided a kind of new methods and techniques in cognitive diagnosis, which might extended the applicable area.

Item Selection Strategies Based on Attribute Mastery Probabilities in CD-CAT

Cognitive diagnostic computerized adaptive testing(CD-CAT) is a popular mode of online testing of cognitive diagnostic assessment(CDA). The key to a CD-CAT system is the item selection strategies. Someof the popular strategies are developed based on Kullback-Leibler information(KL), Shannon entropy(SHE) toselect items in CD-CAT. Typically, during CD-CAT, thesefamiliar methods would use a cutoff point to transform the attribute mastery probabilities' provisional value to binary values, but at the initial stage, the cutoff point method may lead to a larger deviation. A method that can take advantage of the probabilistic information with regard to attributes may offer a better alternative. This paper proposed two item selection strategies based on the provisional value of the attribute mastery probabilities, as follows:(1) the first strategy, which is called as PPWKL(Posterior Probability WeightedKullback-Leibler), is based on the KL information, and it can lead to maximum difference of the sum of attribute mastery probabilities, and it is weighted bythe pattern's posterior probability as well asthe difference of the attribute mastery probabilities between the ?? and any possible latent state;(2) the PPWKL considers the fact that not all the patterns are equally likely, but overlooks the fact that the distances between different patterns and the current estimate are not all of equal importance. Therefore, the PPWKL can be weighed by the inverse of the distance between the ?? and any possible latent state, which is called as PHKL(Posterior Hybrid KullbackLeibler). Then, three simulation studies were carried out, one was the fixed length of CD-CAT, and the secondwas the variant length CD-CAT, and the last wasshort length CD-CAT.Variant item selection strategiesweretaken intoconsideration in these studies, including KL, SHE, PWKL, HKL, MI, PPWKL and PHWKL. The results were compared in terms of pattern or attribute classification correct rate, itemaverage exposure ratio, item maximum exposure ratio, item minimum exposure ratio, average test length, unused item number, number of items with exposure ratioover 20%, test overlap ratio.The simulation results indicate that:(1) the comprehensive performance of PPWKL and PHKLare better than other mentioned strategies in fixed and variant length CD-CAT; as to PHKL and MI, each has different strengths in short length CD-CAT;(2) PHKL and PPWKL can retain a good measurement accuracy, and also improve the utilization ratio of item pool.

Application of Online Calibration Technique in Computerized Adaptive Testing

Computerized Adaptive Testing(CAT) has developed rapidly in recent years,and item replenishing is essential for item bank construction and maintenance in CAT.Calibration of new items is a technical challenge in item replenishing,and the precision of which directly impacts the accuracy of the estimation of examinees' abilities.Now the online calibration technique has been commonly used to calibrate the new items.Studies on online calibration for two kinds of CATs(traditional CAT based on item response theory and cognitive diagnostic CAT(CD-CAT) based on cognitive diagnostic theory) are reviewed in the aspects of online calibration design and online calibration method.There have been relatively abundant research results for the traditional CAT,while online calibration studies of CD-CAT have just started.Future studies could further explore the comparison and combination of different online calibration designs/methods,as well as the online calibration for CD-CAT and multidimensional CAT.

A Comparison of Item Selection Methods for Controlling Exposure Rate in Cognitive Diagnostic Computerized Adaptive Testing

Item exposure rate is the utilization frequency of an item. When the exposure rate is high, examinees will likely share item content. If there are too many over-exposed items, test security hence the validity of the assessment will certainly be compromised. Furthermore, with a lot of under-exposed items having low or zero item-exposure rates, the manpower financial resources spent on item construction will be wasted the item pool construction will become more challenging. Item exposure control is, therefore, an important issue in computerized adaptive testing (CAT). Cognitive diagnostic CAT (CD-CAT) combines makes use of the strengths of cognitive diagnosis theory CAT. The system will be able to provide information on the knowledge competence of the examinees by administering fewer items than traditional assessment. Based on the a-stratified method the item eligibility method in regular CAT, the present study proposed compared the performance of six techniques, namely, (a) the item eligibility (IE) method, (b) the stratified multistage (SM) approach, (c) the stratified multistage-item eligibility (SMIE) method, (d) the restrictive threshold (RT) method, (e) the maximum modified priority index (MMPI) method, (f) the restrictive progress (RPG) method. With noting it that the SM approach is similar to the a-stratified method in item selection steps. The SM approach, however, different with the a-stratified method firstly in that it stratifies the remaining item pool based on the values of item information at the estimated attributed mastery pattern while the a-stratified method is based on the values of item discrimination parameter a. Secondly, in the SM method, the remainder item bank are stratified into a number of levels before the selection of each item, whereas in the a-stratified method, the item pool is stratified only once before the test all the examinees have the same item strata. The SMIE method combines the SM the IE method. MATLAB (R2010a) was used in the simulation experiments to write the CD-CAT code the deterministic inputs, noisy and gate (DINA) model was applied in this study. Results showed that: (a) the SM method used in CD-CAT produced widely distributed item exposure by increasing the exposure rates of most items fully utilizing the item pool but without greatly diminishing the maximum exposure rate measurement accuracy; (b) other than a few items, the exposure rates of the IE method were lower than the setting maximum exposure rate, but most items still had extremely low exposure rates hence resulting in a narrow distribution of item exposure the highest measurement precision; (c) SMIE RT methods behaved similarly in that not only could they increase the utilization frequency of the under-exposed items but they could also decrease the maximum exposure rate to a certain extent; (d) the MMPI the RPG methods performed similarly with almost evenly distributed item exposure but at the great sacrifice of the measurement precision. As a whole, the performances of different methods in the order of their measurement accuracy are IE, SM, SMIE, RT, RPG MMPI. The order in terms of their performances in exposure control is: MMPI, RPG, SMIE, RT, SM IE. All in all, the SMIE RT methods are able to balance measurement accuracy item exposure well.

Mutual Information Item Selection Method in Cognitive Diagnostic Computerized Adaptive Testing With Short Test Length

Cognitive diagnostic computerized adaptive testing (CD-CAT) purports to combine the strengths of both CAT and cognitive diagnosis. Cognitive diagnosis models aim at classifying examinees into the correct mastery profile group so as to pinpoint the strengths and weakness of each examinee whereas CAT algorithms choose items to determine those strengths and weakness as efficiently as possible. Most of the existing CD-CAT item selection algorithms are evaluated when test length is relatively long whereas several applications of CD-CAT, such as in interim assessment, require an item selection algorithm that is able to accurately recover examinees’ mastery profile with short test length. In this article, we introduce the mutual information item selection method in the context of CD-CAT and then provide a computationally easier formula to make the method more amenable in real time. Mutual information is then evaluated against common item selection methods, such as Kullback–Leibler information, posterior weighted Kullback–Leibler information, and Shannon entropy. Based on our simulations, mutual information consistently results in nearly the highest attribute and pattern recovery rate in more than half of the conditions. We conclude by discussing how the number of attributes, Q-matrix structure, correlations among the attributes, and item quality affect estimation accuracy.

The Application of the Monte Carlo Approach to Cognitive Diagnostic Computerized Adaptive Testing With Content Constraints

The Monte Carlo approach which has previously been implemented in traditional computerized adaptive testing (CAT) is applied here to cognitive diagnostic CAT to test the ability of this approach to address multiple content constraints. The performance of the Monte Carlo approach is compared with the performance of the modified maximum global discrimination index (MMGDI) method on simulations in which the only content constraint is on the number of items that measure each attribute. The results of the two simulation experiments show that (a) the Monte Carlo method fulfills all the test requirements and produces satisfactory measurement precision and item exposure results and (b) the Monte Carlo method outperforms the MMGDI method when the Monte Carlo method applies either the posterior-weighted Kullback–Leibler algorithm or the hybrid Kullback–Leibler information as the item selection index. Overall, the recovery rate of the knowledge states, the distribution of the item exposure, and the utilization rate of the item bank are improved when the Monte Carlo method is used.

Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT

Item replenishing is essential for item bank maintenance in cognitive diagnostic computerized adaptive testing (CD-CAT). In regular CAT, online calibration is commonly used to calibrate the new items continuously. However, until now no reference has publicly become available about online calibration for CD-CAT. Thus, this study investigates the possibility to extend some current strategies used in CAT to CD-CAT. Three representative online calibration methods were investigated: Method A (Stocking in Scale drift in on-line calibration. Research Rep. 88-28, 1988), marginal maximum likelihood estimate with one EM cycle (OEM) (Wainer & Mislevy In H. Wainer (ed.) Computerized adaptive testing: A primer, pp. 65–102, 1990) and marginal maximum likelihood estimate with multiple EM cycles (MEM) (Ban, Hanson, Wang, Yi, & Harris in J. Educ. Meas. 38:191–212, 2001). The objective of the current paper is to generalize these methods to the CD-CAT context under certain theoretical justifications, and the new methods are denoted as CD-Method A, CD-OEM and CD-MEM, respectively. Simulation studies are conducted to compare the performance of the three methods in terms of item-parameter recovery, and the results show that all three methods are able to recover item parameters accurately and CD-Method A performs best when the items have smaller slipping and guessing parameters. This research is a starting point of introducing online calibration in CD-CAT, and further studies are proposed for investigations such as different sample sizes, cognitive diagnostic models, and attribute-hierarchical structures.

Improving Cognitive Diagnostic Computerized Adaptive Testing by Balancing Attribute Coverage: The Modified Maximum Global Discrimination Index Method

This article proposes a new item selection method, namely, the modified maximum global discrimination index (MMGDI) method, for cognitive diagnostic computerized adaptive testing (CD-CAT). The new method captures two aspects of the appeal of an item: (a) the amount of contribution it can make toward adequate coverage of every attribute and (b) the amount of contribution it can make toward recovering the latent cognitive profile. A simulation study shows that the new method ensures adequate coverage of every attribute, which improves the validity of the test scores, and defensibility of the proposed uses of the test. Furthermore, compared with the original global discrimination index method, the MMGDI method improves the recovery rate of each attribute and of the entire cognitive profile, especially the latter. Therefore, the new method improves both the validity and reliability of the test scores from a CD-CAT program.