G-DINA Model Research Articles

A cognitive analysis of an English reading test through the G-DINA model

The development of language testing calls for more in-depth diagnostic information of language skills to provide targeted feedback for test developers and test takers as well. Cognitive diagnostic assessment is capable of evaluating language skills (cognitive attributes) involved in a language test and assessing the cognitive features of test takers. Therefore, the introduction of cognitive diagnostic assessment to language testing is of great significance. The study is aimed to make a cognitive analysis of an English reading test for first-year college students through a cognitive diagnostic model named G-DINA. Through the feedback information of cognitive analysis, the study is expected to provide scientific and targeted feedback for students and teachers, helping to improve language testing, teaching and learning.

Spectral Clustering Algorithm for Cognitive Diagnostic Assessment.

In cognitive diagnostic assessment (CDA), clustering analysis is an efficient approach to classify examinees into attribute-homogeneous groups. Many researchers have proposed different methods, such as the nonparametric method with Hamming distance, K-means method, and hierarchical agglomerative cluster analysis, to achieve the classification goal. In this paper, according to their responses, we introduce a spectral clustering algorithm (SCA) to cluster examinees. Simulation studies are used to compare the classification accuracy of the SCA, K-means algorithm, G-DINA model and its related reduced cognitive diagnostic models. A real data analysis is also conducted to evaluate the feasibility of the SCA. Some research directions are discussed in the final section.

Exploring the applicability of sequential G-DINA model for polytomous response data

Exploring the applicability of sequential G-DINA model for polytomous response data

Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured.

Although research in cognitive psychology suggests refraining from investigating cognitive skills inisolation, many cognitive diagnosis model (CDM) examples do not take hierarchical attribute structures into account. When hierarchical relationships among the attributes are not considered, CDM estimates may be biased. The current study, through simulation and real data analyses, examines the impact of different MMLE-EM approaches on the item and person parameter estimates of the G-DINA, DINA and DINO models when attributes have a hierarchical structure. A number of estimation approaches that can result from modifying either the Q-matrix or prior distribution are proposed. Impact of the proposed approaches on item parameter estimation accuracy and attribute classification are investigated. For the G-DINA model estimation, the Q-matrix type (i.e, explicit vs. implicit) has greater impact than structuring the prior distribution. Specifically, explicit Q-matrices result in better item parameter recovery and higher correct classification rates. In contrast, structuring the prior distribution is more influential on item and person parameter estimates for the reduced models. When prior distribution is structured, the Q-matrix type has almost no influence on item and person parameter estimates of the DINA and DINO models. We can conclude that the Q-matrix type has a significant impact on CDM estimation, especially when the estimating model is G-DINA.

Retrofitting Non-diagnostic Reading Comprehension Assessment: Application of the G-DINA Model to a High Stakes Reading Comprehension Test

ABSTRACTWith the advancement of Cognitive Diagnostic Assessment (CDA) and the pertinent statistical models, different domains of large-scale testing and assessment have been examined for the sake of reporting more diagnostic information. Applying the generalized deterministic input, noisy, “and” gate (G-DINA) model, the current study analyzed a high-stakes L2 reading comprehension test as an integral section of the PhD degree’s entrance exam in Iran. It aimed at examining the reading comprehension attributes underlying this high stakes test in an attempt to check the capability of CDA models, (in this case G-DINA), in providing diagnostic information for test developers and users. The items’ response data were analyzed in R, “GDINA” package, version 1.4.2. With data collected from multiple sources including the current literature on the sub-skills of reading comprehension, test specifications, test-takers’ think-aloud verbal protocols, and expert panel’s judgments, an initial Q-matrix, including five sub-skills, was developed and then validated. Data analysis, using the validated version of Q-matrix, showed the sub-skill (attribute) prevalence and its difficulty. Code-related sub-skills were the easiest and the most prevalent ones and the connecting/synthesizing sub-skills were the most difficult and the least prevalent ones for the test-takers. Skill mastery profile results verified the relationship among the sub-skills of reading skill and also the prominence of some of these sub-skills over others.

Category-Level Model Selection for the Sequential G-DINA Model

Solving a constructed-response item usually requires successfully performing a sequence of tasks. Each task could involve different attributes, and those required attributes may be “condensed” in various ways to produce the responses. The sequential generalized deterministic input noisy “and” gate model is a general cognitive diagnosis model (CDM) for graded response items of this type. Although a host of dichotomous CDMs with different condensation rules can be used to parameterize the success probability of each task, specifying the most appropriate one remains challenging. If the CDM specified for each task is not in accordance with the underlying cognitive processes, the validity of the inference could be questionable. This study aims to evaluate whether several hypothesis tests, namely, the Wald test using various variance–covariance matrices, the likelihood ratio (LR) test, and the LR test using approximated parameters, can be used to select the appropriate CDMs for each task of graded response items. Simulation studies are conducted to examine the Type I error and power of the hypothesis tests under varied conditions. A data set from the Trends in International Mathematics and Science Study 2007 mathematics assessment is analyzed as an illustration.

Cognitive Diagnostic Research on Chinese Students’ English Listening Skills and Implications on Skill Training

By analyzing the test data of 2718 secondary school students in Guangzhou China on 15 listening items from Guangzhou English Achievement Examination (2015) through G-DINA model, the study explored the relationships among the listening comprehension skills. Based on the test specifications and listening skill taxonomies in existence, 5 experts in language skills and language testing conducted item content analysis independently for the 15 listening items, defined 5 listening attributes, and constructed the Q-matrix. After analyzing latent classes and their posterior probabilities, the study discovered the relationship among the listening skills. According to the listening skill relationship, the study provides insights on the sequence of listening skill training. The efficiency of training may be improved when closely related listening skills are instructed and practiced at the same time. The study also demonstrates that the compensatory and saturated G-DINA model caters to the characteristics of listening comprehension skills and can be applied to tests involving highly interactive and hierarchical skills.

Estimation of Generalized DINA Model with Order Restrictions

Cognitive diagnostic models provide valuable information on whether a student has mastered each of the attributes a test intends to evaluate. Despite its generality, the generalized DINA model allows for the possibility of lower correct rates for students who master more attributes than those who know less. This paper considers the use of order-constrained parameter space of the G-DINA model to avoid such a counter-intuitive phenomenon and proposes two algorithms, the upward and downward methods, for parameter estimation. Through simulation studies, we compare the accuracy in parameter estimation and in classification of attribute patterns obtained from the proposed two algorithms and the current approach when the restricted parameter space is true. Our results show that the upward method performs the best among the three, and therefore it is recommended for estimation, regardless of the distribution of respondents’ attribute patterns, types of test items, and the sample size of the data.

Evaluation of Model Fit in Cognitive Diagnosis Models

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, ABS(fcor) and MAX(χ2jj′)). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.

The selection of cognitive diagnostic models for a reading comprehension test

Cognitive diagnostic models (CDMs) have great promise for providing diagnostic information to aid learning and instruction, and a large number of CDMs have been proposed. However, the assumptions and performances of different CDMs and their applications in regard to reading comprehension tests are not fully understood. In the present study, we compared the performance of a saturated model (G-DINA), two compensatory models (DINO, ACDM), and two non-compensatory models (DINA, RRUM) with the Michigan English Language Assessment Battery (MELAB) reading test. Compared to the saturated G-DINA model, the ACDM showed comparable model fit and similar skill classification results. The RRUM was slightly worse than the ACDM and G-DINA in terms of model fit and classification results, whereas the more restrictive DINA and DINO performed much worse than the other three models. The findings of this study highlighted the process and considerations pertinent to model selection in applications of CDMs with reading tests.

Evaluating the Wald Test for Item‐Level Comparison of Saturated and Reduced Models in Cognitive Diagnosis

This article used the Wald test to evaluate the item‐level fit of a saturated cognitive diagnosis model (CDM) relative to the fits of the reduced models it subsumes. A simulation study was carried out to examine the Type I error and power of the Wald test in the context of the G‐DINA model. Results show that when the sample size is small and a larger number of attributes are required, the Type I error rate of the Wald test for the DINA and DINO models can be higher than the nominal significance levels, while the Type I error rate of the A‐CDM is closer to the nominal significance levels. However, with larger sample sizes, the Type I error rates for the three models are closer to the nominal significance levels. In addition, the Wald test has excellent statistical power to detect when the true underlying model is none of the reduced models examined even for relatively small sample sizes. The performance of the Wald test was also examined with real data. With an increasing number of CDMs from which to choose, this article provides an important contribution toward advancing the use of CDMs in practical educational settings.

A General Cognitive Diagnosis Model for Expert-Defined Polytomous Attributes

Polytomous attributes, particularly those defined as part of the test development process, can provide additional diagnostic information. The present research proposes the polytomous generalized deterministic inputs, noisy, “and” gate (pG-DINA) model to accommodate such attributes. The pG-DINA model allows input from substantive experts to specify attribute levels and is a general model that subsumes various reduced models. In addition to model formulation, the authors evaluate the viability of the proposed model by examining how well the model parameters can be estimated under various conditions, and compare its classification accuracy against that of the conventional G-DINA model with a modified classification rule. A real-data example is used to illustrate the application of the model in practice.

Model Data Fit Comparison between DINA and G-DINA in Cognitive Diagnostic Models

In this study, item and model data fit indices, calculated by DINA and G-DINA Models using the same sample and Q matrix, are analyzed. Fit indices for these two models from Cognitive Diagnostic Models are analyzed using 2LL, AIC and BIC statistics. Item fit indices are analyzed using residual correlations and probabilities. Analysis results showed G-DINA model had better fit results than DINA model. DINA model could give rather better results to estimate student profile in tests where higher level and progressive behaviors are used together. On the other hand, G-DINA model weights required attributes for an item when estimating student profile. Therefore in items requiring more than one attributes, contributions of attributes to probability that a student answers the item correctly are not equal. This provides an important advantage to testers to evaluate multiple choice items in assessing complex and prerequisite forming patterns.