DINA Models Research Articles

Sufficient and Necessary Conditions for the Identifiability of DINA Models with Polytomous Responses.

Cognitive diagnosis models (CDMs) provide a powerful statistical and psychometric tool for researchers and practitioners to learn fine-grained diagnostic information about respondents' latent attributes. There has been a growing interest in the use of CDMs for polytomous response data, as more and more items with multiple response options become widely used. Similar to many latent variable models, the identifiability of CDMs is critical for accurate parameter estimation and valid statistical inference. However, the existing identifiability results are primarily focused on binary response models and have not adequately addressed the identifiability of CDMs with polytomous responses. This paper addresses this gap by presenting sufficient and necessary conditions for the identifiability of the widely used DINA model with polytomous responses, with the aim to provide a comprehensive understanding of the identifiability of CDMs with polytomous responses and to inform future research in this field.

Investigating the 8th-grade Afghan students’ mathematics status and skills using the cognitive diagnostic model

Cognitive diagnostic models (CDMs) are multidimensional multivariate verification flow models with complex structure. In this research, these models were used to investigate the status of eighth grade high school students in mathematics using the TIMMS questionnaire. The cognitive diagnostic test based on 13 attributes including 32 questions was conducted on a sample of 274 students who were selected based on the multi-stage cluster sampling method among the students of Firuzkoh city. IRT and RESMA models were used to determine the psychometric properties of the questions. Data analysis using DINA and DINO models in cognitive diagnostic modeling of mathematics showed that 13 attributes explain the mathematical performance of eighth-grade students. The result shows that Afghan students have a weak mastery level in most attributes compared to 45 other countries (the countries that were included in the TIMMS questionnaire) also general results show that the examinees perform better in the field of numbers (0.49), while they perform worse in data and chance (0.12). Moreover, there exists some difference in estimating item parameters under the DINA and DINO models, such as Item 3 and Item 27. One possible explanation is that the DINA model is completely compensatory while the DINO model is fully non-compensatory. Similar to the results under the DINA model, the SEs of guessing parameters are lower than those of slipping parameters under the DINO model.

Investigating the status and skills of Afghan students using the Cognitive Diagnostic Model (CDM) of Afghanistan's 8th grade mathematics.

Cognitive diagnostic models (CDMs) are multidimensional multivariate verification flow models with complex structure. In this research, these models were used to investigate the status of eighth grade high school students in mathematics using the TIMMS questionnaire. The cognitive diagnostic test based on 13 attributes including 32 questions was conducted on a sample of 274 students who were selected based on the multi-stage cluster sampling method among the students of Firuzkoh city. IRT and RESMA models were used to determine the psychometric properties of the questions. Data analysis using DINA and DINO models in cognitive diagnostic mathematics modeling showed that 13 attributes explain the mathematical performance of eighth-grade students. The result shows that Afghan students have a weak mastery level in most features compared to 45 other countries (the countries that were included in the TIMMS questionnaire) also, general results show that the examinees perform better in the field of numbers (0.49), while they perform worse in data and chance (0.12 ). Moreover, some difference exists in estimating item parameters under the DINA and DINO model, such as Item 3 and Item 27. One possible explanation is that the DINA model is completely compensatory while the DINO model is fully non-compensatory. Similar to the results under the DINA model, the SEs of guessing parameters are lower than those of slipping parameters under the DINO model.

Consistency and Ability of Students Using DINA and DINO Models

There are two prevalent models for studying cognitive diagnosis Models for appraisement and diagnostics, each of these two models is the definite Input Noisy Output “and” gate and the Deterministic Input Noisy Output “or” gate models. The usage of them is to display various views of the ways cognitive skills are related and the probability of how an responds correctly. The aim of study is to compare the mentioned models and girls and boys with the modeling of cognitive diagnosis. The aims of this study to identify differences in performance of Afghan boys’ and girls’ students in the basic mathematical attributes and cognitive skills of the eighth grade in TIMMS (2011). As well as rankings among the countries that participated in TIMMS. Two commonly used CDMs were employed to fit the response data, including these two models (DINA and DINO). With the assistance of CDMs, we could obtain not only the item parameters, but also the skill profile for each student. Results show that the examinees do best in number domain while do worst in data and chance.

On the Identifiability of Cognitive Diagnostic Models: Diagnosing Students’ Translation Ability

Background. In recent years Cognitive Diagnostic Models (CDMs) have attracted a great deal of attention from researchers in a variety of educational fields. However, they have not been taken into consideration in Translation Quality Assessment (TQA), in the aims of presenting fine-grained information about the strengths and weaknesses of translation students.
 Purpose. The present study compares the ACDM, DINO, DINA, HO-DINA, and G-DINA models, in order to define the strengths and weaknesses of Iranian translation students and to examine whether the required translation attributes are compensatory, non-compensatory, additive, or hierarchical.
 Methods. 200 BA translation students translated a two-English-text translation, which was scored by three experienced translation raters using the Translation Quality Assessment Rubric (TQAR). The professional translators, established the relationships between the TQAR items and the nine proposed target translation attributes by constructing a Q-matrix.
 Results. Based on the results, HO-DINA can be considered the best-fitting model. Bibliography and technical skills, together with work methodology skills, are shown to be the most difficult attributes for translation students.
 Conclusion. HO-DINA is a non-compensatory model, thus the study findings assert that for a correct response to a test item, all measurable attributes need to be mastered.

Test-level and Item-level Model Fit Comparison of General vs. Specific Diagnostic Classification Models: A Case of True DCM

The current study compared the model fit indices, skill mastery probabilities, and classification accuracy of six Diagnostic Classification Models (DCMs): a general model (G-DINA) against five specific models (LLM, RRUM, ACDM, DINA, and DINO). To do so, the response data to the grammar and vocabulary sections of a General English Achievement Test, designed specifically for cognitive diagnostic purposes from scratch, was analyzed. The results of the test-level-model fit values obtained strong evidence in supporting the G-DINA and LLM models possessing the best model fit. In addition, the ACDM and RRUM were almost very identical to that of the G-DINA. The value indices of the DINO and DINA models were very close to each other but larger than those of the G-DINA and LLM. The model fit was also investigated at the item level, and the results revealed that model selection should be performed at the item level rather than the test level, and most of the specific models might perform well for the test. The findings of this study suggested that the relationships among the attributes of grammar and vocabulary are not ‘either-or’ compensatory or non-compensatory but a combination of both.

The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination

PurposeDiagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rarely been reported compared with classical test theory (CTT) and item response theory models. This paper compared the accuracy of high-level attribute mastery between deterministic inputs, noisy “and” gate (DINA) and Rasch models, along with sub-scores based on CTT.MethodsFirst, a simulation study explored the effects of attribute length (number of items per attribute) and the correlations among attributes with respect to the accuracy of mastery. Second, a real-data study examined model and item fit and investigated the consistency of mastery for each attribute among the 3 models using the 2017 Korean Medical Licensing Examination with 360 items.ResultsAccuracy of mastery increased with a higher number of items measuring each attribute across all conditions. The DINA model was more accurate than the CTT and Rasch models for attributes with high correlations (>0.5) and few items. In the real-data analysis, the DINA and Rasch models generally showed better item fits and appropriate model fit. The consistency of mastery between the Rasch and DINA models ranged from 0.541 to 0.633 and the correlations of person attribute scores between the Rasch and DINA models ranged from 0.579 to 0.786.ConclusionAlthough all 3 models provide a mastery decision for each examinee, the individual mastery profile using the DINA model provides more accurate decisions for attributes with high correlations than the CTT and Rasch models. The DINA model can also be directly applied to tests with complex structures, unlike the CTT and Rasch models, and it provides different diagnostic information from the CTT and Rasch models.

Exploring Multiple Strategic Problem Solving Behaviors in Educational Psychology Research by Using Mixture Cognitive Diagnosis Model.

A mixture cognitive diagnosis model (CDM), which is called mixture multiple strategy-Deterministic, Inputs, Noisy “and” Gate (MMS-DINA) model, is proposed to investigate individual differences in the selection of response categories in multiple-strategy items. The MMS-DINA model system is an effective psychometric and statistical approach consisting of multiple strategies for practical skills diagnostic testing, which not only allows for multiple strategies of problem solving, but also allows for different strategies to be associated with different levels of difficulty. A Markov chain Monte Carlo (MCMC) algorithm for parameter estimation is given to estimate model, and four simulation studies are presented to evaluate the performance of the MCMC algorithm. Based on the available MCMC outputs, two Bayesian model selection criteria are computed for guiding the choice of the single strategy DINA model and multiple strategy DINA models. An analysis of fraction subtraction data is provided as an illustration example.

Examining cognitive diagnostic modeling in classroom assessment conditions

Cognitive diagnostic models (CDMs) are a family of psychometric models designed to provide categorical classifications for multiple latent attributes. CDMs provide more granular evidence than other psychometric models and have potential for guiding teaching and learning decisions in the classroom. However, CDMs have primarily been conducted using large samples. This study examines estimating CDMs in small sample conditions to aid formative learning. Three CDMs were compared across simulated classroom assessment conditions: deterministic input, noisy "and" gate (DINA) model, non-parametric cognitive diagnosis (NPCD), and supervised artificial neural network (SANN). We found all models estimated examinee classifications at the smallest sample size. Accuracy of individual attribute mastery classifications was acceptably high for the models under certain conditions. Effective item discrimination was the most important factor to accurately classify. The DINA and NPCD models were more resilient to measurement error than the SANN. Recommendations for application of CDMs in the classroom are provided.

Sample Size Requirements for Applying Diagnostic Classification Models.

Results of a comprehensive simulation study are reported investigating the effects of sample size, test length, number of attributes and base rate of mastery on item parameter recovery and classification accuracy of four DCMs (i.e., C-RUM, DINA, DINO, and LCDMREDUCED). Effects were evaluated using bias and RMSE computed between true (i.e., generating) parameters and estimated parameters. Effects of simulated factors on attribute assignment were also evaluated using the percentage of classification accuracy. More precise estimates of item parameters were obtained with larger sample size and longer test length. Recovery of item parameters decreased as the number of attributes increased from three to five but base rate of mastery had a varying effect on the item recovery. Item parameter and classification accuracy were higher for DINA and DINO models.

Estimating CDMs Using the Slice-Within-Gibbs Sampler.

In this paper, the slice-within-Gibbs sampler has been introduced as a method for estimating cognitive diagnosis models (CDMs). Compared with other Bayesian methods, the slice-within-Gibbs sampler can employ a wide-range of prior specifications; moreover, it can also be applied to complex CDMs with the aid of auxiliary variables, especially when applying different identifiability constraints. To evaluate its performances, two simulation studies were conducted. The first study confirmed the viability of the slice-within-Gibbs sampler in estimating CDMs, mainly including G-DINA and DINA models. The second study compared the slice-within-Gibbs sampler with other commonly used Markov Chain Monte Carlo algorithms, and the results showed that the slice-within-Gibbs sampler converged much faster than the Metropolis-Hastings algorithm and more flexible than the Gibbs sampling in choosing the distributions of priors. Finally, a fraction subtraction dataset was analyzed to illustrate the use of the slice-within-Gibbs sampler in the context of CDMs.

An Evaluation of 4PL IRT and DINA Models for Estimating Pseudo-Guessing and Slipping Parameters

In an achievement test, the examinees with the required knowledge and skill on a test item are expected to answer the item correctly while the examinees with a lack of necessary information on the item are expected to give an incorrect answer. However, an examinee can give a correct answer to the multiple-choice test items through guessing or sometimes give an incorrect response to an easy item due to anxiety or carelessness. Either case may cause a bias estimation of examinee abilities and item parameters. 4PL IRT model and the DINA model can be used to mitigate these negative impacts on the parameter estimations. The current simulation study aims to compare the estimated pseudo-guessing and slipping parameters from the 4PL IRT model and the DINA model under several study conditions. The DINA model was used to simulate the datasets in the study. The study results showed that the bias of the estimated slipping and guessing parameters from both 4PL IRT and DINA models were reasonably small in general although the estimated slipping and guessing parameters were more biased when datasets were analyzed through the 4PL IRT model rather than the DINA model (i.e., the average bias for both guessing and slipping parameters = .00 from DINA model, but .08 from 4PL IRT model). Accordingly, both 4PL IRT and DINA models can be considered for analyzing the datasets contaminated with guessing and slipping effects.

Hybrid cognitive diagnostic model

Cognitive diagnostic models can be classified into two categories based on the type of interaction between attributes: disjunctive and conjunctive. A representative example of the former is the “Deterministic Input Noisy-Or gate” (DINO) model, and of the latter is the “Deterministic Input Noisy-And gate” (DINA) model. However, fixing the interaction form to be either disjunctive or conjunctive may be based on a strong assumption. Therefore, we developed a new hybrid cognitive diagnostic model in which the item response function is represented as a weighted mixture of disjunctive and conjunctive item response functions. This made it possible to estimate the quantitative degree of each interaction type for each item, while keeping the parameters within reasonable limits. The proposed model was formalized as a Bayesian model and estimated using the Hamiltonian Monte Carlo algorithm. A Monte Carlo simulation confirmed adequate parameter recovery of the proposed method. In an empirical application to actual mathematics test data, the proposed model achieved better predictive performance than the DINA and DINO models. The obtained posteriors of the mixture weights were found to be heterogeneous among items, indicating key advantages of the proposed approach.

Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured.

Although research in cognitive psychology suggests refraining from investigating cognitive skills inisolation, many cognitive diagnosis model (CDM) examples do not take hierarchical attribute structures into account. When hierarchical relationships among the attributes are not considered, CDM estimates may be biased. The current study, through simulation and real data analyses, examines the impact of different MMLE-EM approaches on the item and person parameter estimates of the G-DINA, DINA and DINO models when attributes have a hierarchical structure. A number of estimation approaches that can result from modifying either the Q-matrix or prior distribution are proposed. Impact of the proposed approaches on item parameter estimation accuracy and attribute classification are investigated. For the G-DINA model estimation, the Q-matrix type (i.e, explicit vs. implicit) has greater impact than structuring the prior distribution. Specifically, explicit Q-matrices result in better item parameter recovery and higher correct classification rates. In contrast, structuring the prior distribution is more influential on item and person parameter estimates for the reduced models. When prior distribution is structured, the Q-matrix type has almost no influence on item and person parameter estimates of the DINA and DINO models. We can conclude that the Q-matrix type has a significant impact on CDM estimation, especially when the estimating model is G-DINA.

English

English

Comparison of Cognitive Diagnosis Models under Changing Conditions: DINA, RDINA, HODINA and HORDINA

The application of CDMs to fraction subtraction data revealed problems on the classification of examinees, latent class sizes, and the use of higher-order models. Additionally, selecting the most appropriate model assumes critical importance if there are several appropriate models available for the data. In the present study, DINA–RDINA and HODINA–HORDINA models were compared under changing conditions (i.e., number of attributes, g and s item parameter values, and number of items) with simulated and real data. The results show that for conditions where the g–s parameter values and the number of attributes were low (0.1 and 3, respectively), the reparameterized models generated values that were virtually identical to those obtained using DINA models. However, when the g–s parameter values and the number of attributes were increased (0.5 and 5, respectively), the parameter estimations obtained from the models, latent class estimates, AIC, and BIC show differences through the values from the models.

On the Complexity of Item Response Theory Models

ABSTRACTComplexity in item response theory (IRT) has traditionally been quantified by simply counting the number of freely estimated parameters in the model. However, complexity is also contingent upon the functional form of the model. We examined four popular IRT models—exploratory factor analytic, bifactor, DINA, and DINO—with different functional forms but the same number of free parameters. In comparison, a simpler (unidimensional 3PL) model was specified such that it had 1 more parameter than the previous models. All models were then evaluated according to the minimum description length principle. Specifically, each model was fit to 1,000 data sets that were randomly and uniformly sampled from the complete data space and then assessed using global and item-level fit and diagnostic measures. The findings revealed that the factor analytic and bifactor models possess a strong tendency to fit any possible data. The unidimensional 3PL model displayed minimal fitting propensity, despite the fact that it included an additional free parameter. The DINA and DINO models did not demonstrate a proclivity to fit any possible data, but they did fit well to distinct data patterns. Applied researchers and psychometricians should therefore consider functional form—and not goodness-of-fit alone—when selecting an IRT model.

Assessing Change in Latent Skills Across Time With Longitudinal Cognitive Diagnosis Modeling: An Evaluation of Model Performance.

Cognitive diagnosis models are diagnostic models used to classify respondents into homogenous groups based on multiple categorical latent variables representing the measured cognitive attributes. This study aims to present longitudinal models for cognitive diagnosis modeling, which can be applied to repeated measurements in order to monitor attribute stability of individuals and to account for respondent dependence. Models based on combining latent transition analysis modeling and the DINA and DINO cognitive diagnosis models were developed and then evaluated through a Monte Carlo simulation study. The study results indicate that the proposed models provide adequate convergence and correct classification rates.

Identifiability of Diagnostic Classification Models.

Diagnostic classification models (DCMs) are important statistical tools in cognitive diagnosis. In this paper, we consider the issue of their identifiability. In particular, we focus on one basic and popular model, the DINA model. We propose sufficient and necessary conditions under which the model parameters are identifiable from the data. The consequences, in terms of the consistency of parameter estimates, of fulfilling or failing to fulfill these conditions are illustrated via simulation. The results can be easily extended to the DINO model through the duality of the DINA and DINO models. Moreover, the proposed theoretical framework could be applied to study the identifiability issue of other DCMs.

Evaluating the Wald Test for Item‐Level Comparison of Saturated and Reduced Models in Cognitive Diagnosis

This article used the Wald test to evaluate the item‐level fit of a saturated cognitive diagnosis model (CDM) relative to the fits of the reduced models it subsumes. A simulation study was carried out to examine the Type I error and power of the Wald test in the context of the G‐DINA model. Results show that when the sample size is small and a larger number of attributes are required, the Type I error rate of the Wald test for the DINA and DINO models can be higher than the nominal significance levels, while the Type I error rate of the A‐CDM is closer to the nominal significance levels. However, with larger sample sizes, the Type I error rates for the three models are closer to the nominal significance levels. In addition, the Wald test has excellent statistical power to detect when the true underlying model is none of the reduced models examined even for relatively small sample sizes. The performance of the Wald test was also examined with real data. With an increasing number of CDMs from which to choose, this article provides an important contribution toward advancing the use of CDMs in practical educational settings.