DINA Model Research Articles

Reinforcement of Core Foundational Science Concepts during Clinical Rotations: A Strategy to Enhance Medical Student Learning and Diagnostic Reasoning.

Cognitive science principles can facilitate integrating and retaining basic science concepts during the clinical years of undergraduate medical education. We hypothesized that reinforcing foundational science concepts during the core clerkship experiences fosters the development of clinical reasoning in medical students. A patient simulation session on diabetic ketoacidosis (DKA) in the pediatric clerkship was chosen to pilot the program. We introduced an intervention in which a team of clinical and foundational science faculty facilitates student discussions, helping them connect important physiological and pathophysiological concepts to the disorder's clinical presentation, diagnosis, and management. The student survey reported that 87% strongly agreed that it helped them reinforce their understanding of foundational science concepts related to clinical cases. The NBME subject exam results of the pediatric clerkships from the corresponding years were subjected to a cognitive diagnostic assessment called the DINA model analysis. The student content mastery in acid-base disorders was improved following the intervention (student mastery of skills, 0.73 vs 0.80, P=0.035). Similarly, analysis of the Step 2 Clinical Knowledge (CK) skills from the NBME subject exam item analysis reports showed improvement in the skill "diagnosis" on the topic of diabetes (probability value- before and after the intervention, respectively, 0.73 and 0.85, p=0.04) following implementation. Our study shows the impact of foundational science reinforcement during clinical clerkships in developing undergraduate medical students' clinical reasoning and diagnostic skills.

MCMC stopping rules in latent variable modelling.

Bayesian analysis relies heavily on the Markov chain Monte Carlo (MCMC) algorithm to obtain random samples from posterior distributions. In this study, we compare the performance of MCMC stopping rules and provide a guideline for determining the termination point of the MCMC algorithm in latent variable models. In simulation studies, we examine the performance of four different MCMC stopping rules: potential scale reduction factor (PSRF), fixed-width stopping rule, Geweke's diagnostic, and effective sample size. Specifically, we evaluate these stopping rules in the context of the DINA model and the bifactor item response theory model, two commonly used latent variable models in educational and psychological measurement. Our simulation study findings suggest that single-chain approaches outperform multiple-chain approaches in terms of item parameter accuracy. However, when it comes to person parameter estimates, the effect of stopping rules diminishes. We caution against relying solely on the univariate PSRF, which is the most popular method, as it may terminate the algorithm prematurely and produce biased item parameter estimates if the cut-off value is not chosen carefully. Our research offers guidance to practitioners on choosing suitable stopping rules to improve the precision of the MCMC algorithm in models involving latent variables.

Investigating The Performance of Item Selection Algorithms in Cognitive Diagnosis Computerized Adaptive Testing

In this study, it was aimed to examine the performances of item selection algorithms in terms of measurement accuracy and computational time in fixed-length CD-CAT and average test lengths in the variable-length CD-CAT through factors such as test length, number of attributes and item quality in CD-CAT. In the research, two different simulation studies were conducted for the fixed and variable-length tests. Item responses were generated according to DINA model. Two item banks which consisted of 480 items for 5 and 6 attributes were generated and the item banks were used for both the fixed and variable-length test. Q-matrix was generated item by item and attribute by attribute. In the study, 3000 examinees were generated in such a way that each examinee had a 50% chance of achieving each attribute. Cognitive patterns of examinees were estimated by using MAP. In the variable-length CD-CAT, the first highest posterior probability threshold is 0.80, and the second highest posterior probability threshold is 0.10. R 3.6.1 was used to code CD-CAT administration and carry out to other analyses. At the end of the study in which the fixed-length CD-CAT was used, it was concluded that as the number of attributes increased, the pattern recovery rates of the item selection algorithms decreased, and these rates increased as the item quality and test length increased. The highest values in terms of pattern recovery rate were obtained from JSD and MPWKL algorithms. In the variable-length CD-CAT, it was concluded that the average test length increased as the number of attributes increased, and the average test length decreased as the item quality increased. In all conditions, the shortest average test length was obtained with the JSD. It has been determined that the GDI performs the fastest calculation in all conditions, while the MPWKL algorithm performs the slowest calculation.

Diagnostic Classification Models for Testlets: Methods and Theory.

Diagnostic classification models (DCMs) have seen wide applications in educational and psychological measurement, especially in formative assessment. DCMs in the presence of testlets have been studied in recent literature. A key ingredient in the statistical modeling and analysis of testlet-based DCMs is the superposition of two latent structures, the attribute profile and the testlet effect. This paper extends the standard testlet DINA (T-DINA) model to accommodate the potential correlation between the two latent structures. Model identifiability is studied and a set of sufficient conditions are proposed. As a byproduct, the identifiability of the standard T-DINA is also established. The proposed model is applied to a dataset from the 2015 Programme for International Student Assessment. Comparisons are made with DINA and T-DINA, showing that there is substantial improvement in terms of the goodness of fit. Simulations are conducted to assess the performance of the new method under various settings.

Boosting Neural Cognitive Diagnosis with Student’s Affective State Modeling

Cognitive Diagnosis Modeling aims to infer students' proficiency level on knowledge concepts from their response logs. Existing methods typically model students’ response processes as the interaction between students and exercises or concepts based on hand-crafted or deeply-learned interaction functions. Despite their promising achievements, they fail to consider the relationship between students' cognitive states and affective states in learning, e.g., the feelings of frustration, boredom, or confusion with the learning content, which is insufficient for comprehensive cognitive diagnosis in intelligent education. To fill the research gap, we propose a novel Affect-aware Cognitive Diagnosis (ACD) model which can effectively diagnose the knowledge proficiency levels of students by taking into consideration the affective factors. Specifically, we first design a student affect perception module under the assumption that the affective state is jointly influenced by the student's affect trait and the difficulty of the exercise. Then, our inferred affective distribution is further used to estimate the student's subjective factors, i.e., guessing and slipping, respectively. Finally, we integrate the estimated guessing and slipping parameters with the basic neural cognitive diagnosis framework based on the DINA model, which facilitates the modeling of complex exercising interactions in a more accurate and interpretable fashion. Besides, we also extend our affect perception module in an unsupervised learning setting based on contrastive learning, thus significantly improving the compatibility of our ACD. To the best of our knowledge, we are the first to unify the cognition modeling and affect modeling into the same framework for student cognitive diagnosis. Extensive experiments on real-world datasets clearly demonstrate the effectiveness of our ACD. Our code is available at https://github.com/zeng-zhen/ACD.

Sufficient and Necessary Conditions for the Identifiability of DINA Models with Polytomous Responses.

Cognitive diagnosis models (CDMs) provide a powerful statistical and psychometric tool for researchers and practitioners to learn fine-grained diagnostic information about respondents' latent attributes. There has been a growing interest in the use of CDMs for polytomous response data, as more and more items with multiple response options become widely used. Similar to many latent variable models, the identifiability of CDMs is critical for accurate parameter estimation and valid statistical inference. However, the existing identifiability results are primarily focused on binary response models and have not adequately addressed the identifiability of CDMs with polytomous responses. This paper addresses this gap by presenting sufficient and necessary conditions for the identifiability of the widely used DINA model with polytomous responses, with the aim to provide a comprehensive understanding of the identifiability of CDMs with polytomous responses and to inform future research in this field.

Explanatory Cognitive Diagnosis Models Incorporating Item Features.

Item quality is crucial to psychometric analyses for cognitive diagnosis. In cognitive diagnosis models (CDMs), item quality is often quantified in terms of item parameters (e.g., guessing and slipping parameters). Calibrating the item parameters with only item response data, as a common practice, could result in challenges in identifying the cause of low-quality items (e.g., the correct answer is easy to be guessed) or devising an effective plan to improve the item quality. To resolve these challenges, we propose the item explanatory CDMs where the CDM item parameters are explained with item features such that item features can serve as an additional source of information for item parameters. The utility of the proposed models is demonstrated with the Trends in International Mathematics and Science Study (TIMSS)-released items and response data: around 20 item linguistic features were extracted from the item stem with natural language processing techniques, and the item feature engineering process is elaborated in the paper. The proposed models are used to examine the relationships between the guessing/slipping item parameters of the higher-order DINA model and eight of the item features. The findings from a follow-up simulation study are presented, which corroborate the validity of the inferences drawn from the empirical data analysis. Finally, future research directions are discussed.

Selecting an Appropriate Cognitive Diagnostic Model for Eighth Grade Math Questions: TIMSS 2019 Study

Background and Objective: Traditional assessment methods have been criticized for neglecting the cognitive processes required by test-takers to provide correct responses. To address these issues, cognitive diagnostic models have been introduced by researchers. This study aimed to explore and apply cognitive diagnostic models to estimate the psychometric properties of published eighth-grade math questions from TIMSS 2019. Methods and Materials: The research method employed in this study is mixed-methods, conducted with a retrofitting approach to existing tests. The statistical population of this study consisted of all Iranian eighth-grade students who participated in the TIMSS 2019 study. The population size included 1,095,026 students from 23,895 schools. The sample consisted of 5,980 Iranian eighth-grade students from 220 schools, selected through a two-stage cluster sampling method, with an average age reported as 14.1 years. Findings: Using a qualitative method, 16 skills across four content domains—knowledge, application, and reasoning—were identified and the Q-matrix was developed. Then, in the quantitative section, the fit of the DINA model as a compensatory model, the DINO model as a non-compensatory model, and the G-DINA model as a general model were examined with data from students' responses to math questions in blocks 1, 2, 3, and 5. The findings showed that in terms of model fit, the G-DINA model had the weakest fit, while the DINA model had the best fit. Absolute fit indices of the models also indicated that the DINA model was better, and the DINO model did not fit. Relative fit indices of the models showed no significant difference between the G-DINA model and the other two models, and overall, the non-compensatory DINA model demonstrated the best fit. Conclusion: The results indicated that the DINA model had the best fit with the data and can be considered the optimal model for analyzing the eighth-grade mathematics questions in TIMSS 2019.

Investigating the 8th-grade Afghan students’ mathematics status and skills using the cognitive diagnostic model

Cognitive diagnostic models (CDMs) are multidimensional multivariate verification flow models with complex structure. In this research, these models were used to investigate the status of eighth grade high school students in mathematics using the TIMMS questionnaire. The cognitive diagnostic test based on 13 attributes including 32 questions was conducted on a sample of 274 students who were selected based on the multi-stage cluster sampling method among the students of Firuzkoh city. IRT and RESMA models were used to determine the psychometric properties of the questions. Data analysis using DINA and DINO models in cognitive diagnostic modeling of mathematics showed that 13 attributes explain the mathematical performance of eighth-grade students. The result shows that Afghan students have a weak mastery level in most attributes compared to 45 other countries (the countries that were included in the TIMMS questionnaire) also general results show that the examinees perform better in the field of numbers (0.49), while they perform worse in data and chance (0.12). Moreover, there exists some difference in estimating item parameters under the DINA and DINO models, such as Item 3 and Item 27. One possible explanation is that the DINA model is completely compensatory while the DINO model is fully non-compensatory. Similar to the results under the DINA model, the SEs of guessing parameters are lower than those of slipping parameters under the DINO model.

Investigating the status and skills of Afghan students using the Cognitive Diagnostic Model (CDM) of Afghanistan's 8th grade mathematics.

Cognitive diagnostic models (CDMs) are multidimensional multivariate verification flow models with complex structure. In this research, these models were used to investigate the status of eighth grade high school students in mathematics using the TIMMS questionnaire. The cognitive diagnostic test based on 13 attributes including 32 questions was conducted on a sample of 274 students who were selected based on the multi-stage cluster sampling method among the students of Firuzkoh city. IRT and RESMA models were used to determine the psychometric properties of the questions. Data analysis using DINA and DINO models in cognitive diagnostic mathematics modeling showed that 13 attributes explain the mathematical performance of eighth-grade students. The result shows that Afghan students have a weak mastery level in most features compared to 45 other countries (the countries that were included in the TIMMS questionnaire) also, general results show that the examinees perform better in the field of numbers (0.49), while they perform worse in data and chance (0.12 ). Moreover, some difference exists in estimating item parameters under the DINA and DINO model, such as Item 3 and Item 27. One possible explanation is that the DINA model is completely compensatory while the DINO model is fully non-compensatory. Similar to the results under the DINA model, the SEs of guessing parameters are lower than those of slipping parameters under the DINO model.

An application of cognitive diagnostic models in the evaluation of reading comprehension in Chilean university students

Reading Comprehension skills are fundamental to the academic performance of university students. These abilities are not innate; therefore, systematic training is required before entering university. The main objective of this study was to evaluate the reading comprehension skills of Chilean University Students. Method: A sample of 579 students participated in the study, responding to four texts that comprise the Reading Comprehension Strategies for University Students (ELCU). An automated platform was used to apply and correct the test with immediate results and feedback. The tool allowed the use of a broad and representative sample of the various regions of Chile. The ELCU was developed using a cognitive diagnostic assessment (CDA) considering cognitive, metacognitive and linguistic strategies. The analyses were performed using two different approaches: The Linear Logistic Test Model (LLTM) and the deterministic inputs, noisy “and” gate (DINA) model. Results: The rejection of the LLTM implies that the strategies do not differ in cognitive complexity (difficulty) while the adequate goodness of fit to the DINA model allows to characterise the students according to their reading abilities. Conclusion: The ELCU test provides reliable and valid information on the reading comprehension ability and metacognitive awareness of university students.

DINA Model with Entropy Penalization

The cognitive diagnosis model (CDM) is an effective statistical tool for extracting the discrete attributes of individuals based on their responses to diagnostic tests. When dealing with cases that involve small sample sizes or highly correlated attributes, not all attribute profiles may be present. The standard method, which accounts for all attribute profiles, not only increases the complexity of the model but also complicates the calculation. Thus, it is important to identify the empty attribute profiles. This paper proposes an entropy-penalized likelihood method to eliminate the empty attribute profiles. In addition, the relation between attribute profiles and the parameter space of item parameters is discussed, and two modified expectation–maximization (EM) algorithms are designed to estimate the model parameters. Simulations are conducted to demonstrate the performance of the proposed method, and a real data application based on the fraction–subtraction data is presented to showcase the practical implications of the proposed method.

Measuring Learning Loss Due to Disruptions from COVID-19: Perspectives from the Concept of Fractions

Ineffective learning during Covid-19 can be a major cause of learning loss. Through polytomous item response theory analysis, this study seeks to identify learning loss of fractions and investigate the reliability and validity of the DINA model diagnostic test. This study involved 177 Grade 8 students from six junior high schools in Banda Aceh, Indonesia, to obtain information about the validity and reliability of the diagnostic test and learning loss of fraction due to Covid-19. The research was conducted by designing diagnostic test, expert validation, and empirical testing. Multiple choice problems with polytomous options were developed. This study produced 20 valid problems based on the level of item discriminant, item difficulty, and slip and guess, with a reliability of 0.899. This study revealed that learning loss was experienced by 62 students from low-level junior high schools (100%), 51 students from middle-level junior high schools (98%), and 54 students from high-level junior high schools (85.7%). It can be said that low-level schools have the highest percentage of students with learning loss. It is recommended that future studies evaluate the effectiveness of diagnostic tests in identifying learning weaknesses in mathematics before semester exams using the items generated in this study.

An Explicit Form With Continuous Attribute Profile of the Partial Mastery DINA Model

Cognitive diagnosis models (CDMs) are the statistical framework for cognitive diagnostic assessment in education and psychology. They generally assume that subjects’ latent attributes are dichotomous—mastery or nonmastery, which seems quite deterministic. As an alternative to dichotomous attribute mastery, attention is drawn to the use of a continuous attribute mastery format in recent literature. To obtain subjects’ finer-grained attribute mastery for more precise diagnosis and guidance, an equivalent but more explicit form of the partial-mastery-deterministic inputs, noisy “and” gate (DINA) model (termed continuous attribute profile [CAP]-DINA form) is proposed in this article. Its parameters estimation algorithm based on this form using Bayesian techniques with Markov chain Monte Carlo algorithm is also presented. Two simulation studies are conducted then to explore its parameter recovery and model misspecification, and the results demonstrate that the CAP-DINA form performs robustly with satisfactory efficiency in these two aspects. A real data study of the English test also indicates it has a better model fit than DINA.

Tracking Ordinal Development of Skills with a Longitudinal DINA Model with Polytomous Attributes

ABSTRACT Many educational and psychological studies have shown that the development of students is generally step-by-step (i.e. ordinal development) to a specific level. This study proposed a novel longitudinal learning diagnosis model with polytomous attributes to track students’ ordinal development in learning. Using the concept of polytomous attributes in the proposed model, the learning process of a specific skill, from non-mastery to mastery, can be divided into multiple ordinal steps in order to better characterize the learning trajectory. The results of an empirical study conducted to explore the performance of the proposed model indicated that it could adequately diagnose the ordinal development of skills in longitudinal assessments. A simulation study was also conducted to examine the estimation accuracy of general ability and the classification accuracy of attributes of the proposed model in different simulated conditions.

Cognitive diagnostic analysis of students' mathematical competency based on the DINA model

AbstractConsidering the importance of mathematics in modern society, it is crucial to understand the cognitive processes involved in the acquisition of complex mathematical competency. As a new generation of evaluation theory, cognitive diagnosis has its unique advantages in personalized evaluation. Based on the mathematical cognitive framework of Trends in International Mathematics and Science Study (TIMSS)‐2011 and the Chinese mathematics curriculum, this research has formed a mathematical competency model composed of seven cognitive attributes. Sixty‐seven released mathematical items for the fourth grade in TIMSS‐2011 were used as assessment tools in this study. The deterministic inputs, noisy, “and” gate model was selected as the cognitive diagnosis model in this research, and the parameters of the model were evaluated according to the response data, based on which the effectiveness of the assessment tool was further verified, forming the evaluation framework of students' mathematical competency. This framework was used to analyze the data of 573 students' mathematical competency from Shanghai, China, specifically from three aspects: attribute mastery probability, learning path, and knowledge structure. Results show that students' performance in mastering attributes of mathematical cognition is excellent on the whole; some students' learning paths are leapfrog; there are certain differences in students' knowledge structure despite that they have the same total score. This research is performed as a systematic case study of the evaluation of students' mathematical competency and also provides a new perspective in assessing other knowledge and skills.

Cognitive diagnosis and algorithmic cultural analysis of fourth-grade Yi students' mathematical skills in China: A case study of several primary schools in Puge county, Liangshan prefecture.

The Liangshan Yi Autonomous Prefecture is the largest area in China inhabited by the Yi people, and the original Yi characteristics and culture are well maintained. The Yi also have a high degree of ethnic and cultural intermingling with Tibetans, Han and other ethnic groups. The level of mathematical abilities directly determines the quality of mathematical learning of Yi students. Primary four is the stage of "concrete operations," and is a critical point in the development of mathematical symbolic awareness. In this study, the geographical location of the school and the financial income of the township in which the school is located were used as the basis for sampling, and the DINA model was used to diagnose the mathematical ability of fourth grade students in three rural Yi primary schools in Puge County. The study found that there was individual variability in the mathematical abilities of fourth grade Yi students, with 21 different types of cognitive error patterns identified, the main ones being five. In addition, the state of knowledge of fourth grade Yi students in arithmetic revealed that their overall level of mathematical ability was low, showing a lag, with none of the knowledge attributes of arithmetic being fully mastered. Cultural differences between the Chinese and Yi languages contribute to the difficulties that Yi students have in learning mathematical operations, including differences in understanding the place value system, zero, decimal expressions, and differences in the perception of multiplication and division. The above research can inform the implementation of targeted remediation for teaching and learning.

Generic Identifiability of the DINA Model and Blessing of Latent Dependence.

Cognitive diagnostic models are a powerful family of fine-grained discrete latent variable models in psychometrics. Within this family, the DINA model is a fundamental and parsimonious one that has received significant attention. Similar to other complex latent variable models, identifiability is an important issue for CDMs, including the DINA model. Gu and Xu (Psychometrika 84(2):468-483, 2019) established the necessary and sufficient conditions for strict identifiability of the DINA model. Despite being the strongest possible notion of identifiability, strict identifiability may impose overly stringent requirements on designing the cognitive diagnostic tests. This work studies a slightly weaker yet very useful notion, generic identifiability, which means parameters are identifiable almost everywhere in the parameter space, excluding only a negligible subset of measure zero. We propose transparent generic identifiability conditions for the DINA model, relaxing existing conditions in nontrivial ways. Under generic identifiability, we also explicitly characterize the forms of the measure-zero sets where identifiability breaks down. In addition, we reveal an interesting blessing-of-latent-dependence phenomenon under DINA-that is, dependence between the latent attributes can restore identifiability under some otherwise unidentifiable [Formula: see text]-matrix designs. The blessing of latent dependence provides useful practical implications and reassurance for real-world designs of cognitive diagnostic assessments.

Representation of Object Provenance for Research on Natural Science Objects: Samples, parts and derivatives in DINA-compliant collection data management

Collection objects in natural science collections span a diverse set of object types of substantially different origin, physical composition, and relevance for different fields and methodologies of research and application. Object provenance is often characterized by elaborate series of interventions from collecting or observing originals in a natural state to generating derived objects that can be physically persistent or are suitable for a given use. This sequence of events gives rise to intermediate objects or object states that can be of a persistent or ephemeral nature in their own right. Detailed metadata on object provenance is vital to enable informed use of collection objects for research and other application areas. Providing the ability to generate, maintain, update and access such accounts is an important requirement for Collection Management Software (CMS). DINA (Digital Information System for Natural History Data, Glöckler et al. 2020)-compliant collection management software meets this challenge by using process- and state-based representation of object histories and modular application architecture as the main conceptual and architectural principles, respectively (Bölling et al. 2021). In applying these principles, we showcase how object provenance can be represented in the DINA system in cases where multiple objects, possibly of varying types, are derived from a single object, objects consist of parts of different biological individuals, object histories involve different types of objects such as living biological individuals, samples, and preserved specimens. multiple objects, possibly of varying types, are derived from a single object, objects consist of parts of different biological individuals, object histories involve different types of objects such as living biological individuals, samples, and preserved specimens. We highlight how the abstractions and categories used in the DINA model can be used to meet a variety of challenging use cases for representing collection object provenance. For instance, while the connections and relationships between living, preserved, and even destructively processed samples can be documented in DINA, these are ordinarily difficult to accommodate in a single information system.

Commentary on "Extending the Basic Local Independence Model to Polytomous Data" by Stefanutti, de Chiusole, Anselmi, and Spoto.

The Polytomous Local Independence Model (PoLIM) by Stefanutti, de Chiusole, Anselmi, and Spoto, is an extension of the Basic Local Independence Model (BLIM) to accommodate polytomous items. BLIM, a model for analyzing responses to binary items, is based on Knowledge Space Theory, a framework developed by cognitive scientists and mathematical psychologists for modeling human knowledge acquisition and representation. The purpose of this commentary is to show that PoLIM is simply a paraphrase of a DINA model in cognitive diagnosis for polytomous items. Specifically, BLIM is shown to be equivalent to the DINA model when the BLIM-items are conceived as binary single-attribute items, each with a distinct attribute; thus, PoLIM is equivalent to the DINA for polytomous single-attribute items, each with a distinct attribute.