Cognitive Diagnosis Models Research Articles

Diagnosing middle school students’ proficiency in constructing scientific explanations with the integration of chemical reactions and patterns: a cognitive diagnostic modeling approach

ABSTRACT Scientific explanation has been widely recognised as one of the primary proficiencies in global science education. From research and practical perspectives, investigating the cognitive process of constructing scientific explanations by integrating core ideas is essential to make sense of phenomena or solve problems. This study applied the cognitive diagnosis modeling (CDM) approach to diagnose students’ cognitive patterns of constructing scientific explanations by integrating chemical reactions and patterns. We localised three well-designed assessment tasks and correspondingly developed five cognitive attributes. Responses from 244 Grade 9 students in two middle schools were collected, scored, and analyzed. Results show that the five developed cognitive attributes are reliable and effective in identifying students’ cognitive challenges in constructing scientific explanations in a content-specific domain. In addition, we articulated students’ potential cognitive process of constructing scientific explanations. We identified a primary cognitive pathway that students may follow: making correct claims, finding sufficient evidence based on data patterns, and providing scientific principles for the reasoning process. The cognitive diagnosis results could be used to guide teachers in selecting teaching materials and strategies and arranging lesson sequences to support the development of students’ scientific explanations by integrating chemical reactions and patterns.

Applying support vector machines to a diagnostic classification model for polytomous attributes in small-sample contexts.

Over several years, the evaluation of polytomous attributes in small-sample settings has posed a challenge to the application of cognitive diagnosis models. To enhance classification precision, the support vector machine (SVM) was introduced for estimating polytomous attribution, given its proven feasibility for dichotomous cases. Two simulation studies and an empirical study assessed the impact of various factors on SVM classification performance, including training sample size, attribute structures, guessing/slipping levels, number of attributes, number of attribute levels, and number of items. The results indicated that SVM outperformed the pG-DINA model in classification accuracy under dependent attribute structures and small sample sizes. SVM performance improved with an increased number of items but declined with higher guessing/slipping levels, more attributes, and more attribute levels. Empirical data further validated the application and advantages of SVMs.

Cognitive diagnostic assessment: A Q-matrix constraint-based neural network method.

Cognitive diagnosis is a crucial element of intelligent education that aims to assess the proficiency of specific skills or traits in students at a refined level and provide insights into their strengths and weaknesses for personalized learning. Researchers have developed numerous cognitive diagnostic models. However, previous studies indicate that diagnostic accuracy can be significantly influenced by the appropriateness of the model and the sample size. Thus, designing a general model that can adapt to different assumptions and sample sizes remains a considerable challenge. Artificial neural networks have been proposed as a promising approach in some studies. In this paper, we propose a cognitive diagnosis model of a neural network constrained by a Q-matrix and named QNN. Specifically, we employ the Q-matrix to determine the connections between neurons and the width and depth of the neural network. Moreover, to reduce the human effort in the training algorithm, we designed a self-organizing map-based cognitive diagnosis training framework called SOM-NN, which enables the QNN to be trained unsupervised. Extensive experimental results on simulated and real datasets demonstrate that our approaches are effective in both accuracy and interpretability. Notably, under unsupervised conditions, our approach has significant advantages on small sample datasets with high levels of guessing and slipping, especially on the pattern-wise agreement rates. This work bridges the gap between psychometrics and machine learning and provides a realistic and implementable reference solution for classroom instructional assessment and the cold start of personalized and adaptive assessment systems.

Leveraging large language models through natural language processing to provide interpretable machine learning predictions of mental deterioration in real time

AbstractBased on official estimates, 50 million people worldwide are affected by dementia, and this number increases by 10 million new patients every year. Without a cure, clinical prognostication and early intervention represent the most effective ways to delay its progression. To this end, artificial intelligence and computational linguistics can be exploited for natural language analysis, personalized assessment, monitoring, and treatment. However, traditional approaches need more semantic knowledge management and explicability capabilities. Moreover, using large language models (llms) for cognitive decline diagnosis is still scarce, even though these models represent the most advanced way for clinical–patient communication using intelligent systems. Consequently, we leverage an llm using the latest natural language processing (nlp) techniques in a chatbot solution to provide interpretable machine learning prediction of cognitive decline in real-time. Linguistic-conceptual features are exploited for appropriate natural language analysis. Through explainability, we aim to fight potential biases of the models and improve their potential to help clinical workers in their diagnosis decisions. More in detail, the proposed pipeline is composed of (i) data extraction employing nlp-based prompt engineering; (ii) stream-based data processing including feature engineering, analysis, and selection; (iii) real-time classification; and (iv) the explainability dashboard to provide visual and natural language descriptions of the prediction outcome. Classification results exceed 80% in all evaluation metrics, with a recall value for the mental deterioration class about 85%. To sum up, we contribute with an affordable, flexible, non-invasive, personalized diagnostic system to this work.

The choice between cognitive diagnosis and item response theory: A case study from medical education

Feedback is a powerful instructional tool for motivating learning. But effective feedback, requires that instructors have accurate information about their students’ current knowledge status and their learning progress. In modern educational measurement, two major theoretical perspectives on student ability and proficiency can be distinguished. Latent trait models identify ability as a continuous uni- or multi-dimensional construct, with unidimensional item response theoretic (IRT) models presumably the most popular type of latent trait models. They report a single ability score that allows for locating examinees relative to their peers on the latent ability dimension targeted by the test. Latent trait models have been criticized for lacking diagnostic information on students’ specific skills, their strengths and weaknesses in a knowledge domain. Cognitive diagnosis (CD) models, in contrast, describe ability as a combination of discrete skills (called “attributes”) that constitute (partially) ordered latent classes of proficiency. The focus of CD is on collecting information about the learning progress for immediate feedback to students in terms of skills they have mastered and those needing study. CD has been underused in education; performance assessment still mostly relies on latent-trait-based methods. The motivation for the study reported here arose from the desire to conduct a side-by-side evaluation of the two seemingly disparate psychometric frameworks, CD and IRT. Data from a biochemistry end-of-term exam were used for illustration. They were fitted with multiple CD and IRT models, among them also HO-GDINA models that permit for a close approximation to several unidimensional IRT models.

Determining the number of attributes in the GDINA model.

Exploratory cognitive diagnosis models have been widely used in psychology, education and other fields. This paper focuses on determining the number of attributes in a widely used cognitive diagnosis model, the GDINA model. Under some conditions of cognitive diagnosis models, we prove that there exists a special structure for the covariance matrix of observed data. Due to the special structure of the covariance matrix, an estimator based on eigen-decomposition is proposed for the number of attributes for the GDINA model. The performance of the proposed estimator is verified by simulation studies. Finally, the proposed estimator is applied to two real data sets Examination for the Certificate of Proficiency in English (ECPE) and Big Five Personality (BFP).

Combining regularization and logistic regression model to validate the Q-matrix for cognitive diagnosis model.

Q-matrix is an important component of most cognitive diagnosis models (CDMs); however, it mainly relies on subject matter experts' judgements in empirical studies, which introduces the possibility of misspecified q-entries. To address this, statistical Q-matrix validation methods have been proposed to aid experts' judgement. A few of these methods, including the multiple logistic regression-based (MLR-B) method and the Hull method, can be applied to general CDMs, but they are either time-consuming or lack accuracy under certain conditions. In this study, we combine the L1 regularization and MLR model to validate the Q-matrix. Specifically, an L1 penalty term is imposed on the log-likelihood of the MLR model to select the necessary attributes for each item. A simulation study with various factors was conducted to examine the performance of the new method against the two existing methods. The results show that the regularized MLR-B method (a) produces the highest Q-matrix recovery rate (QRR) and true positive rate (TPR) for most conditions, especially with a small sample size; (b) yields a slightly higher true negative rate (TNR) than either the MLR-B or the Hull method for most conditions; and (c) requires less computation time than the MLR-B method and similar computation time as the Hull method. A real data set is analysed for illustration purposes.

Modeling question difficulty for unbiased cognitive diagnosis: A causal perspective

Cognitive diagnosis is an intelligent education task that aims to learn students’ cognitive states on knowledge concepts based on historical answering logs over questions. Existing studies focus on modeling the interactions between students and questions through either manual-designed functions (e.g., logistic function) or complex neural network structures. However, such studies neglect the question difficulty bias, i.e., questions exhibit uneven distribution on the answering frequency, as simple questions are answered more times than difficult ones for the given concept. To tackle this issue, we present a Causal Cognitive Diagnosis Framework (CausalCDF), which considers the question difficulty bias and could be readily integrated with traditional diagnostic models for better cognitive diagnosis. Specifically, we first analyze the effect of question difficulty (acting as the confounder) on student performance via a causal graph. Then we eliminate the bad effect of the confounding difficulty bias via causal intervention in model training. We instantiate CausalCDF on five representative diagnostic models and perform extensive experiments on two real-world datasets. Empirical studies prove the effectiveness of CausalCDF compared to existing studies.

Boosting Neural Cognitive Diagnosis with Student’s Affective State Modeling

Cognitive Diagnosis Modeling aims to infer students' proficiency level on knowledge concepts from their response logs. Existing methods typically model students’ response processes as the interaction between students and exercises or concepts based on hand-crafted or deeply-learned interaction functions. Despite their promising achievements, they fail to consider the relationship between students' cognitive states and affective states in learning, e.g., the feelings of frustration, boredom, or confusion with the learning content, which is insufficient for comprehensive cognitive diagnosis in intelligent education. To fill the research gap, we propose a novel Affect-aware Cognitive Diagnosis (ACD) model which can effectively diagnose the knowledge proficiency levels of students by taking into consideration the affective factors. Specifically, we first design a student affect perception module under the assumption that the affective state is jointly influenced by the student's affect trait and the difficulty of the exercise. Then, our inferred affective distribution is further used to estimate the student's subjective factors, i.e., guessing and slipping, respectively. Finally, we integrate the estimated guessing and slipping parameters with the basic neural cognitive diagnosis framework based on the DINA model, which facilitates the modeling of complex exercising interactions in a more accurate and interpretable fashion. Besides, we also extend our affect perception module in an unsupervised learning setting based on contrastive learning, thus significantly improving the compatibility of our ACD. To the best of our knowledge, we are the first to unify the cognition modeling and affect modeling into the same framework for student cognitive diagnosis. Extensive experiments on real-world datasets clearly demonstrate the effectiveness of our ACD. Our code is available at https://github.com/zeng-zhen/ACD.

Sufficient and Necessary Conditions for the Identifiability of DINA Models with Polytomous Responses.

Cognitive diagnosis models (CDMs) provide a powerful statistical and psychometric tool for researchers and practitioners to learn fine-grained diagnostic information about respondents' latent attributes. There has been a growing interest in the use of CDMs for polytomous response data, as more and more items with multiple response options become widely used. Similar to many latent variable models, the identifiability of CDMs is critical for accurate parameter estimation and valid statistical inference. However, the existing identifiability results are primarily focused on binary response models and have not adequately addressed the identifiability of CDMs with polytomous responses. This paper addresses this gap by presenting sufficient and necessary conditions for the identifiability of the widely used DINA model with polytomous responses, with the aim to provide a comprehensive understanding of the identifiability of CDMs with polytomous responses and to inform future research in this field.

Explanatory Cognitive Diagnosis Models Incorporating Item Features.

Item quality is crucial to psychometric analyses for cognitive diagnosis. In cognitive diagnosis models (CDMs), item quality is often quantified in terms of item parameters (e.g., guessing and slipping parameters). Calibrating the item parameters with only item response data, as a common practice, could result in challenges in identifying the cause of low-quality items (e.g., the correct answer is easy to be guessed) or devising an effective plan to improve the item quality. To resolve these challenges, we propose the item explanatory CDMs where the CDM item parameters are explained with item features such that item features can serve as an additional source of information for item parameters. The utility of the proposed models is demonstrated with the Trends in International Mathematics and Science Study (TIMSS)-released items and response data: around 20 item linguistic features were extracted from the item stem with natural language processing techniques, and the item feature engineering process is elaborated in the paper. The proposed models are used to examine the relationships between the guessing/slipping item parameters of the higher-order DINA model and eight of the item features. The findings from a follow-up simulation study are presented, which corroborate the validity of the inferences drawn from the empirical data analysis. Finally, future research directions are discussed.

A Mixture Fluency model using responses and response times with cognitive diagnosis model framework.

Random guessing behaviors are frequently observed in low-stakes assessments, often attributed to factors such as test-takers lacking motivation or experiencing time constraints and fatigue. Existing research suggests that responses stemming from random guessing behaviors introduce biases into the constructs and relationships of interest. This is particularly problematic when estimating the relationship between speed and ability. This study introduces a Mixture Fluency model designed to account for random guessing behaviors while utilizing valid response accuracy and response time to uncover students' latent attribute profiles. The model directly addresses a limitation present in the Fluency cognitive diagnostic model (Wang & Chen, Psychometrika, 85, 600-629, (2020), which assumes that test-takers consistently employ solution behaviors when answering questions. To investigate the effectiveness of the proposed Mixture Fluency model, we conducted a simulation study encompassing various simulation conditions. Results from this study not only confirm the model's ability to detect potential random guessing behaviors but also demonstrate its capacity to enhance the inference of targeted latent constructs within the assessment. Additionally, we showcase the practical utility of the proposed model through an application to real data.

Knowledge Integration in Science Learning: Tracking Students' Knowledge Development and Skill Acquisition with Cognitive Diagnosis Models

AbstractIn scientific literacy, knowledge integration (KI) is a scaffolding‐based theory to assist students' scientific inquiry learning. To drive students to be self‐directed, many courses have been developed based on KI framework. However, few efforts have been made to evaluate the outcome of students' learning under KI instruction. Moreover, finer‐grained information has been pursued to better understand students' learning and how it progresses over time. In this article, a normative procedure of building and choosing cognitive diagnosis models (CDMs) and attribute hierarchies was formulated under KI theory. We examined the utility of CDMs for evaluating students' knowledge status in KI learning. The results of the data analysis confirmed an intuitive assumption of the hierarchical structure of KI components. Furthermore, analysis of pre‐ and posttests using a higher‐order, hidden Markov model tracked students' skill acquisition while integrating knowledge. Results showed that students make significant progress after using the web‐based inquiry science environment (WISE) platform.

Properties and performance of the one-parameter log-linear cognitive diagnosis model

Diagnostic classification models (DCMs) are psychometric models that yield probabilistic classifications of respondents according to a set of discrete latent variables. The current study examines the recently introduced one-parameter log-linear cognitive diagnosis model (1-PLCDM), which has increased interpretability compared with general DCMs due to useful measurement properties like sum score sufficiency and invariance properties. We demonstrate its equivalence with the Latent Class/Rasch Model and discuss interpretational consequences. The model is further examined in a DCM framework. We demonstrate the sum score sufficiency property and we derive an expression for the cut score for mastery classification. It is shown by means of a simulation study that the 1-PLCDM is fairly robust to model constraint violations in terms of classification accuracy and reliability. This robustness in combination with useful measurement properties and ease of interpretation can make the model attractive for stakeholders to apply in various assessment settings.

Research and Practice of Intelligent Improvement Strategies for Teacher-Student Relationships Toward Educational Community

Abstract A good teacher-student relationship is an important part of the educational community and a symbol of spiritual civilization. Effective teaching in the classroom requires even better communication and interactive actions between teachers and students, and measures to improve teacher-student relationships are proposed for the sake of classroom teaching. In this paper, an adaptive learning system based on cognitive diagnosis is constructed, which utilizes the DINA cognitive diagnosis model to obtain the learner’s cognitive structure, perceive the changes in the student’s cognitive structure in real time, and allow the learning task to make adaptive adjustments. The K-means clustering algorithm randomly selects the number of K objects as the center of mass for the initial clustering, and after the output of clustering and layering results, the students will be classified according to the categorization of the student’s learning needs. The students are divided into three groups that are similar within the group. By doing this, the learning task becomes adaptive, and appropriate learning tasks are adjusted for different levels of learners. Most of them believe that the adaptive learning system can help learners improve their learning efficiency and are very satisfied with the effectiveness of the system. The proportion of teachers who were satisfied with overall perception reached 89.1%, and the proportion of students who were satisfied was also high. The overall perceived effect of the improved teacher-student relationship is excellent, suggesting that the adaptive learning system can effectively improve the teacher-student relationship.

BNMI-DINA: A Bayesian Cognitive Diagnosis Model for Enhanced Personalized Learning

In the field of education, cognitive diagnosis is crucial for achieving personalized learning. The widely adopted DINA (Deterministic Inputs, Noisy And gate) model uncovers students’ mastery of essential skills necessary to answer questions correctly. However, existing DINA-based approaches overlook the dependency between knowledge points, and their model training process is computationally inefficient for large datasets. In this paper, we propose a new cognitive diagnosis model called BNMI-DINA, which stands for Bayesian Network-based Multiprocess Incremental DINA. Our proposed model aims to enhance personalized learning by providing accurate and detailed assessments of students’ cognitive abilities. By incorporating a Bayesian network, BNMI-DINA establishes the dependency relationship between knowledge points, enabling more accurate evaluations of students’ mastery levels. To enhance model convergence speed, key steps of our proposed algorithm are parallelized. We also provide theoretical proof of the convergence of BNMI-DINA. Extensive experiments demonstrate that our approach effectively enhances model accuracy and reduces computational time compared to state-of-the-art cognitive diagnosis models.

Research on the selection of cognitive diagnosis model from the perspective of experts

Research on the selection of cognitive diagnosis model from the perspective of experts

Knowledge Graph-Enhanced Intelligent Tutoring System Based on Exercise Representativeness and Informativeness

In the realm of online tutoring intelligent systems, e-learners are exposed to a substantial volume of learning content. The extraction and organization of exercises and skills hold significant importance in establishing clear learning objectives and providing appropriate exercise recommendations. Presently, knowledge graph-based recommendation algorithms have garnered considerable attention among researchers. However, these algorithms solely consider knowledge graphs with single relationships and do not effectively model exercise-rich features, such as exercise representativeness and informativeness. Consequently, this paper proposes a framework, namely, the Knowledge Graph Importance-Exercise Representativeness and Informativeness Framework, to address these two issues. The framework consists of four intricate components and a novel cognitive diagnosis model called the Neural Attentive Cognitive Diagnosis model to recommend the proper exercises. These components encompass the informativeness component, exercise representation component, knowledge importance component, and exercise representativeness component. The informativeness component evaluates the informational value of each exercise and identifies the candidate exercise set E C that exhibits the highest exercise informativeness. Moreover, the exercise representation component utilizes a graph neural network to process student records. The output of the graph neural network serves as the input for exercise-level attention and skill-level attention, ultimately generating exercise embeddings and skill embeddings. Furthermore, the skill embeddings are employed as input for the knowledge importance component. This component transforms a one-dimensional knowledge graph into a multidimensional one through four class relations and calculates skill importance weights based on novelty and popularity. Subsequently, the exercise representativeness component incorporates exercise weight knowledge coverage to select exercises from the candidate exercise set for the tested exercise set. Lastly, the cognitive diagnosis model leverages exercise representation and skill importance weights to predict student performance on the test set and estimate their knowledge state. To evaluate the effectiveness of our selection strategy, extensive experiments were conducted on two types of publicly available educational datasets. The experimental results demonstrate that our framework can recommend appropriate exercises to students, leading to improved student performance.

A Multistrategy Cognitive Diagnosis Model Incorporating Item Response Times Based on Strategy Selection Theories

Response times (RTs) facilitate the quantification of underlying cognitive processes in problem-solving behavior. To provide more comprehensive diagnostic feedback on strategy selection and attribute profiles with multistrategy cognitive diagnosis model (CDM) and utilize additional information for item RTs, this study develops a multistrategy cognitive diagnosis modeling framework combined with RTs. The proposed model integrates individual response accuracy and RT into a unified framework to define strategy selection and make it closer to the individual’s strategy selection process. Simulation studies demonstrated that the proposed model had reasonable parameter recovery and attribute classification accuracy and outperformed the existing multistrategy CDMs and single-strategy CDMs in terms of performance. Empirical results further illustrated the practical application and the advantages of the proposed model.

DINA Model with Entropy Penalization

The cognitive diagnosis model (CDM) is an effective statistical tool for extracting the discrete attributes of individuals based on their responses to diagnostic tests. When dealing with cases that involve small sample sizes or highly correlated attributes, not all attribute profiles may be present. The standard method, which accounts for all attribute profiles, not only increases the complexity of the model but also complicates the calculation. Thus, it is important to identify the empty attribute profiles. This paper proposes an entropy-penalized likelihood method to eliminate the empty attribute profiles. In addition, the relation between attribute profiles and the parameter space of item parameters is discussed, and two modified expectation–maximization (EM) algorithms are designed to estimate the model parameters. Simulations are conducted to demonstrate the performance of the proposed method, and a real data application based on the fraction–subtraction data is presented to showcase the practical implications of the proposed method.