Cognitive Diagnosis Models Research Articles

Enhancing performance factor analysis through skill profile and item similarity integration via an attention mechanism of artificial intelligence

IntroductionFrequent formative assessment is essential for accurately evaluating student learning, enhancing engagement, and providing personalized feedback. In STEM education, understanding the relationship between skills that students have internalized (mastered) and those they are developing (emergent) is crucial. Traditional models, including item response and cognitive diagnosis models, primarily focus on emergent skills, often overlooking internalized skills. Moreover, new tools like large language models lack a complete approach for tracking knowledge and capturing complex skill relationships.MethodsThis study incorporates artificial intelligence, specifically attention mechanisms, into educational assessment to evaluate both emergent and internalized skills. We propose a modified version of Performance Factor Analysis (PFA), which assesses student abilities by analyzing past responses and comparing them with peer performance on the same items, using parameters from a sigmoid function. This model leverages attention mechanisms to capture item order-based similarity and decay principles, providing a nuanced view of student skill profiles.ResultsThe Modified Performance Factor Analysis model significantly improved discriminative power, accuracy, precision, recall, and F1 scores across various skill areas compared to traditional PFA models.DiscussionThese results indicate that the Modified Performance Factor Analysis model allows for a more accurate and comprehensive evaluation of student performance, effectively identifying both emergent and internalized skills. By integrating AI into assessment, educators gain deeper insights, enabling them to refine teaching strategies and better support students' mastery of both types of skills.

Regularized Bayesian algorithms for Q-matrix inference based on saturated cognitive diagnosis modelling.

Q-matrices are crucial components of cognitive diagnosis models (CDMs), which are used to provide diagnostic information and classify examinees according to their attribute profiles. The absence of an appropriate Q-matrix that correctly reflects item-attribute relationships often limits the widespread use of CDMs. Rather than relying on expert judgment for specification and post-hoc methods for validation, there has been a notable shift towards Q-matrix estimation by adopting Bayesian methods. Nevertheless, their dependency on Markov chain Monte Carlo (MCMC) estimation requires substantial computational burdens and their exploratory tendency is unscalable to large-scale settings. As a scalable and efficient alternative, this study introduces the partially confirmatory framework within a saturated CDM, where the Q-matrix can be partially defined by experts and partially inferred from data. To address the dual needs of accuracy and efficiency, the proposed framework accommodates two estimation algorithms-an MCMC algorithm and a Variational Bayesian Expectation Maximization (VBEM) algorithm. This dual-channel approach extends the model's applicability across a variety of settings. Based on simulated and real data, the proposed framework demonstrated its robustness in Q-matrix inference.

Diagnosing middle school students’ proficiency in constructing scientific explanations with the integration of chemical reactions and patterns: a cognitive diagnostic modeling approach

ABSTRACT Scientific explanation has been widely recognised as one of the primary proficiencies in global science education. From research and practical perspectives, investigating the cognitive process of constructing scientific explanations by integrating core ideas is essential to make sense of phenomena or solve problems. This study applied the cognitive diagnosis modeling (CDM) approach to diagnose students’ cognitive patterns of constructing scientific explanations by integrating chemical reactions and patterns. We localised three well-designed assessment tasks and correspondingly developed five cognitive attributes. Responses from 244 Grade 9 students in two middle schools were collected, scored, and analyzed. Results show that the five developed cognitive attributes are reliable and effective in identifying students’ cognitive challenges in constructing scientific explanations in a content-specific domain. In addition, we articulated students’ potential cognitive process of constructing scientific explanations. We identified a primary cognitive pathway that students may follow: making correct claims, finding sufficient evidence based on data patterns, and providing scientific principles for the reasoning process. The cognitive diagnosis results could be used to guide teachers in selecting teaching materials and strategies and arranging lesson sequences to support the development of students’ scientific explanations by integrating chemical reactions and patterns.

A Two-Step Q-Matrix Estimation Method.

Cognitive Diagnosis Models in educational measurement are restricted latent class models that describe ability in a knowledge domain as a composite of latent skills an examinee may have mastered or failed. Different combinations of skills define distinct latent proficiency classes to which examinees are assigned based on test performance. Items of cognitively diagnostic assessments are characterized by skill profiles specifying which skills are required for a correct item response. The item-skill profiles of a test form its Q-matrix. The validity of cognitive diagnosis depends crucially on the correct specification of the Q-matrix. Typically, Q-matrices are determined by curricular experts. However, expert judgment is fallible. Data-driven estimation methods have been developed with the promise of greater accuracy in identifying the Q-matrix of a test. Yet, many of the extant methods encounter computational feasibility issues either in the form of excessive amounts of CPU times or inadmissible estimates. In this article, a two-step algorithm for estimating the Q-matrix is proposed that can be used with any cognitive diagnosis model. Simulations showed that the new method outperformed extant estimation algorithms and was computationally more efficient. It was also applied to Tatsuoka's famous fraction-subtraction data. The paper concludes with a discussion of theoretical and practical implications of the findings.

Applying support vector machines to a diagnostic classification model for polytomous attributes in small-sample contexts.

Over several years, the evaluation of polytomous attributes in small-sample settings has posed a challenge to the application of cognitive diagnosis models. To enhance classification precision, the support vector machine (SVM) was introduced for estimating polytomous attribution, given its proven feasibility for dichotomous cases. Two simulation studies and an empirical study assessed the impact of various factors on SVM classification performance, including training sample size, attribute structures, guessing/slipping levels, number of attributes, number of attribute levels, and number of items. The results indicated that SVM outperformed the pG-DINA model in classification accuracy under dependent attribute structures and small sample sizes. SVM performance improved with an increased number of items but declined with higher guessing/slipping levels, more attributes, and more attribute levels. Empirical data further validated the application and advantages of SVMs.

Cognitive diagnostic assessment: A Q-matrix constraint-based neural network method.

Cognitive diagnosis is a crucial element of intelligent education that aims to assess the proficiency of specific skills or traits in students at a refined level and provide insights into their strengths and weaknesses for personalized learning. Researchers have developed numerous cognitive diagnostic models. However, previous studies indicate that diagnostic accuracy can be significantly influenced by the appropriateness of the model and the sample size. Thus, designing a general model that can adapt to different assumptions and sample sizes remains a considerable challenge. Artificial neural networks have been proposed as a promising approach in some studies. In this paper, we propose a cognitive diagnosis model of a neural network constrained by a Q-matrix and named QNN. Specifically, we employ the Q-matrix to determine the connections between neurons and the width and depth of the neural network. Moreover, to reduce the human effort in the training algorithm, we designed a self-organizing map-based cognitive diagnosis training framework called SOM-NN, which enables the QNN to be trained unsupervised. Extensive experimental results on simulated and real datasets demonstrate that our approaches are effective in both accuracy and interpretability. Notably, under unsupervised conditions, our approach has significant advantages on small sample datasets with high levels of guessing and slipping, especially on the pattern-wise agreement rates. This work bridges the gap between psychometrics and machine learning and provides a realistic and implementable reference solution for classroom instructional assessment and the cold start of personalized and adaptive assessment systems.

Leveraging large language models through natural language processing to provide interpretable machine learning predictions of mental deterioration in real time

AbstractBased on official estimates, 50 million people worldwide are affected by dementia, and this number increases by 10 million new patients every year. Without a cure, clinical prognostication and early intervention represent the most effective ways to delay its progression. To this end, artificial intelligence and computational linguistics can be exploited for natural language analysis, personalized assessment, monitoring, and treatment. However, traditional approaches need more semantic knowledge management and explicability capabilities. Moreover, using large language models (llms) for cognitive decline diagnosis is still scarce, even though these models represent the most advanced way for clinical–patient communication using intelligent systems. Consequently, we leverage an llm using the latest natural language processing (nlp) techniques in a chatbot solution to provide interpretable machine learning prediction of cognitive decline in real-time. Linguistic-conceptual features are exploited for appropriate natural language analysis. Through explainability, we aim to fight potential biases of the models and improve their potential to help clinical workers in their diagnosis decisions. More in detail, the proposed pipeline is composed of (i) data extraction employing nlp-based prompt engineering; (ii) stream-based data processing including feature engineering, analysis, and selection; (iii) real-time classification; and (iv) the explainability dashboard to provide visual and natural language descriptions of the prediction outcome. Classification results exceed 80% in all evaluation metrics, with a recall value for the mental deterioration class about 85%. To sum up, we contribute with an affordable, flexible, non-invasive, personalized diagnostic system to this work.

The choice between cognitive diagnosis and item response theory: A case study from medical education

Feedback is a powerful instructional tool for motivating learning. But effective feedback, requires that instructors have accurate information about their students’ current knowledge status and their learning progress. In modern educational measurement, two major theoretical perspectives on student ability and proficiency can be distinguished. Latent trait models identify ability as a continuous uni- or multi-dimensional construct, with unidimensional item response theoretic (IRT) models presumably the most popular type of latent trait models. They report a single ability score that allows for locating examinees relative to their peers on the latent ability dimension targeted by the test. Latent trait models have been criticized for lacking diagnostic information on students’ specific skills, their strengths and weaknesses in a knowledge domain. Cognitive diagnosis (CD) models, in contrast, describe ability as a combination of discrete skills (called “attributes”) that constitute (partially) ordered latent classes of proficiency. The focus of CD is on collecting information about the learning progress for immediate feedback to students in terms of skills they have mastered and those needing study. CD has been underused in education; performance assessment still mostly relies on latent-trait-based methods. The motivation for the study reported here arose from the desire to conduct a side-by-side evaluation of the two seemingly disparate psychometric frameworks, CD and IRT. Data from a biochemistry end-of-term exam were used for illustration. They were fitted with multiple CD and IRT models, among them also HO-GDINA models that permit for a close approximation to several unidimensional IRT models.

Determining the number of attributes in the GDINA model.

Exploratory cognitive diagnosis models have been widely used in psychology, education and other fields. This paper focuses on determining the number of attributes in a widely used cognitive diagnosis model, the GDINA model. Under some conditions of cognitive diagnosis models, we prove that there exists a special structure for the covariance matrix of observed data. Due to the special structure of the covariance matrix, an estimator based on eigen-decomposition is proposed for the number of attributes for the GDINA model. The performance of the proposed estimator is verified by simulation studies. Finally, the proposed estimator is applied to two real data sets Examination for the Certificate of Proficiency in English (ECPE) and Big Five Personality (BFP).

Combining regularization and logistic regression model to validate the Q-matrix for cognitive diagnosis model.

Q-matrix is an important component of most cognitive diagnosis models (CDMs); however, it mainly relies on subject matter experts' judgements in empirical studies, which introduces the possibility of misspecified q-entries. To address this, statistical Q-matrix validation methods have been proposed to aid experts' judgement. A few of these methods, including the multiple logistic regression-based (MLR-B) method and the Hull method, can be applied to general CDMs, but they are either time-consuming or lack accuracy under certain conditions. In this study, we combine the L1 regularization and MLR model to validate the Q-matrix. Specifically, an L1 penalty term is imposed on the log-likelihood of the MLR model to select the necessary attributes for each item. A simulation study with various factors was conducted to examine the performance of the new method against the two existing methods. The results show that the regularized MLR-B method (a) produces the highest Q-matrix recovery rate (QRR) and true positive rate (TPR) for most conditions, especially with a small sample size; (b) yields a slightly higher true negative rate (TNR) than either the MLR-B or the Hull method for most conditions; and (c) requires less computation time than the MLR-B method and similar computation time as the Hull method. A real data set is analysed for illustration purposes.

Modeling question difficulty for unbiased cognitive diagnosis: A causal perspective

Cognitive diagnosis is an intelligent education task that aims to learn students’ cognitive states on knowledge concepts based on historical answering logs over questions. Existing studies focus on modeling the interactions between students and questions through either manual-designed functions (e.g., logistic function) or complex neural network structures. However, such studies neglect the question difficulty bias, i.e., questions exhibit uneven distribution on the answering frequency, as simple questions are answered more times than difficult ones for the given concept. To tackle this issue, we present a Causal Cognitive Diagnosis Framework (CausalCDF), which considers the question difficulty bias and could be readily integrated with traditional diagnostic models for better cognitive diagnosis. Specifically, we first analyze the effect of question difficulty (acting as the confounder) on student performance via a causal graph. Then we eliminate the bad effect of the confounding difficulty bias via causal intervention in model training. We instantiate CausalCDF on five representative diagnostic models and perform extensive experiments on two real-world datasets. Empirical studies prove the effectiveness of CausalCDF compared to existing studies.

Boosting Neural Cognitive Diagnosis with Student’s Affective State Modeling

Cognitive Diagnosis Modeling aims to infer students' proficiency level on knowledge concepts from their response logs. Existing methods typically model students’ response processes as the interaction between students and exercises or concepts based on hand-crafted or deeply-learned interaction functions. Despite their promising achievements, they fail to consider the relationship between students' cognitive states and affective states in learning, e.g., the feelings of frustration, boredom, or confusion with the learning content, which is insufficient for comprehensive cognitive diagnosis in intelligent education. To fill the research gap, we propose a novel Affect-aware Cognitive Diagnosis (ACD) model which can effectively diagnose the knowledge proficiency levels of students by taking into consideration the affective factors. Specifically, we first design a student affect perception module under the assumption that the affective state is jointly influenced by the student's affect trait and the difficulty of the exercise. Then, our inferred affective distribution is further used to estimate the student's subjective factors, i.e., guessing and slipping, respectively. Finally, we integrate the estimated guessing and slipping parameters with the basic neural cognitive diagnosis framework based on the DINA model, which facilitates the modeling of complex exercising interactions in a more accurate and interpretable fashion. Besides, we also extend our affect perception module in an unsupervised learning setting based on contrastive learning, thus significantly improving the compatibility of our ACD. To the best of our knowledge, we are the first to unify the cognition modeling and affect modeling into the same framework for student cognitive diagnosis. Extensive experiments on real-world datasets clearly demonstrate the effectiveness of our ACD. Our code is available at https://github.com/zeng-zhen/ACD.

Sufficient and Necessary Conditions for the Identifiability of DINA Models with Polytomous Responses.

Cognitive diagnosis models (CDMs) provide a powerful statistical and psychometric tool for researchers and practitioners to learn fine-grained diagnostic information about respondents' latent attributes. There has been a growing interest in the use of CDMs for polytomous response data, as more and more items with multiple response options become widely used. Similar to many latent variable models, the identifiability of CDMs is critical for accurate parameter estimation and valid statistical inference. However, the existing identifiability results are primarily focused on binary response models and have not adequately addressed the identifiability of CDMs with polytomous responses. This paper addresses this gap by presenting sufficient and necessary conditions for the identifiability of the widely used DINA model with polytomous responses, with the aim to provide a comprehensive understanding of the identifiability of CDMs with polytomous responses and to inform future research in this field.

Explanatory Cognitive Diagnosis Models Incorporating Item Features.

Item quality is crucial to psychometric analyses for cognitive diagnosis. In cognitive diagnosis models (CDMs), item quality is often quantified in terms of item parameters (e.g., guessing and slipping parameters). Calibrating the item parameters with only item response data, as a common practice, could result in challenges in identifying the cause of low-quality items (e.g., the correct answer is easy to be guessed) or devising an effective plan to improve the item quality. To resolve these challenges, we propose the item explanatory CDMs where the CDM item parameters are explained with item features such that item features can serve as an additional source of information for item parameters. The utility of the proposed models is demonstrated with the Trends in International Mathematics and Science Study (TIMSS)-released items and response data: around 20 item linguistic features were extracted from the item stem with natural language processing techniques, and the item feature engineering process is elaborated in the paper. The proposed models are used to examine the relationships between the guessing/slipping item parameters of the higher-order DINA model and eight of the item features. The findings from a follow-up simulation study are presented, which corroborate the validity of the inferences drawn from the empirical data analysis. Finally, future research directions are discussed.

A Mixture Fluency model using responses and response times with cognitive diagnosis model framework.

Random guessing behaviors are frequently observed in low-stakes assessments, often attributed to factors such as test-takers lacking motivation or experiencing time constraints and fatigue. Existing research suggests that responses stemming from random guessing behaviors introduce biases into the constructs and relationships of interest. This is particularly problematic when estimating the relationship between speed and ability. This study introduces a Mixture Fluency model designed to account for random guessing behaviors while utilizing valid response accuracy and response time to uncover students' latent attribute profiles. The model directly addresses a limitation present in the Fluency cognitive diagnostic model (Wang & Chen, Psychometrika, 85, 600-629, (2020), which assumes that test-takers consistently employ solution behaviors when answering questions. To investigate the effectiveness of the proposed Mixture Fluency model, we conducted a simulation study encompassing various simulation conditions. Results from this study not only confirm the model's ability to detect potential random guessing behaviors but also demonstrate its capacity to enhance the inference of targeted latent constructs within the assessment. Additionally, we showcase the practical utility of the proposed model through an application to real data.

Knowledge Integration in Science Learning: Tracking Students' Knowledge Development and Skill Acquisition with Cognitive Diagnosis Models

AbstractIn scientific literacy, knowledge integration (KI) is a scaffolding‐based theory to assist students' scientific inquiry learning. To drive students to be self‐directed, many courses have been developed based on KI framework. However, few efforts have been made to evaluate the outcome of students' learning under KI instruction. Moreover, finer‐grained information has been pursued to better understand students' learning and how it progresses over time. In this article, a normative procedure of building and choosing cognitive diagnosis models (CDMs) and attribute hierarchies was formulated under KI theory. We examined the utility of CDMs for evaluating students' knowledge status in KI learning. The results of the data analysis confirmed an intuitive assumption of the hierarchical structure of KI components. Furthermore, analysis of pre‐ and posttests using a higher‐order, hidden Markov model tracked students' skill acquisition while integrating knowledge. Results showed that students make significant progress after using the web‐based inquiry science environment (WISE) platform.

Properties and performance of the one-parameter log-linear cognitive diagnosis model

Diagnostic classification models (DCMs) are psychometric models that yield probabilistic classifications of respondents according to a set of discrete latent variables. The current study examines the recently introduced one-parameter log-linear cognitive diagnosis model (1-PLCDM), which has increased interpretability compared with general DCMs due to useful measurement properties like sum score sufficiency and invariance properties. We demonstrate its equivalence with the Latent Class/Rasch Model and discuss interpretational consequences. The model is further examined in a DCM framework. We demonstrate the sum score sufficiency property and we derive an expression for the cut score for mastery classification. It is shown by means of a simulation study that the 1-PLCDM is fairly robust to model constraint violations in terms of classification accuracy and reliability. This robustness in combination with useful measurement properties and ease of interpretation can make the model attractive for stakeholders to apply in various assessment settings.

Research and Practice of Intelligent Improvement Strategies for Teacher-Student Relationships Toward Educational Community

Abstract A good teacher-student relationship is an important part of the educational community and a symbol of spiritual civilization. Effective teaching in the classroom requires even better communication and interactive actions between teachers and students, and measures to improve teacher-student relationships are proposed for the sake of classroom teaching. In this paper, an adaptive learning system based on cognitive diagnosis is constructed, which utilizes the DINA cognitive diagnosis model to obtain the learner’s cognitive structure, perceive the changes in the student’s cognitive structure in real time, and allow the learning task to make adaptive adjustments. The K-means clustering algorithm randomly selects the number of K objects as the center of mass for the initial clustering, and after the output of clustering and layering results, the students will be classified according to the categorization of the student’s learning needs. The students are divided into three groups that are similar within the group. By doing this, the learning task becomes adaptive, and appropriate learning tasks are adjusted for different levels of learners. Most of them believe that the adaptive learning system can help learners improve their learning efficiency and are very satisfied with the effectiveness of the system. The proportion of teachers who were satisfied with overall perception reached 89.1%, and the proportion of students who were satisfied was also high. The overall perceived effect of the improved teacher-student relationship is excellent, suggesting that the adaptive learning system can effectively improve the teacher-student relationship.

BNMI-DINA: A Bayesian Cognitive Diagnosis Model for Enhanced Personalized Learning

In the field of education, cognitive diagnosis is crucial for achieving personalized learning. The widely adopted DINA (Deterministic Inputs, Noisy And gate) model uncovers students’ mastery of essential skills necessary to answer questions correctly. However, existing DINA-based approaches overlook the dependency between knowledge points, and their model training process is computationally inefficient for large datasets. In this paper, we propose a new cognitive diagnosis model called BNMI-DINA, which stands for Bayesian Network-based Multiprocess Incremental DINA. Our proposed model aims to enhance personalized learning by providing accurate and detailed assessments of students’ cognitive abilities. By incorporating a Bayesian network, BNMI-DINA establishes the dependency relationship between knowledge points, enabling more accurate evaluations of students’ mastery levels. To enhance model convergence speed, key steps of our proposed algorithm are parallelized. We also provide theoretical proof of the convergence of BNMI-DINA. Extensive experiments demonstrate that our approach effectively enhances model accuracy and reduces computational time compared to state-of-the-art cognitive diagnosis models.

Research on the selection of cognitive diagnosis model from the perspective of experts

Research on the selection of cognitive diagnosis model from the perspective of experts