Cognitive Diagnostic Assessment Research Articles

Development of a Cognitive Assessment Checklist for First-Grade Mathematics: Utilizing Hierarchical Cognitive Diagnostic Modeling in Elementary Education

This study aimed to address the need for a comprehensive assessment tool to evaluate the mathematical abilities of first-grade students through cognitive diagnostic assessment (CDA). The primary challenge involved in this endeavor was to delineate the specific cognitive skills and sub-skills pertinent to first-grade mathematics (FG-M) and to determine the most suitable model for assessment. Employing a mixed-methods approach, the research identified nine cognitive attributes essential for FG-M and developed a Q-matrix delineating the relationship between these attributes. A preliminary version of the FG-M checklist comprising 74 items was formulated. Subsequently, the assessment tool was administered to 1018 first-grade students, with their teachers utilizing the FG-M checklist for evaluation. Results indicated a commendable accuracy in classifying the nine cognitive attributes, ranging from 0.68 to 0.92, with an average accuracy of 0.76. Furthermore, the validity of the scores was corroborated by a substantial correlation (r = 0.78) between scores from the FG-M checklist and the arithmetic subtest of the Wechsler Intelligence Scale for Children, Fourth Edition (WISC-IV). Overall, the study concluded that the scores from the FG-M checklist are reliable for assessing the mathematical skills of first-grade students. The findings underscored the students’ proficiency in fundamental mathematical concepts such as cardinality, addition, subtraction, weight, and statistics. However, areas of weakness were identified in concepts related to multiplication, time, symmetry, and geometry. This research contributes significantly to the advancement of CDA in educational contexts, providing educators with a valuable tool for identifying students’ strengths and weaknesses in mathematical cognition.

A cognitive diagnostic approach to IELTS speaking test: unveiling the subskills and test-takers’ perceptions

Although the IELTS speaking test assesses English verbal-communication competence accurately and reliably, its traditional, retroactive scoring framework stops short of providing detailed diagnostic information about individual test-takers’ specific domains of proficiency or deficiency. This study employed a cognitive diagnostic assessment (CDA) framework, grounded in cognitive psychology and diagnostic, formative feedback theories, to identify the specific cognitive (sub)skills and attributes required to successfully perform on the IELTS speaking test and examine the test-takers’ perceptions regarding the CDA and IELTS-speaking-assessment interface. The purpose was to integrate CDA and feedback literacy into the IELTS assessment. Adopting a cognitive/psychometric perspective, we conceptualized language proficiency as encompassing separate cognitive subskills that can be systematically measured and diagnosed. To this end, we analyzed speaking data from 500 pre-IELTS candidates and conducted 20 expert interviews. The thematic analysis employed through the MAXQDA software helped identify five specific subskills related to the IELTS speaking test. Using a similar thematic analysis procedure, we extracted five perceptive themes regarding an integrated IELTS-CDA framework from the interview data of 151 participants. The results have important implications for the practicality and efficacy of employing CDA to improve IELTS speaking interpretation and feedback.

Cognitive diagnostic assessment: A Q-matrix constraint-based neural network method.

Cognitive diagnosis is a crucial element of intelligent education that aims to assess the proficiency of specific skills or traits in students at a refined level and provide insights into their strengths and weaknesses for personalized learning. Researchers have developed numerous cognitive diagnostic models. However, previous studies indicate that diagnostic accuracy can be significantly influenced by the appropriateness of the model and the sample size. Thus, designing a general model that can adapt to different assumptions and sample sizes remains a considerable challenge. Artificial neural networks have been proposed as a promising approach in some studies. In this paper, we propose a cognitive diagnosis model of a neural network constrained by a Q-matrix and named QNN. Specifically, we employ the Q-matrix to determine the connections between neurons and the width and depth of the neural network. Moreover, to reduce the human effort in the training algorithm, we designed a self-organizing map-based cognitive diagnosis training framework called SOM-NN, which enables the QNN to be trained unsupervised. Extensive experimental results on simulated and real datasets demonstrate that our approaches are effective in both accuracy and interpretability. Notably, under unsupervised conditions, our approach has significant advantages on small sample datasets with high levels of guessing and slipping, especially on the pattern-wise agreement rates. This work bridges the gap between psychometrics and machine learning and provides a realistic and implementable reference solution for classroom instructional assessment and the cold start of personalized and adaptive assessment systems.

Modeling Skipped Items in Cognitive Diagnostic Assessments

In cognitive diagnosis assessments, examinees frequently skip some items due to various reasons, such as time constraints, lack of confidence, or the perception of item difficulty. These skipped items are usually due to lacking specific cognitive attributes or knowledge structures. However, many research studies conventionally rely on complete-case analysis, potentially compromising the validity of results by ignoring skipped items. This omission can lead to invalid inferences about the attributes profiles of examinees and introduce bias into item parameters. This study aims to develop statistical models for effectively handling missing data resulting from skipped items. Specifically, we employ an item response theory model for missing indicators and utilize the deterministic inputs, noisy “and” gate (DINA) model to describe cognitive item responses. Furthermore, we introduce a higher-order structure to describe the correlation between the higher-order ability parameters and skipping propensity parameters and the correlation of the item parameters. The proposed new model fills the gaps in missing data handling, providing a more precise evaluation for the examinees who skip items and yielding more accurate research results. This enhances our understanding of cognitive processes and strengthens support for educational policies, aligning them better with student needs and optimizing decision-making.

An exploratory Q-matrix estimation method based on sparse non-negative matrix factorization.

Cognitive diagnostic assessment (CDA) is widely used because it can provide refined diagnostic information. The Q-matrix is the basis of CDA, and can be specified by domain experts or by data-driven estimation methods based on observed response data. The data-driven Q-matrix estimation methods have become a research hotspot because of their objectivity, accuracy, and low calibration cost. However, most of the existing data-driven methods require known prior knowledge, such as initial Q-matrix, partial q-vector, or the number of attributes. Under the G-DINA model, we propose to estimate the number of attributes and Q-matrix elements simultaneously without any prior knowledge by the sparse non-negative matrix factorization (SNMF) method, which has the advantage of high scalability and universality. Simulation studies are carried out to investigate the performance of the SNMF. The results under a wide variety of simulation conditions indicate that the SNMF has good performance in the accuracy of attribute number and Q-matrix elements estimation. In addition, a set of real data is taken as an example to illustrate its application. Finally, we discuss the limitations of the current study and directions for future research.

Simulation study: Seeking fairness in a high-stakes placement test

The primary objective of Cognitive Diagnostic Assessment (CDA) lies in problem solving processes to systematically categorize test takers in mastery/non-mastery groups. What is of concern is using some attributes in reading comprehension test item options which are suspected to bias, and is simulating unbiased test items in a reversed engineering attempt. To this end, 4200 PhD candidates who sat for the examination were randomly selected. A Q-matrix was constructed, and raw data were fed into the R-studio software. Two groups of female and male differences were considered for the Differential Item Functioning (DIF) detection based on the Deterministic Inputs, Noisy “and” Gate (DINA) model. Results of the simulation study revealed that in 100 times of item generations, items were non-significant in favor of gender bias. In conclusion, this study was an attempt to determine whether an item identified with DIF is a correct detection or failed to be identified in fine-grained details.

Asesmen Diagnostik Sebagai Penilaian Pembelajaran Dalam Kurikulum Merdeka di MIN 2 Boyolali

Diagnostic assessment is an assessment of learning assessment in the Merdeka Curriculum, which teachers conduct on students before the teacher designs learning. However, teachers' understanding of diagnostic assessment is still lacking, so it is challenging to compile diagnostic assessments. This study aims to describe diagnostic assessment as a learning assessment in the Merdeka Curriculum in elementary schools. The research method uses a literature study by collecting data from the results of exploring various literature in books and scientific articles from national and internationally reputable journals. The data analysis technique uses content analysis. The results obtained in this study indicate that there are two parts to the diagnostic assessment: the non-cognitive and cognitive diagnostic assessment. The non-cognitive diagnostic assessment aims to obtain information regarding family conditions, learning styles, characteristics, and students' learning interests. The cognitive diagnostic assessment aims to provide an overview of the initial abilities possessed by students in a subject topic. Teachers need to carry out diagnostic assessments of students to identify the abilities, strengths, and weaknesses so that teachers can design learning that is tailored to the abilities and characteristics of students.

Modeling Partial Knowledge in Multiple-Choice Cognitive Diagnostic Assessment

Diagnostic tests are typically administered in a multiple-choice (MC) format due to their advantages of objectivity and time efficiency. The MC-deterministic input, noisy “and” gate (DINA) family of models, a representative class of cognitive diagnostic models for MC items, efficiently and parsimoniously estimates the mastery profiles of examinees. However, the existing models often overestimate the latent traits of examinees when they respond with partial knowledge, which is often observed in educational assessment. Therefore, the novel models of the MC-DINA family that can appropriately handle such responses were developed in this study. Unlike the existing models, the proposed models placed no restrictions on the Q-vector, which represents attribute specifications. Simulation and empirical studies verified that the proposed approach could resolve the overestimation problem.

Cognitive Diagnostic Assessment Instrument with Individual Feedback for Language Receptive Learning on Multimodal Text

Cognitive Diagnostic Assessment Instrument with Individual Feedback for Language Receptive Learning on Multimodal Text

Analisis Hasil Asesmen Diagnostiksebagai Dasar Pelaksanaan Pembelajaran Berdiferensiasi Di Sekolah Dasar

Students have different characteristics and a diversity of learning needs. Educators need to map students' learning needs according to students' learning abilities, interests and learning talents. Cognitive and non-cognitive diagnostic assessments make it easier for teachers to map students' learning needs through tests or expanded questionnaires. Diagnostic assessments are carried out with the aim of reviewing students' initial abilities in learning readiness. So it can be a solution in applying differentiated learning. The purpose of this observation is to analyze the number of diagnostic assessments as a benchmark for differentiated learning for Pancasila Education subjects in class IV. The research method used is descriptive qualitative. This data was collected using non-participatory observation from carrying out cognitive diagnostic assessment tests. Data analysis techniques use the Milles and Huberman models, including value reduction, data presentation and drawing conclusions. Research in class IV A. The results of the research show that educators have mapped groups based on students' learning readiness using cognitive diagnostic assessment tests. Groupings are made from developing, and advanced groups. Content differentiated learning can be applied to provide learning material based on proficiency level so that students' learning needs are met.

Using Regularized Methods to Validate Q-Matrix in Cognitive Diagnostic Assessment

One of the central components of cognitive diagnostic assessment is the Q-matrix, which is an essential loading indicator matrix and is typically constructed by subject matter experts. Nonetheless, to a large extent, the construction of Q-matrix remains a subjective process and might lead to misspecifications. Many researchers have recognized the importance of estimating or validating the Q-matrix, but most of them focus on the conditions of relatively large sample sizes. This article aims to explore Q-matrix validation possibilities under small sample conditions and uses regularized methods to validate the Q-matrix based on the compensatory reparametrized unified model and generalized deterministic inputs, noisy “and” gate models. Simulation studies were conducted to evaluate the viability of the modified least absolute shrinkage and selection operator (Lasso) and modified smoothly clipped absolute deviation (SCAD) methods, comparing them with existing methods. Results show that the modified Lasso and the modified SCAD methods outperform the stepwise, Hull, and MLR-B methods in general, especially under the conditions of small sample sizes. While good recovery in all small sample size conditions is not guaranteed, the modified methods demonstrate advantages across various item quality conditions. Also, a real data set is analyzed to illustrate the application of the modified methods.

Exploration of polytomous-attribute Q-matrix validation in cognitive diagnostic assessment

Compared with typical binary attributes, polytomous attributes can take three or more values (corresponding to different levels of mastery of a respondent or measurement of an item). They can indicate whether a respondent possesses the attributes of interest and mastery levels. Therefore, the test with polytomous-attribute Q-matrix can become more informative and provide respondents with richer diagnostic information than the test based on the dichotomous-attribute Q-matrix. This paper extends the S-statistic and the residual method applicable for the Q-matrix of binary attributes to validate the polytomous-attribute Q-matrix. Under two common scenarios in real-world applications, two associated validation algorithms: the joint validation (JV) algorithm and the online validation (OV) algorithm, are proposed. Both simulation studies and an empirical data example were employed to assess the robustness and usefulness of these two methods under various conditions. Results indicate that the JV algorithm is suitable for validating a Q-matrix predefined by subject matter experts. Especially when the Q-matrix contains fewer misspecifications, while the OV algorithm can be applied to define the attribute vector of "new items". Based on a certain number of "operational items", the OV algorithm can achieve a promising performance for obtaining the specification of the new items.

Analisis asesmen diagnostik kognitif materi geometri di kelas IV sekolah dasar

The study aims to describe the stages in the cognitive diagnostic assessment of geometry material in grade IV of elementary school which consists of the preparation, implementation and follow up stages. Through a cognitive diagnostic assessment, the teacher can identify the level of student learning readiness regarding geometry material. The research method used is descriptive analysis with a qualitative approach. Data collection techniques obtained through observation, interviews, and documentation. The research was conducted in 2 schools, namely at one of the SDN in Cihideung district and one of the SDN in Tawang district. Based on the research conducted, the results obtained regarding the cognitive diagnostic assessment of geometry material in grade IV of elementary school are as follows: (1) the preparation stage, the class teacher makes a grid of questions and teaching modules as a guide in making questions, (2) the implementation stage, the class teacher delivering geometry material before the test begins, (3) follow up stage, the class teacher provides follow up in the form of remedial and additional lessons. Thus the class teacher can find out the learning needs of students based on cognitive diagnostic assessment of geometry material.

Utilizing Cognitive Diagnostic Assessments to Identify and Address Student Needs in Differentiated Classrooms

Utilizing Cognitive Diagnostic Assessments to Identify and Address Student Needs in Differentiated Classrooms

Preferred Learning styles among Junior High School students Using non Cognitive Diagnostic Assessment

Every student possesses individual learning styles, making it crucial to examine their preferred approaches as a means of designing courses and implementing effective instructional methods. The aim of this research endeavor was to identify the predominant learning style among junior high school students, along with discerning any gender-related associations with specific learning styles. Employing a descriptive quantitative research methodology, data were collected from 97 participants via non-cognitive diagnostic assessments. The results revealed that the visual learning style emerged as the most favored choice, with a prevalence rate of 60.8%. It was followed by the auditory style (12.4%), kinesthetic style (9.3%), visual-auditory style (7.2%), visual-kinesthetic style (7.2%), and auditory-kinesthetic style (3.1%). In terms of gender distribution, a significant majority of male students exhibited a preference for the visual style (42.3%), while the visual style also predominated among female students (18.6%). These findings hold valuable implications for fostering pedagogical excellence. Based on these results, educators are strongly advised to embrace diverse teaching methods and design multifaceted activities that encompass and accommodate the different learning styles exhibited among students.

Research on English Teaching Reading Quality Assessment Based on Cognitive Diagnostic Assessment

Research on English Teaching Reading Quality Assessment Based on Cognitive Diagnostic Assessment

Q-matrix theory and its applications in cognitive diagnostic assessment

Q-matrix theory and its applications in cognitive diagnostic assessment

Cognitive diagnostic assessment based on signal detection theory: Modeling and application

Cognitive diagnostic assessment based on signal detection theory: Modeling and application

Analysis of Cognitive Diagnostic Assessment (CDA) in Differentiated Learning on English Subject Phase D

This article presents a meticulous examination of Cognitive Diagnostic Assessment (CDA) in the context of Phase D differentiated learning, with a specific emphasis on English subjects. The study investigates the practice of diagnostic assessment in differentiated learning in Phase D English diagnostic courses and evaluate the challenges teachers face in implementing cognitive diagnostic assessment in differentiated learning. Through a qualitative inquiry, the research administrated questioner and an interview by involving 3 (three) English teacher at the seventh-grade students as the techniques of collecting data, and further analyzed the data by conducting data coding, interpretation, and triangulation. The study concluded that the practice of CDA was based on their prior knowledge of the assessment, and their believe that cognitive diagnostic assessment needs to be conducted in the differentiated learning on English subject phase D. Further, the assessment is distributed by using rubrics and checklist diagnostic tools, and rely on observation method. The respondent believed that CDA strongly informs their differentiated instructional practice and see moderate impact on their instructional practice in conducting it, and they need support from the institution to strengthen their capability in conducting CDA. Related to the challenging of CDA, the most challenges aspects are in struggling with time limitations in implementing CDA, dealing with student reluctance or opposition, interpreting assessment outcomes, and training on cognitive diagnostic tools.

인지진단평가 영어 시험 적용에서 Q행렬 구축 방법과 후속 학습 효과 연구

In cognitive diagnostic modeling research, one area that has not had enough research interests is remedial learning or instruction based on the information provided by cognitive diagnostic assessment (CDA). The present study tries to address this research gap by exploring how effective subsequent self-directed learning will be after applying cognitive diagnostic assessment to diagnosing English grammar ability, which was not addressed in the existing CDA research. The significance of the present study is that it provides an in-depth self-study guide to investigate its effectiveness. To this end, the study administers a diagnostic test and provides detailed descriptions of their mastery levels of grammar for a small group of students who will follow through with the guided self-regulated learning. The post-test reveals that the guided self-study was notably effective with the improved total score and heightened skill mastery probabilities. The current research also examines the effectiveness of distractor analysis in optimally modifying Q-matrix for CDA. The result indicates the usefulness of distractor analysis for Q-matrix refinement when the response data are available. The article ends with some suggestions for future research based on the implications of the findings and limitations of the study.