Cognitive Diagnosis Models Research Articles

Kayıp Veri Varlığında DINA Model Madde-Özellik İlişkisinin Parametre Kestirimine Etkisi

The objective of this study is to investigate the relation between the number of items and attributes and to analyze the manner in which the different rates of missing data affect the model estimations based on the simulation data. A Q-matrix contains 24 items, and data are generated using four attributes. A dataset of n = 3000 is generated by associating the first, middle, and final eight items in the Q-matrix with one, two, and three attributes, respectively, and 5%, 10%, and 15% of the data have been randomly deleted from the first, middle, and final eight-item blocks in the Q-matrix, respectively. Subsequently, imputation was performed using the multiple imputation (MI) method with these datasets, 100 replication was performed for each condition. The values obtained from these datasets were compared with the values obtained from the full dataset. Thus, it can be observed that an increase in the amount of missing data negatively affects the consistency of the DINA parameters and the latent class estimations. Further, the latent class consistency becomes less affected by the missing data as the number of attributes associated with the items increase. With an increase in the number of attributes associated with the items, the missing data in these items affect the consistency level of the g parameter (guessing) less and the s parameter (slip) more. Furthermore, it can be observed from the results that the test developers using the cognitive diagnosis models should specifically consider the item–attribute relation in items with missing data.

Characterizing and Diagnosing Complex Professional Competencies—An Example of Intrapreneurship

AbstractWe describe an approach to characterizing and diagnosing complex professional competencies (CPCs) for the field of Intrapreneurship, i.e. activities of an entrepreneurial nature engaged by employees within their existing organizations. Our approach draws upon prior conceptual, empirical, and analytical efforts by researchers in Germany. Results are presented from an application of a cognitive diagnostic modeling approach to the performance of late stage apprentices on tasks derived from a previously developed competence model of Intrapreneurship. The results are discussed in terms of the type of cognitive diagnosis model (CDM) most appropriate for the domain and task battery, and patterns of performance are presented for seven diagnosable Intrapreneurship skills. By interpreting the assessment task response data in terms of a CDM, diagnostic, skill‐based information is obtained which verifies the strengths and weaknesses of the apprentices at a late stage in their training and has the potential to provide feedback to training programs triggering the improvement of individual apprentice learning and subsequent work‐related performance.

Evaluating the Fit of Sequential G-DINA Model Using Limited-Information Measures.

Limited-information fit measures appear to be promising in assessing the goodness-of-fit of dichotomous response cognitive diagnosis models (CDMs), but their performance has not been examined for polytomous response CDMs. This study investigates the performance of the M ord statistic and standardized root mean square residual (SRMSR) for an ordinal response CDM-the sequential generalized deterministic inputs, noisy "and" gate model. Simulation studies showed that the M ord statistic had well-calibrated Type I error rates, but the correct detection rates were influenced by various factors such as item quality, sample size, and the number of response categories. In addition, the SRMSR was also influenced by many factors and the common practice of comparing the SRMSR against a prespecified cut-off (e.g., .05) may not be appropriate. A set of real data was analyzed as well to illustrate the use of M ord statistic and SRMSR in practice.

A multilevel logistic hidden Markov model for learning under cognitive diagnosis.

Students who wish to learn a specific skill have increasing access to a growing number of online courses and open-source educational repositories of instructional tools, including videos, slides, and exercises. Navigating these tools is time-consuming and the search itself can hinder the learning of the skill. Educators are hence interested in aiding students by selecting the optimal content sequence for individual learners, specifically which skill one should learn next and which material one should use to study. Such adaptive selection would rely on pre-knowledge of how the learners' and the instructional tools' characteristics jointly affect the probability of acquiring a certain skill. Building upon previous research on Latent Transition Analysis and Learning Trajectories, we propose a multilevel logistic hidden Markov model for learning based on cognitive diagnosis models, where the probability that a learner acquires the target skill depends not only on the general difficulty of the skill and the learner's mastery of other skills in the curriculum but also on the effectiveness of the particular learning tool and its interaction with mastery of other skills, captured by random slopes and intercepts for each learning tool. A Bayesian modeling framework and an MCMC algorithm for parameter estimation are proposed and evaluated using a simulation study.

A Partial Mastery, Higher-Order Latent Structural Model for Polytomous Attributes in Cognitive Diagnostic Assessments

The latent attribute space in cognitive diagnosis models (CDMs) is often assumed to be unstructured or saturated. In recent years, the number of latent attributes in real tests has often been found to be large, and polytomous latent attributes have been advocated. Therefore, it is preferable to adopt substantive theories to connect seemingly unrelated latent attributes, to replace the unstructured or saturated latent structural models (LSMs) with structured or parsimonious ones, with simplified parameter estimation. In the present study, we developed a partial mastery, higher-order LSM for polytomous attributes, which was built upon the framework of adjacent-category logit models to account for a higher-order latent structure of multiple polytomous attributes. The proposed model can be incorporated into many existing CDMs. We conducted simulations to evaluate the psychometric properties of the proposed model and obtained good parameter recovery. We then provided an empirical example to demonstrate the applications and the advantages of the proposed model.

Effects of Computerized Formative Assessment with the Cognitive Diagnosis Model on Achievement and Affective Attitude for Math

본 연구는 평가의 설계 및 개발 단계에서부터 인지진단이론에 의해 제작된 컴퓨터화 형성평가 도구가 실제 중학생들의 수학 성취도와 수학에 대한 정의적 특성에 어떠한 영향을 미치는지 확인하고 그에 따른 시사점을 제공하고자 하는 목적을 가진다. 성태제(2018)가 개발한 인지진단이론에 의한 형성평가 도구를 활용하였으며, 지필검사와 피드백이 없는 집단, 지필검사와 지연된 피드백을 준 집단, 컴퓨터화 검사와 즉각적 피드백을 준 집단으로 나누어 수학 성취도와 수학에 대한 정의적 특성을 분석하였다. 분석 결과, 지필검사 실시 집단은 사전과 사후검사 점수 차이가 통계적으로 유의하지 않았던 반면 컴퓨터화 검사 실시 집단은 총점, 덧셈, 뺄셈, 곱셈, 나눗셈 점수가 사후검사에서 통계적으로 유의하게 증가하였다. 공분산분석 결과 총점과 뺄셈 점수에서 집단간 유의한 차이가 나타났으며, 컴퓨터화 검사 집단과 지필검사 두 집단의 복합 대비가 통계적으로 유의하였고 실질적인 점수 차이 또한 나타났다. 수학에 대한 정의적 특성에서는 ‘이해지각’에 대한 집단간 차이가 유의하였다. 연구 결과를 종합하면 형성평가의 실시 형태와 피드백의 제공 시기와 방법, 상세한 피드백이 수학 성취도와 수학에 대한 정의적 특성에 영향을 미침을 알 수 있다. 특히 상세한 수준의 피드백은 인지진단이론을 바탕으로 검사를 설계함으로써 제공 가능한 것으로, 인지요소 수준에서의 평가 설계 및 개발과 그에 따른 상세한 피드백, 즉각적인 피드백 제공이 가능한 컴퓨터화 검사 환경의 구축이 학교 현장에서 필요함을 시사한다.The purpose of this research is to investigate effects of computerized formative assessment with cognitive diagnostic model on students’ achievement and affective attitude for mathematics. To achieve goal of this study, computerized formative assessment was developed first, and the effect of computerized formative assessment and feedback on students’ math achievement and affective attitude were investigated. Especially, computerized formative assessment provided the feedbacks of correct answer and a reason of choosing wrong answer. Students were grouped according to test form and feedback. Students in control group took a paper-pencil test and were not provided feedback. Students in experimental group Ⅰ took a paper-pencil test and were provided a postponded feedback. Students in experimental group Ⅱ took a computerized test and were provided a immediate feedback. The result of this study showed that there was no significant difference of the pre and post-test scores in the control group and experimental groupⅠ. However, there was significant progress of ‘addition’, ‘subtraction’, ‘multiplication’, and ‘division’ score compared to pre test score in experimental groupⅡ. Next, the result of ANCOVA showed that there was a significant difference in post-test score and attribute score among three groups. And there was significant difference of attitude which is the perception of comprehension for mathmatics among three groups. Based on the results, computerized formative assessment with cognitive diagnostic model and immediate feedback have impacts on students’ achievement and affective attitude for math. Therefore, this study suggests to apply the computerized formative assessment with cognitive diagnostic model and immediate feedbacks for math education.

Conditional Independence and Dimensionality of Cognitive Diagnostic Models: a Test for Model Fit

Nonparametric cognitive diagnosis methods are useful in cognitive diagnosis modeling for calibration efficiency, especially when sample size is small or large, or the latent attributes are more complex. This article proposes the Mantel-Haenszel chi-squared statistic as an index for detecting the misspecification of latent attributes as well as testlet effects in nonparametric cognitive diagnosis methods. The proposed theoretical considerations are augmented by simulation studies conducted to assess the performance of the Mantel-Haenszel statistic under various conditions within the nonparametric diagnosis framework, with a special focus on situations were the set of latent abilities assumed to underlie the data was underspecified.

Using the log-linear cognitive diagnosis model to classify individuals in career and educational assessment

The log-linear cognitive diagnosis model (LCDM) is a modern technique that dichotomously classifies individuals (e.g., possession and non-possession) on attributes of a multidimensional construct, which is particularly suited to diagnostic and formative assessments where a classification decision is desired (e.g., to assign intervention or not). This article provides a tutorial for conducting analyses using the LCDM and illustrates an application in career assessment with simulated data to showcase the unique types of diagnostic feedback the LCDM provides. The article concludes with a discussion of the ideal contexts in which to apply the model and its implications in career and educational assessment.

A Sequential Higher Order Latent Structural Model for Hierarchical Attributes in Cognitive Diagnostic Assessments.

The higher-order structure and attribute hierarchical structure are two popular approaches to defining the latent attribute space in cognitive diagnosis models. However, to our knowledge, it is still impossible to integrate them to accommodate the higher-order latent trait and hierarchical attributes simultaneously. To address this issue, this article proposed a sequential higher-order latent structural model (LSM) by incorporating various hierarchical structures into a higher-order latent structure. The feasibility of the proposed higher-order LSM was examined using simulated data. Results indicated that, in conjunction with the deterministic-inputs, noisy "and" gate model, the sequential higher-order LSM produced considerable improvement in person classification accuracy compared with the conventional higher-order LSM, when a certain attribute hierarchy existed. An empirical example was presented as well to illustrate the application of the proposed LSM.

Digital Module 04: Diagnostic Measurement: Modeling Checklists for Practitioners https://ncme.elevate.commpartners.com

AbstractIn this ITEMS module, we provide a didactic overview of the specification, estimation, evaluation, and interpretation steps for diagnostic measurement/classification models (DCMs), which are a promising psychometric modeling approach. These models can provide detailed skill‐ or attribute‐specific feedback to respondents along multiple latent dimensions and hold theoretical and practical appeal for a variety of fields. We use a current unified modeling framework—the log‐linear cognitive diagnosis model (LCDM)—as well as a series of quality‐control checklists for data analysts and scientific users to review the foundational concepts, practical steps, and interpretational principles for these models. We demonstrate how the models and checklists can be applied in real‐life data‐analysis contexts. A library of macros and supporting files for Excel, SAS, and Mplus are provided along with video tutorials for key practices.

Utilizing response times in cognitive diagnostic computerized adaptive testing under the higher-order deterministic input, noisy 'and' gate model.

Methods of cognitive diagnostic computerized adaptive testing (CD-CAT) under higher-order cognitive diagnosis models have been developed to simultaneously provide estimates of the attribute mastery statuses of examinees for formative assessment and estimates of a latent continuous trait for overall summative evaluation. In a typical CD-CAT environment, examinees are often subject to a time limit, and the examinees' response times (RTs) for specific test items can be routinely recorded by custom-made programs. Because examinees are individually administered tailored sets of test items from the item pool, they may experience different levels of speededness during testing and different levels of risk of running out of time. In this study, RTs were considered during the item-selection procedure to control the test speededness and the RTs were treated as useful information for improving latent trait estimation in CD-CAT under the higher-order deterministic input, noisy 'and' gate (DINA) model. A modified posterior-weighted Kullback-Leibler (PWKL) method that maximizes the item information per time unit and a shadow-test method that assembles a provisional test subject to a specified time constraint were developed. Two simulation studies were conducted to assess the effects of the proposed methods on the quality of CD-CAT for fixed- and variable-length exams. The results show that, compared with the traditional PWKL method, the proposed methods preserve a lower risk of running out of time while ensuring satisfactory attribute estimation and providing more accurate estimates of the latent trait and speed parameters. Finally, several suggestions for future research are proposed.

Cognitive diagnosis models for multiple strategies.

Cognitive diagnosis models (CDMs) have been used as psychometric tools in educational assessments to estimate students' proficiency profiles. However, most CDMs assume that all students adopt the same strategy when approaching problems in an assessment, which may not be the case in practice. This study develops a generalized multiple-strategy CDM for dichotomous response data. The proposed model provides a unified framework to accommodate various condensation rules (e.g., conjunctive, disjunctive, and additive) and different strategy selection approaches (i.e., probability-matching, over-matching, and maximizing). Model parameters are estimated using the marginal maximum likelihood estimation via expectation-maximization algorithm. Simulation studies showed that the parameters of the proposed model can be adequately recovered and that the proposed model was relatively robust to some types of model misspecifications. A set of real data was analysed as well to illustrate the use of the proposed model in practice.

Using JAGS for Bayesian Cognitive Diagnosis Modeling: A Tutorial

In this article, we systematically introduce the just another Gibbs sampler (JAGS) software program to fit common Bayesian cognitive diagnosis models (CDMs) including the deterministic inputs, noisy “and” gate model; the deterministic inputs, noisy “or” gate model; the linear logistic model; the reduced reparameterized unified model; and the log-linear CDM (LCDM). Further, we introduce the unstructured latent structural model and the higher order latent structural model. We also show how to extend these models to consider polytomous attributes, the testlet effect, and longitudinal diagnosis. Finally, we present an empirical example as a tutorial to illustrate how to use JAGS codes in R.

An empirical Q-matrix validation method for the sequential generalized DINA model.

As a core component of most cognitive diagnosis models, the Q-matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-matrix empirically because a misspecified Q-matrix could result in erroneous attribute estimation. Most existing Q-matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-matrix for graded response data based on the sequential generalized deterministic inputs, noisy 'and' gate (G-DINA) model. The proposed Q-matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.

Reconsidering Cutoff Points in the General Method of Empirical Q-Matrix Validation.

Cognitive diagnosis models (CDMs) are latent class multidimensional statistical models that help classify people accurately by using a set of discrete latent variables, commonly referred to as attributes. These models require a Q-matrix that indicates the attributes involved in each item. A potential problem is that the Q-matrix construction process, typically performed by domain experts, is subjective in nature. This might lead to the existence of Q-matrix misspecifications that can lead to inaccurate classifications. For this reason, several empirical Q-matrix validation methods have been developed in the recent years. de la Torre and Chiu proposed one of the most popular methods, based on a discrimination index. However, some questions related to the usefulness of the method with empirical data remained open due the restricted number of conditions examined, and the use of a unique cutoff point (EPS) regardless of the data conditions. This article includes two simulation studies to test this validation method under a wider range of conditions, with the purpose of providing it with a higher generalization, and to empirically determine the most suitable EPS considering the data conditions. Results show a good overall performance of the method, the relevance of the different studied factors, and that using a single indiscriminate EPS is not acceptable. Specific guidelines for selecting an appropriate EPS are provided in the discussion.

The CDA-BPD: retrofitting a traditional borderline personality questionnaire under the cognitive diagnosis model framework

To obtain rich information about the cognitive diagnosis of borderline personality disorder (BPD), this study attempted to retrofit a traditional borderline personality questionnaire so that the improved assessment (called CDA-BPD) could provide more diagnostic information. The retrofitting processes included the following steps: (1) applied an cognitive diagnosis model to analyze the psychometric characteristics of the traditional questionnaire; (2) under the guidance of cognitive diagnosis assessment (CDA), high-quality items were chosen to develop the CDA-BPD and tested on 1,097 subjects; (3) the quality of the CDA-BPD was evaluated; (4) the structure of the CDA-BPD was analyzed. Results indicated that: (1) the CDA-BPD had acceptable reliability and validity; (2) the CDA-BPD had sensitivity of 0.985 and specificity of 0.853 with area under curve (AUC) = 0.956; (3) the two structural factors of the traditional questionnaire were confirmed in the CDA-BPD; χ2 was 83.01 with df = 26, p < .0001, comparative fit index (CFI) = 0.97, root mean square error of approximation (RMSEA) = 0.045. It was concluded that the practice of retrofitting a traditional borderline personality assessment for cognitive diagnostic purpose was feasible. Most importantly, under the cognitive diagnosis model framework, CDA-BPD could simultaneously provide general-level information and the detailed symptom criteria-level information about the posterior probability of satisfying each symptom criterion in the Diagnostic and Statistical Manual of Mental Disorders (5th edition; DSM-5; American Psychiatric Association, 2013 ) for each individual, which gave further insight into tailoring individual-specific treatments for borderline personality disorder.

An Investigation of a Country-Level Diagnostic Assessment Model for the TIMSS

국제 수학·과학 성취도 추이 연구와 같은 교육평가의 목적은 국가와 같은 집단을 비교하는 것이다. 이와 같이 평가의 단위가 학생보다 상위 수준일 때 집단을 비교하기 위해서는 다층 문항반응이론 기반의 집단 수준 진단평가 모형을 고려할 수 있다. 이 연구에서는 Park과 Bolt(2008)가 제안한 문항특성모형을 수정하여 새로운 다층 문항반응이론 기반의 집단 수준 진단평가 모형을 제안하였다. 이 수정된 모형은 2015년에 실시된 국제 수학·과학 성취도 추이 연구의 8학년 수학 평가 자료에 적용되었다. 그 결과 9개의 인지적 요소에 대한 국가들의 프로파일을 구할 수 있었다. 이 연구 결과는 국가 간 비교 또는 요소 간 상호 비교에 이용될 수 있다. 이와 같이 잔차항을 설명할 수 있도록 수정된 모형은 보다 높은 신뢰도를 지니며 국가 간 비교에 보다 정확한 정보를 제공할 수 있을 것으로 기대된다. 예를 이용하여 연구 결과를 어떻게 해석할 수 있는지 제시하였으며, 연구의 제한점과 후속 연구에 대해서도 논의하였다.The purpose of educational assessments such as the Trends in International Mathematics and Science Study (TIMSS) is to compare groups such as countries. When the unit of measurement is above the student level, group-level diagnostic assessment based on multilevel item response theory (ML-IRT) can be considered just as cognitive diagnosis models are developed from item response theory. This study suggests an ML-IRT-based group-level diagnostic assessment model by modifying an item feature model by Park and bolt (2008). The model is illustrated on the recently released TIMSS 2015 Grade 8 mathematics assessment. The results provide skill profiles for the studied countries and the nine cognitive attributes; that is, the attribute effects can be compared across the countries and also across the attributes. By controlling unexplained variance, the suggested model may provide more reliable and more informative group-level comparisons. The results are interpreted using an example. Limitations and directions for future research are also discussed.

A Sequential Process Model for Cognitive Diagnostic Assessment With Repeated Attempts.

When diagnostic assessments are administered to examinees, the mastery status of each examinee on a set of specified cognitive skills or attributes can be directly evaluated using cognitive diagnosis models (CDMs). Under certain circumstances, allowing the examinees to have at least one opportunity to correctly answer the questions and assessments, with repeated attempts on the items, provides many potential benefits. A sequential process model can be extended to model repeated attempts in diagnostic assessments. Two formulations of the sequential generalized deterministic-input noisy-"and"-gate (G-DINA) model were developed in this study. The first extension uses the latent transition analysis (LTA) approach to model changes in the attributes over attempts, and the second extension constructs a higher order structure of latent continuous variables and latent attributes to account for the dependences of the attributes over attempts. Accurate model parameter estimation and correct classifications of attributes were observed in a series of simulations using Bayesian estimation. The effectiveness of the developed sequential G-DINA model was demonstrated by fitting real data from a longitudinal mathematical test to the developed model and the longitudinal G-DINA model using the LTA approach. Finally, this article closes by discussing several important issues associated with the developed models and providing suggestions for future directions.

Q-Matrix Refinement Based on Item Fit Statistic RMSEA.

A Q-matrix, which reflects how attributes are measured for each item, is necessary when applying a cognitive diagnosis model to an assessment. In most cases, the Q-matrix is constructed by experts in the field and may be subjective and incorrect. One efficient method to refine the Q-matrix is to employ a suitable statistic that is calculated using response data. However, this approach is limited by its need to estimate all items in the Q-matrix even if only some are incorrect. To address this challenge, this study proposes an item fit statistic root mean square error approximation (RMSEA) for validating a Q-matrix with the deterministic inputs, noisy, "and" (DINA) model. Using a search algorithm, two simulation studies were performed to evaluate the effectiveness and efficiency of the proposed method at recovering Q-matrices. Results showed that using RMSEA can help define attributes in a Q-matrix. A comparison with the existing Delta method and residual sum of squares (RSS) method revealed that the proposed method had higher mean recovery rates and can be used to identify and correct Q-matrix misspecifications. When no error exists in the Q-matrix, the proposed method does not modify the correct Q-matrix.

Estimating the Cognitive Diagnosis [Formula: see text] Matrix with Expert Knowledge: Application to the Fraction-Subtraction Dataset.

Cognitive diagnosis models (CDMs) are an important psychometric framework for classifying students in terms of attribute and/or skill mastery. The [Formula: see text] matrix, which specifies the required attributes for each item, is central to implementing CDMs. The general unavailability of [Formula: see text] for most content areas and datasets poses a barrier to widespread applications of CDMs, and recent research accordingly developed fully exploratory methods to estimate Q. However, current methods do not always offer clear interpretations of the uncovered skills and existing exploratory methods do not use expert knowledge to estimate Q. We consider Bayesian estimation of [Formula: see text] using a prior based upon expert knowledge using a fully Bayesian formulation for a general diagnostic model. The developed method can be used to validate which of the underlying attributes are predicted by experts and to identify residual attributes that remain unexplained by expert knowledge. We report Monte Carlo evidence about the accuracy of selecting active expert-predictors and present an application using Tatsuoka's fraction-subtraction dataset.