Log-linear Cognitive Diagnosis Model Research Articles

Properties and performance of the one-parameter log-linear cognitive diagnosis model

Diagnostic classification models (DCMs) are psychometric models that yield probabilistic classifications of respondents according to a set of discrete latent variables. The current study examines the recently introduced one-parameter log-linear cognitive diagnosis model (1-PLCDM), which has increased interpretability compared with general DCMs due to useful measurement properties like sum score sufficiency and invariance properties. We demonstrate its equivalence with the Latent Class/Rasch Model and discuss interpretational consequences. The model is further examined in a DCM framework. We demonstrate the sum score sufficiency property and we derive an expression for the cut score for mastery classification. It is shown by means of a simulation study that the 1-PLCDM is fairly robust to model constraint violations in terms of classification accuracy and reliability. This robustness in combination with useful measurement properties and ease of interpretation can make the model attractive for stakeholders to apply in various assessment settings.

Approximate Invariance Testing in Diagnostic Classification Models in the Presence of Attribute Hierarchies: A Bayesian Network Approach

This paper demonstrates the process of invariance testing in diagnostic classification models in the presence of attribute hierarchies via an extension of the log-linear cognitive diagnosis model (LCDM). This extension allows researchers to test for measurement (item) invariance as well as attribute (structural) invariance simultaneously in a single analysis. The structural model of the LCDM was parameterized as a Bayesian network, which allows attribute hierarchies to be modeled and tested for attribute invariance via a series of latent regression models. We illustrate the steps for carrying out the invariance analyses through an in-depth case study with an empirical dataset and provide JAGS code for carrying out the analysis within the Bayesian framework. The analysis revealed that a subset of the items exhibit partial invariance, and evidence of full invariance was found at the structural level.

A diagnostic comparison of Spanish and Turkish middle school students’ proportional reasoning

This small-scale comparative study investigated Spanish and Turkish middle school students’ mastery of four core cognitive skills required for proportional reasoning using the log-linear cognitive diagnosis model. We compared students in the two samples for their proportional reasoning and provided a diagnostic assessment of their strengths and weaknesses. The study included 314 Spanish eighth grade and 282 Turkish seventh grade students. The students were administered a proportional reasoning test with 22 multiple-choice items. We found significant differences between the two samples in two of the cognitive skills. While the Spanish sample were better at understanding nonproportional relationships, the Turkish sample were better at understanding the concept of ratio. Furthermore, while the Spanish sample had a more dispersed distribution of students among different attribute profiles, the Turkish sample mostly consisted of either low-performing or high-performing students. To better understand the similarities and differences between the two samples, we compared students’ performance on some specific test items. The study shows how cognitive diagnostic models, which are not widely used in small-scale comparative studies, can be very useful in revealing similarities and differences between two samples that cannot be detected by traditional psychometric models.

A Gibbs Sampling Algorithm with Monotonicity Constraints for Diagnostic Classification Models

Diagnostic classification models (DCMs) are restricted latent class models with a set of cross-class equality constraints and additional monotonicity constraints on their item parameters, both of which are needed to ensure the meaning of classes and model parameters. In this paper, we develop an efficient, Gibbs sampling-based Bayesian Markov chain Monte Carlo estimation method for general DCMs with monotonicity constraints. A simulation study was conducted to evaluate parameter recovery of the algorithm which showed accurate estimation of model parameters. Moreover, the proposed algorithm was compared to a previously developed Gibbs sampling algorithm which imposed constraints on only the main effect item parameters of the log-linear cognitive diagnosis model. The newly proposed algorithm showed less bias and faster convergence. An analysis of the 2000 Programme for International Student Assessment reading assessment data using this algorithm was also conducted.

Cognitive Diagnostic Models for Rater Effects.

In recent decades, cognitive diagnostic models (CDMs) have been intensively researched and applied to various educational and psychological tests. However, because existing CDMs fail to consider rater effects, the application of CDMs to constructed-response (CR) items that involve human raters is seriously limited. Given the popularity of CR items, it is desirable to develop new CDMs that are capable of describing and estimating rater effects on CR items. In this study, we developed such new CDMs within the frameworks of facets models and hierarchical rater models, using the log-linear cognitive diagnosis model as a template. The parameters of the new models were estimated with the Markov chain Monte Carlo methods implemented in the freeware JAGS. Simulations were conducted to evaluate the parameter recovery of the new models. Results showed that the parameters were recovered fairly well and the more data there were, the better the recovery. Implications and applications of the new models were illustrated with an empirical study that adopted a fine-grained checklist to assess English academic essays.

A diagnostic assessment to middle school students’ proportional reasoning

This study investigated Turkish middle school students’ proportional reasoning and provided a diagnostic assessment of their strengths and weaknesses on the ratio and proportion concepts. A proportional reasoning test with 22 multiple-choice items was developed from the context of the log-linear cognitive diagnosis model. The test was developed around four core cognitive skills (attributes) that required in solving middle school ratio and proportion problems. These skills included understanding ratios, directly, inversely, and nonproportional relationships. The test was applied to 282 seventh grade students, and the collected data were analyzed using the Mplus software. The analysis showed that approximately 62% of the students were able to recognize directly proportional relationships. Whereas, roughly 48% of them were able to recognize inversely proportional relationships. Moreover, while 25% of the students did not master any of the four cognitive skills, 39.1% mastered all four of these skills. In addition, many students had difficulty distinguishing proportional relationships from nonproportional relationships. Diagnostic feedbacks on the students’ strengths and weaknesses were provided based on the findings.

Diagnosing Preservice Teachers’ Understanding of Statistics and Probability: Developing a Test for Cognitive Assessment

This study investigates preservice middle school mathematics teachers’ understanding of statistics and probability and provides a cognitive diagnostic assessment of their strengths and weaknesses on these subjects. A statistical reasoning test that included 15 multiple-choice and 5 open-ended items was developed from the perspective of the log-linear cognitive diagnosis model, which is a general form of the cognitive diagnosis models. The statistical reasoning test was applied to 456 preservice teachers from 4 universities in 3 different regions of Turkey. The collected data were analyzed using the Mplus 6.12 software, and diagnostic feedback on the preservice teachers’ responses was provided based on the findings. The analysis suggested that although many preservice teachers were able to represent and interpret the given data, most experienced difficulty in drawing inferences about populations based on samples, selecting and using appropriate statistical methods, and understanding and applying the basic concepts of probability. In addition, preservice teachers had difficulty answering open-ended items. Implications for teaching are also discussed.

Using the log-linear cognitive diagnosis model to classify individuals in career and educational assessment

The log-linear cognitive diagnosis model (LCDM) is a modern technique that dichotomously classifies individuals (e.g., possession and non-possession) on attributes of a multidimensional construct, which is particularly suited to diagnostic and formative assessments where a classification decision is desired (e.g., to assign intervention or not). This article provides a tutorial for conducting analyses using the LCDM and illustrates an application in career assessment with simulated data to showcase the unique types of diagnostic feedback the LCDM provides. The article concludes with a discussion of the ideal contexts in which to apply the model and its implications in career and educational assessment.

Digital Module 04: Diagnostic Measurement: Modeling Checklists for Practitioners https://ncme.elevate.commpartners.com

AbstractIn this ITEMS module, we provide a didactic overview of the specification, estimation, evaluation, and interpretation steps for diagnostic measurement/classification models (DCMs), which are a promising psychometric modeling approach. These models can provide detailed skill‐ or attribute‐specific feedback to respondents along multiple latent dimensions and hold theoretical and practical appeal for a variety of fields. We use a current unified modeling framework—the log‐linear cognitive diagnosis model (LCDM)—as well as a series of quality‐control checklists for data analysts and scientific users to review the foundational concepts, practical steps, and interpretational principles for these models. We demonstrate how the models and checklists can be applied in real‐life data‐analysis contexts. A library of macros and supporting files for Excel, SAS, and Mplus are provided along with video tutorials for key practices.

Effects of Item Calibration Errors on Computerized Adaptive Testing under Cognitive Diagnosis Models

In a cognitive diagnostic computerized adaptive testing (CD-CAT) exam, an item pool that consists of items with calibrated item parameters is used for item selection and attribute estimation. The parameter estimates for the items in the item pool are often treated as if they were the true population parameters, and therefore, the calibration errors are ignored. The purpose of this study was to investigate the effects of calibration errors on the attribute classification accuracy, the measurement precision of attribute mastery classification, and the test information under the log-linear cognitive diagnosis model (LCDM) framework. The deterministic input, noisy “and” gate (DINA) model and the compensatory re-parameterized unified model (C-RUM) were used in fixed-length CD-CAT simulations. The results showed that high levels of calibration errors were associated with low classification accuracy, low test information, and misleading estimation of measurement precision. The effects of calibration errors decreased as the test length increased, and the DINA model appeared to be more vulnerable in the presence of calibration errors. The C-RUM was less influenced by calibration errors because of its additive characteristics in the LCDM framework. The same conclusions applied when item exposure control was incorporated and when different item selection methods were used. Finally, the use of a larger calibration sample size to calibrate the item pool was found to reduce the magnitudes of error variances and increase the attribute classification accuracy.

Assessing Growth in a Diagnostic Classification Model Framework.

A common assessment research design is the single-group pre-test/post-test design in which examinees are administered an assessment before instruction and then another assessment after instruction. In this type of study, the primary objective is to measure growth in examinees, individually and collectively. In an item response theory (IRT) framework, longitudinal IRT models can be used to assess growth in examinee ability over time. In a diagnostic classification model (DCM) framework, assessing growth translates to measuring changes in attribute mastery status over time, thereby providing a categorical, criterion-referenced interpretation of growth. This study introduces the Transition Diagnostic Classification Model (TDCM), which combines latent transition analysis with the log-linear cognitive diagnosis model to provide methodology for analyzing growth in a general DCM framework. Simulation study results indicate that the proposed model is flexible, provides accurate and reliable classifications, and is quite robust to violations to measurement invariance over time. The TDCM is used to analyze pre-test/post-test data from a diagnostic mathematics assessment.

Using Hamiltonian Monte Carlo to estimate the log-linear cognitive diagnosis model via Stan.

The Bayesian literature has shown that the Hamiltonian Monte Carlo (HMC) algorithm is powerful and efficient for statistical model estimation, especially for complicated models. Stan, a software program built upon HMC, has been introduced as a means of psychometric modeling estimation. However, there are no systemic guidelines for implementing Stan with the log-linear cognitive diagnosis model (LCDM), which is the saturated version of many cognitive diagnostic model (CDM) variants. This article bridges the gap between Stan application and Bayesian LCDM estimation: Both the modeling procedures and Stan code are demonstrated in detail, such that this strategy can be extended to other CDMs straightforwardly.

Comparison of Relative Fit Indices for Diagnostic Model Selection.

The purpose of this study was to thoroughly examine the performance of three information-based fit indices-Akaike's Information Criterion (AIC), Bayesian Information Criterion (BIC), and sample-size-adjusted BIC (SABIC)-using the log-linear cognitive diagnosis model and a set of well-known item response theory (IRT) models. Two simulation studies were conducted to examine the extent to which relative fit indices can identify the generating model under a variety of data conditions and model misspecifications. Generally, indices performed better when item quality was stronger. When the IRT was the generating model, all three indices correctly selected the IRT model for all replications. When the true model was a diagnostic classification model, for all three fit indices, the multidimensional IRT model was incorrectly selected as frequently as 70% of the replications. The results of this study identify situations for researchers where commonly used-and typically well-performing-fit indices may not be appropriate to compare models for selection.

Joint Maximum Likelihood Estimation for Diagnostic Classification Models.

Joint maximum likelihood estimation (JMLE) is developed for diagnostic classification models (DCMs). JMLE has been barely used in Psychometrics because JMLE parameter estimators typically lack statistical consistency. The JMLE procedure presented here resolves the consistency issue by incorporating an external, statistically consistent estimator of examinees' proficiency class membership into the joint likelihood function, which subsequently allows for the construction of item parameter estimators that also have the consistency property. Consistency of the JMLE parameter estimators is established within the framework of general DCMs: The JMLE parameter estimators are derived for the Loglinear Cognitive Diagnosis Model (LCDM). Two consistency theorems are proven for the LCDM. Using the framework of general DCMs makes the results and proofs also applicable to DCMs that can be expressed as submodels of the LCDM. Simulation studies are reported for evaluating the performance of JMLE when used with tests of varying length and different numbers of attributes. As a practical application, JMLE is also used with "real world" educational data collected with a language proficiency test.

An Application of M2 Statistic to Evaluate the Fit of Cognitive Diagnostic Models

The fit of cognitive diagnostic models (CDMs) to response data needs to be evaluated, since CDMs might yield misleading results when they do not fit the data well. Limited-information statistic M2 and the associated root mean square error of approximation (RMSEA2) in item factor analysis were extended to evaluate the fit of CDMs. The findings suggested that the M2 statistic has proper empirical Type I error rates and good statistical power, and it could be used as a general statistical tool. More importantly, we found that there was a strong linear relationship between mean marginal misclassification rates and RMSEA2 when there was model–data misfit. The evidence demonstrated that .030 and .045 could be reasonable thresholds for excellent and good fit, respectively, under the saturated log-linear cognitive diagnosis model.

Invariance Properties for General Diagnostic Classification Models

In item response theory (IRT), the invariance property states that item parameter estimates are independent of the examinee sample, and examinee ability estimates are independent of the test items. While this property has long been established and understood by the measurement community for IRT models, the same cannot be said for diagnostic classification models (DCMs). DCMs are a newer class of psychometric models that are designed to classify examinees according to levels of categorical latent traits. We examined the invariance property for general DCMs using the log-linear cognitive diagnosis model (LCDM) framework. We conducted a simulation study to examine the degree to which theoretical invariance of LCDM classifications and item parameter estimates can be observed under various sample and test characteristics. Results illustrated that LCDM classifications and item parameter estimates show clear invariance when adequate model data fit is present. To demonstrate the implications of this important property, we conducted additional analyses to show that using pre-calibrated tests to classify examinees provided consistent classifications across calibration samples with varying mastery profile distributions and across tests with varying difficulties.

A Family of Generalized Diagnostic Classification Models for Multiple Choice Option-Based Scoring.

This article proposes a new family of diagnostic classification models (DCM) called the Generalized Diagnostic Classification Models for Multiple Choice Option-Based Scoring (GDCM-MC). The GDCM-MC is created for multiple choice assessments with response options designed to attract particular kinds of student thinking and understanding, both desired (correct) thinking and problematic (incorrect or partially correct) thinking. Key features that combine to distinguish GDCM-MC are: (a) an expanded latent space that can include both desirable and problematic facets of thinking, (b) an expanded Q matrix that includes a row for each response option and that uses a three-valued coding scheme to specify which latent states are strongly attracted to that option, (c) a guessing component that responds to the forced choice aspect of multiple choice questions, and (d) a general modeling framework that can incorporate the diagnostic modeling functionality of almost any dichotomous DCM, such as deterministic input, noisy ``and'' gate (DINA), reparameterized unified model (RUM), loglinear cognitive diagnosis model (LCDM), or general diagnostic model (GDM). The article discusses these four components and presents the GDCM-MC model equation as a mixture of cognitive and guessing components. Two identifiability theorems are presented. A Bayesian Markov Chain Monte Carlo (MCMC) model estimation algorithm is discussed, and real and simulated data studies are reported.

The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model.

Diagnostic classification models are psychometric models that aim to classify examinees according to their mastery or non-mastery of specified latent characteristics. These models are well-suited for providing diagnostic feedback on educational assessments because of their practical efficiency and increased reliability when compared with other multidimensional measurement models. A priori specifications of which latent characteristics or attributes are measured by each item are a core element of the diagnostic assessment design. This item-attribute alignment, expressed in a Q-matrix, precedes and supports any inference resulting from the application of the diagnostic classification model. This study investigates the effects of Q-matrix design on classification accuracy for the log-linear cognitive diagnosis model. Results indicate that classification accuracy, reliability, and convergence rates improve when the Q-matrix contains isolated information from each measured attribute.

The Use and Misuse of Psychometric Models

The purpose of our paper entitled Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies (Templin & Bradshaw, 2014) was two-fold: to create a psychometric model and framework that would enable attribute hierarchies to be parameterized as dependent binary latent traits, and to formulate an empirically driven hypothesis test for the purpose of falsifying proposed attribute hierarchies. The methodological contributions of this paper were motivated by a curious result in the analysis of a real data set using the log-linear cognitive diagnosis model, or LCDM (Henson, Templin, & Willse, 2009). In the analysis of the Examination for Certification of Proficiency in English (ECPE; Templin & Hoffman, 2013), results indicated that few, if any, examinees were classified into four of the possible eight attribute profiles that are hypothesized in the LCDM for a test of three binary latent attributes. Further, when considering the four profiles lacking examinees, it appeared that some attributes must be mastered before others, suggesting what is commonly called an attribute hierarchy (e.g., Leighton, Gierl, & Hunka, 2004). Although the data analysis alerted us to the notion that such a data structure might be present, we lacked the methodological tools to falsify the presence of such an attribute hierarchy. As such, we developed the Hierarchical Diagnostic Classification Model, or HDCM, in an attempt to fill the need for such tools. We note that the driving force behind the HDCM is one of seeking a simpler, or more parsimonious, solution when model data misfit is either evident from LCDM results or implied by the hypothesized theories underlying the assessed constructs. As a consequence of the ECPE data results, we worked to develop a more broadly defined set of models that would allow for empirical evaluation of hypothesized attribute hierarchies. We felt our work was timely, as a number of methods, both new and old, are now using implied attribute hierarchies to assess examinees in many large scale analyses—from so-called intelligent tutoring systems (e.g., Cen, Koedinger, & Junker, 2006) to large scale state assessment systems for alternative assessments using instructionally imbedded items (e.g. the Dynamic Learning Maps Alternate Assessment System Consortium Grant, 2010–2015). Moreover, such large scale analyses are based on tremendously large data sets, many of which simply cannot fit with the types of (mainly unidimensional) models often used in current large scale testing situations. Furthermore, newly developed standards in education have incorporated ideas of learning progressions which indirectly imply the existence of hierarchically structured attributes (e.g., Progressions for the Common Core State Standards in Mathematics, Common Core State Standards Writing Team, 2012). In short, the current and

Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies

Although latent attributes that follow a hierarchical structure are anticipated in many areas of educational and psychological assessment, current psychometric models are limited in their capacity to objectively evaluate the presence of such attribute hierarchies. This paper introduces the Hierarchical Diagnostic Classification Model (HDCM), which adapts the Log-linear Cognitive Diagnosis Model to cases where attribute hierarchies are present. The utility of the HDCM is demonstrated through simulation and by an empirical example. Simulation study results show the HDCM is efficiently estimated and can accurately test for the presence of an attribute hierarchy statistically, a feature not possible when using more commonly used DCMs. Empirically, the HDCM is used to test for the presence of a suspected attribute hierarchy in a test of English grammar, confirming the data is more adequately represented by hierarchical attribute structure when compared to a crossed, or nonhierarchical structure.