Q-matrix Validation Research Articles

A new Q-matrix validation method based on signal detection theory.

The Q-matrix is a crucial component of cognitive diagnostic theory and an important basis for the research and practical application of cognitive diagnosis. In practice, the Q-matrix is typically developed by domain experts and may contain some misspecifications, so it needs to be refined using Q-matrix validation methods. Based on signal detection theory, this paper puts forward a new Q-matrix validation method (i.e., method) and then conducts a simulation study to compare the new method with existing methods. The results show that when the model is DINA (deterministic inputs, noisy 'and' gate), the method outperforms the existing methods under all conditions; under the generalized DINA (G-DINA) model, the method still has the highest validation rate when the sample size is small, and the item quality is high or the rate of Q-matrix misspecification is ≥.4. Finally, a sub-dataset of the PISA 2000 reading assessment is analysed to evaluate the reliability of the method.

Combining regularization and logistic regression model to validate the Q-matrix for cognitive diagnosis model.

Q-matrix is an important component of most cognitive diagnosis models (CDMs); however, it mainly relies on subject matter experts' judgements in empirical studies, which introduces the possibility of misspecified q-entries. To address this, statistical Q-matrix validation methods have been proposed to aid experts' judgement. A few of these methods, including the multiple logistic regression-based (MLR-B) method and the Hull method, can be applied to general CDMs, but they are either time-consuming or lack accuracy under certain conditions. In this study, we combine the L1 regularization and MLR model to validate the Q-matrix. Specifically, an L1 penalty term is imposed on the log-likelihood of the MLR model to select the necessary attributes for each item. A simulation study with various factors was conducted to examine the performance of the new method against the two existing methods. The results show that the regularized MLR-B method (a) produces the highest Q-matrix recovery rate (QRR) and true positive rate (TPR) for most conditions, especially with a small sample size; (b) yields a slightly higher true negative rate (TNR) than either the MLR-B or the Hull method for most conditions; and (c) requires less computation time than the MLR-B method and similar computation time as the Hull method. A real data set is analysed for illustration purposes.

Using Regularized Methods to Validate Q-Matrix in Cognitive Diagnostic Assessment

One of the central components of cognitive diagnostic assessment is the Q-matrix, which is an essential loading indicator matrix and is typically constructed by subject matter experts. Nonetheless, to a large extent, the construction of Q-matrix remains a subjective process and might lead to misspecifications. Many researchers have recognized the importance of estimating or validating the Q-matrix, but most of them focus on the conditions of relatively large sample sizes. This article aims to explore Q-matrix validation possibilities under small sample conditions and uses regularized methods to validate the Q-matrix based on the compensatory reparametrized unified model and generalized deterministic inputs, noisy “and” gate models. Simulation studies were conducted to evaluate the viability of the modified least absolute shrinkage and selection operator (Lasso) and modified smoothly clipped absolute deviation (SCAD) methods, comparing them with existing methods. Results show that the modified Lasso and the modified SCAD methods outperform the stepwise, Hull, and MLR-B methods in general, especially under the conditions of small sample sizes. While good recovery in all small sample size conditions is not guaranteed, the modified methods demonstrate advantages across various item quality conditions. Also, a real data set is analyzed to illustrate the application of the modified methods.

A Method of Empirical Q-Matrix Validation for Multidimensional Item Response Theory

ABSTRACT A Q-matrix is a binary matrix that defines the relationship between items and latent variables and is widely used in diagnostic classification models (DCMs), and can also be adopted in multidimensional item response theory (MIRT) models. The construction process of the Q-matrix is typically carried out by experts in the subject area of the items and statistical procedures can be used to verify its suitability. In DCMs, different approaches have been proposed for validating the Q-matrix through iterative algorithms. This article presents a method of empirical Q-matrix validation for MIRT models. A simulation study and an application to real data of morphology skills of elementary students are conducted to examine the viability of the method. Relevant issues regarding the implementation of the method and the results obtained are discussed.

Exploration of polytomous-attribute Q-matrix validation in cognitive diagnostic assessment

Compared with typical binary attributes, polytomous attributes can take three or more values (corresponding to different levels of mastery of a respondent or measurement of an item). They can indicate whether a respondent possesses the attributes of interest and mastery levels. Therefore, the test with polytomous-attribute Q-matrix can become more informative and provide respondents with richer diagnostic information than the test based on the dichotomous-attribute Q-matrix. This paper extends the S-statistic and the residual method applicable for the Q-matrix of binary attributes to validate the polytomous-attribute Q-matrix. Under two common scenarios in real-world applications, two associated validation algorithms: the joint validation (JV) algorithm and the online validation (OV) algorithm, are proposed. Both simulation studies and an empirical data example were employed to assess the robustness and usefulness of these two methods under various conditions. Results indicate that the JV algorithm is suitable for validating a Q-matrix predefined by subject matter experts. Especially when the Q-matrix contains fewer misspecifications, while the OV algorithm can be applied to define the attribute vector of "new items". Based on a certain number of "operational items", the OV algorithm can achieve a promising performance for obtaining the specification of the new items.

Using machine learning to improve Q-matrix validation.

The Q-matrix, which specifies the relationship between items and attributes, is a crucial component of cognitive diagnostic models (CDMs). A precisely specified Q-matrix allows for valid cognitive diagnostic assessments. In practice, a Q-matrix is usually developed by domain experts, and noted as being subjective and potentially containing misspecifications which can decrease the classification accuracy of examinees. To overcome this, some promising validation methods have been proposed, such as the general discrimination index (GDI) method and the Hull method. In this article, we propose four new methods for Q-matrix validation based on random forest and feed-forward neural network techniques. Proportion of variance accounted for (PVAF) and coefficient of determination (i.e., the McFadden pseudo-R2) are used as input features for developing the machine learning models. Two simulation studies are carried out to examine the feasibility of the proposed methods. Finally, a sub-dataset of the PISA 2000 reading assessment is analyzed as illustration.

An empirical Q-matrix validation method using complete information matrix in cognitive diagnostic models

An empirical Q-matrix validation method using complete information matrix in cognitive diagnostic models

Performance of the Q-Matrix Validation Methods in the DINA Model

All cognitive diagnostic models that evaluate educational test data require a Q-matrix that combines every item in a test with the required cognitive skills for each item to be answered correctly. Generally, the Q-matrix is constructed by education experts' judgment, leading to some uncertainty in its elements. Various statistical methods are suggested to validate misspecifications in the Q-matrix. This paper evaluates the performance of the Q-matrix validation methods, thesequential expectation-maximization-based δ-method (SEM δ-method), and the Q-matrix refinement (QMR) method using a study with real data and simulations.The simulation design results showed that the misspecification percentage and the length of the test had a small or no effect on the mean q-entry recovery rates (MRRs) of both methods, while the increase in sample size had an improving effect. The MRRs of both methods decreased when the number of attributes and guessing (g) -slip (s) parameters increased. According to simulation study results, the QMR method performed better than the SEM δ-method. For the q-matrix validation, it can be suggested that CDM practitioners prioritize the QMR method and use a sample size of 1,000. On the other hand, the real data results revealed that the MRRs of both methods were at the base rates. This result highlights the need for further research on method comparison, specifically for real-world data applications where the number of attributes is relatively large and the test duration is short.

A multiple logistic regression-based (MLR-B) Q-matrix validation method for cognitive diagnosis models:A confirmatory approach.

Q-matrix is an essential component specifying the relationship between attributes and items, which plays a key role in cognitive diagnosis assessment. The Q-matrix is usually developed by domain experts and its specifications tend to be subjective and might have misspecifications. Many existing pieces of research concentrate on the validation of Q-matrix; however, few of them can be applied to saturated cognitive diagnosis models. This paper proposes a general and effective Q-matrix validation method by employing multiple logistic regression model. Simulation studies are carried out to investigate the performance of the proposed method and compare it with four existing methods. Simulation results indicate the proposed method outperforms the existing methods in terms of validation accuracy. In addition, a set of real data is used as an example to illustrate its application. Finally, we discuss the limitations of the current study and the directions of future studies.

A simple and effective new method of Q-matrix validation

A simple and effective new method of Q-matrix validation

An Empirical Q-Matrix Validation Method for the Polytomous G-DINA Model.

A number of empirically based Q-matrix validation methods are available in the literature, all of which were developed for cognitive diagnosis models (CDMs) involving dichotomous attributes. However, in many applications, it is more instructionally relevant to classify students into more than two categories (e.g., no mastery, basic mastery, and advanced mastery). To extend the practical utility of CDMs, methods for validating the Q-matrix for CDMs that measure polytomous attributes are needed. This study focuses on validating the Q-matrix of the generalized deterministic input, noisy, "and" gate model for polytomous attributes (pG-DINA). The pGDI, an extension of the G-DINA model discrimination index, is proposed for polytomous attributes. The pGDI serves as the basis of a validation method that can be used not only to identify potential misspecified q-entries, but also to suggest more appropriate attribute-level specifications. The theoretical properties of the pGDI are underpinned by several mathematical proofs, whereas its practical viability is examined using simulation studies covering various conditions. The results show that the method can accurately identify misspecified q-entries and suggest the correct attribute-level specifications, particularly when high-quality items are involved. The pGDI is applied to a proportional reasoning test that measures several polytomous attributes.

An Application of Cognitive Diagnostic Modeling to a Listening Comprehension Test

인지진단모형은 학생 개개인의 지식에 대한 숙달 여부를 판정하여 교수·학습 과정의 다양한 의사 결정에 유용한 정보를 제공할 수 있다. 최근 인지진단모형은 다양한 모형의 개발, 효율적인 엘고리듬을 통한 모형의 일관성 있는 추정과 비교, Q-행렬의 통계적 타당도 검증 등에서 많은 진전을 이루었다. 이러한 기술적 발전과 잠재적 가능성에도 불구하고 실제 인지진단모형의 적용은 전문가들에 의한 대규모 시험 자료의 분석에 주로 국한되는 문제점을 갖고 있다. 이 연구에서는 일반 단위 학교에서 흔히 볼 수 있는 규모의 영어 듣기평가 자료에 인지진단모형을 적용하여 교수·학습 과정에 유용한 진단 정보를 도출할 수 있는지 조사하였다. 연구 결과에 따르면 소규모 시험 반응 자료에도 인지진단모형의 적용이 가능하나 모형의 축소와 일부 모수 추정값의 불안정성과 같은 문제점을 관찰할 수 있었다. 이 연구에서는 이러한 가능성과 제한성을 바탕으로 인지진단모형의 현장 적용에 관한 문제점과 개선 사항을 논의하고 있다.Cognitive diagnostic modeling(CDM) is intended for fine-grained diagnostic feedback that can facilitate learning and teaching by providing valuable information about students’ current mastery or nonmastry status on a set of skills in a subject domain. CDM has now matured into a comprehensive framework in which researchers and practitioners can find rich and sophisticated models with attribute hierarchies and problem-solving strategies integrated, fast estimation algorithms, efficient model fit checking and comparisons, statistical Q-matrix validation and powerful free softwares specifically developed for CDM. CDM has also received increasing research interest in language testing for the last 20 years or so. Despite these recent developments and inherent potential in CDM, its applications to subject domains including language testing have been carried out by a limited number of researchers who somehow obtained access to large-scale test data from testing programs. This study was to apply a comprehensive cognitive diagnostic model(GDINA) and its nested models to a small data set from an English listening comprehension test and to explore whether a meaningful diagnostic information could be drawn from data that are commonly available in a unit school. The results show that CDM can be successfully applied to the small size data and provide useful diagnostic information. The study also reports and discusses problems as well as issues encountered in the process of CDM applications.

Balancing fit and parsimony to improve Q-matrix validation.

The Q-matrix identifies the subset of attributes measured by each item in the cognitive diagnosis modelling framework. Usually constructed by domain experts, the Q-matrix might contain some misspecifications, disrupting classification accuracy. Empirical Q-matrix validation methods such as the general discrimination index (GDI) and Wald have shown promising results in addressing this problem. However, a cut-off point is used in both methods, which might be suboptimal. To address this limitation, the Hull method is proposed and evaluated in the present study. This method aims to find the optimal balance between fit and parsimony, and it is flexible enough to be used either with a measure of item discrimination (the proportion of variance accounted for, PVAF) or a coefficient of determination (pseudo-R2 ). Results from a simulation study showed that the Hull method consistently showed the best performance and shortest computation time, especially when used with the PVAF. The Wald method also performed very well overall, while the GDI method obtained poor results when the number of attributes was high. The absence of a cut-off point provides greater flexibility to the Hull method, and it places it as a comprehensive solution to the Q-matrix specification problem in applied settings. This proposal is illustrated using real data.

Improving Robustness in Q-Matrix Validation Using an Iterative and Dynamic Procedure.

In the context of cognitive diagnosis models (CDMs), a Q-matrix reflects the correspondence between attributes and items. The Q-matrix construction process is typically subjective in nature, which may lead to misspecifications. All this can negatively affect the attribute classification accuracy. In response, several methods of empirical Q-matrix validation have been developed. The general discrimination index (GDI) method has some relevant advantages such as the possibility of being applied to several CDMs. However, the estimation of the GDI relies on the estimation of the latent group sizes and success probabilities, which is made with the original (possibly misspecified) Q-matrix. This can be a problem, especially in those situations in which there is a great uncertainty about the Q-matrix specification. To address this, the present study investigates the iterative application of the GDI method, where only one item is modified at each step of the iterative procedure, and the required cutoff is updated considering the new parameter estimates. A simulation study was conducted to test the performance of the new procedure. Results showed that the performance of the GDI method improved when the application was iterative at the item level and an appropriate cutoff point was used. This was most notable when the original Q-matrix misspecification rate was high, where the proposed procedure performed better 96.5% of the times. The results are illustrated using Tatsuoka's fraction-subtraction data set.

基于类别水平的多级计分认知诊断Q矩阵修正：相对拟合统计量视角

基于类别水平的多级计分认知诊断Q矩阵修正：相对拟合统计量视角

An empirical Q-matrix validation method for the sequential generalized DINA model.

As a core component of most cognitive diagnosis models, the Q-matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-matrix empirically because a misspecified Q-matrix could result in erroneous attribute estimation. Most existing Q-matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-matrix for graded response data based on the sequential generalized deterministic inputs, noisy 'and' gate (G-DINA) model. The proposed Q-matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.

Reconsidering Cutoff Points in the General Method of Empirical Q-Matrix Validation.

Cognitive diagnosis models (CDMs) are latent class multidimensional statistical models that help classify people accurately by using a set of discrete latent variables, commonly referred to as attributes. These models require a Q-matrix that indicates the attributes involved in each item. A potential problem is that the Q-matrix construction process, typically performed by domain experts, is subjective in nature. This might lead to the existence of Q-matrix misspecifications that can lead to inaccurate classifications. For this reason, several empirical Q-matrix validation methods have been developed in the recent years. de la Torre and Chiu proposed one of the most popular methods, based on a discrimination index. However, some questions related to the usefulness of the method with empirical data remained open due the restricted number of conditions examined, and the use of a unique cutoff point (EPS) regardless of the data conditions. This article includes two simulation studies to test this validation method under a wider range of conditions, with the purpose of providing it with a higher generalization, and to empirically determine the most suitable EPS considering the data conditions. Results show a good overall performance of the method, the relevance of the different studied factors, and that using a single indiscriminate EPS is not acceptable. Specific guidelines for selecting an appropriate EPS are provided in the discussion.

A Nondiagnostic Assessment for Diagnostic Purposes: Q-Matrix Validation and Item-Based Model Fit Evaluation for the TIMSS 2011 Assessment

Large-scale assessments are generally designed for summative purposes to compare achievement among participating countries. However, these nondiagnostic assessments have also been adapted in the context of cognitive diagnostic assessment for diagnostic purposes. Following the large amount of investments in these assessments, it would be cost-effective to draw finer-grained inferences about the attribute mastery. Nonetheless, the correctness of attribute specifications in the Q-matrix has not been verified, despite being designed by domain experts. Furthermore, the underlying process of TIMSS (Trends in International Mathematics and Science Study) assessment is unknown as it was not developed for diagnostic purposes. Thus, this study suggests an initial validating attribute specifications in the Q-matrix and thereafter defining specific reduced or saturated models for each item. In doing so, the two analyses were validated across 20 countries that were selected randomly for TIMSS 2011 data. Results show that attribute specifications can differ from expert opinions and the underlying model for each item can vary.

Investigation of Missing Responses in Q-Matrix Validation.

Missing data can be a serious issue for practitioners and researchers who are tasked with Q-matrix validation analysis in implementation of cognitive diagnostic models. The article investigates the impact of missing responses, and four common approaches (treat as incorrect, logistic regression, listwise deletion, and expectation-maximization [EM] imputation) for dealing with them, on the performance of two major Q-matrix validation methods (the EM-based δ-method and the nonparametric Q-matrix refinement method) across multiple factors. Results of the simulation study show that both validation methods perform better when missing responses are imputed using EM imputation or logistic regression instead of being treated as incorrect and using listwise deletion. The nonparametric Q-matrix validation method outperforms the EM-based δ-method in most conditions. Higher missing rates yield poorer performance of both methods. Number of attributes and items have an impact on performance of both methods as well. Results of a real data example are also discussed in the study.

An EM-Based Method for Q-Matrix Validation.

With the purpose to assist the subject matter experts in specifying their Q-matrices, the authors used expectation-maximization (EM)-based algorithm to investigate three alternative Q-matrix validation methods, namely, the maximum likelihood estimation (MLE), the marginal maximum likelihood estimation (MMLE), and the intersection and difference (ID) method. Their efficiency was compared, respectively, with that of the sequential EM-based δ method and its extension (ς2), the γ method, and the nonparametric method in terms of correct recovery rate, true negative rate, and true positive rate under the deterministic-inputs, noisy "and" gate (DINA) model and the reduced reparameterized unified model (rRUM). Simulation results showed that for the rRUM, the MLE performed better for low-quality tests, whereas the MMLE worked better for high-quality tests. For the DINA model, the ID method tended to produce better quality Q-matrix estimates than other methods for large sample sizes (i.e., 500 or 1,000). In addition, the Q-matrix was more precisely estimated under the discrete uniform distribution than under the multivariate normal threshold model for all the above methods. On average, the ς2 and ID method with higher true negative rates are better for correcting misspecified Q-entries, whereas the MLE with higher true positive rates is better for retaining the correct Q-entries. Experiment results on real data set confirmed the effectiveness of the MLE.