Nonparametric methods for cognitive diagnosis to multiple-choice test items

Abstract

<p id="p00005">Cognitive diagnostic assessment (CDA) focuses on evaluating students' advantages and disadvantages in knowledge mastering, providing an opportunity for individualized teaching. Therefore, CDA has attracted attention of many scholars, teachers, and students at domestic and overseas. In CDA and a large number of standardized tests, multiple-choice (MC) are typical item types, which have the advantages of not being affected by subjective errors, improving test reliability, being easy to review, scoring quickly, and meeting the needs of content balance. To fulfil the potential of MC items for CDA, researchers proposed the MC-cognitive diagnosis models (MC-CDMs). However, these MC-CDMs pertain to parameter methods, which need a large sample size to obtain accurate parameter estimation. They are not suitable for small samples at class level, and the MCMC algorithm is very time-consuming. In this study, three nonparametric MC cognitive diagnosis methods based on hamming-distance are proposed, aiming at maximizing the diagnostic efficacy of MC items and being suitable for the diagnosis target of a small sample. <p id="p00010">Simulation study 1 considered four factors: sample size (30, 50, 100), test length (10, 20, 30), item quality (high and low), and the true model (MC-S-DINA1, MC-S-DINA2). Three nonparametric MC methods and two parametric models were compared. The results showed that in most conditions, the pattern accuracy rates and average attribute accuracy rates of the nonparametric MC method(<inline-formula>${{d}_{\text{h}-\text{MC}}}$</inline-formula>) were higher than those of parametric models, especially when the test length was short or item quality was low. <p id="p00015">In a real test situation, the quality of different items in a test may vary greatly. Based on this, simulation study 2 set the first half of the items at high quality and the remaining items at low quality. The results showed that the pattern accuracy rates and average attribute accuracy rates of the nonparametric MC method (<inline-formula>${{d}_{\text{ph}-\text{MC}}}$</inline-formula>) were higher than those of the parametric models in all conditions. <p id="p00020">In an empirical study, the nonparametric MC methods and the parametric models were used to analyze a set of real data simultaneously. The results showed that nonparametric MC methods and parametric models presented high classification consistency rates. Furthermore, the <inline-formula>${{d}_{\text{ph}-\text{MC}}}$</inline-formula> method had satisfactory estimations. <p id="p00025">In sum, <inline-formula>${{d}_{\text{h}-\text{MC}}}$</inline-formula> was suitable in most conditions, especially when the test length was short or the item quality was low When the quality of different items was quite diverse, <inline-formula>${{d}_{\text{ph}-\text{MC}}}$</inline-formula> was a better choice compared with parameteric approaches.