Automated Test Assembly for Multistage Testing With Cognitive Diagnosis.

Dongbo Tu,Xuliang Gao,Daxun Wang,Guiyu Li,Yan Cai

doi:10.3389/fpsyg.2021.509844

Abstract

Computer multistage adaptive test (MST) combines the advantages of paper and pencil-based test (P&P) and computer-adaptive test (CAT). As CAT, MST is adaptive based on modules; as P&P, MST can meet the need of test developers to manage test forms and keep test forms parallel. Cognitive diagnosis (CD) can accurately measure students’ knowledge states (KSs) and provide diagnostic information, which is conducive to student’s self-learning and teacher’s targeted teaching. Although MST and CD have a lot of advantages, many factors prevent MST from applying to CD. In this study, we first attempt to employ automated test assembly (ATA) to achieve the objectives of MST in the application of CD (called CD-MST) via heuristic algorithms. The mean correct response probability of all KSs for each item is used to describe the item difficulty of CD. The attribute reliability in CD is defined as the test quantitative target. A simulation study with the G-DINA model (generalized deterministic input noisy “and” gate model) was carried out to investigate the proposed CD-MST, and the results showed that the assembled panels of CD-MST satisfied the statistical and the non-statistical constraints.

Highlights

The computer multistage adaptive test (MST), as a “balanced compromise” between computer-adaptive test (CAT) and paper and pencilbased test (P&P), can provide high measurement accuracy as CAT (Kim et al, 2015) and can meet the need of test developers to manage test forms and keep test forms parallel
Some studies provided on-the-fly MST (OMST; Zheng and Chang, 2015), which may be a practical method of Cognitive diagnosis (CD)-MST, this may lead to many problems, such as (1) the test developer having difficulty in managing tests before administering, (2) the parallel of the test is difficult to ensure, (3) and the non-statistical constraint is difficult to satisfy
To address the above issues, a CD-MST framework that provides rich diagnostic information about the candidates and retains the inherent advantages of MST was proposed in this paper

Summary

Introduction

The computer multistage adaptive test (MST), as a “balanced compromise” between CAT and P&P, can provide high measurement accuracy as CAT (Kim et al, 2015) and can meet the need of test developers to manage test forms and keep test forms parallel. MST allows subjects to modify the item answers in the current stage, which is beneficial to reduce the examinees’ test anxiety and improve the measurement accuracy. MST can allow examinees to modify their item answers in the current stage, which helps alleviate test anxiety while avoiding measurement mistakes caused by errors. (4) Compared with CAT online testing, MST preassembles a test before performing the test administration, which can help test developers better manage a test. Because of these benefits, many high-stake tests have switched from the CAT mode to the MST mode (Wang et al, 2015), such as the United States National Education Progress Assessment (NAEP), the US Graduate Entrance Examination (GRE), the Program for the International Assessment of Adult Competencies (PIAAC), and other large examinations (Yamamoto et al, 2018)

Objectives

Results

Conclusion