Model Selection for Cogitative Diagnostic Analysis of the Reading Comprehension Test.

Hui Liu,Yufang Bian

doi:10.3389/fpsyg.2021.644764

Abstract

Reading subskills are generally regarded as continuous variables, while most models used in the previous reading diagnoses have the hypothesis that the latent variables are dichotomous. Considering that the multidimensional item response theory (MIRT) model has continuous latent variables and can be used for diagnostic purposes, this study compared the performances of MIRT with two representatives of traditionally widely used models in reading diagnoses [reduced reparametrized unified model (R-RUM) and generalized deterministic, noisy, and gate (G-DINA)]. The comparison was carried out with both empirical and simulated data. First, model-data fit indices were used to evaluate whether MIRT was more appropriate than R-RUM and G-DINA with real data. Then, with the simulated data, relations between the estimated scores from MIRT, R-RUM, and G-DINA and the true scores were compared to examine whether the true abilities were well-represented, correct classification rates under different research conditions for MIRT, R-RUM, and G-DINA were calculated to examine the person parameter recovery, and the frequency distributions of subskill mastery probability were also compared to show the deviation of the estimated subskill mastery probabilities from the true values in the general value distribution. The MIRT obtained better model-data fit, gained estimated scores being a more reasonable representation for the true abilities, had an advantage on correct classification rates, and showed less deviation from the true values in frequency distributions of subskill mastery probabilities, which means it can produce more accurate diagnostic information about the reading abilities of the test-takers. Considering that more accurate diagnostic information has greater guiding value for the remedial teaching and learning, and in reading diagnoses, the score interpretation will be more reasonable with the MIRT model, this study recommended MIRT as a new methodology for future reading diagnostic analyses.

Highlights

In the area of language testing, it is reasonable to expect diagnostic information because any language assessment has the potential to provide some diagnostic information (Bachman, 1990; Mousavi, 2002), and there is a series of reading diagnostic studies that have successfully been conducted
The correct classification rates for the three models were explored under different research conditions in the simulation section, with the result that multidimensional item response theory (MIRT) achieved the highest pattern correct classification rate (PCCR) and subskill correct classification rate (SCCR) under all conditions, and its improvement over the other two models increased as the subskill correlations increased
The estimated frequency distributions of the subskill mastery probability were compared to the true distribution, and the results revealed that the estimated frequency distributions of the subskill mastery probability from MIRT were more similar to the true values whereas, for reparametrized unified model (R-reparametrized unified model (RUM)) and G-DINA, the frequency distributions were very different from the corresponding true values

Summary

Introduction

In the area of language testing, it is reasonable to expect diagnostic information because any language assessment has the potential to provide some diagnostic information (Bachman, 1990; Mousavi, 2002), and there is a series of reading diagnostic studies that have successfully been conducted. Model Selection for Reading Diagnosis continuous nature of reading subskills has been provided, though reading subskills are generally regarded as continuous variables (Griffin and Nix, 1991; Lumley, 1993; Grosjean, 2001; Smith, 2004). Buck and Tatsuoka (1998) noted that dichotomizing continuous variables are a problem of their diagnostic study, diagnostic studies focusing on the continuous nature of reading comprehension test data have remained elusive. Reading comprehension test data consisting of continuous latent variables are the target data of this present study, and we investigate the performances of different models when they are used in reading diagnoses, aiming to determine whether a diagnostic model with continuous latent variables will gain an advantage when used in reading diagnostic analyses

Objectives

Methods

Results

Conclusion