Psychometric Models for a New State Science Assessment Aligned to the Next Generation Science Standards

Abstract

AbstractThe complexity of the Next Generation Science Standards (NGSS) poses significant task design, psychometric, and practical challenges for assessments. This study focuses on the psychometric challenges and explores an appropriate measurement model to interpret scores for an NGSS-aligned state science assessment. Multiple item response theory (IRT) models based on content specifications were applied to the data collected from a pilot test of the newly developed science assessment to identify the most appropriate model. Results suggest that although the three-dimensional IRT model that aligns with the NGSS dimensions provides slightly better overall model fit than the unidimensional IRT model and the testlet model, the item-level fit of the three-dimensional model is poor. Implementing multidimensional IRT (MIRT) models requires large sample sizes and a much longer estimation time, which poses challenges in an operational setting. Future studies can be conducted to further evaluate the need for using MIRT models and the robustness of a unidimensional model under various test conditions.KeywordsMultidimensional science assessment designNGSS-aligned assessmentMultidimensional IRT models