Optimizing Implementation of Artificial‐Intelligence‐Based Automated Scoring: An Evidence Centered Design Approach for Designing Assessments for AI‐based Scoring

Abstract

AbstractArtificial‐intelligence‐based automated scoring is often an afterthought and is considered after assessments have been developed, resulting in nonoptimal possibility of implementing automated scoring solutions. In this article, we provide a review of Artificial intelligence (AI)‐based methodologies for scoring in educational assessments. We then propose an evidence‐centered design framework for developing assessments to align conceptualization, scoring, and ultimate assessment interpretation and use with the advantages and limitations of AI‐based scoring in mind. We provide recommendations for defining construct, task, and evidence models to guide task and assessment design that optimize the development and implementation of AI‐based automated scoring of constructed response items and support the validity of inferences from and uses of scores.