Embeddings for Automatic Short Answer Grading: A Scoping Review

Abstract

Automatic grading of short answers is an important task in computer-assisted assessment (CAA). Recently, embeddings, as semantic-rich textual representations, have been increasingly used to represent short answers and predict the grade. Despite the recent trend of applying embeddings in automatic short answer grading (ASAG), there are no systematic reviews of literature reporting on their usage. Therefore, following the PRISMA-ScR guidelines, this scoping review summarises relevant literature on the use of embeddings in ASAG, and reports on the current state of the art in that research area and on the identified knowledge gaps. We searched seven research databases for the relevant journal, conference, and workshop papers published from 2016 to July 2021. The inclusion criteria were based on the type of publication, its venue ranking, study focus, and evaluation methods. Upon the full-text screening, 17 articles were included in the scoping review. Among these, most of the articles used word embeddings, mainly to estimate the similarity of student and model answers using the cosine similarity measure or to initialise a neural network-based classification model. The contribution of embeddings to the performance of ASAG models compared to non-embedding features is inconclusive. Models employing embeddings were mostly evaluated on four public ASAG datasets using earlier ASAG methods as baselines. We summarise the reported evaluation results and draw conclusions on the performance of the state-of-the-art ASAG models.