Transformer Model Fine-Tuning for Indonesian Automated Essay Scoring with Semantic Textual Similarity

Abstract

The quality of exam assessment is a very important part of education. Because the assessment process plays a role in various aspects of education. The results of the assessment are used to measure students’ abilities, as a reference for achieving learning targets, evaluating educational curriculum, evaluating learning processes and others. In manual assessment, several problems arise when the amount of data that must be assessed is large. Manual assessments are time-consuming, subjective and will potentially lead to unbalanced assessments, especially if there are multiple raters involved. Automated Essay Scoring (AES) emerged as a new research field to address this problem. Many researchers have conducted research with various methods to overcome some of these problems. The most recent technology namely neural network has recently given fantastic results in NLP task. However, most of these AES systems use large datasets, so it's relatively common to get good results. Bidirectional Encoder Representations from Transformers (BERT)-based approach can improve NLP tasks with limited training data. But still has some drawbacks when used specifically in AES. In this paper, we propose a transformer-based AES model which is optimized by fine-tuning and hyperparameter-optimization methods to produce more accurate scoring. The results obtained based on the Quadratic Weighted Kappa (QWK) measurement are 93% and the accuracy is 92 %.