Automated Essay Scoring System Research Articles

Using Automated Procedures to Score Educational Essays Written in Three Languages

AbstractThe purpose of this study is to describe and evaluate a multilingual automated essay scoring (AES) system for grading essays in three languages. Two different sentence embedding models were evaluated within the AES system, multilingual BERT (mBERT) and language‐agnostic BERT sentence embedding (LaBSE). German, Italian, and Czech essays were holistically scored using the Common European Framework of Reference of Languages. The AES system with mBERT produced results that were consistent with human raters overall across all three language groups. The system also produced accurate predictions for some but not all of the score levels within each language. The AES system with LaBSE produced results that were even more consistent with the human raters overall across all three language groups compared to mBERT. In addition, the system produced accurate predictions for the majority of the score levels within each language. The performance differences between mBERT and LaBSE can be explained by considering how each language embedding model is implemented. Implications of this study for educational testing are also discussed.

DeBERTa with hats makes Automated Essay Scoring system better

Automated Essay Scoring (AES) is a rapidly growing field that applies natural language processing (NLP) and machine learning techniques to the analysis and evaluation of academic essays. By automating the process of evaluating essay quality, AES not only greatly reduces the workload of human graders but also ensures consistency and objectivity in the evaluation process. AES systems can evaluate essays based on multiple criteria, including organization, coherence, and content. With the advent of deep learning, AES has shown significant improvements in accuracy and reliability. AES systems have numerous applications in education, particularly in large-scale assessment and feedback loops. In this article, we delve into the use of an improved Bidirectional Encoder Representations from Transformers (BERT) architecture with disentangled attention mechanism known as DeBERTa for student question-based summarization. This is one of the downstream tasks within AES, which is of great significance for student learning assessment. The organic combination of DeBERTa-v3 and diverse hats like Light Gradient Boosting Machine (LGBM) algorithm and Extreme Gradient Boosting algorithm (XGBoost) has proven to be highly effective in achieving excellent results in this task, indicating their significant potential in real-world AES systems.

A crowdsourcing-based incremental learning framework for automated essays scoring

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training are taken into consideration. This paper aims to tackle this problem by proposing the Crowdsourcing-based Automated Essay Scoring (CAES) framework. The proposed framework gradually collects data through crowdsourcing and incrementally trains the AES models. In particular, we propose the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to simultaneously tackle catastrophic forgetting and concept drift. The proposed approach dynamically updates the exemplar set by the Dynamic Exemplar Herding algorithm to obtain the best approximation of the overall data distribution and selectively apply knowledge distillation on the model outputs by Linear Outlier Suppression loss to retain the learned knowledge. Moreover, we use a lightweight AES model for effective and efficient essay scoring. The experimental results show that our proposed ILDEH approach outperforms other strong baseline approaches for the AES task. Moreover, the CAES framework is able to steadily improve the AES performance in the crowdsourcing environment with only 10.6% training time of the conventional approach. Further analysis shows that one single CPU server can support daily updates of more than 300 AES models, which is sufficient for most practical AES systems.

Exploring the Efficacy of Automated Essay Scoring Systems in the Context of Korean EFL High School

Despite growing attention on the Automated Essay Scoring (AES) systems, it is difficult to find their actual use in Korean EFL high schools. Among various free and easily accessible AES systems available, this study aims to investigate the efficacy of three AES systems in the Korean EFL high school context: Grammarly, MS Editor, and ProWritingAid. To this end, the three AES systems were compared with two human raters in three aspects: scores reported on Korean EFL high school students’ writings, errors detected per error category, and errors detected per student writing. The t-test, ANOVA, and correlation analysis results have led to questions and doubts on the validity of the AES programs and their role as essay evaluators. While AES programs have shown growth, the AES programs are not yet comparable to human raters and teacher feedback. However, the AES programs do show potential as a supplementary tool in the Korean EFL context. Thus, a comparison of the three AES systems was examined in the hopes of finding the optimal choice of AES system. The findings provide information and insights on how AES systems may be implemented in Korean EFL high school classrooms, and further emphasized the importance of teachers’ role in implementing a new tool.

The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation

Ethical considerations, including transparency, play an important role when using artificial intelligence (AI) in education. Explainable AI has been coined as a solution to provide more insight into the inner workings of AI algorithms. However, carefully designed user studies on how to design explanations for AI in education are still limited. The current study aimed to identify the effect of explanations of an automated essay scoring system on students’ trust and motivation. The explanations were designed using a needs-elicitation study with students in combination with guidelines and frameworks of explainable AI. Two types of explanations were tested: full-text global explanations and an accuracy statement. The results showed that both explanations did not have an effect on student trust or motivation compared to no explanations. Interestingly, the grade provided by the system, and especially the difference between the student’s self-estimated grade and the system grade, showed a large influence. Hence, it is important to consider the effects of the outcome of the system (here: grade) when considering the effect of explanations of AI in education.

Using Active Learning Methods to Strategically Select Essays for Automated Scoring

AbstractResearch on Automated Essay Scoring has become increasing important because it serves as a method for evaluating students’ written responses at scale. Scalable methods for scoring written responses are needed as students migrate to online learning environments resulting in the need to evaluate large numbers of written‐response assessments. The purpose of this study is to describe and evaluate three active learning methods that can be used to minimize the number of essays that must be scored by human raters while still providing the data needed to train a modern Automated Essay Scoring system. The three active learning methods are the uncertainty‐based, the topological‐based, and the hybrid method. These three methods were used to select essays included in the Automated Student Assessment Prize competition that were then classified using a scoring model that was trained with the bidirectional encoder representations from a transformer language model. All three active learning methods produced strong results, with the topological‐based method producing the most efficient classification. Growth rate accuracy was also evaluated. The active learning methods produced different levels of efficiency under different sample size allocations but, overall, all three methods were highly efficient and produced classifications that were similar to one another.

Examining Human and Automated Ratings of Elementary Students’ Writing Quality: A Multivariate Generalizability Theory Application

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students ( n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.

An efficient and enhancement of recent approaches to build an automated essay scoring system

In this paper, we present an efficient system (models) to automate the task of essay grading through a comprehensive study and enhancement of recent approaches namely multiple linear regression (MLR) and gradient boosted regression (Xg-boost) which can positively impact the operational efficiency, time consumption, labour dependency, Unbiased grading and consistency. We also illustrate the efficiency of the proposed systems(models) through a fair comparison of recent models with respect to the evaluation metric, features applied and limitations.

Big Data Analysis: Innovation of College English Writing Teaching: A Quantitative Research on “One Million Essays on Same Topics” English Writing Activity

Guided by the College English Curriculum Requirements ( published by National Foreign Languages Teaching Advisory Board in the Ministry of Education),this the paper aims at exploring methods to improve college English writing teaching based on data analysis. This paper uses the most popular automated essay scoring system and sentence correcting platform in China to launch a case study. Authors of this paper made quantitative research and data analysis on essays submitted in “One Million Essays on Same Topics” English Writing Activity, then summarized students’ writing characteristics and their shortcomings. Finally, authors of this paper provided suggestions on effective data-driven precise teaching model to improve college English writing teaching.

Penggunaan Penskor Jawaban Esai Otomatis dalam Pengukuran Pengetahuan Guru

Measuring the ability of teachers is part of the evaluation in the national education system. The development of the instrument in the form of essay questions has become an alternative because the instrument is considered to have better authenticity than the form of multiple choice questions. The implications of using the essay instrument were scored consistency and efficient resources. The purpose of this study was to examine the inter-rater reliability between the automated essay scoring system and human raters in measuring teachers’ knowledge using the essay instrument. The research was conducted randomly with 200 junior high school science teachers. The quantitative method design was applied to investigate the intra-class correlation coefficient (ICC) and Pearson's correlation (r) as indicators of the ability of automated essay scoring (UKARA) that had been used. The main data in this study were test answers from participants in the form of limited essay answers that have been distributed online. The inter-rater reliability coefficient between the UKARA and the human rater in this study was in the high category (more than 0.7) for all items or means that the score given by UKARA has a strong correlation with the score given by human rater. These results mean that UKARA has adequate capability as an automated essay scoring system on the measurement of science teacher knowledge.

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

AbstractThis paper focuses on automatically assessing language proficiency levels according to linguistic complexity in learner English. We implement a supervised learning approach as part of an automatic essay scoring system. The objective is to uncover Common European Framework of Reference for Languages (CEFR) criterial features in writings by learners of English as a foreign language. Our method relies on the concept of microsystems with features related to learner-specific linguistic systems in which several forms operate paradigmatically. Results on internal data show that different microsystems help classify writings from A1 to C2 levels (82% balanced accuracy). Overall results on external data show that a combination of lexical, syntactic, cohesive and accuracy features yields the most efficient classification across several corpora (59.2% balanced accuracy).

Automated Cross-prompt Scoring of Essay Traits

The majority of current research in Automated Essay Scoring (AES) focuses on prompt-specific scoring of either the overall quality of an essay or the quality with regards to certain traits. In real-world applications obtaining labelled data for a target essay prompt is often expensive or unfeasible, requiring the AES system to be able to perform well when predicting scores for essays from unseen prompts. As a result, some recent research has been dedicated to cross-prompt AES. However, this line of research has thus far only been concerned with holistic, overall scoring, with no exploration into the scoring of different traits. As users of AES systems often require feedback with regards to different aspects of their writing, trait scoring is a necessary component of an effective AES system. Therefore, to address this need, we introduce a new task named Automated Cross-prompt Scoring of Essay Traits, which requires the model to be trained solely on non-target-prompt essays and to predict the holistic, overall score as well as scores for a number of specific traits for target-prompt essays. This task challenges the model's ability to generalize in order to score essays from a novel domain as well as its ability to represent the quality of essays from multiple different aspects. In addition, we introduce a new, innovative approach which builds on top of a state-of-the-art method for cross-prompt AES. Our method utilizes a trait-attention mechanism and a multi-task architecture that leverages the relationships between each trait to simultaneously predict the overall score and the score of each individual trait. We conduct extensive experiments on the widely used ASAP and ASAP++ datasets and demonstrate that our approach is able to outperform leading prompt-specific trait scoring and cross-prompt AES methods.

Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined?

This article investigates the feasibility of using automated scoring methods to evaluate the quality of student-written essays. In 2012, Kaggle hosted an Automated Student Assessment Prize contest to find effective solutions to automated testing and grading. This article: a) analyzes the datasets from the contest – which contained hand-graded essays – to measure their suitability for developing competent automated grading tools; b) evaluates the potential for deep learning in automated essay scoring (AES) to produce sophisticated testing and grading algorithms; c) advocates for thorough and transparent performance reports on AES research, which will facilitate fairer comparisons among various AES systems and permit study replication; d) uses both deep neural networks and state-of-the-art NLP tools to predict finer-grained rubric scores, to illustrate how rubric scores are determined from a linguistic perspective, and to uncover important features of an effective rubric scoring model. This study’s findings first highlight the level of agreement that exists between two human raters for each rubric as captured in the investigated essay dataset, that is, 0.60 on average as measured by the quadratic weighted kappa (QWK). Only one related study has been found in the literature which also performed rubric score predictions through models trained on the same dataset. At best, the predictive models had an average agreement level (QWK) of 0.53 with the human raters, below the level of agreement among human raters. In contrast, this research’s findings report an average agreement level per rubric with the two human raters’ resolved scores of 0.72 (QWK), well beyond the agreement level between the two human raters. Further, the AES system proposed in this article predicts holistic essay scores through its predicted rubric scores and produces a QWK of 0.78, a competitive performance according to recent literature where cutting-edge AES tools generate agreement levels between 0.77 and 0.81, results computed as per the same procedure as in this article. This study’s AES system goes one step further toward interpretability and the provision of high-level explanations to justify the predicted holistic and rubric scores. It contends that predicting rubric scores is essential to automated essay scoring, because it reveals the reasoning behind AIED-based AES systems. Will building AIED accountability improve the trustworthiness of the formative feedback generated by AES? Will AIED-empowered AES systems thoroughly mimic, or even outperform, a competent human rater? Will such machine-grading systems be subjected to verification by human raters, thus paving the way for a human-in-the-loop assessment mechanism? Will trust in new generations of AES systems be improved with the addition of models that explain the inner workings of a deep learning black box? This study seeks to expand these horizons of AES to make the technique practical, explainable, and trustable.

Implications of AES System of Pigai for Self-regulated Learning

Recent preoccupation with Chinese English-as-the-foreign-language (EFL) learners’ unsatisfying writing performance leads to a growing interest in the use of automatic essay scoring system (AES) in EFL writing in China. Pigai (www.pigai.org), being an online AES system specialised for Chinese EFL learners, has been popularised. Yet, research into Pigai is underdeveloped and constrained by simply evaluating its linguistic effectiveness; while this research innovates in investigating the efficacy of Pigai as a self-regulated learning (SRL) instrument to judge whether it should be promoted to benefit Chinese EFL learners’ writing. This study analysed and compared the changes in the quality of texts after revising via Pigai and contrasted learners’ tendency to English writing before and after the use of Pigai, ultimately examining its qualification as a SRL instrument along the sociological, pedagogical and psychological dimensions. Data for this study was collected via an experiment and follow-up interviews with undergraduate Chinese EFL learners. Results suggest that Pigai users achieved improvements in the quality of the texts and showed greater passion and persistence, as well as confidence, oriented to EFL writing. It is thus inferred that Pigai is qualified enough as a SRL instrument and could be applied into Chinese learners’ English writing.

Automated Chinese Essay Scoring based on Deep Learning

Writing is an important part of language learning and is considered the best approach to demonstrate the comprehensive language skills of students. Manually grading student essays is a time-consuming task; however, it is necessary. An automated essay scoring system can not only greatly improve the efficiency of essay scoring, but also provide more objective score. Therefore, many researchers have been exploring automated essay scoring techniques and tools. However, the technique of scoring Chinese essays is still limited, and its accuracy needs to be enhanced further. To improve the accuracy of the scoring model for a Chinese essay, we propose an automated scoring approach based on a deep learning model and validate its effect by conducting two comparison experiments. The experimental results indicate that the accuracy of the proposed model is significantly higher than that of multiple linear regression (MLR), which was commonly used in the past. The three accuracy rates of the proposed model are comparable to those of the novice teacher. The root mean square error (RMSE) of the proposed model is slightly lower than that of the novice teacher, and the correlation coefficient of the proposed model is also significantly higher than that of the novice teacher. Besides, when the predicted scores are not very low or very high, the two predicted models are as good as a novice teacher. However, when the predicted score is very high or very low, the results should be treated with caution.

A Trait-based Deep Learning Automated Essay Scoring System with Adaptive Feedback

Numerous Automated Essay Scoring (AES) systems have been developed over the past years. Recent advances in deep learning have shown that applying neural network approaches to AES systems has accomplished state-of-the-art solutions. Most neural-based AES systems assign an overall score to given essays, even if they depend on analytical rubrics/traits. The trait evaluation/scoring helps to identify learners’ levels of performance. Besides, providing feedback to learners about their writing performance is as important as assessing their level. Producing adaptive feedback to the learners requires identifying the strengths/weaknesses and the magnitude of influence of each trait. In this paper, we develop a framework that strengthens the validity and enhances the accuracy of a baseline neural-based AES model with respect to traits evaluation/scoring. We extend the model to present a method based on essay traits prediction to give trait-specific adaptive feedback. We explored multiple deep learning models for the automatic essay scoring task, and we performed several analyses to get some indicators from these models. The results show that Long Short-Term Memory (LSTM) based system outperformed the baseline study by 4.6% in terms of quadratic weighted Kappa (QWK). Moreover, the prediction of the traits scores enhance the efficiency of the prediction of the overall score. Our extended model is used in the iAssistant, an educational module that provides trait-specific adaptive feedback to learners.

Mitigating Examination Malpractices in Nigerian Universities using an Enhanced Automated Essay Scoring System

Mitigating Examination Malpractices in Nigerian Universities using an Enhanced Automated Essay Scoring System

College students’ perceptions of using an online automated essay scoring system

With an ever-increasing use of technology in language learning, a vast variety of online Automated Essay Scoring (AES) systems has been developed and applied in English as a Foreign Language (EFL) writing class. By providing instant feedback to the learners in their writing process, the system is used in and outside of the class to improve students’ writing skills. Among all the systems, the PIGAI system is the most commonly used writing tool in China. This study aimed to investigate college students’ perceptions and writing performances following the use of the PIGAI system in their English writing class. The current research adopted both quantitative as well as qualitative methods on 25 college students taking English Writing Course. The results indicated that students took a positive attitude toward using the PIGAI system, especially in the aspects of vocabulary and grammar. It is interesting to note that students at a higher level of English proficiency argued the system sometimes gave them confusing information and scores lower than expected. Moreover, they pointed out that the teacher’s feedback is equally necessary for their writing process. By contrast, students at a lower English level of proficiency considered that it was a good writing tool to help them boost their writing confidence before submitting an essay to the teacher. Our empirical findings highlight the importance of teachers and researchers appreciating the need for proper, balanced use of both the AES system and the teacher’s feedback in a writing class when designing the course plan.

Generalizability of automated scores of writing quality in Grades 3–5.

The present study examined issues pertaining to the reliability of writing assessment in the elementary grades, and among samples of struggling and nonstruggling writers. The present study also extended nascent research on the reliability and the practical applications of automated essay scoring (AES) systems in Response to Intervention frameworks aimed at preventing and remediating writing difficulties (RTI-W). Students in Grade 3 (n = 185), Grade 4 (n = 192), and Grade 5 (n = 193) responded to six writing prompts, two prompts each in the three genres emphasized in the Common Core and similar “Next Generation” academic standards: narrative, informative, and persuasive. Prompts were scored using an AES system called Project Essay Grade (PEG). Generalizability theory was used to examine the following sources of variation in PEG’s quality scores: prompts, genres, and the interaction among those facets and the object of measurement: students. Separate generalizability and decision studies were conducted for each grade level and for subsamples of nonstruggling and struggling writers identified using a composite measure of writing skill. Low-stakes decisions (reliability ≥ .80) could be made by averaging scores from a single prompt per genre (i.e., 3 total) or 2 prompts per genre if administered to struggling writers (i.e., 6 total). High-stakes decisions (reliability ≥ .90) could be made by averaging across two prompts per genre (6 total) or 4–5 prompts per genre if administered to struggling writers (12–15 total). Implications for use of AES within RTI-W and the construct validity of AES writing quality scores are discussed. (PsycINFO Database Record (c) 2019 APA, all rights reserved)

College Students’ Perceptions of Using an Online Automated Essay Scoring System

With an ever-increasing use of technology in language learning, a vast variety of online Automated Essay Scoring (AES) systems has been developed and applied in English as a Foreign Language (EFL) writing class. By providing instant feedback to the learners in their writing process, the system is used in and outside of the class to improve students’ writing skills. Among all the systems, the PIGAI system is the most commonly used writing tool in China. This study aimed to investigate college students’ perceptions and writing performances following the use of the PIGAI system in their English writing class. The current research adopted both quantitative as well as qualitative methods on 25 college students taking English Writing Course. The results indicated that students took a positive attitude toward using the PIGAI system, especially in the aspects of vocabulary and grammar. It is interesting to note that students at a higher level of English proficiency argued the system sometimes gave them confusing information and scores lower than expected. Moreover, they pointed out that the teacher’s feedback is equally necessary for their writing process. By contrast, students at a lower English level of proficiency considered that it was a good writing tool to help them boost their writing confidence before submitting an essay to the teacher. Our empirical findings highlight the importance of teachers and researchers appreciating the need for proper, balanced use of both the AES system and the teacher’s feedback in a writing class when designing the course plan.