Automated Essay Scoring Systems Research Articles

AI-powered language learning: The role of NLP in grammar, spelling, and pronunciation feedback

Abstract. The introduction of artificial intelligence (AI) and natural language processing (NLP) in language learning has transformed the practices of education. This paper analyzes the role of AI-powered tools in English language learning with specific focus on automated essay scoring (AES), speech recognition, and advanced grammar and spelling checkers. By examining the accuracy of the most effective AES systems, it is evident that automated feedback has increased the quality of writing by providing detailed, consistent, and objective feedback on submitted essays. Speech recognition is another tool that is being used to assess pronunciation and fluency of learners. The software is constantly improving, with advances in NLP and machine learning. The technology for fluency tests is becoming more accurate, and is helping to improve learner outcomes significantly. Lastly, we examine advanced grammar and spelling checkers, which are increasingly used for correcting the contextual errors of individual learners. The AI systems adapt to the user and their needs, which make the corrections more effective. This paper offers an overview of how AI is leading to personalized and effective language learning, ultimately enhancing the educational experience of students and embracing future possibilities in education technology.

DeBERTa with hats makes Automated Essay Scoring system better

Automated Essay Scoring (AES) is a rapidly growing field that applies natural language processing (NLP) and machine learning techniques to the analysis and evaluation of academic essays. By automating the process of evaluating essay quality, AES not only greatly reduces the workload of human graders but also ensures consistency and objectivity in the evaluation process. AES systems can evaluate essays based on multiple criteria, including organization, coherence, and content. With the advent of deep learning, AES has shown significant improvements in accuracy and reliability. AES systems have numerous applications in education, particularly in large-scale assessment and feedback loops. In this article, we delve into the use of an improved Bidirectional Encoder Representations from Transformers (BERT) architecture with disentangled attention mechanism known as DeBERTa for student question-based summarization. This is one of the downstream tasks within AES, which is of great significance for student learning assessment. The organic combination of DeBERTa-v3 and diverse hats like Light Gradient Boosting Machine (LGBM) algorithm and Extreme Gradient Boosting algorithm (XGBoost) has proven to be highly effective in achieving excellent results in this task, indicating their significant potential in real-world AES systems.

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

Language models in automated essay scoring: Insights for the Turkish language

The proliferation of large language models represents a paradigm shift in the landscape of automated essay scoring (AES) systems, fundamentally elevating their accuracy and efficacy. This study presents an extensive examination of large language models, with a particular emphasis on the transformative influence of transformer-based models, such as BERT, mBERT, LaBSE, and GPT, in augmenting the accuracy of multilingual AES systems. The exploration of these advancements within the context of the Turkish language serves as a compelling illustration of the potential for harnessing large language models to elevate AES performance in in low-resource linguistic environments. Our study provides valuable insights for the ongoing discourse on the intersection of artificial intelligence and educational assessment.

Exploring effective methods for automated essay scoring of non-native speakers

Automated essay scoring (AES) has become a valuable tool in educational settings, providing efficient and objective evaluations of student essays. However, the majority of AES systems have primarily focused on native English speakers, leaving a critical gap in the evaluation of non-native speakers’ writing skills. This research addresses this gap by exploring the effectiveness of automated essay-scoring methods specifically designed for non-native speakers. The study acknowledges the unique challenges posed by variations in language proficiency, cultural differences, and linguistic complexities when assessing non-native speakers’ writing abilities. This work focuses on the automated student assessment prize and Khon Kaen University academic English language test dataset and presents an approach that leverages variants of the long short-term memory network model to learn features and compare results with the Kappa coefficient. The findings demonstrate that the proposed framework and approach, which involve joint learning of different essay representations, yield significant benefits and achieve results comparable to state-of-the-art deep learning models. These results suggest that the novel text representation proposed in this paper holds promise as a new and effective choice for assessing the writing tasks of non-native speakers. The result of this study can apply to advance educational assessment practices and promote equitable opportunities for language learners worldwide by enhancing the evaluation process for non-native speakers

A crowdsourcing-based incremental learning framework for automated essays scoring

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training are taken into consideration. This paper aims to tackle this problem by proposing the Crowdsourcing-based Automated Essay Scoring (CAES) framework. The proposed framework gradually collects data through crowdsourcing and incrementally trains the AES models. In particular, we propose the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to simultaneously tackle catastrophic forgetting and concept drift. The proposed approach dynamically updates the exemplar set by the Dynamic Exemplar Herding algorithm to obtain the best approximation of the overall data distribution and selectively apply knowledge distillation on the model outputs by Linear Outlier Suppression loss to retain the learned knowledge. Moreover, we use a lightweight AES model for effective and efficient essay scoring. The experimental results show that our proposed ILDEH approach outperforms other strong baseline approaches for the AES task. Moreover, the CAES framework is able to steadily improve the AES performance in the crowdsourcing environment with only 10.6% training time of the conventional approach. Further analysis shows that one single CPU server can support daily updates of more than 300 AES models, which is sufficient for most practical AES systems.

Exploring the Efficacy of Automated Essay Scoring Systems in the Context of Korean EFL High School

Despite growing attention on the Automated Essay Scoring (AES) systems, it is difficult to find their actual use in Korean EFL high schools. Among various free and easily accessible AES systems available, this study aims to investigate the efficacy of three AES systems in the Korean EFL high school context: Grammarly, MS Editor, and ProWritingAid. To this end, the three AES systems were compared with two human raters in three aspects: scores reported on Korean EFL high school students’ writings, errors detected per error category, and errors detected per student writing. The t-test, ANOVA, and correlation analysis results have led to questions and doubts on the validity of the AES programs and their role as essay evaluators. While AES programs have shown growth, the AES programs are not yet comparable to human raters and teacher feedback. However, the AES programs do show potential as a supplementary tool in the Korean EFL context. Thus, a comparison of the three AES systems was examined in the hopes of finding the optimal choice of AES system. The findings provide information and insights on how AES systems may be implemented in Korean EFL high school classrooms, and further emphasized the importance of teachers’ role in implementing a new tool.

The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation

Ethical considerations, including transparency, play an important role when using artificial intelligence (AI) in education. Explainable AI has been coined as a solution to provide more insight into the inner workings of AI algorithms. However, carefully designed user studies on how to design explanations for AI in education are still limited. The current study aimed to identify the effect of explanations of an automated essay scoring system on students’ trust and motivation. The explanations were designed using a needs-elicitation study with students in combination with guidelines and frameworks of explainable AI. Two types of explanations were tested: full-text global explanations and an accuracy statement. The results showed that both explanations did not have an effect on student trust or motivation compared to no explanations. Interestingly, the grade provided by the system, and especially the difference between the student’s self-estimated grade and the system grade, showed a large influence. Hence, it is important to consider the effects of the outcome of the system (here: grade) when considering the effect of explanations of AI in education.

EssayGAN: Essay Data Augmentation Based on Generative Adversarial Networks for Automated Essay Scoring

In large-scale testing and e-learning environments, automated essay scoring (AES) can relieve the burden upon human raters by replacing human grading with machine grading. However, building AES systems based on deep learning requires a training dataset consisting of essays that are manually rated with scores. In this study, we introduce EssayGAN, an automatic essay generator based on generative adversarial networks (GANs). To generate essays rated with scores, EssayGAN has multiple generators for each score range and one discriminator. Each generator is dedicated to a specific score and can generate an essay rated with that score. Therefore, the generators can focus only on generating a realistic-looking essay that can fool the discriminator without considering the target score. Although ordinary-text GANs generate text on a word basis, EssayGAN generates essays on a sentence basis. Therefore, EssayGAN can compose not only a long essay by predicting a sentence instead of a word at each step, but can also compose a score-rated essay by adopting multiple generators dedicated to the target score. Since EssayGAN can generate score-rated essays, the generated essays can be used in the supervised learning process for AES systems. Experimental results show that data augmentation using augmented essays helps to improve the performance of AES systems. We conclude that EssayGAN can generate essays not only consisting of multiple sentences but also maintaining coherence between sentences in essays.

A Comprehensive Review of Automated Essay Scoring (AES) Research and Development

Automated Essay Scoring (AES) is a service or software that can predictively grade essay based on a pre-trained computational model. It has gained a lot of research interest in educational institutions as it expedites the process and reduces the effort of human raters in grading the essays as close to humans’ decisions. Despite the strong appeal, its implementation varies widely according to researchers’ preferences. This critical review examines various AES development milestones specifically on different methodologies and attributes used in deriving essay scores. To generalize existing AES systems according to their constructs, we attempted to fit all of them into three frameworks which are content similarity, machine learning and hybrid. In addition, we presented and compared various common evaluation metrics in measuring the efficiency of AES and proposed Quadratic Weighted Kappa (QWK) as standard evaluation metric since it corrects the agreement purely by chance when estimate the degree of agreement between two raters. In conclusion, the paper proposes hybrid framework standard as the potential upcoming AES framework as it capable to aggregate both style and content to predict essay grades Thus, the main objective of this study is to discuss various critical issues pertaining to the current development of AES which yielded our recommendations on the future AES development.

Automated Cross-prompt Scoring of Essay Traits

The majority of current research in Automated Essay Scoring (AES) focuses on prompt-specific scoring of either the overall quality of an essay or the quality with regards to certain traits. In real-world applications obtaining labelled data for a target essay prompt is often expensive or unfeasible, requiring the AES system to be able to perform well when predicting scores for essays from unseen prompts. As a result, some recent research has been dedicated to cross-prompt AES. However, this line of research has thus far only been concerned with holistic, overall scoring, with no exploration into the scoring of different traits. As users of AES systems often require feedback with regards to different aspects of their writing, trait scoring is a necessary component of an effective AES system. Therefore, to address this need, we introduce a new task named Automated Cross-prompt Scoring of Essay Traits, which requires the model to be trained solely on non-target-prompt essays and to predict the holistic, overall score as well as scores for a number of specific traits for target-prompt essays. This task challenges the model's ability to generalize in order to score essays from a novel domain as well as its ability to represent the quality of essays from multiple different aspects. In addition, we introduce a new, innovative approach which builds on top of a state-of-the-art method for cross-prompt AES. Our method utilizes a trait-attention mechanism and a multi-task architecture that leverages the relationships between each trait to simultaneously predict the overall score and the score of each individual trait. We conduct extensive experiments on the widely used ASAP and ASAP++ datasets and demonstrate that our approach is able to outperform leading prompt-specific trait scoring and cross-prompt AES methods.

Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined?

This article investigates the feasibility of using automated scoring methods to evaluate the quality of student-written essays. In 2012, Kaggle hosted an Automated Student Assessment Prize contest to find effective solutions to automated testing and grading. This article: a) analyzes the datasets from the contest – which contained hand-graded essays – to measure their suitability for developing competent automated grading tools; b) evaluates the potential for deep learning in automated essay scoring (AES) to produce sophisticated testing and grading algorithms; c) advocates for thorough and transparent performance reports on AES research, which will facilitate fairer comparisons among various AES systems and permit study replication; d) uses both deep neural networks and state-of-the-art NLP tools to predict finer-grained rubric scores, to illustrate how rubric scores are determined from a linguistic perspective, and to uncover important features of an effective rubric scoring model. This study’s findings first highlight the level of agreement that exists between two human raters for each rubric as captured in the investigated essay dataset, that is, 0.60 on average as measured by the quadratic weighted kappa (QWK). Only one related study has been found in the literature which also performed rubric score predictions through models trained on the same dataset. At best, the predictive models had an average agreement level (QWK) of 0.53 with the human raters, below the level of agreement among human raters. In contrast, this research’s findings report an average agreement level per rubric with the two human raters’ resolved scores of 0.72 (QWK), well beyond the agreement level between the two human raters. Further, the AES system proposed in this article predicts holistic essay scores through its predicted rubric scores and produces a QWK of 0.78, a competitive performance according to recent literature where cutting-edge AES tools generate agreement levels between 0.77 and 0.81, results computed as per the same procedure as in this article. This study’s AES system goes one step further toward interpretability and the provision of high-level explanations to justify the predicted holistic and rubric scores. It contends that predicting rubric scores is essential to automated essay scoring, because it reveals the reasoning behind AIED-based AES systems. Will building AIED accountability improve the trustworthiness of the formative feedback generated by AES? Will AIED-empowered AES systems thoroughly mimic, or even outperform, a competent human rater? Will such machine-grading systems be subjected to verification by human raters, thus paving the way for a human-in-the-loop assessment mechanism? Will trust in new generations of AES systems be improved with the addition of models that explain the inner workings of a deep learning black box? This study seeks to expand these horizons of AES to make the technique practical, explainable, and trustable.

Automated scoring of junior and senior high essays using Coh-Metrix features: Implications for large-scale language testing

An automated essay scoring (AES) program is a software system that uses techniques from corpus and computational linguistics and machine learning to grade essays. In this study, we aimed to describe and evaluate particular language features of Coh-Metrix for a novel AES program that would score junior and senior high school students’ essays from their large-scale assessments. Specifically, we studied nine categories of Coh-Metrix features for developing prompt-specific AES scoring models for our sample. We developed the models by capitalizing on the nine features’ informativeness as a function of dimensionality reduction. We used a three-staged scoring framework. The machine scores were validated against a “gold standard” of ratings, that is, those assigned by two human raters. The nine language features reliably captured the construct of the students’ writing quality. We performed a secondary analysis to see how the scoring models performed in relation to other, already established AES systems, and there was no systematic pattern of scoring discrepancy. However, for essays with widely divergent human ratings, the scoring models were disadvantaged owing to the inherent unreliability of the human scores.

A Trait-based Deep Learning Automated Essay Scoring System with Adaptive Feedback

Numerous Automated Essay Scoring (AES) systems have been developed over the past years. Recent advances in deep learning have shown that applying neural network approaches to AES systems has accomplished state-of-the-art solutions. Most neural-based AES systems assign an overall score to given essays, even if they depend on analytical rubrics/traits. The trait evaluation/scoring helps to identify learners’ levels of performance. Besides, providing feedback to learners about their writing performance is as important as assessing their level. Producing adaptive feedback to the learners requires identifying the strengths/weaknesses and the magnitude of influence of each trait. In this paper, we develop a framework that strengthens the validity and enhances the accuracy of a baseline neural-based AES model with respect to traits evaluation/scoring. We extend the model to present a method based on essay traits prediction to give trait-specific adaptive feedback. We explored multiple deep learning models for the automatic essay scoring task, and we performed several analyses to get some indicators from these models. The results show that Long Short-Term Memory (LSTM) based system outperformed the baseline study by 4.6% in terms of quadratic weighted Kappa (QWK). Moreover, the prediction of the traits scores enhance the efficiency of the prediction of the overall score. Our extended model is used in the iAssistant, an educational module that provides trait-specific adaptive feedback to learners.

Automated language essay scoring systems: a literature review.

BackgroundWriting composition is a significant factor for measuring test-takers’ ability in any language exam. However, the assessment (scoring) of these writing compositions or essays is a very challenging process in terms of reliability and time. The need for objective and quick scores has raised the need for a computer system that can automatically grade essay questions targeting specific prompts. Automated Essay Scoring (AES) systems are used to overcome the challenges of scoring writing tasks by using Natural Language Processing (NLP) and machine learning techniques. The purpose of this paper is to review the literature for the AES systems used for grading the essay questions.MethodologyWe have reviewed the existing literature using Google Scholar, EBSCO and ERIC to search for the terms “AES”, “Automated Essay Scoring”, “Automated Essay Grading”, or “Automatic Essay” for essays written in English language. Two categories have been identified: handcrafted features and automatically featured AES systems. The systems of the former category are closely bonded to the quality of the designed features. On the other hand, the systems of the latter category are based on the automatic learning of the features and relations between an essay and its score without any handcrafted features. We reviewed the systems of the two categories in terms of system primary focus, technique(s) used in the system, the need for training data, instructional application (feedback system), and the correlation between e-scores and human scores. The paper includes three main sections. First, we present a structured literature review of the available Handcrafted Features AES systems. Second, we present a structured literature review of the available Automatic Featuring AES systems. Finally, we draw a set of discussions and conclusions.ResultsAES models have been found to utilize a broad range of manually-tuned shallow and deep linguistic features. AES systems have many strengths in reducing labor-intensive marking activities, ensuring a consistent application of scoring criteria, and ensuring the objectivity of scoring. Although many techniques have been implemented to improve the AES systems, three primary challenges have been identified. The challenges are lacking of the sense of the rater as a person, the potential that the systems can be deceived into giving a lower or higher score to an essay than it deserves, and the limited ability to assess the creativity of the ideas and propositions and evaluate their practicality. Many techniques have only been used to address the first two challenges.

Appraising the scoring performance of automated essay scoring systems—Some additional considerations: Which essays? Which human raters? Which scores?

ABSTRACTThe literature on Automated Essay Scoring (AES) systems has provided useful validation frameworks for any assessment that includes AES scoring. Furthermore, evidence for the scoring fidelity of AES systems is accumulating. Yet questions remain when appraising the scoring performance of AES systems. These questions include: (a) which essays are used to calibrate and test AES systems; (b) which human raters provided the scores on these essays; and (c) given that multiple human raters are generally used for this purpose, which human scores should ultimately be used when there are score disagreements? This article provides commentary on the first two questions and an empirical investigation into the third question. The authors suggest that addressing these three questions strengthens the scoring component of the validity argument for any assessment that includes AES scoring.

Discovering the Predictive Power of Five Baseline Writing Competences

Background: A shift of focus has been marked in recent years in the development of automated essay scoring systems (AES) passing from merely assigning a holistic score to an essay to providing constructive feedback over it. Despite all the major advances in the domain, many objections persist concerning their credibility and readiness to replace human scoring in high-stakes writing assessments. The purpose of this study is to shed light on how to build a relatively simple AES system based on five baseline writing features. The study shows that the proposed AES system compares very well with other state-of-the-art systems despite its obvious limitations. Literature Review: In 2012, ASAP (Automated Student Assessment Prize) launched a demonstration to benchmark the performance of state-of-the-art AES systems using eight hand-graded essay datasets originating from state writing assessments. These datasets are still used today to measure the accuracy of new AES systems. Recently, Zupanc and Bosnic (2017) developed and evaluated another state-of-the-art AES system, called SAGE, which enclosed new semantic and consistency features and provided for the first time an automatic semantic feedback. SAGE’s agreement level between machine and human scores for ASAP dataset #8 (the dataset also of interest in this study) was measured and had a quadratic weighted kappa of 0.81, while it ranged for 10 other state-of-the-art systems between 0.60 and 0.73 (Chen et al., 2012; Shermis, 2014). Finally, this section discusses the limitations of AES, which come mainly from its omission to assess higher-order thinking skills that all writing constructs are ultimately designed to assess. Research Questions: The research questions that guide this study are as follows: RQ1: What is the power of the writing analytics tool’s five-variable model (spelling accuracy, grammatical accuracy, semantic similarity, connectivity, lexical diversity) to predict the holistic scores of Grade 10 narrative essays (ASAP dataset #8)? RQ2: What is the agreement level between the computer rater based on the regression model obtained in RQ1 and the human raters who scored the 723 narrative essays written by Grade 10 students (ASAP dataset #8)? Methodology: ASAP dataset #8 was used to train the predictive model of the writing analytics tool introduced in this study. Each essay was graded by two teachers. In case of disagreement between the two raters, the scoring was resolved by a third rater. Basically, essay scores were the weighted sums of four rubric scores. A multiple linear regression analysis was conducted to determine the extent to which a five-variable model (selected from a set of 86 writing features) was effective to predict essay scores. Results: The regression model in this study accounted for 57% of the essay score variability. The correlation (Pearson), the percentage of perfect matches, the percentage of adjacent matches (±2), and the quadratic weighted kappa between the resolved scores and predicted essay scores were 0.76, 10%, 49%, and 0.73, respectively. The results were measured on an integer scale of resolved essay scores between 10-60. Discussion: When measuring the accuracy of an AES system, it is important to take into account several metrics to better understand how predicted essay scores are distributed along the distribution of human scores. Using average ranking over correlation, exact/adjacent agreement, quadratic weighted kappa, and distributional characteristics such as standard deviation and mean, this study’s regression model ranks 4th out of 10 AES systems. Despite its relatively good rank, the predictions of the proposed AES system remain imprecise and do not even look optimal to identify poor-quality essays (binary condition) smaller than or equal to a 65% threshold (71% precision and 92% recall). Conclusions: This study sheds light on the implementation process and the evaluation of a new simple AES system comparable to the state of the art and reveals that the generally obscure state-of-the-art AES system is most likely concerned only with shallow assessment of text production features. Consequently, the authors advocate greater transparency in the development and publication of AES systems. In addition, the relationship between the explanation of essay score variability and the inter-rater agreement level should be further investigated to better represent the changes in terms of level of agreement when a new variable is added to a regression model. This study should also be replicated at a larger scale in several different writing settings for more robust results.

Automated Essay Scoring feedback for second language writers: How does it compare to instructor feedback?

Abstract Writing is an essential component of students’ academic English development, yet it requires a considerable amount of time and effort on the part of both students and teachers. In an effort to reduce their workload, many instructors are looking into the use of Automated Essay Scoring (AES) systems to complement more traditional ways of providing feedback. This paper investigates the use of an AES system in a college ESL writing classroom. Participants included 14 advanced students from various linguistic backgrounds who wrote on three prompts and received feedback from the instructor and the AES system (Criterion). Instructor feedback on the drafts (n = 37) was compared to AES feedback and analyzed both quantitatively and qualitatively across the feedback categories of grammar (e.g., subject-verb agreement, ill-formed verbs), usage (e.g., incorrect articles, prepositions), mechanics (e.g., spelling, capitalization), and perceived quality by an additional ESL instructor. Data were triangulated with opinion surveys regarding student perceptions of the feedback received. The results show large discrepancies between the two feedback types (the instructor provided more and better quality feedback) and suggest important pedagogical implications by providing ESL writing instructors with insights regarding the use of AES systems in their classrooms.

Ethical Implementation of an Automated Essay Scoring (AES) System: A Case Study of Student and Instructor Use, Satisfaction, and Perceptions of AES In a Business Law Course

A pilot study of a vendor provided automated grading system was conducted in a Business Law class of 27 students. Students answered a business law fact pattern question which was reviewed and graded by the textbook vendor utilizing artificial intelligence software. Students were surveyed on their use, satisfaction, perceptions and technical issues utilizing the Write Experience automated essay scoring (AES) software. The instructor also chronicles the adoption, set up and use of an AES. Also detailed are the advantages and disadvantages of utilizing such software in an undergraduate course environment where some students may not be technologically adept or may lack motivation to experiment with a new testing procedure. Automated grading of student assignments is part of the next wave of textbook enhancements that vendors will be providing to instructors in the near future. Several vendors are conducting beta testing with instructors in hope of offering automated grading as part of their textbook and course support. Of course, such services will be an additional cost to the text. Exactly what will be charged for such services remains to be seen. The vast majority of previous research in the area of AES has been limited to its use in grading assignments in the STEM fields. Computer science instructors have been experimenting with self-created automated grading software for several decades. Recently automated grading software has been implemented by vendors to score essay questions in online tests such as the Graduate Management Admission Test (GMAT). The experience of one business law class in using automated grading software as part of such a beta test can give valuable insight into the needs and expectations of the three stakeholders in this situation – instructor, student and vendor. Implementation of an AES raises numerous issues. Should an instructor be willing to relinquish control of the grading process to an outside entity? Are the student needs for feedback and grading fairness being met? Is the technology advanced enough to replace the human element always present in grading assignments? What will be the economic impact on students if such software is adopted? The implications for the various stakeholders are discussed and addressed. The author comes to conclusions about the pedagogical usefulness of AES systems and offers suggestions for best practices to be employed by instructors interested in implementing such software in their courses.

Experimenting with a Japanese automated essay scoring system in the L2 Japanese environment

The purpose of this study is to provide an empirical analysis of the performance of an L1 Japanese automated essay scoring system which was on L2 Japanese compositions. In particular, this study concerns the use of such a system in formal L2 Japanese classes by the teachers (not in standardised tests). Thus experiments were designed accordingly. For this study, Jess, Japanese essay scoring system, was trialled using L2 Japanese compositions (n = 50). While Jess performed very well, being comparable with human raters in that the correlation between Jess and the average of the nine human raters is at least as high as the correlation between the 9 human raters, we also found: 1) that the performance of Jess is not as good as the reported performance of English automated essay scoring systems in the L2 environment and 2) that the very good compositions tend to be under-scored by Jess, indicating that Jess still has possible room for improvement.