Short Answer Scoring Research Articles

A Trustworthy Automated Short-Answer Scoring System Using a New Dataset and Hybrid Transfer Learning Method

To measure the quality of student learning, teachers must conduct evaluations. One of the most efficient modes of evaluation is the short answer question. However, there can be inconsistencies in teacher-performed manual evaluations due to an excessive number of students, time demands, fatigue, etc. Consequently, teachers require a trustworthy system capable of autonomously and accurately evaluating student answers. Using hybrid transfer learning and student answer dataset, we aim to create a reliable automated short answer scoring system called Hybrid Transfer Learning for Automated Short Answer Scoring (HTL-ASAS). HTL-ASAS combines multiple tokenizers from a pretrained model with the bidirectional encoder representations from transformers. Based on our evaluation of the training model, we determined that HTL-ASAS has a higher evaluation accuracy than models used in previous studies. The accuracy of HTL-ASAS for datasets containing responses to questions pertaining to introductory information technology courses reaches 99.6%. With an accuracy close to one hundred percent, the developed model can undoubtedly serve as the foundation for a trustworthy ASAS system.

Deep Learning Based Automatic Answer Scoring Through Bi-Directional LSTM

Due to population expansion and the increasing importance of education, it is becoming increasingly difficult for assessors to evaluate the correctness and relevance of the responses provided by students. The LSTM model was initially used to build the answer-scoring system. The Bi-LSTM model has been designed with callbacks to acquire the student answer scoring system due to the LSTM's limitations for optimal scoring. The proposed system has been implemented using the ASAP Short Answer Scoring dataset. The results show that the system developed using Bi-LSTM displays better performance than LSTM.

M-Sim: Multi-level Semantic Inference Model for Chinese short answer scoring in low-resource scenarios

Short answer scoring is a significant task in natural language processing. On datasets comprising numerous explicit or implicit symbols and quantization entities, the existing approaches continue to perform poorly. Additionally, the majority of relevant datasets contain few-shot samples, reducing model efficacy in low-resource scenarios. To solve the above issues, we propose a Multi-level Semantic Inference Model (M-Sim), which obtains features at multiple scales to fully consider the explicit or implicit entity information contained in the data. We then design a prompt-based data augmentation to construct the simulated datasets, which effectively enhance model performance in low-resource scenarios. Our M-Sim outperforms the best competitor models by an average of 1.48 percent in the F1 score. The data augmentation significantly increases all approaches’ performance by an average of 0.036 in correlation coefficient scores.

On the Effectiveness of Curriculum Learning in Educational Text Scoring

Automatic Text Scoring (ATS) is a widely-investigated task in education. Existing approaches often stressed the structure design of an ATS model and neglected the training process of the model. Considering the difficult nature of this task, we argued that the performance of an ATS model could be potentially boosted by carefully selecting data of varying complexities in the training process. Therefore, we aimed to investigate the effectiveness of curriculum learning (CL) in scoring educational text. Specifically, we designed two types of difficulty measurers: (i) pre-defined, calculated by measuring a sample's readability, length, the number of grammatical errors or unique words it contains; and (ii) automatic, calculated based on whether a model in a training epoch can accurately score the samples. These measurers were tested in both the easy-to-hard to hard-to-easy training paradigms. Through extensive evaluations on two widely-used datasets (one for short answer scoring and the other for long essay scoring), we demonstrated that (a) CL indeed could boost the performance of state-of-the-art ATS models, and the maximum improvement could be up to 4.5%, but most improvements were achieved when assessing short and easy answers; (b) the pre-defined measurer calculated based on the number of grammatical errors contained in a text sample tended to outperform the other difficulty measurers across different training paradigms.

Smart Essay Grading

Abstract: With the onset of the technological revolution, the efficiency of the current manual systems has been improved drastically and the accuracy of the products produced are also increasing. The most affected of the fields from this technological change is Education. The combination of Education with technology has been coined as a new term of edtech. Assessment in the Education system plays a significant role in judging student performance. Humans are becoming more interested in using automated tools. Consequently, in the last few years, the use of automatic assessment methods in the education system and student response evaluation has increased significantly. There is currently no adequate evaluation mechanism for grading essays and short responses; the computer-based evaluation system only works for multiple-choice questions. For the past few decades, many researchers have been working on automated essay grading and short answer scoring, but evaluating an essay while taking into account all the criteria, such as the content's relevance to the prompt, the development of ideas, cohesion, and coherence, is still a difficult task. We examined the limits of the most recent studies and research trends while studying the Artificial Intelligence and Machine Learning approaches used to assess computerized essay grading. In this project, we have studied different uses of Machine Learning and how we can improve the efficiency of the Essay Scoring using AI and other algorithms. In this project we also aim to highlight the problems faced by teachers and the possible solution which could be designed to overcome the problem.

A Multitask Learning System for Trait-based Automated Short Answer Scoring

A Multitask Learning System for Trait-based Automated Short Answer Scoring

Enhancing GAN-LCS Performance Using an Abbreviations Checker in Automatic Short Answer Scoring

Automatic short answer scoring methods have been developed with various algorithms over the decades. In the Indonesian language, the string-based similarity is more commonly used. This method is difficult to accurately measure the similarity of two sentences with significantly different word lengths. This problem has been handled by the Geometric Average Normalized-Longest Common Subsequence (GAN-LCS) method by eliminating non-contributive words utilizing the Longest Common Subsequence method. However, students’ answers may vary not only in character length but also in the words they choose. For instance, some students tend only to write the abbreviations or acronyms of the phrase instead of writing meaningful words. As a result, it will reduce the intersection character between the reference answer and the student answer. Moreover, it can change the sentence structure even though it has the same meaning by definition. Therefore, this study aims to improve GAN-LCS method performance by incorporating the abbreviation checker to handle the abbreviations or acronyms found in the reference answer or student answer. The dataset used in this study consisted of 10 questions with 1 reference answer for each question and 585 student answers. The experimental results show an improvement in GAN-LCS performance that could run 34.43% faster. Meanwhile, the Root Mean Square Error (RSME) value became lower by 7.65% and the correlation value was increased by 8%. Looking forward, future studies may continue to investigate a method for automatically generate the abbreviations dictionary.

Automated Short Answer Scoring Using an Ensemble of Neural Networks and Latent Semantic Analysis Classifiers

Automated Short Answer Scoring Using an Ensemble of Neural Networks and Latent Semantic Analysis Classifiers

AraScore: A deep learning-based system for Arabic short answer scoring

In the past years, Arabic NLP has been significantly lagging behind its English counterpart, but recent advancements in Natural Language Processing have made it possible for Arabic to catch up and show promising results for a multitude of tasks. Complex tasks such as short answer scoring, have been widely researched mainly for English, leveraging machine learning and state-of-the-art deep learning techniques. In this paper, we introduce the first deep learning-based system for Arabic short answer scoring, in efforts to provide a reliable system that can help teachers in the Arab world better utilize their time in other teaching activities that would elevate the quality of learning in the region. We empirically study different techniques and propose the best performing system based on our results, where we have achieved state-of-the-art performance, achieving a QWK score of 0.78 and showing how powerful and robust recent Arabic NLP tools have become.

Stacking Neural Network Models for Automatic Short Answer Scoring

Automatic short answer scoring is one of the text classification problems to assess students’ answers during exams automatically. Several challenges can arise in making an automatic short answer scoring system, one of which is the quantity and quality of the data. The data labeling process is not easy because it requires a human annotator who is an expert in their field. Further, the data imbalance process is also a challenge because the number of labels for correct answers is always much less than the wrong answers. In this paper, we propose the use of a stacking model based on neural network and XGBoost for classification process with sentence embedding feature. We also propose to use data upsampling method to handle imbalance classes and hyperparameters optimization algorithm to find a robust model automatically. We use Ukara 1.0 Challenge dataset and our best model obtained an F1-score of 0.821 exceeding the previous work at the same dataset.

AraScore: Investigating Response-Based Arabic Short Answer Scoring

There are more than 80 million students in the Arab world. Students take tests and examinations as an intrinsic part of their educational journey to assess the quality of learning and understanding of the examined material. Exams require an immense amount of resources to be conducted, but even a bigger amount of resources to be scored. Scoring of exams and papers takes up enormous time and effort from teachers all over the Arab world. This time and effort can be better utilized in other teaching activities that would elevate the quality of learning in the region. For that reason, the work presented in this paper investigates multiple supervised learning approaches for the domain of Arabic short answer scoring. Moreover, due to the scarcity of Arabic datasets, and to our knowledge, there is only one publicly available dataset for this specific task(AR-ASAG). Thus, we introduce a new publicly available dataset AraScore-Dataset. The model proposed is evaluated on 3 different datasets: 1-AraScore-Dataset 2-AR-ASAG 3-Two translated answer sets taken from the Hewlett Foundation SAS dataset. Based on the investigation, the most effective approach for Arabic automated short answer scoring is proposed, and the performance of using our dataset is compared to other publicly available datasets. The aim is to scale efficient and unbiased content scoring applications across different Arabic educational domains. The results showed that our newly developed response-based system and AraScore-Dataset have achieved state-of-the-art performance on Arabic Automated Short Answer Scoring.

ESAS: Towards Practical and Explainable Short Answer Scoring (Student Abstract)

Motivated by the mandate to design and deploy a practical, real-world educational tool for grading, we extensively explore linguistic patterns for Short Answer Scoring (SAS) as well as authorship feedback. We approach the SAS task via a multipronged approach that employs linguistic context features for capturing domain-specific knowledge while emphasizing on domain agnostic grading and detailed feedback via an ensemble of explainable statistical models. Our methodology quantitatively supersedes multiple automatic short answer scoring systems.

Multiple Data Augmentation Strategies for Improving Performance on Automatic Short Answer Scoring

Automatic short answer scoring (ASAS) is a research subject of intelligent education, which is a hot field of natural language understanding. Many experiments have confirmed that the ASAS system is not good enough, because its performance is limited by the training data. Focusing on the problem, we propose MDA-ASAS, multiple data augmentation strategies for improving performance on automatic short answer scoring. MDA-ASAS is designed to learn language representation enhanced by data augmentation strategies, which includes back-translation, correct answer as reference answer, and swap content. We argue that external knowledge has a profound impact on the ASAS process. Meanwhile, the Bidirectional Encoder Representations from Transformers (BERT) model has been shown to be effective for improving many natural language processing tasks, which acquires more semantic, grammatical and other features in large amounts of unsupervised data, and actually adds external knowledge. Combining with the latest BERT model, our experimental results on the ASAS dataset show that MDA-ASAS brings a significant gain over state-of-art. We also perform extensive ablation studies and suggest parameters for practical use.

Machine learning algorithm for grading open-ended physics questions in Turkish

Worldwide, open-ended questions that require short answers have been used in many exams in fields of science, such as the International Student Assessment Program (PISA), the International Science and Maths Trends Research (TIMSS). However, multiple-choice questions are used for many exams at the national level in Turkey, especially high school and university entrance exams. This study aims to develop an objective and useful automatic scoring model for open-ended questions using machine learning algorithms. Within the scope of this aim, an automated scoring model construction study was conducted on four Physics questions at a University level course with the participation of 246 undergraduate students. The short-answer scoring was handled through an approach that addresses students’ answers in Turkish. Model performing machine learning classification techniques such as SVM (Support Vector Machines), Gini, KNN (k-Nearest Neighbors), and Bagging and Boosting were applied after data preprocessing. The score indicated the accuracy, precision and F1-Score of each predictive model of which the AdaBoost.M1 technique had the best performance. In this paper, we report on a short answer grading system in Turkish, based on a machine learning approach using a constructed dataset about a Physics course in Turkish. This study is also the first study in the field of open-ended exam scoring in Turkish.

Get IT Scored Using AutoSAS — An Automated System for Scoring Short Answers

In the era of MOOCs, online exams are taken by millions of candidates, where scoring short answers is an integral part. It becomes intractable to evaluate them by human graders. Thus, a generic automated system capable of grading these responses should be designed and deployed. In this paper, we present a fast, scalable, and accurate approach towards automated Short Answer Scoring (SAS). We propose and explain the design and development of a system for SAS, namely AutoSAS. Given a question along with its graded samples, AutoSAS can learn to grade that prompt successfully. This paper further lays down the features such as lexical diversity, Word2Vec, prompt, and content overlap that plays a pivotal role in building our proposed model. We also present a methodology for indicating the factors responsible for scoring an answer. The trained model is evaluated on an extensively used public dataset, namely Automated Student Assessment Prize Short Answer Scoring (ASAP-SAS). AutoSAS shows state-of-the-art performance and achieves better results by over 8% in some of the question prompts as measured by Quadratic Weighted Kappa (QWK), showing performance comparable to humans.

Quantum-based automated short answer scoring system

Answer Scoring is defined as an act of assigning a score to an answer by a human grader. This scoring technique is costly and requires deep logical efforts and it depends on less-than-perfect human assessment. However, the Automated Scoring (AS) System has its importance in providing the student with a score as well as feedback within seconds. This paper describes an AS system in which scores are assigned to essays automatically based upon predefined algorithms. Most of the educational sectors carry out an important examination process, i.e. to examine and assess the capabilities of the student based on his/her given answers. To accomplish this process, the human graders can apply this Automated Answer Scoring system. The paper goes through the existing techniques for automated answer scoring systems and then goes on to explain the newly developed system in which scoring is done by the statistical method adopting and integrating rule-based semantic quantum-based features analysis resulting in more accuracy. It is in a way a hybrid system suitable for short answer type scoring. It also presents the methodology and architecture of AS.

Short answer scoring system using automatic reference answer generation and geometric average normalized-longest common subsequence (GAN-LCS)

The Automatic Short Answer Scoring (ASAS) system is one of the tools that can be used to conduct assessment process on e-learning system. One of the methods applied in the ASAS system is a method for measuring similarities between the reference and student answers. There are two issues to be considered in the assessment process using this method. First, this method should be able to provide a variety of reference answers that can handle the diversity of student answers. Secondly, this method should be able to provide an accurate sentence similarity between the reference answers and student answers. Therefore, two methods are proposed to solve both problems. The first method is to generate a variety of reference answers automatically using Maximum Marginal Relevance (MMR) method, which obtains an accuracy of 91.95%. The second method is to measure accurately sentence similarity between student answers and reference answers that have significantly different length using GAN-LCS. The performance of the proposed method shows an improvement of the Root Mean Square Error (RMSE) value of 0.884 and a correlation value of 0.468.

Short answer scoring using W-Bleu for regular assessment in vocational high school

The assessment process in a vocational school needs tools that can integrate both knowledge assessment and skill assessment, among which are the construct response question. One example of the construct response question is automated short answer scoring. However, there is a problem with automated short-answer scoring that is how to overcome the difference in sentence length between student answer and reference answer that has the same meaning. The difference in the length of these sentences occurs because each student provides answers with different additional information. This paper proposes a method for handling the problem by weighting the key words that make up the sentence in the reference answer. This weighting process is done by modifying the Bleu method into weighted bleu (W-Bleu). The proposed method produces a process of automated short answer scoring that obtains a correlation value of 0.64 and root mean square error of 1.01.

Short answer scoring using W-Bleu for regular assessment in vocational high school

The assessment process in a vocational school needs tools that can integrate both knowledge assessment and skill assessment, among which are the construct response question. One example of the construct response question is automated short answer scoring. However, there is a problem with automated short-answer scoring that is how to overcome the difference in sentence length between student answer and reference answer that has the same meaning. The difference in the length of these sentences occurs because each student provides answers with different additional information. This paper proposes a method for handling the problem by weighting the key words that make up the sentence in the reference answer. This weighting process is done by modifying the Bleu method into weighted bleu (W-Bleu). The proposed method produces a process of automated short answer scoring that obtains a correlation value of 0.64 and root mean square error of 1.01.

Automatic Short Answer Scoring based on Paragraph Embeddings

Automatic scoring systems for students’ short answers can eliminate from instructors the burden of grading large number of test questions and facilitate performing even more assessments during lectures especially when number of students is large. This paper presents a supervised learning approach for short answer automatic scoring based on paragraph embeddings. We review significant deep learning based models for generating paragraph embeddings and present a detailed empirical study of how the choice of paragraph embedding model influences accuracy in the task of automatic scoring.