Automated Scoring Models Research Articles

Enhancing the Performance of Automated Scoring Model for Kinematic Graph Answers Using Synthetic Graph Images

Enhancing the Performance of Automated Scoring Model for Kinematic Graph Answers Using Synthetic Graph Images

Examining the responsible use of zero-shot AI approaches to scoring essays

The promise of AI to alleviate the burdens of grading and potentially enhance writing instruction is an exciting prospect. However, we believe it is crucial to emphasize that the accuracy of AI is only one component of its responsible use in education. Various governmental agencies, such as NIST in the US, and non-governmental agencies like the UN, UNESCO, and OECD have published guidance on the responsible use of AI, which we have synthesized to come up with our principles for the responsible use of AI in assessments at ETS. Our principles include fairness and bias mitigation; privacy & security; transparency, explainability, and accountability; educational impact & integrity; and continuous improvement. The accuracy of AI-scoring is one component of our principles related to educational impact & integrity. In this work, we share our thoughts on fairness & bias mitigation, and transparency & explainability. We demonstrate an empirical evaluation of zero-shot scoring using GTP-4o, with an emphasis on fairness evaluations and explainability of these automated scoring models.

Strengths and weaknesses of automated scoring of free-text student answers

AbstractFree-text tasks, where students need to write a short answer to a specific question, serve as a well-established method for assessing learner knowledge. To address the high cost of manually scoring these tasks, automated scoring models can be used. Such models come in various types, each with its own strengths and weaknesses. Comparing these models helps in selecting the most suitable one for a given problem. Depending on the assessment context, this decision can be driven by ethical or legal considerations. When implemented successfully, a scoring model has the potential to substantially reduce costs and enhance the reliability of the scoring process. This article compares the different categories of scoring models across a set of crucial criteria that have immediate relevance to model employment in practice.

Facilitating self-directed language learning in real-life scene description tasks with automated evaluation

Engaging children in describing real-life scenes provides an effective approach to fostering language production and developing their language skills, enabling them to establish meaningful connections between their language proficiency and authentic contexts. However, for such learning tasks, there has been a lack of research focusing on promoting self-directed language learning by using artificial intelligence techniques, primarily due to the challenges of handling multimodal information involved in such tasks. To address this gap, this study introduced a two-stage automated evaluation method that employed emerging cross-modal matching AI techniques. Firstly, an automated scoring model was developed to evaluate the quality of students' responses to scene description tasks. Compared with manually assigned human scores, our model scored students' descriptions accurately, as evidenced by a small testing mean absolute error of 0.3969 for a total score of 10 points. Based on the scoring results, immediate feedback was then provided to students by generating targeted comments and suggestions. The goal of this feedback was to assist students in progressively improving their descriptions of daily-life scenes, thereby enabling them to practice their language skills independently. To assess the effectiveness of the feedback, a comprehensive investigation was conducted involving 157 students from middle schools in China, and both qualitative and quantitative experimental data were collected from the students. It is found that the quality of students' descriptions was improved significantly with the assistance of immediate feedback. On average, students achieved an increase of 1.48 points in their scores after making revisions based on the feedback. In addition, students reported positive learning experiences and expressed favorable opinions regarding the language learning tasks with the automated evaluation. The findings of this study have significant implications for future research and educational practice. They not only highlighted the potential of emerging cross-modal matching AI techniques in automatically evaluating learning tasks involving multimodal data but also suggested that providing immediate targeted feedback based on automated scoring results can effectively promote students’ self-directed language learning.

Clinical validation of automated depth camera-based measurement of the Fugl-Meyer assessment for upper extremity

Objective Depth camera-based measurement has demonstrated efficacy in automated assessment of upper limb Fugl-Meyer Assessment for paralysis rehabilitation. However, there is a lack of adequately sized studies to provide clinical support. Thus, we developed an automated system utilizing depth camera and machine learning, and assessed its feasibility and validity in a clinical setting. Design Validation and feasibility study of a measurement instrument based on single cross-sectional data. Setting Rehabilitation unit in a general hospital Participants Ninety-five patients with hemiparesis admitted for inpatient rehabilitation unit (2021–2023). Main measures Scores for each item, excluding those related to reflexes, were computed utilizing machine learning models trained on participant videos and readouts from force test devices, while the remaining reflex scores were derived through regression algorithms. Concurrent criterion validity was evaluated using sensitivity, specificity, percent agreement and Cohen's Kappa coefficient for ordinal scores of individual items, as well as correlations and intraclass correlation coefficients for total scores. Video-based manual assessment was also conducted and compared to the automated tools. Result The majority of patients completed the assessment without therapist intervention. The automated scoring models demonstrated superior validity compared to video-based manual assessment across most items. The total scores derived from the automated assessment exhibited a high coefficient of 0.960. However, the validity of force test items utilizing force sensing resistors was relatively low. Conclusion The integration of depth camera technology and machine learning models for automated Fugl-Meyer Assessment demonstrated acceptable validity and feasibility, suggesting its potential as a valuable tool in rehabilitation assessment.

한국어 논·서술 답안 자동채점 프로그램 PASTA-I 개발

In this study, we developed PASTA-I, an automated scoring program for Korean essays, to solve the problems of scoring burden and objectivity and consistency of scoring. PASTA-I builds a scoring model based on data on three topics from the ‘Essay Writing Evaluation Data’ released by the National Information society Agency(NIA) The scoring was done with 3 points scale across 10 scoring areas, and accordingly, the automated scoring model was implemented in this study so that scoring was done for each of the 10 scoring areas. The algorithm used to implement the scoring model was electra-kor-base, which was trained on Korean data based on the ELECTRA model, and the performance of the automated scoring model was verified using testing data set on three topics, including ‘The biography’, ‘Movie/reading review,’ and ‘My personality strengths and weaknesses,’ and the results showed an accuracy of .7 or higher in almost all areas.

Writing Quality Predictive Modeling: Integrating Register-Related Factors

The primary purpose of this study is to investigate the degree to which register knowledge, register-specific motivation, and diverse linguistic features are predictive of human judgment of writing quality in three registers—narrative, informative, and opinion. The secondary purpose is to compare the evaluation metrics of register-partitioned automated writing evaluation models in three conditions: (1) register-related factors alone, (2) linguistic features alone, and (3) the combination of these two. A total of 1006 essays ( n = 327, 342, and 337 for informative, narrative, and opinion, respectively) written by 92 fourth- and fifth-graders were examined. A series of hierarchical linear regression analyses controlling for the effects of demographics were conducted to select the most useful features to capture text quality, scored by humans, in the three registers. These features were in turn entered into automated writing evaluation predictive models with tuning of the parameters in a tenfold cross-validation procedure. The average validity coefficients (i.e., quadratic-weighed kappa, Pearson correlation r, standardized mean score difference, score deviation analysis) were computed. The results demonstrate that (1) diverse feature sets are utilized to predict quality in the three registers, and (2) the combination of register-related factors and linguistic features increases the accuracy and validity of all human and automated scoring models, especially for the registers of informative and opinion writing. The findings from this study suggest that students’ register knowledge and register-specific motivation add additional predictive information when evaluating writing quality across registers beyond that afforded by linguistic features of the paper itself, whether using human scoring or automated evaluation. These findings have practical implications for educational practitioners and scholars in that they can help strengthen consideration of register-specific writing skills and cognitive and motivational forces that are essential components of effective writing instruction and assessment.

Machine Learning Mixed Methods Text Analysis: An Illustration From Automated Scoring Models of Student Writing in Biology Education

Assessing student knowledge based on their writing using traditional qualitative methods is time-consuming. To improve speed and consistency of text analysis, we present our mixed methods development of a machine learning predictive model to analyze student writing. Our approach involves two stages: first an exploratory sequential design, and second an iterative complex design. We first trained our predictive model using qualitative coding of categories (ideas) in student writing. We next revised our model based on feedback from instructor-users. The model itself highlighted categories in need of revision. The contribution to mixed methods research lies in our innovative use of the machine learning tool as a rapid, consistent additional coder, and a resource that can predict codes for new student writing.

채점자질의 적용이 영어 자동채점 모델의 성능에 미치는 영향

Purpose: This study attempted to compare the performance of an automated scoring model based on scoring features and three other automated scoring models that do not use the scoring feature. Methods: The data used in this study were 300 essays written by native English speakers in the tenth grade, which were divided into training and validating data at a ratio of 7:3. The RF model was used to predict the scores of the essays based on the analysis of 106 linguistic features extracted from Coh-Metrix. The accuracy of this model was compared to that of three deep learning models― RNN, LSTM, and GRU―which do not use those scoring features. Lastly, in the case of the RF model, scoring features mainly affecting accuracy prediction were further analyzed. Results: RNN which is a type of deep learning had an agreement with human scoring results ranging from .39 to .69 (mean=.58) in each rating domain, while LSTM showed an agreement between .59 and .72 (mean=.64), and GRU’s agreement was between .59 and .72 (mean=.64). However, when using the RF model based on the scoring features, its average accuracy was between .60 to .76 (mean=.69), which is the highest agreement among the four models. In the case of the RF model, ‘word count,’ ‘sentence count,’ and ‘vocabulary density’ were common characteristics across all scoring domains, but several scoring features reflecting the characteristics of each scoring domain were also found. Conclusion: When large-scale data is not available, the RF model showed relatively high automated scoring performance, and the RF model that provides learners with pedagogical feedback based on scoring features would be more useful compared to the other three automated scoring models in terms of the utilization of the automated scoring result.

The relationship between features of source text use and integrated writing quality

Abstract Academic writing ability is an important aspect of success in higher education. Recently, standardized academic language proficiency tests (such as the TOEFL) have begun to include integrated writing tasks, which ask test-takers to read and/or listen to a passage and construct a response that reflects the information in the passage(s). Arguably, integrated tasks more closely resemble authentic academic tasks than independent tasks, and therefore increase the construct validity of assessment tools that include them (Cumming et al., 2005; Taylor & Angelis, 2008). A number of recent studies have investigated differences between the products and processes of responding to independent and integrated tasks (Guo et al., 2013; Kyle & Crossley, 2016; Plakans, 2009a; Plakans & Gebril, 2013). In this study, the relationship between automated indices of source text use and holistic quality scores is investigated, building on Plakans and Gebril (2013). The results indicate that a number of indices related to content word overlap and n-gram overlap explained a substantial portion of the variance in holistic scores. These results generally align with the findings of Plakans and Gebril (2013), and provide important implications for increasing the construct coverage of automated scoring models (such as e-rater).

Machine learning algorithm for grading open-ended physics questions in Turkish

Worldwide, open-ended questions that require short answers have been used in many exams in fields of science, such as the International Student Assessment Program (PISA), the International Science and Maths Trends Research (TIMSS). However, multiple-choice questions are used for many exams at the national level in Turkey, especially high school and university entrance exams. This study aims to develop an objective and useful automatic scoring model for open-ended questions using machine learning algorithms. Within the scope of this aim, an automated scoring model construction study was conducted on four Physics questions at a University level course with the participation of 246 undergraduate students. The short-answer scoring was handled through an approach that addresses students’ answers in Turkish. Model performing machine learning classification techniques such as SVM (Support Vector Machines), Gini, KNN (k-Nearest Neighbors), and Bagging and Boosting were applied after data preprocessing. The score indicated the accuracy, precision and F1-Score of each predictive model of which the AdaBoost.M1 technique had the best performance. In this paper, we report on a short answer grading system in Turkish, based on a machine learning approach using a constructed dataset about a Physics course in Turkish. This study is also the first study in the field of open-ended exam scoring in Turkish.

Automated Essay Scoring at Scale: A Case Study in Switzerland and Germany

AbstractIn this research report, we describe the design and empirical findings for a large‐scale study of essay writing ability with approximately 2,500 high school students in Germany and Switzerland on the basis of 2 tasks with 2 associated prompts, each from a standardized writing assessment whose scoring involved both human and automated components. For the human scoring aspect, we describe the methodology for training and monitoring human raters as well as for collecting their ratings within a customized platform. For the automated scoring aspect, we describe the methodology for training, evaluating, and selecting appropriate automated scoring models as well as correlational patterns of resulting task scores with scores from secondary measures. Analyses show that the human ratings were highly reliable and that effective prompt‐specific automated scoring models could be built with state‐of‐the‐art features and machine learning methods, which resulted in correlational patterns with secondary measures that were in line with general expectations. In closing, we discuss the methodological implications for conducting this kind of work at scale in the future.

Home-Grown Automated Essay Scoring in the Literature Classroom: A Solution for Managing the Crowd?

Managing crowded classes in terms of classroom assessment is a difficult task due to the amount of time which needs to be devoted to providing feedback to student products. In this respect, the present study aimed to develop an automated essay scoring environment as a potential means to overcome this problem. Secondarily, the study aimed to test if automatically-given scores would correlate with the scores given by a human rater. A quantitative research design employing a machine learning approach was preferred to meet the aims of the study. The data set to be used for machine learning consisted of 160 scored literary analysis essays written in an English Literature course, each essay analyzing a theme in a given literary work. To train the automated scoring model, LightSide software was used. First, textual features were extracted and filtered. Then, Logistic Regression, SMO, SVO, Logistic Tree and Naive Bayes text classification algorithms were tested by using 10-Fold Cross-Validation to reach the most accurate model. To see if the scores given by the computer correlated with the scores given by the human rater, Spearman’s Rank Order Correlation Coefficient was calculated. The results showed that none of the algorithms were sufficiently accurate in terms of the scores of the essays within the data set. It was also seen that the scores given by the computer were not significantly correlated with the scores given by the human rater. The findings implied that the size of the data collected in an authentic classroom environment was too small for classification algorithms in terms of automated essay scoring for classroom assessment.

The applicability of credit scoring models in emerging economies: an evidence from Jordan

PurposeThe purpose of this study is to propose an objective and efficient method for assessing credit risk by introducing and investigating to a greater extent the applicability of credit scoring models in the Jordanian banks and to what range they can be used to achieve their strategic and business objectives.Design/methodology/approachThe research methodology comprises two phases. The first phase is the model development. Three modeling techniques are used to build the scoring models, namely, logistic regression (LR), artificial neural network (NN) and support vector machine (SVM), and the best performing model is selected for next stage. The second phase is two-fold: linking the credit expert knowledge in a way that can enhance the outcomes of the scoring model and a profitability test to explore if the selected model is efficient in meeting banks’ strategic and business objectives.FindingsThe findings showed that LR model outperformed both ANN and SVM across various performance indicators. The LR model also fits best with achieving the bank’s strategic and business objectives.Originality/valueTo the best of the authors’ knowledge, this study is the first that applied several modeling and classification techniques for Jordanian banks and calibrated the best model in terms of its strategic and business objectives. Furthermore, credit experts’ knowledge was engaged with the scoring model to determine its efficiency and reliability against the sole use of an automated scoring model in the hope to encourage the application of credit scoring models as an advisory tool for credit decisions.

A comparison of grammatical proficiency measures in the automated assessment of spontaneous speech

We developed new measures that assess the level of grammatical proficiency for an automated speech proficiency scoring system. The new measures assess the range and sophistication in grammar usage based on natural language processing technology and a large corpus of learners’ spoken responses. First, we automatically identified a set of grammatical expressions associated with each proficiency level from the corpus. Next, we predicted the level of grammatical proficiency based on the similarity in the grammatical expression distribution between a learner's response and the corpus. We evaluated the strength of the association between the new measures and proficiency levels using spontaneous responses from an international English language assessment. The Pearson correlation test results showed that compared to commonly used syntactic complexity measures the proposed measures had stronger relationships with proficiency. We also explored the impact of system errors from a multi-stage automated process and found that the new measures were robust against the errors. Finally, we developed an automated scoring model which predicted the holistic oral proficiency scores. The new measures led to statistically significant improvement in agreement between human and machine scores over the previous system.

Designing An Automated Assessment of Public Speaking Skills Using Multimodal Cues

Traditional assessments of public speaking skills rely on human scoring. We report an initial study on the development of an automated scoring model for public speaking performances using multimodal technologies. Task design, rubric development, and human rating were conducted according to standards in educational assessment. An initial corpus of 17 speakers with 4 speaking tasks was collected using audio, video, and 3D motion capturing devices. A scoring model based on basic features in the speech content, speech delivery, and hand, body, and head movements significantly predicts human rating, suggesting the feasibility of using multimodal technologies in the assessment of public speaking skills.

RSMTool: collection of tools building and evaluating automated scoring models

RSMTool: collection of tools building and evaluating automated scoring models

Constructing Automated Scoring Model for Human Translation with Multidisciplinary Technologies

This study constructed a computer scoring model for Chinese EFL learners’ English-to-Chinese translations using multidisciplinary techniques in corpus linguistics, natural language processing, information retrieval and statistics. The proposed model, once implemented as computer software, can score English-to-Chinese translations in large-scale examinations. This study built five tentative scoring models with 50, 100, 130, 150 and 180 translations as the training set for 300 translations of an expository writing. The correlation coefficients between the computed scores of these models and human-assigned scores were above 0.8. The results further indicated that the computed scores with 130 training translations were closest to human-assigned scores. Therefore, it was concluded that the text features extracted in this research were effective and the finalized model can produce reliable scores for Chinese EFL learners’ English-to-Chinese expository translations.

INVESTIGATING THE SUITABILITY OF IMPLEMENTING THE E‐RATER® SCORING ENGINE IN A LARGE‐SCALE ENGLISH LANGUAGE TESTING PROGRAM

ABSTRACTIn this research, we investigated the suitability of implementing e‐rater® automated essay scoring in a high‐stakes large‐scale English language testing program. We examined the effectiveness of generic scoring and 2 variants of prompt‐based scoring approaches. Effectiveness was evaluated on a number of dimensions, including agreement between the automated and the human score and relations with criterion variables. Results showed that the sample size was generally not sufficient for prompt‐specific scoring. For the generic scoring model, automated scores agreed with human raters as strongly as, or more strongly than, human raters agreed with one another for more than 97% of the prompts. The impact of substituting e‐rater for the second human rater made no practically important impact on test takers' scores at both the item and total test score levels. However, neither automated scoring models nor human raters performed invariantly across all prompts or across different test countries/territories. Further investigation indicated homogeneity in the examinee population, possibly nested within test countries/territories as one potential cause of this lack of invariance. Among other limitations, findings may not be generalizable beyond the examinee population investigated in this study.