Automated Writing Evaluation Feedback Research Articles

Timed second language writing performance: effects of perceived teacher vs perceived automated feedback

Automated writing evaluation (AWE) software has received much attention by researchers and practitioners as it has the potential of reducing the time necessary for providing second language (L2) students with written corrective feedback on their writing. While more practitioners are taking advantage of AWE affordances, research has indicated students engaged in untimed writing may still possess reservations about its usefulness as a feedback provider compared to that of L2 writing teachers. As trust affects student writer behavior, it is important to understand how the potential relationship between trust and writing outcomes are related to the source of writing feedback. This is especially important in contexts in which high stakes timed writing is the most frequent type of writing that L2 writers find themselves engaging in. Using a quasi-experimental design, we addressed these issues by examining the effects that perceived feedback source had on two groups of high-intermediate second language writers (n = 61 perceived teacher feedback group; n = 60 perceived AWE feedback group) enrolled in academic reading and writing courses in Northern Taiwan. Both groups received the same type of instruction and feedback for 18 weeks with the only caveat of perceiving the feedback source as coming from only the teacher or the AWE software. Results showed that the perceived AWE feedback group significantly outperformed the perceived teacher feedback group in their L2 writing by the end of the course. In addition, the perceived AWE feedback group put more trust in feedback on grammar rules and lexical choices than the perceived teacher feedback group. This indicates that high-intermediate L2 writers may trust AWE software more for specific types of feedback, such as those related to grammar and vocabulary. These and other results suggest decisions on providing AWE or teacher feedback should be made based on error type.

Effects of Automated Feedback on English as a Foreign Language Learners’ Writing Performance: Evidence from a Quasi-experiment

Automated writing evaluation (AWE) is increasingly used to provide formative feedback on second language (L2) students’ writing. A key factor influencing the effectiveness of AWE feedback on L2 writing performance is the learners’ revision behaviors as they process the feedback. Adopting a quasi-experiment, this study aims to evaluate the impacts of Criterion automated corrective feedback (ACF) on English as a foreign language (EFL) students’ writing performance based on two measures of accuracy: overall writing accuracy and accuracy of English article usage. Learners’ textual operations in response to Criterion ACF were examined for possible explanations for recorded gains (if any) in their writing accuracy. The main findings indicate a lack of intervention and retention effects on learners’ accuracy over the semester during which Criterion ACF was incorporated to supplement the writing instructor's feedback on organization and content. In addition, across four writing entries conducted on Criterion, learners’ revisions to their essays following Criterion ACF were primarily at the local level, dominated by addition, deletion, or substitution of individual words or short phrases rather than substantive revisions to their scripts. About one third of all Criterion feedback points did not result in textual changes to the first drafts, indicating a moderate uptake rate of the feedback. Implications related to formative feedback practices in the EFL writing classroom and the adaptation of Criterion's technical capacities are accordingly presented.

Impact of Research-Informed Integration of AWE Among Chinese EFL Undergraduates

Recent years have seen growing attention to the use of automated writing evaluation (AWE) in the L2/EFL writing classrooms. While it is generally agreed that teachers should provide scaffolding to students when introducing AWE to the class, a paucity of research has an explicit focus on investigating methods for optimizing the integration of AWE feedback. To fill the gap, the present study proposed a research-informed integration of AWE based on the literature and explored empirically its impact on students’ writing performance and their perceived usefulness of AWE feedback in the context of Chinese EFL undergraduates. Data of the study include writing scores and student responses to a questionnaire. The study found that the student participants made significant improvements in content, organization, and holistic score, but not in accuracy, and they seemed to perceive the automated feedback more positively after the intervention. The findings of the study can contribute to knowledge regarding the integration of AWE feedback and provide insights to teachers who are interested in utilizing AWE in L2 writing classrooms.

ENHANCING WRITING SKILLS OF EFL LEARNERS THROUGH AUTOMATED FEEDBACK: AN EMPIRICAL INVESTIGATION

This study examines the effectiveness of an automated writing evaluation (AWE) tool in improving the writing skills of lower-proficiency EFL learners. The study involved 34 non-English major university students who received AWE feedback from the WhiteSmoke Writing Assistant during their writing practice sessions. The quality of their writing was evaluated using pre-and post-tests, and their perceptions of the feedback were gathered through a questionnaire. The results showed a significant improvement in the quality of students' writing after using AWE feedback. Additionally, the students perceived the AWE feedback as positively impacting the quality of their writing. These findings suggest that AWE feedback is a promising tool that can be incorporated into the writing class program to enhance the quality of students' writing. Writing teachers can use the tool to evaluate students' compositions and provide targeted feedback to help them improve their writing skills. Overall, the study highlights the potential of technology-based writing tools to facilitate language learning and improve EFL learners' writing proficiency.

STUDENTS’ ATTITUDES AND PERCEPTIONS OF E-FEEDBACK TYPES: ONLINE TEACHER FEEDBACK (OTF), ONLINE PEER FEEDBACK (OPF), AND AUTOMATED WRITING EVALUATION FEEDBACK (AWE)

This study investigated students’ attitudes and perceptions of online teacher feedback (OTF), online peer feedback (OPF), and automated writing evaluation (AWE) feedback and their impact on the writing performance of EFL learners. It was conducted over two semesters with freshman ELT students. The progress writing technique was implemented. Data were collected through a questionnaire with 20 open ended questions administered to the participants, and writing samples were collected at various stages of the writing process and semi-structured interviews with volunteer students. 65 students were involved in the questionnaire and 10 students participated in the interviews. The results showed that the use of OTF, OPF, and AWE feedback all had a positive impact on learners' writing performance in various areas, including grammar, organization, coherence, and vocabulary. Participants reported that they found the e-feedback, especially the one provided by the instructor helpful and that it improved their writing most while the one provided by the peers was not as effective as the others. The study suggests that the use of written e-feedback from multiple sources can be effective in improving EFL learners' writing performance, and that incorporating OTF, OPF, and AWE feedback can be a valuable addition to the traditional writing process.

Pedagogic Exploration Into Adapting Automated Writing Evaluation and Peer Review Integrated Feedback Into Large-Sized University Writing Classes

Automated Writing Evaluation (AWE) software has been viewed as a promising tool for assisting writing. This study integrated AWE and its combination with peer review discussion into writing practice in a large college writing class in the Asian context. Adopting a mixed-method approach, this study employed a quantitative questionnaire to investigate how students perceived the integration at different writing stages and a qualitative interview to examine what prompted their quantitative decisions. The results show that the integration of AWE and peer review feedback significantly reinforced the perceived usefulness of AWE at the revision stage, and students with different proficiency levels had notably different attitudes toward AWE. Their writing anxieties intensified as they became increasingly aware of their own weaknesses during the writing process. Therefore, student writing challenges involve not only language proficiency and writing skills but also psychological variables.

The role of digital literacy in student engagement with automated writing evaluation (AWE) feedback on second language writing

Research on second language (L2) writing suggests that student engagement with automated writing evaluation (AWE) feedback is influenced by various individual and contextual factors. Little attention, however, has been given to the role that students’ digital literacy can play in this process. Increasingly, digital literacy is becoming indispensable in language teaching and learning as reading, writing, and communicating are conducted and mediated by digital technologies. This study explores how seven L2 students engaged with AWE feedback on their L2 writing in a Chinese university, analysing data collected from student written texts, AWE feedback, and student interviews. We found that the students engaged with AWE feedback in different ways in terms of their perceptions, attitudes, and revisions. The study shows that students’ digital literacy has considerable impact on how they engaged with AWE feedback regardless of their L2 proficiency. We argue that L2 students’ digital literacy, characterised by an awareness of the affordances and constraints of technologies, an ability to evaluate digital information, and a willingness to use digital technologies for peer collaboration, can be a key factor in effective student engagement with AWE feedback on L2 writing.

Learner Uptake with Automated Feedback and Teacher Feedback in an Online EFL Writing Context

The Automated Writing Evaluation (AWE) system has garnered growing attention in recent years. It is a powerful complement to traditional teacher feedback. Drawing on the data collected from student written texts, AWE feedback, teacher feedback and student interviews in a Chinese university, this study aims to investigates the impact of man-machine feedback on the practice of teaching English writing in an online English as a foreign language (EFL) context as well as students’ engagement with it. The results show that compared with teacher, AWE provides more feedback items and focuses on surface-level such as vocabulary, mechanics and grammar, while teacher pays attention to both surface-level and meaning-level, including organization, content, coherence, etc. Although there is not much difference in the uptake rate between teacher feedback and AWE feedback, the uptake rate of meaning-level feedback is much higher than surface-level feedback. Furthermore, students believe that uptake rate is influenced by various factors such as English competence, feedback quality, score, available time, personal preference, motivation, interest in English learning, and so on. Additionally, students deem that the combination of AWE feedback and teacher feedback has a positive impact on stimulating their writing enthusiasm and improving their writing abilities.

Student engagement with automated feedback on academic writing: a study on Uyghur ethnic minority students in China

ABSTRACT Aided by big-data technology and artificial intelligence, automated writing evaluation (AWE) systems aim to help students engage in self-regulated learning and improve their academic writing in the digital era. While much research on student engagement with AWE systems has been conducted in mainstream classrooms, little attention has been paid to ethnic minority students who have different educational experiences and sociocultural backgrounds. This study investigates how a group of ethnic minority students from Xinjiang Uyghur Autonomous Region engaged with an AWE system in academic writing during an 18-week semester at a comprehensive Chinese university. Data were collected from multiple sources including student multiple-draft assignments, AWE feedback, and retrospective interviews with the students. The participants were found to engage with the AWE system behaviourally, cognitively, and affectively in their writing and revision, but the extent to which they engaged in the process is influenced by their multilingual proficiency, familiarity with digital technologies, learning beliefs and sociocultural identities. Those who were more actively engaged and obtained more learning gains tended to capitalise on their linguistic and cultural capital in their engagement with AWE feedback in their writing. The article concludes with educational implications for ethnic minority students in higher education.

The effects of AWE and peer feedback on cohesion and coherence in continuation writing

The past two decades have seen the development and popularity of automated writing evaluation (AWE) systems in second language writing courses. The present study investigates the effectiveness of AWE feedback provided by the iWrite system and peer feedback on improving cohesion and coherence in a continuation writing task. We recruited 96 students from four classes in an academic writing course in mainland China, with 46 students receiving AWE feedback and 50 receiving peer feedback. Quantitative and qualitative data were collected from students’ two writing drafts and questionnaires, respectively, to investigate (1) the development of cohesion and coherence in the writings and (2) students’ perceptions of the usefulness of the target feedback. The results reveal that, compared to students receiving AWE feedback, those treated with peer feedback had greater success increasing the use of cohesive devices and improving cohesive chain formation in their revised drafts. Students considered the benefits and challenges of the two types of feedback in response to open-ended questions in the questionnaire. The evaluation of AWE and peer feedback provided empirical evidence concerning the use of feedback to improve cohesion and coherence in continuation writing tasks.

The Effects of Automated Writing Evaluation (AWE) Feedback on Students’ English Writing Quality: A Systematic Literature Review

The purpose of this review is to examine the effects of automated writing evaluation (AWE) feedback on students’ English writing performance. We systematically reviewed studies that have empirically focused on this purpose. This review uses several combinations of key words to search in the databases of JSTOR, SSCI, and ERIC for peer-reviewed articles published from 2005 to April 2020. The systematic review produced 22 eligible studies categorized as within-group and between group studies based on Stevenson and Phakiti’s (2014) categorization. The results indicated that AWE feedback might be helpful for student writing under certain conditions. Specifically, the feedback was helpful when it was provided for one single group of students. The feedback was also helpful when the writing performance of a group of students receiving the feedback was compared to the writing performance of the other group of students receiving no such feedback. Moreover, AWE feedback should be continuously offered to help students benefit most from it. This review is an update about the effects of AWE feedback on student writing and may serve as a guide for researchers and instructional practitioners through informing them of the latest research on AWE feedback.

Understanding AWE Feedback and English Writing of Learners with Different Proficiency Levels in an EFL Classroom: A Sociocultural Perspective

Automated writing evaluation (AWE) systems have been increasingly incorporated in L2 writing instruction. However, how the instructional use of AWE influences learners’ writing has remained inconclusive partly due to variations in heterogeneous learners. Underpinned by a sociocultural theory, this classroom-based study was to understand how English learners of different proficiency levels differed in their written products measured by complexity, accuracy and fluency, multiple revisions, and the perceived usefulness after 15-week instructional use of AWE. The results showed that disparity in writing accuracy of learners with different proficiency levels greatly reduced after the mandatory use of AWE feedback, but syntactic complexity and fluency remained unchanged. Meanwhile, perceived usefulness of AWE feedback was contingent on language proficiency. Limitations and pedagogical implications were also discussed.

L2 student engagement with automated feedback on writing: Potential for learning and issues of trust

Recent case-study research on L2 student engagement with automated writing evaluation (AWE) feedback shows it is not simply a matter of understanding and implementing suggested changes but can instead be strongly influenced by individual and contextual factors. The present multiple-case study builds on this work by investigating the extent to which engagement supports learning from, and is influenced by trust in, AWE feedback. Six Mandarin L1 university students at different levels of English writing instruction participated, each providing a text they had written for a current course, which was submitted to the tool Grammarly for feedback. Screen-capture recordings, stimulated recalls, and interviews were used to collect information about how much of the feedback participants viewed, which of it they accepted, which they ignored or modified, and why they did so. Findings showed participants adopting a proofreading as opposed to a learning orientation toward the feedback, perhaps because most of it included specific corrections, thus requiring little work on the user’s part. Whereas previous studies attributed more successful engagement with AWE feedback to higher language proficiency, the present research showed different dimensions of trust in Grammarly’s feedback to be determining factors.

Automated writing evaluation (AWE) feedback: a systematic investigation of college students’ acceptance

Automated writing evaluation (AWE) has been used increasingly to provide feedback on student writing. Previous research typically focused on its inter-rater reliability with human graders and validation frameworks. The limited body of research has only discussed students’ attitudes or perceptions in general. A systematic investigation of the driving factors contributing to students’ acceptance is still lacking. This study proposes an extended technology acceptance model (TAM) to identify the environmental, individual, educational, and systemic factors that influence college students’ acceptance of AWE feedback and examine how they affect college students’ usage intention. Structural equation modeling (SEM) was used to analyze the quantitative survey data from 448 Chinese college students who had used AWE feedback for at least one semester. Results revealed that students’ behavioral intention to use AWE feedback was affected by the subjective norm, facilitating conditions, perceived trust, AWE self-efficacy, cognitive feedback, and system characteristics. Among them, subjective norm, perceived trust, and cognitive feedback positively influenced perceived usefulness; facilitating conditions, AWE self-efficacy, and system characteristics were significant determinants of perceived ease of use; anxiety played no role for experienced users. Implications from these findings to AWE developers and practitioners are further elaborated.

Effect of Automated Writing Evaluation Feedback on Korean University Students’ Revision Behavior

This study investigated the effect of automated writing evaluation (AWE) feedback on Korean university students’ revision behavior. Specifically, it explored the extent to which the quality of essays differed between first drafts and second drafts. Furthermore, the way individual students dealt with incorrect feedback in revising essays was examined. In this study, ETS’ Criterion program, one of the most popular AWE programs, was employed. In cases where incorrect feedback was detected, the first and second drafts were closely compared to find out how students altered the text in response to the incorrect feedback, and this was classified into one of three categories: successful changes, unsuccessful changes, and text deletion. The results showed that students’ second drafts were rated significantly higher than first drafts by two NS raters, which indicates that many students followed the revision suggestions made by the Criterion program. As students’ Criterion scores increased, their ability to identify and make successful changes in response to incorrect feedback improved. The findings of the study can advance our understanding of AWE use in an EFL context and should contribute to broader examination of how Korean university students engage in revision of their essays.

Tapping the Potential of Pigai Automated Writing Evaluation (AWE) Program to Give Feedback on EFL Writing

With the rapid development of educational technology, studies on automated writing evaluation (AWE) program have been gaining more ground in EFL writing education, particularly due to its potential in giving continuous, formative feedback on students' writing performance. However, little research has paid attention to the effectiveness of giving and receiving AWE feedback on improving EFL learners' writing performance from cognitive, constructive and sociocultural dimensions. By drawing on the Cognitive Constructivism theory, particularly the Cognitive process writing theory, and Sociocultural theory in particular the notion of the Zone of Proximal Development (ZPD) and Scaffolding, this article reviews past studies done on the use of AWE programs in teaching English writing and this article also attempts to propose for an AWE program called Pigai to be embedded as a teaching paradigm for EFL teachers to apply in writing classes for Chinese EFL students in China in giving continuous and formative feedback to their students, in order to improve English writing teaching and positively affect EFL students' writing performance. At the end of the article, the authors also offer strategies of the implementation of the Pigai AWE program through a framework for Pigai to be used with EFL learners.

The Long-Term Effect of Automated Writing Evaluation Feedback on Writing Development

The Long-Term Effect of Automated Writing Evaluation Feedback on Writing Development

Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions

Abstract The use of automated writing evaluation (AWE) systems in second language (L2) writing has garnered growing attention in recent years. Most previous studies, however, are primarily concerned with the effectiveness of AWE feedback (e.g., scoring mechanism, validity and reliability issues) on L2 student writing, while giving scant attention to how L2 students engage with AWE feedback in the revision process. It is crucial to note that it is the student engagement with AWE feedback rather than the AWE feedback itself that contributes to student learning. Drawing on the data collected from student written texts, AWE feedback, and student interviews in a Chinese university, this study shows that individual students differed from each other in their engagement with AWE feedback reflected in their perceptions of AWE feedback and use of revision operations in the revision process, which was influenced by individual and contextual factors. The study suggests that compared to a narrow focus on accuracy improvement in L2 student writing, it is more meaningful to examine how L2 students perceive the role of AWE feedback in the revision process and how they make use of different cognitive strategies to revise their writing through their engagement with AWE feedback.

What Barriers do Students Perceive to Engagement with Automated Immediate Formative Feedback

A small preliminary PhD research project used OpenEssayist, a web based automated writing evaluation (AWE) system designed to provide immediate formative feedback to students, to gain insights into how students use such systems. One of the themes which emerged from the data analysis was that most of the students on the module did not make use of OpenEssayist, which raised the question of why? Are there barriers to student use of immediate AWE feedback? The low uptake of use of OpenEssayist reflects the findings of (Attali, 2004), whose research on the Criterion AWE system found that 71% of students did not make use of the redrafting facilities of Criterion and were excluded from his data. All thirty (n = 30) students on the module subject of the preliminary research had the opportunity to use OpenEssayist, regardless of whether they participated in the research, only four students chose to do so. Two students who did not use OpenEssayist were interviewed. The first did not use OpenEssayist for technical reasons, the second did not have enough time to learn about the software. It is not known why the other students did not make use of OpenEssayist. This short paper reports on the preliminary findings of non-use of AWE and outlines how that has led to a research question for a PhD project, which is a work-in-progress. The question is: Are there barriers to student use of immediate AWE feedback?

Investigating Chinese EFL College Students’ Writing Through the Web-Automatic Writing Evaluation Program

WWE-pigai is a kind of upgraded automated writing evaluation (AWE) system and there are 444,877,400 essays submitted and corrected on this platform. Some previous research on AWE system indicates that students do not tend to utilize AWE feedback to revise essays and improve writing abilities. The major objective of this study is to investigate Chinese EFL college students’ writing through the comparison of WWE-pigai and traditional writing method. The study lasts two terms and 120 Chinese colleges students participate in the research. The findings reveal that WWE-pigai can motivate EFL students to revise and resubmit their essays more than ten times, improve the scores, increase students’ grammar accuracy and vocabulary richness. The surface-level spelling errors (including punctuation mark misuse) are the most common for freshmen. WWE-pigai is not very effective to correct certain grammatical errors besides spelling and conjugation errors. For certain grammatical errors that the students cannot correct by themselves, the assistance of EFL teachers is necessary. We argue that the results reached through this study can offer useful implications for the usage of EFL writing strategies.