Essay Quality Research Articles

When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models (LLMs) such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that LLMs and transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that LLMs and transformer PLMs tend to score GPT-generated text 10-20% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that even though the LLMs and transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly (in part) due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.

ChatGPT Affordances and Indonesian EFL Students’ Perceptions in L2 Writing: A Collaborative Reflexive Thematic Analysis

ABSTRACT This study explores ChatGPT affordances, students’ use of prompts, and their perceptions of using the tool in EFL writing. With screen capture technology to observe students’ interactions with ChatGPT during writing process and semi-structured interviews, the data were analysed using a collaborative reflective thematic analysis. The results revealed that students’ use of ChatGPT has the potential to improve their essay quality in terms of lexical resources, grammatical range and accuracy, and coherence and cohesion. ChatGPT also provides such affordances as brainstorming, essay outlining, idea development, and feedback on hedging devices. Overall, students considered ChatGPT a valuable personal learning tutor that may stimulate student autonomy by offering immediate feedback in a less anxiety-provoking learning environment. However, some experienced challenges in producing effective prompts and understanding ChatGPT’s feedback. The study concludes that while the use of ChatGPT can enhance L2 writing development, teachers’ guidance is necessary during students’ interaction with the tool.

Designing Politics and IR Assessments in the Era of AI: An Empirical Investigation into ChatGPT’s Output Across Bloom’s Revised Taxonomy

AI breakthroughs have sent shockwaves through the HE sector. Reactions have ranged from concern over academic integrity to excitement that large language models (LLM) might allow automating lower-stakes intellectual tasks. This paper tests some of these arguments using a mixed-methods research design to analyze an original dataset of 120 ChatGPT-generated essays on far-right politics based on prompts on different levels of Bloom’s revised taxonomy. Our findings echo results that LLM-generated essays generally achieve solid to good marks. However, we find no linear negative trend between the level on Bloom and the quality of essays produced. Essays’ quality was weaker on the lowest and highest levels of the taxonomy and strongest on intermediate levels, even when controlling for the version of ChatGPT and the round of data collection. The in-depth reading revealed distinctive limitations in argument construction and empirical detail—particularly regarding “hallucinations” and source misattributions—which could only be marginally improved through prompt engineering. In turn, ChatGPT was better at suggesting “places to look” for helpful information, meaning it might be more accurately compared to a signpost rather than a calculator. The implications for politics assessments and teaching are discussed.

Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest.

Artificial intelligence (AI) large language models (LLMs) now produce human-like general text and images. LLMs' ability to generate persuasive scientific essays that undergo evaluation under traditional peer review has not been systematically studied. To measure perceptions of quality and the nature of authorship, we conducted a competitive essay contest in 2024 with both human and AI participants. Human authors and 4 distinct LLMs generated essays on controversial topics in stroke care and outcomes research. A panel of Stroke Editorial Board members (mostly vascular neurologists), blinded to author identity and with varying levels of AI expertise, rated the essays for quality, persuasiveness, best in topic, and author type. Among 34 submissions (22 human and 12 LLM) scored by 38 reviewers, human and AI essays received mostly similar ratings, though AI essays were rated higher for composition quality. Author type was accurately identified only 50% of the time, with prior LLM experience associated with improved accuracy. In multivariable analyses adjusted for author attributes and essay quality, only persuasiveness was independently associated with odds of a reviewer assigning AI as author type (adjusted odds ratio, 1.53 [95% CI, 1.09-2.16]; P=0.01). In conclusion, a group of experienced editorial board members struggled to distinguish human versus AI authorship, with a bias against best in topic for essays judged to be AI generated. Scientific journals may benefit from educating reviewers on the types and uses of AI in scientific writing and developing thoughtful policies on the appropriate use of AI in authoring manuscripts.

Readers’ awareness in the use of intertextual strategies when writing from multiple texts

Metacognitive reading awareness, involving cognitive process control and reading strategies, is linked to better comprehension and performance, but its relationship with intertextual integration strategies and the quality of argumentative essays remains unexplored. This study aimed to investigate the role of metacognition in employing integration strategies when reading conflicting texts. 69 undergraduate students participated in an online reading-writing activity, where they wrote argumentative essays based on conflicting texts about red meat consumption. We examined the students' use of intertextual integration strategies (refutation, weighing, synthesizing) and assessed their metacognitive awareness through their reflections on these strategies. The quality of the argumentative essays served as a measure of multiple text comprehension. The results indicated a lack of metacognitive awareness regarding integration strategies, with students overestimating their ability to employ these strategies. However, they demonstrated better understanding of refutational strategies based on the examples provided in their essays. Interestingly, students who were aware of and utilized these strategies in their essays performed better in the multiple-text comprehension task.

Kurdish EFL Undergraduates’ Academic Writing: Discussing the Main Writing Difficulties and Potential Solutions

English as a Foreign Language (EFL) lecturers in the Kurdistan Region - Iraq (KRI) often report poor academic essay quality among Kurdish EFL undergraduates. The purpose of this study was therefore to determine the challenges and difficulties Kurdish English-majoring undergraduates encounter in their academic writing. Through writing 60 essays by the undergraduates in three public universities in KRI and interviewing 12 lecturers, we explored writing difficulties and potential suggestions to enhance undergraduates’ writing skills. We found that students made numerous grammatical, spelling, and capitalization errors. Additionally, many of the students also struggled to write proper introduction and conclusion paragraphs in their essays. The outcomes of this article suggest that students need to read more frequently and organize their ideas for effective writing. Suggesting university programmes, various ways of teaching by lecturers in EFL classes, and writing modules were the most repetitive recommendations by the interviewees meant to enhance students’ academic writing skills. Practical implications, Limitations, further research considerations, and implications are discussed in the article.

Feedback sources in essay writing: peer-generated or AI-generated feedback?

Peer feedback is introduced as an effective learning strategy, especially in large-size classes where teachers face high workloads. However, for complex tasks such as writing an argumentative essay, without support peers may not provide high-quality feedback since it requires a high level of cognitive processing, critical thinking skills, and a deep understanding of the subject. With the promising developments in Artificial Intelligence (AI), particularly after the emergence of ChatGPT, there is a global argument that whether AI tools can be seen as a new source of feedback or not for complex tasks. The answer to this question is not completely clear yet as there are limited studies and our understanding remains constrained. In this study, we used ChatGPT as a source of feedback for students’ argumentative essay writing tasks and we compared the quality of ChatGPT-generated feedback with peer feedback. The participant pool consisted of 74 graduate students from a Dutch university. The study unfolded in two phases: firstly, students’ essay data were collected as they composed essays on one of the given topics; subsequently, peer feedback and ChatGPT-generated feedback data were collected through engaging peers in a feedback process and using ChatGPT as a feedback source. Two coding schemes including coding schemes for essay analysis and coding schemes for feedback analysis were used to measure the quality of essays and feedback. Then, a MANOVA analysis was employed to determine any distinctions between the feedback generated by peers and ChatGPT. Additionally, Spearman’s correlation was utilized to explore potential links between the essay quality and the feedback generated by peers and ChatGPT. The results showed a significant difference between feedback generated by ChatGPT and peers. While ChatGPT provided more descriptive feedback including information about how the essay is written, peers provided feedback including information about identification of the problem in the essay. The overarching look at the results suggests a potential complementary role for ChatGPT and students in the feedback process. Regarding the relationship between the quality of essays and the quality of the feedback provided by ChatGPT and peers, we found no overall significant relationship. These findings imply that the quality of the essays does not impact both ChatGPT and peer feedback quality. The implications of this study are valuable, shedding light on the prospective use of ChatGPT as a feedback source, particularly for complex tasks like argumentative essay writing. We discussed the findings and delved into the implications for future research and practical applications in educational contexts.

Investigating the element distribution of argumentative essay and its quality: The case of Indonesian EFL students’ writing

Although many research studies discussed argumentative essays, little is known about argumentative essays discussing the dialogical exchange of argumentation. This study aims to investigate the distribution of elements and the quality of argumentative essays produced by EFL students. The content analysis was employed to examine their argumentative essays. The data were garnered from essay writing tests for forty students of the English Department at a state university in Palangka Raya, Indonesia. They were assigned to write an argumentative essay about "Online learning during the COVID-19 pandemic.” The data analysis comprised collecting, categorizing, and displaying the data and conclusion drawing. The results indicated that (1) a greater number of the students were not able to supply all elements of an argumentative essay, such as an explanation of an issue, thesis statement, counterargument, refutation, and conclusion, and (2) most of them failed to obtain excellence qualification. The study findings imply how teachers can redesign the materials, find effective strategies for teaching an argumentative essay, and provide many practice opportunities. The teachers should focus on teaching an argumentative essay element by allowing the students to understand the goal of each element and integrate them to form a well-developed argumentative essay.

Half-baked essays: the embedded use of writing exemplars to encourage active learning for radiography students

This paper discusses the implementation of exemplar essays within an undergraduate first-year diagnostic radiography module to encourage active learning and the development of academic literacies. Nine essays (~400 words each) were provided over six sessions, using lecturer-guided discussion to explore the academic virtues and pitfalls demonstrated alongside assessment criteria. Exemplars were based upon a fictitious baking profession to illicit student interest and eliminate the risk of imitation or plagiarism. The quality of essays varied, with overt examples of common mistakes in spelling, essay structure and selection of appropriate literature. Upon completion of the sessions, and prior to summative assessment, students were surveyed with regard to their opinion of the teaching intervention. Feedback was obtained from 65% of the cohort (n=31/48), who reported an improvement in essay writing confidence. Students found the marking of the essays in class useful and the baking analogy was well received. Not all students found the classroom activities engaging though, with some wishing greater interactivity due to boredom and repetition. Furthermore, there was a slight reduction in pass rates and average marks across the cohort when compared to the previous module delivery. Nevertheless, there was a substantial increase in students attaining marks of 80% or more. This study highlights exemplar essays as a low-tech, low-cost option to increase assessment literacy but accepts the wide diversity in student learning preferences and experience. The efficacy of exemplar essays cannot be guaranteed, and teaching staff must be cognisant of individual student learning needs.

DeBERTa with hats makes Automated Essay Scoring system better

Automated Essay Scoring (AES) is a rapidly growing field that applies natural language processing (NLP) and machine learning techniques to the analysis and evaluation of academic essays. By automating the process of evaluating essay quality, AES not only greatly reduces the workload of human graders but also ensures consistency and objectivity in the evaluation process. AES systems can evaluate essays based on multiple criteria, including organization, coherence, and content. With the advent of deep learning, AES has shown significant improvements in accuracy and reliability. AES systems have numerous applications in education, particularly in large-scale assessment and feedback loops. In this article, we delve into the use of an improved Bidirectional Encoder Representations from Transformers (BERT) architecture with disentangled attention mechanism known as DeBERTa for student question-based summarization. This is one of the downstream tasks within AES, which is of great significance for student learning assessment. The organic combination of DeBERTa-v3 and diverse hats like Light Gradient Boosting Machine (LGBM) algorithm and Extreme Gradient Boosting algorithm (XGBoost) has proven to be highly effective in achieving excellent results in this task, indicating their significant potential in real-world AES systems.

AN INVESTIGATION ON DIFFICULTIES ENCOUNTERED BY ENGLISH-MAJORED STUDENTS AT DONG NAI TECHNOLOGY UNIVERSITY IN WRITING ACADEMIC ESSAYS

University students in Vietnam are facing a growing demand to use English in their academic and future career endeavors. Among four skills, English writing is a significant challenge for students, particularly those pursuing a major in academic English language, as they are expected to demonstrate the proficiency in a diverse range of writing styles. Consequently, writing difficulties among students have consistently emerged as a primary concern in writing courses. Nevertheless, Dong Nai Technology University has not yet done any research study to examine this particular element. The objective of this study was to examine the challenges encountered by students when attempting to meet the criteria of an academic English essay. Subsequently, some solutions were applied to enhance the quality of students' essays. The author of this study employs a questionnaire and an interview to investigate the English-majored students’ essay writing practice and challenges at Dong Nai Technology University. The survey was sent to a total of 140 students, and the findings revealed that students had not only a low frequency of writing practice but also encountered many barriers to meet writing criteria including task achievement, lexical resource, grammatical range and accuracy, coherence and cohesion . In addition, the interview’ results provided a more comprehensive picture of students’ challenges in writing academic essays. The study also indicates the importance of teachers’ feedback and students’ reading and writing practice in enhancing students’ writing proficiency.

Effects of Self-Regulated Strategy Instruction in an Accelerated Developmental English Course: A Quasi-experimental Study

This study examines the effects of a curriculum based on self-regulated strategy instruction in an accelerated developmental education (DE) English course in a community college. Faculty at the college had established a four-week, two-credit compressed course that enabled students to enroll in an eleven-week first-year composition (FYC) course in the same semester, reducing remediation from fifteen weeks to just four weeks. The course focused on writing argumentative essays using sources. The study used a quasi-experimental design with five instructors and sixty-six students to compare the experimental curriculum to a business-as-usual control condition. In the experimental curriculum, students learned strategies for writing using sources, including strategies for critical reading and for planning and revising. In addition to writing and reading strategies, students also learned metacognitive, self-regulation strategies, such as goal setting, task management, and reflection. The study found a large positive effect (ES = .96) of the treatment on quality of an argument essay written using sources. However, no significant effects were found on a summary outline, self-efficacy, or completion of the subsequent FYC course. The study demonstrates the value of strategy instruction in DE English courses; it is the first experimental study of strategy instruction in an accelerated DE course. Further research is needed to evaluate the effects of strategy instruction in corequisite courses and in FYC.

Automated College Application Essay Grading

In recent years, computer programs used to score essays have been explored extensively, with many different approaches being developed. Most of these approaches use Natural Language Processing (NLP) techniques (Ade-Ibijola et al., 2012), a field of machine learning often used to analyze and understand text. These approaches fall under the name of Automated Essay Scoring (AES), which typically assesses essay quality with a single score (Ke and Ng, 2019). This paper proposes a natural language processing (NLP) model which predicts the quality of a college application essay, which is proximally measured through a college’s acceptance rate. Key essay factors include the number of grammar mistakes, sophistication of writing, repetition, and the text of the essay. Multiple different models were tested. A Random Forest Classifier relying solely on grammar, sophistication of writing, and repetition metrics achieved the best performance, yielding an accuracy of 89.7%. The second-best model was a combination of an LSTM and a logistic regression model. Other models significantly underperformed, yielding accuracies in the range of 40%-60%. Ultimately, our model may help a number of students going through the college application process to understand where their essay may stand compared to other students.

Examining L2 studentsʼ development of global cohesion and its relationship with their argumentative essay quality

Abstract Cohesion is vital for connecting messages in writing, but there has been a dearth of studies focusing on the development of cohesion. Taking a Complex Dynamic Systems Theory (CDST) perspective, this longitudinal case study investigated the development of L2 learnersʼ use of global cohesion and its relationship with their argumentative writing quality over an academic year. Four English-as-a-foreign-language (EFL) learners at a university in Western China were invited to participate in this study and requested to write 31 argumentative essays each over an academic year from March 2019 to January 2020. The resampling and Monte Carlo simulations were utilized to examine the characteristics demonstrated in the development path of and the interaction between global cohesion devices as well as the relationship between global cohesion and the text quality of argumentative essays. Statistical results showed inter- and intra-individual variability in the learnersʼ development of global cohesion devices and their interactions, as well as the relationship between global cohesion and L2 writing quality. The findings suggest that the studentsʼ development of global cohesion in argumentative writing is a slow process and necessitates teaching interventions.

School Reform: New Future-Ready Quality Outcomes and Proposed Measures

As we increasingly emphasise the importance of developing future-ready outcomes for learners, we will need to also expand new capabilities to measure such outcomes. AI, big data, and analytics are examples of such new capabilities. Ideation is one of six habits of practice we have identified that will prepare students for the future. In this paper, we present a means to computationally appraise ideation quality as one such capability. We have developed a heuristic to appraise the ideation quality of university student essays using natural language processing, a branch of artificial intelligence concerned with the understanding of human languages. Our heuristic allows for ideation quality to be quickly quantified in the form of an ideation score. So, instead of going about the process blindly, we now have a means to provide a point of reference to allow students to give measured consideration to their ideation. Unlike a learning outcome, a future-ready habit is more of a predisposition. Consequently, it is not coherent with conventional assessments, which rather seek to evaluate than to guide. This heuristic represents an outcome of our evaluation of a new problem space in education and is, at the same time, a novel expansion into a space that exploits new capabilities.

Book Review: Abrams, A. (2023). Shortchanged. How Advanced Placement Cheats Students. Baltimore: Johns Hopkins University Press.

A much-needed thorough analysis of the validity of the powerful cultural and institutional role of the Advanced Placement exams is clarified right from the start: “Much of today’s discussion about the decline of the liberal arts centers on the undergraduate experience: tracking applications to liberal arts colleges and enrollments in majors, navigating the relationship between humanities and STEM requirements.” (pp. 3-4) This is how Annie Abrams begins her highly engaging and argumentative presentation of a contemporary issue regarding the mission of the Advanced Placement (AP) programs. With a quick self-assessment of her experience as a college instructor, the author recalls that “teaching writing was social,” which also relates beautifully to basic values embedded in student writing: “Clarity, innovation, investment, and enthusiasm all contributed to an essay quality.” (p. 4)

Analyzing Students’ Difficulties in Writing English Essay

Analyzing students' difficulties in writing essays is a valuable tool for educators to enhance their teaching methods, provide personalized support, and ultimately improve students' writing skills. It also allows educators to be responsive to the needs of their students and create a more effective and engaging English learning experience. This study investigates the quality of English essays written by college students. The research aims to understand the difficulties of students in writing English essays in particularly in writing opinion essay. A qualitative method approach was employed by analyzing 23 students’ English essays. The findings reveal that while students generally demonstrate competence in structuring their essays and presenting arguments, there are common difficulties related to thesis development, evidence integration, and coherence. Therefore, it is suggested that students always implement the stages of writing an essay to be able to present their opinion in English easily and well.

EFL Intermediate Students’ Attitudes towards the Impacts of Teaching Morphological Knowledge on the Degree of Lexical Complexity and the Quality of Academic Essays

Writing is usually considered the most challenging skill for the majority of students in the Vietnamese context. Several previous studies revealed that the instruction of morphological knowledge has positive impacts on expanding students’ vocabulary and indirectly improves the quality of academic writing. This current study was implemented in order to investigate EFL intermediate students’ attitudes towards the impacts of teaching morphological knowledge on the degree of lexical complexity and the quality of academic essays in the context of an English language center in the Mekong Delta. This 14-week experimental research employed the use of pre-questionnaire and post-questionnaire, with the participation of two groups of EFL intermediate students who were studying at a similar level and progress. The results indicated that EFL intermediate students in the experimental group had positive attitudes towards the impacts of teaching morphological knowledge on the degree of lexical complexity and the quality of their academic essays throughout the research period, whereas no positive changes regarding attitudes could be found within the ones in the control group. From that, several recommendations and pedagogical implications were drawn with the attempt to contribute to the innovation of teaching vocabulary and teaching writing in the context of secondary schools and high schools in the Mekong Delta.

Impacts of Morphological Knowledge on the Degree of Lexical Complexity and the Quality of Academic Essays of EFL Intermediate Students

In the context of learning English in secondary schools and high schools in Vietnam in general and in Mekong Delta in particular, writing is usually considered the most challenging skill for the majority of students. These students frequently encounter a lot of obstacles in their writing practices, such as high-standard conventions in academic writing, complicated grammatical structures, lack of ideas, an insufficient amount of time spent on writing in class, as well as poor and noncomplex vocabulary. Such challenges do not only result in low performances and low scores but also students’ fear and loss of interest in writing lessons. Several previous studies indicated that the teaching of morphology had positive impacts on expanding students’ vocabulary, and indirectly improved the quality of their writing. This current study was conducted in order to examine the impacts of teaching morphology to EFL intermediate students on the degree of lexical complexity and the quality of their academic essays. This study employed the experimental research approach, which utilized three different research instruments: writing tests (before and after the study), questionnaires (before and after the study), and focus group interviews (after the study). This 14-week study was carried out at a private English center in Mekong Delta, with the participation of two groups of EFL students who were studying at the same level and progress. The results revealed that the instruction of morphology to EFL intermediate students had positive impacts on the degree of lexical complexity and the quality of their academic essays. Moreover, the results of the questionnaires and interviews also pointed out that these students had positive attitudes towards the instruction of morphology in their academic essay writing lessons. Finally, several recommendations and pedagogical implications were concluded with the attempt of contributing to the innovation of teaching vocabulary and teaching writing in the context of secondary schools and high schools in Mekong Delta particularly and Vietnam generally.

Assessing the content quality of essays in content and language integrated learning: Exploring the construct from subject specialists’ perspectives

Assessing the content of learners’ compositions is a common practice in second language (L2) writing assessment. However, the construct definition of content in L2 writing assessment potentially underrepresents the target competence in content and language integrated learning (CLIL), which aims to foster not only L2 proficiency but also critical thinking skills and subject knowledge. This study aims to conceptualize the construct of content in CLIL by exploring subject specialists’ perspectives on essays’ content quality in a CLIL context. Eleven researchers of English as a lingua franca (ELF) rated the content quality of research-based argumentative essays on ELF submitted in a CLIL course and produced think-aloud protocols. This study explored some essay features that have not been considered relevant in language assessment but are essential in the CLIL context, including the accuracy of the content, presence and quality of research, and presence of elements required in academic essays. Furthermore, the findings of this study confirmed that the components of content often addressed in language assessment (e.g., elaboration and logicality) are pertinent to writing assessment in CLIL. The manner in which subject specialists construe the content quality of essays on their specialized discipline can deepen the current understanding of content in CLIL.