Essay Quality Research Articles

Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus

Recently, comparative judgement has been introduced as an alternative method for scoring essays. Although this method is promising in terms of obtaining reliable scores, empirical evidence concerning its validity is lacking. The current study examines implications resulting from two critical assumptions underpinning the use of comparative judgement, namely: its holistic characteristic and how the final rank order reflects the shared consensus on what makes for a good essay. Judges’ justifications that underpin their decisions are qualitatively analysed to obtain insight into the dimensions of academic writing they take into account. The results show that most arguments are directly related to the competence description. However, judges also use their expertise in order to judge the quality of essays. Additionally, judges differ in terms of how they conceptualise writing quality, and regarding the extent to which they tap into their own expertise. Finally, this study explores diverging conceptualisation of misfitting judges.

The relationship between lexical sophistication and independent and source-based writing

Lexical sophistication is an important component of writing proficiency. New lexical indices related to range, n-gram frequency, psycholinguistic word information, academic language, polysemy, and hypernymy have yielded new insights into the construct of lexical sophistication and its relationship with second language (L2) acquisition and writing. For example, recent studies have suggested that range and bigram indices are stronger indicators of lexical sophistication than frequency in the context of L2 acquisition and L2 writing and speaking proficiency. This study explores the relationship between these newly developed indices of lexical sophistication and holistic scores of writing proficiency in both independent and source-based writing tasks. The results suggest that range and bigrams are important predictors of essay quality in independent tasks, but that lexical sophistication indices are not strong predictors of essay quality in source-based tasks. The results also indicate that responses to source-based tasks tend to include more sophisticated lexical items than responses to independent tasks. Implications for second language writing assessment and pedagogy are discussed.

That noun phrase may be beneficial and this may not be: discourse cohesion in reading and writing

This paper examines the effects of attended and unattended demonstratives on text processing, comprehension, and writing quality in two studies. In the first study, participants (n = 45) read 64 mini-stories in a self-paced reading task and identified the main referent in the clauses. The sentences varied in the type of demonstratives (i.e., this, that, these, and those) contained in the sentences and whether the referent was followed by a demonstrative determiner and noun (i.e., an attended demonstrative) or a demonstrative pronoun (i.e., an unattended demonstrative). In the second study, 173 persuasive essays written by high school students were rated by expert judges on overall writing quality using a standardized rubric. Expert coders manually counted the number and types of demonstratives (attended and unattended demonstratives) in each essay. These counts were used to predict the human scores of essay quality. The findings demonstrate that the use of unattended demonstratives as anaphoric references is disadvantageous to both reading time and referent identification. However, these disadvantages become advantages in terms of essay quality likely because linguistic complexity is a strong indicator of high proficiency writing. From a text processing and comprehension viewpoint, the findings indicate, then, that anaphoric reference is not always beneficial and does not always create a more cohesive text. In contrast, from a writing context, the use of unattended demonstratives leads to a more linguistically complex text, which generally equates to a higher quality text.

The Effect of Working in Heterogeneous and Homogeneous Pairs on the Studentss Writing Skill

This study investigated the effect of working in homogeneous and heterogeneous pairs on students’ writing skill using quasi experimental research design involving 40 students. The first group consisted of 20 homogeneous pairs (paired with similar pre-test scores) while the second group consisted of 20 heterogeneous pairs (participants with high scores are paired with those having low pretest score). Then they were assigned to work on their pairs to write descriptive essays three times. After that they were assigned to write descriptive essays individually. The data was analyzed using independent sample t test to compare the students’ post-test scores. The result shows that after the experiment the quality of descriptive essays produced individually by the students from heterogeneous group was on the average significantly better than the quality of descriptive essays produced by the students from the homogeneous group. Further investigation on comparing the quality of descriptive essays produced individually by the high achievers from heterogeneous group and high achievers from homogenous group shows that the quality was not significantly different. While the quality of descriptive essays produced individually by low achievers from heterogeneous group is significantly better that the quality of descriptive essays produced individually by low achievers from homogenous group. It means that the heterogeneous grouping was more effective than the homogeneous grouping only for low achievers but not for high achievers. Furthermore, the questionnaire result found that the participants in heterogeneous group had more positive attitude toward the collaborative work than those in homogeneous group.

Are Individuals Sensitive to Changes in Performance when Interrupted?

Although there is an abundance of research showing how interruptions disrupt performance, there is little evidence regarding whether individuals believe they are personally affected by interruptions. The goal for this research was to determine whether individuals are sensitive to changes in performance when interrupted. Thirty-two participants wrote essays while interrupted or not interrupted and rated the quality of their essay on a 7-point Likert scale. Afterwards, the participants answered a separate 7-point Likert scale question that directly asked whether they believed that the interruptions negatively affected their performance. Participants' subjective ratings of their essay quality were not affected by the presence or absence of an interruption even though actual essay quality suffered from the interruptions. Additionally, a majority of the participants claimed that the interruptions had no or little impact on performance. Our data suggest individuals may not be sensitive to differences in their actual performance when interrupted.

Same language, different functions: A cross-genre analysis of Chinese EFL learners’ writing performance

Secondary-school learners of English as a foreign language (EFL) in China constitute a rapidly growing yet understudied population. This study examined Chinese secondary-school EFL learners’ writing performance in two genres, argumentative essays and narratives. Crossgenre research on native language writing has documented that, at the macro-text level, secondary-school students’ narratives are typically of higher quality than their written essays; while at the lexico-syntactic level, essays display higher complexity than narratives. To investigate cross-genre differences in EFL learners, 200 English written texts (100 essays; 100 narratives) were collected from 100 EFL Chinese secondary school learners and scored for quality, lexico-syntactic, and genrespecific discourse features. Unlike prior research on native language writing, no significant differences in quality ratings were found across the two genres. However, in line with prior research, results revealed that argumentative essays displayed a higher lexico-syntactic complexity. Regression analyses identified distinct sets of predictors of writing quality ratings for each genre. Controlling for length, lexico-syntactic complexity and diversity of organizational markers were identified as predictors of argumentative essay quality. Conversely, controlling for length, narrative quality was only predicted by the frequency of stance markers. Results are discussed in relation to pedagogical implications and directions for future research.

Idea Generation in Student Writing

Idea generation is an important component of most major theories of writing. However, few studies have linked idea generation in writing samples to assessments of writing quality or examined links between linguistic features in a text and idea generation. This study uses human ratings of idea generation, such as idea fluency, idea flexibility, idea originality, and idea elaboration, to analyze the extent to which idea generation relates to human judgments of essay quality in a corpus of college student essays. In conjunction with this analysis, linguistic features extracted from the essays are used to develop a predictive model of idea generation to further understand relations between the language features in an essay and the idea generation scores assigned to that essay. The results indicate that essays rated as containing a greater number of ideas that were flexible, original, and elaborated were judged to be of higher quality. Two of these features (elaboration and originality) were significant predictors of essay quality scores in a regression analysis that explained 33% of the variance in human scores. The results also indicate that idea generation is strongly linked to language features in essays. Specifically, the use of unique multiword units, more difficult words, semantic but not lexical similarities between paragraphs, and fewer word repetitions explained 80% of the variance in human scores of idea generation. These results have implications for writing theories and writing practice.

Civilians at War: From the Fifteenth Century to the Present, edited by Gunner Lind

Civilians at War: From the Fifteenth Century to the Present, edited by Gunner Lind. Copenhagen, Museum Tusculanum Press, 2014. 250 pp. $36.00 US (cloth). This uneven volume presents some of the fruits of a series of seminars conducted between 2005 and 2007. Edited by Gunner Lind, a prolific historian of early modern Europe, the anthology also includes the work of sociologists, philosophers, and researchers in peace and conflict studies. The funding for the early stages of the group's work, several of the organizers, and two and a half of the six chapters are Danish or Scandinavian, and so the volume has that distinct emphasis. It also does not pretend, other than in the title, to present a connected or representative view of the experience of war over the whole period. Instead one finds essays on the social theory of war (Lars Bo Kaspersen), a synthetic re-examination of the emergence of the concept of civilians in European history (Gunner Lind), a look at the norms and practices of war as related to non-combatants in late medieval and early modern Denmark (Jeppe Buchert Netterstrom), Danish public and media reaction to World War II in 1939 and 1940 (Palle Roslyng-Jensen), philosophical reflections on the impact of rape in war (primarily derived from the Yugoslav wars of the 1990s, by Robyn May Schott), and a policy-oriented study of the problems of demobilizing armed civilians in post-conflict transitions based on case studies in modern Guatemala and South Africa (Steffen Jensen and Finn Stepputat). Lind's preface suggests a uniting theme of investigating the meaning, significance, or appropriateness of the classical dichotomy between the combatant and the civilian, and several of the essays do take that as their central mission, but whatever other value they may have, Kaspersen's and Roslyng-Jensen's do not. In addition, the length and quality of the individual essays vary greatly, from fourteen pages to sixty-two, and from original research in a very small niche to truly insightful synthetic thinking. This straying from the central theme, the heavy Scandinavian focus, the lack of real temporal or geographical coverage, constitute the unevenness. Even so, there is much to learn here. Space limitations suggest focusing on those essays by historians. Lind's overview of the emergence of the concept of a civilian as distinct from a theologically defined non-combatant is an excellent overview of the complex interactions of attitudes toward war, the institutionalization of armed force, and the emergence of the modern state. It immediately recommended itself for teaching purposes, although it fails to engage David Bell's parallel and contrasting argument in The First Total War: Napoleon's Europe and the Birth of Warfare as We Know It (Boston, 2007). …

Linguistic Features of Humor in Academic Writing

A corpus of 313 freshman college essays was analyzed in order to better understand the forms and functions of humor in academic writing. Human ratings of humor and wordplay were statistically aggregated using Factor Analysis to provide an overall Humor component score for each essay in the corpus. In addition, the essays were also scored for overall writing quality by human raters, which correlated (r = .195) with the humor component score. Correlations between the humor component scores and linguistic features were examined. To investigate the potential for linguistic features to predict the Humor component scores, regression analysis identified four linguistic indices that accounted for approximately 17.5% of the variance in humor scores. These indices were related to text descriptiveness (i.e., more adjective and adverb use), lower cohesion (i.e., less paragraph-to-paragraph similarity), and lexical sophistication (lower word frequency). The findings suggest that humor can be partially predicted by linguistic features in the text. Furthermore, there was a small but significant correlation between the humor and essay quality scores, suggesting a positive relation between humor and writing quality. Keywords : humor, academic writing, text analysis, essay score, human rating

The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality

An important topic in writing research has been the use of cohesive features. Much of this research has focused on local and text cohesion. The few studies that have studied global cohesion have been restricted to first language writing. This study investigates the development of local, global, and text cohesion in the writing of 57s language (L2) university students and examines the effects of these cohesion types on judgments of L2 writing quality Growth is observed in the use of a number of local, global, and text cohesive features across a semester-long upper-level English for Academic Purposes (EAP) course. Local, global, and text features also predicted whether an essay was written at the beginning or the end of the semester with an accuracy of 71%. In addition, the use of local, global, and text cohesive features explains 36% of the variance in human judgments of text cohesion and 42% of the variance in overall judgments of writing quality. This study has important implications for second language acquisition, writing development, and writing pedagogy.

Building e‐rater® Scoring Models Using Machine Learning Methods

AbstractThe e‐rater® automated scoring engine used at Educational Testing Service (ETS) scores the writing quality of essays. In the current practice, e‐rater scores are generated via a multiple linear regression (MLR) model as a linear combination of various features evaluated for each essay and human scores as the outcome variable. This study evaluates alternative scoring models based on several additional machine learning algorithms, including support vector machines (SVM), random forests (RF), and k‐nearest neighbor regression (k‐NN). The results suggest that models based on the SVM algorithm outperform MLR models in predicting human scores. Specifically, SVM‐based models yielded the highest agreement between human and e‐rater scores. Furthermore, compared with MLR, SVM‐based models improved the agreement between human and e‐rater scores at the ends of the score scale. In addition, the high correlation between SVM‐based e‐rater scores with external measures such as examinee's scores on the other parts of the test provided some validity evidence for SVM‐based e‐rater scores. Future research is encouraged to explore the generalizability of these findings.

Keys to Detecting Writing Flexibility Over Time: Entropy and Natural Language Processing

Writing researchers have suggested that students who are perceived as strong writers (i.e., those who generate texts that are rated as high quality) demonstrate flexibility in their writing style. While anecdotally this has been a commonly held belief among researchers, scientists, and educators, there is little empirical research to support this claim. This study further investigates this hypothesis by examining how students vary in their use of two linguistic features (i.e., narrativity and cohesion) across 16 prompt-based essays. Forty-five high school students wrote 16 essays across 8 sessions within an Automated Writing Evaluation (AWE) system. Natural language processing (NLP) techniques and Entropy analyses were used to calculate how rigid or flexible students were in their use of narrative and cohesive linguistic features over time and how this trait related to individual differences in literacy abilities (i.e., vocabulary knowledge and comprehension ability), prior world knowledge, and essay quality. For instance, through the unique combination of NLP and Entropy, we found that patterns of narrative flexibility (or rigidity) was significantly and reliably related to students’ prior reading comprehension ability after 2 sessions (4 essays). Conversely, students’ flexible (or rigid) use of cohesive features was reliably related to their prior reading comprehension ability after 5 sessions (10 essays). These exploratory methodologies are important for researchers and educators, as they indicate that writing flexibility is indeed a trait of strong writers and can be detected rather quickly using the combination of textual features and dynamic analyses.

Say more and be more coherent: How text elaboration and cohesion can increase writing quality

This study examines links between essay quality and text elaboration and text cohesion. For this study, 35 students wrote two essays (on two different prompts) and for each, were given 15 minutes to elaborate on their original text. An expert in discourse comprehension then modified the original and elaborated essays to increase cohesion, resulting in a 2 (prompt) x 2 (original content, elaborated content) x 2 (original cohesion, improved cohesion) design. Expert raters scored the essays for overall quality and text coherence. In terms of overall essay quality, increasing text content (i.e., elaboration) and improving cohesion both led to significant gains in expert judgments of writing quality, and a combination of both elaboration and improved cohesion led to increased scores over increased cohesion alone. Judgments of text coherence were increased by improved cohesion (but not elaboration); and a combination of both elaboration and improved cohesion led to higher human ratings of coherence in comparison to the original and elaborated versions. The results have important implications for writing theories, writing success, writing pedagogy, and standardized testing.

FRUIT OR CANDY? EASE OF JUSTIFICATION AS A MODERATOR OF THE EFFECT OF ASSORTMENT SIZE ON CONSUMER CHOICE

When going out to eat or to the movies people often struggle to reach a final decision. Within these choices individuals are constantly faced with the dilemma of choosing between vices and virtues.It has been shown that assortment size has an impact on whether consumers make a choice, but does it also influence the kind of options that are chosen?When selecting lunch, one could choose between tasty but unhealthy options, or healthier but less gratifying alternatives (Sela, Berger, & Liu, 2009). So, could a larger menu really make us more prone to choose a salad? Conversely, is there a mechanism which might encourage the selection of more self-indulgent choices?The present paper explores existing work on the influence of assortment size, character of options, and ease of justification in the consumer's decision-making process before providing findings in two experiments. In E1, individuals' decisions were modified by assortment size and character of options while E2 also adds accessible justifications (for choosing vices) created by subjects' prior altruistic decisions.Variety and Decision-MakingThe variety of alternatives shapes consumer in multiple ways. Prior research suggests that larger assortment size in creases satisfaction (Sela et al., 2009). However, while many of the earlier studies on variety argued that more options are better, many current researchers recently highlighted the downsides of much choice (Berger, Draganska, & Simonson, 2007). This stream of research suggests that may become overwhelming and choosing from larger assortments increases difficulty and in some cases even regret (Berger et al., 2007).Decision DifficultyIyengar and Lepper (2000) found empirical evidence for decision difficulty (choice overload) in a study in which the quality of essays decreased if the number of topics students could choose from increased. It is not only the motivation to choose which decreases when facing too many options, but also the overall satisfaction. Too many alternatives cause distress. As a result, decision making becomes hard, almost unbearable. This argument is later strengthened by Scheibehenne, Greifeneder and Todd (2010) in their meta-analytic review of overload (a collection of 50 published and unpublished studies), which explains that facing too many options to choose from ultimately decreases the motivation to choose any of them.Necessary Preconditions of Choice OverloadResearchers who focused on overload in the past have commonly argued that negative effects do not always occur but rather depend on certain necessary preconditions. One of these important preconditions is lack of familiarity with, or prior preferences for, items in the assortment (Iyengar & Lepper, 2000).Additionally, Chernev (2003) found that consumers with clear prior preferences prefer to choose from larger assortments. The same author also showed that consumers' probability and satisfaction increased with the number of options to choose from, which is the opposite of overload. Similar results were presented by Mogilner, Rudnick and Iyengar (2008) where a negative relationship between assortment size and satisfaction was only found for those who were relatively less familiar with the domain.For this reason, experiments on overload have typically used options participants were not very familiar with to prevent strong prior preferences for a specific option. If, however, the participants were familiar with some options, it might result in a highly selective search process that would allow them to ignore most of the assortment. This study also used unfamiliar options (relatively unknown brands of sweets in E1, and types of computers in E2) to eliminate this effect.Character of Options and Decision-MakingAnother relevant factor that plays a part in decision-making is the character of options. …

The Use of Audio and Visual Cues of Audience and Their Effects on Persuasive Writing

The researcher in this study attempted to discover whether the inclusion of audio and visual cues of the audience in the writing context will lead to an increase in the quality of persuasive essays among English 2 students of Mindanao State University (MSU) in Marawi City, Philippines. Specifically, the study focused on finding the mean difference between the pretest and the posttest results of the respondents’ quality of persuasive writing under three conditions, namely: unspecified audience, the specified audience without audio and visual cues, and specified audience with audio and visual cues. To measure the quality of student’s writing, a holistic score was obtained using a Focused Holistic Scoring Guide. The score reflected the quality of the test in terms of communication effectiveness and persuasiveness. The findings of the study revealed that the quality of the students’ persuasive essays has significantly improved when the writing task was presented with visual and audio cues of the audience. The results manifested some positive effects of visual and audio cues when used as instructional aids or meditative tools in writing classes.

Deeper Learning With Advances in Discourse Science and Technology

The need for improving learning at deeper levels has been recognized in the national and international arena because language literacy, numeracy, and other core skills will not go the distance in handling 21st-century tasks. The challenge is aggravated by major gaps in school curricula and professional development of instructors. There is inadequate training of strategies for comprehending technical material, generating inferences, metacognition, integrating information from multiple documents, detecting misinformation, self-regulated learning, communicating through different media, collaborative problem solving, and other 21st-century skills. Some of these gaps can be filled with advances in discourse science and technology. Discourse scientists have identified the mental representations and processing mechanisms associated with different discourse levels. These levels stretch beyond word decomposition, vocabulary, and syntax (called basic literacy skills) into the deeper realms of semantics, situation (mental) models, rhetorical structure, and pragmatics. Computer tools automatically scale discourse on many levels so that individuals can get immediate multilevel feedback on text complexity, essay quality, and answers to questions. Discourse scientists have also identified effective strategies for training students how to improve comprehension of difficult material and mastery of a broad array of 21st-century competencies. These strategies have been implemented in intelligent, adaptive computer technologies, including those with conversational agents that take on different roles (e.g., peers, tutors, mentors). Future challenges lie in creating a curriculum for 21st-century skills that embrace discourse science and in scaling up discourse technologies that facilitate deeper learning.

The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion.

This study introduces the Tool for the Automatic Analysis of Cohesion (TAACO), a freely available text analysis tool that is easy to use, works on most operating systems (Windows, Mac, and Linux), is housed on a user's hard drive (rather than having an Internet interface), allows for the batch processing of text files, and incorporates over 150 classic and recently developed indices related to text cohesion. The study validates TAACO by investigating how its indices related to local, global, and overall text cohesion can predict expert judgments of text coherence and essay quality. The findings of this study provide predictive validation of TAACO and support the notion that expert judgments of text coherence and quality are either negatively correlated or not predicted by local and overall text cohesion indices, but are positively predicted by global indices of cohesion. Combined, these findings provide supporting evidence that coherence for expert raters is a property of global cohesion and not of local cohesion, and that expert ratings of text quality are positively related to global cohesion.

Contributions of Morphological Skill to Children's Essay Writing.

Morphological skills have previously been found to reliably predict reading skill, including word reading, vocabulary, and comprehension. However, less is known about how morphological skills might contribute to writing skill, aside from its well-documented role in the development of spelling. This correlational study examines whether morphological skill, as measured by a sentence generation task tapping both derivational morphology and meta-syntactic skills, predicts performance on a standardized essay writing task for fifth- and eighth-grade U.S. students (N = 233), after controlling for grade level, comprehension, and writing fluency. Multilevel analyses indicated that morphological skill and writing fluency were each uniquely predictive of essay quality, and this finding was consistent regardless of whether accurate spelling was required in the morphological task. Our results suggest that morphological skills play an important role in writing, as has been previously documented in reading and spelling.

Emphasizing Planning for Essay Writing With a Computer-Based Graphic Organizer

The authors conducted a multiple-baseline study to investigate the effects of a computer-based graphic organizer (CBGO) with embedded self-regulated learning strategies on the quantity and quality of persuasive essay writing by students with high-incidence disabilities. Ten seventh- and eighth-grade students with learning disabilities, emotional and behavioral disorders, attention deficit hyperactivity disorder, and autism spectrum disorder participated in the study. The authors collected data on the number of words, sentences, transition words, essay parts, and holistic writing quality score across (a) baseline (writing on the computer without the CBGO), (b) CBGO-use phase (writing on the computer with the CBGO), and (c) maintenance (writing on the computer without the CBGO). In a visual analysis, all participants improved the quality of their writing, and the majority of students also increased the quantity of their writing. Student interview data are presented to discuss the social validity of this innovative technology-based intervention.

Argument-Counterargument Structure in Indonesian EFL Learners’ English Argumentative Essays: A Dialogic Concept of Writing

This study investigates the potential use of the argument-counterargument structure in English L2 essays written by Indonesian EFL learners. It examines whether L2 proficiency affects the use of opposing views in their essays, and measures whether there is a correlation between the use of the rhetorical structure and the participants’ overall essay quality. In order to achieve this purpose, the study asks participants to write an English argumentative essay and measures their English L2 proficiency. The researcher also requests that the participants write in Indonesian (L1) and uses the data to measure whether there is consistency in terms of the use of the argument-counterargument structure in their L1 and L2 essays. The results show that the majority of the students developed a one-sided model of argumentation in their L1 and L2 essays by focusing only on how to state their main claim and providing relevant justification for it. The students’ relatively low L2 proficiency was found to be one of the possible factors behind their inability to include other-side views in their essays. In addition, the participants’ use of rhetorical features affected the overall quality of their essays. Pedagogical implications of these findings will be discussed.