Essay Grading Research Articles

Knowledge Consolidation Analysis

Researchers studying technology development often examine how rhetorical activity contributes to technologies' design, implementation, and stabilization. This article offers a possible methodology for studying one role of rhetorical activity in technology development: knowledge consolidation analysis. Applying this method to an exemplar case, the author describes how explanations of Project Essay Grade (PEG), the first initiative to computerize student essay assessment, made knowledge available about this technology project. More specifically, technologist Ellis Page and his coauthors reworked a key explanatory argument, a justification of PEG's functioning, during the course of several decades, refining and clarifying its key contrasts and strengthening its presentation for generalist educators in particular. Analysis suggests that late presentations of the argument reveal that knowledge was successfully consolidated about a technical procedure Page and his coauthors called “rating simulation.” The conclusion discusses the key advantages and limitations of the method.

COGNITIVELY INSPIRED NLP-BASED KNOWLEDGE REPRESENTATIONS: FURTHER EXPLORATIONS OF LATENT SEMANTIC ANALYSIS

Natural-language based knowledge representations borrow their expressiveness from the semantics of language. One such knowledge representation technique is Latent semantic analysis (LSA), a statistical, corpus-based method for representing knowledge. It has been successfully used in a variety of applications including intelligent tutoring systems, essay grading and coherence metrics. The advantage of LSA is that it is efficient in representing world knowledge without the need for manual coding of relations and that it has in fact been considered to simulate aspects of human knowledge representation. An overview of LSA applications will be given, followed by some further explorations of the use of LSA. These explorations focus on the idea that the power of LSA can be amplified by considering semantic fields of text units instead of pairs of text units. Examples are given for semantic networks, category membership, typicality, spatiality and temporality, showing new evidence for LSA as a mechanism for knowledge representation. The results of such tests show that while the mechanism behind LSA is unique, it is flexible enough to replicate results in different corpora and languages.

Self‐Assessed Intelligence and Academic Performance

This paper reports the results of a two‐year longitudinal study of the relationship between self‐assessed intelligence (SAI) and academic performance (AP) in a sample of 184 British undergraduate students. Results showed significant correlations between SAI (both before and after taking an IQ test) and academic exam marks obtained two years later, even when IQ scores were partialled out. Several continuous assessment indicators (notably attendance, oral expression, and motivation) were also significantly correlated with SAI, even when IQ scores were controlled. A series of hierarchical regressions indicated that although exam grades were best predicted by IQ, SAI showed significant incremental validity in the prediction of AP, accounting for an additional 3% of exam, 9% of continuous assessment, and 2% of essay grades. Results are discussed with regard to current trends to integrate individual differences underlying AP.

What price presentation? The effects of typographic variables on essay grades

The marks awarded to 130 second‐year undergraduate word‐processed essays were assessed in relation to their use of different typographic features. By and large most of these features had little effect on the essay grades. However, essays printed in 12‐point type gained significantly higher marks (mean = 56.8) than essays printed in 10‐point type (mean = 52.7), and essays using a combination of popular features (Times Roman, 12‐point, double‐spaced unjustified text, and a line‐space to denote new paragraphs) gained significantly higher marks (mean = 59.6) than those using other combinations (mean = 55.8).

Automated Essay Scoring.

The impacts of computers on writing have been widely studied for three decades. Even basic computers functions, i.e. word processing, have been of great assistance to writers in modifying their essays. The research on Automated Essay Scoring (AES) has revealed that computers have the capacity to function as a more effective cognitive tool (Attali, 2004). AES is defined as the computer technology that evaluates and scores the written prose (Shermis & Barrera, 2002; Shermis & Burstein, 2003; Shermis, Raymat, & Barrera, 2003). Revision and feedback are essential aspects of the writing process. Students need to receive feedback in order to increase their writing quality. However, responding to student papers can be a burden for teachers. Particularly if they have large number of students and if they assign frequent writing assignments, providing individual feedback to student essays might be quite time consuming. AES systems can be very useful because they can provide the student with a score as well as feedback within seconds (Page, 2003). Four types of AES systems, which are widely used by testing companies, universities, and public schools: Project Essay Grader (PEG), Intelligent Essay Assessor (IEA), E-rater, and IntelliMetric. AES is a developing technology. Many AES systems are used to overcome time, cost, and generalizability issues in writing assessment. The accuracy and reliability of these systems have been proven to be high. The search for excellence in machine scoring of essays is continuing and numerous studies are being conducted to improve the effectiveness of the AES systems.

The Validity of Human and Computerized Writing Assessment

This paper summarizes an experiment designed to assess the validity of essay grading between holistic and analytic human graders and a computerized grader based on latent semantic analysis. The validity of the grade was gauged by the extent to which the student's knowledge of the topic correlated with the grader's expert knowledge. To assess knowledge, Pathfinder networks were generated by the student essay writers, the holistic and analytic graders, and the computerized grader. It was found that the computer generated grades more closely matched the definition of valid grading than did human generated grades.

A Model of Rater Behavior in Essay Grading Based on Signal Detection Theory

An approach to essay grading based on signal detection theory (SDT) is presented. SDT offers a basis for understanding rater behavior with respect to the scoring of construct responses, in that it provides a theory of psychological processes underlying the raters' behavior. The approach also provides measures of the precision of the raters and the accuracy of classifications. An application of latent class SDT to essay grading is detailed, and similarities to and differences from item response theory (IRT) are noted. The validity and utility of classifications obtained from the SDT model and scores obtained from IRT models are compared. Validity coefficients were found to be about equal in magnitude across SDT and IRT models. Results from a simulation study of a 5‐class SDT model with eight raters are also presented.

A framework for the computerized assessment of university student essays

This paper presents an approach for automatic grading of essays. Student essays are compared against a model or key essay provided by the teacher. The similarity between a student essay and the model essay is measured by the cosine of their contained angle in an n-dimensional semantic space. The model essay is preprocessed by removing stopwords, extracting keywords, assigning weights to keywords to reflect their importance and finally by linking every keyword to a subject-oriented synonym list. The student essay, by comparison, is preprocessed by removing stopwords and then by extracting keywords. The keywords extracted from the model essay and the keywords extracted from students essays together with weights provided by teacher are used to build feature vectors for teacher and students essays. The obtained grade depends on the similarity between these vectors (calculated by using the cosine formula). A simulator was implemented to test the viability of the proposed approach. It was fed with student essays (at the university level) gathered from database management course over three semesters. The results were very encouraging and the agreement between the auto-grader and human grader was as good as the agreement between human graders.

Automatically Grading Essays with Markit©

An international association advancing the multidisciplinary study of informing systems. Founded in 1998, the Informing Science Institute (ISI) is a global community of academics shaping the future of informing science.

Human and Computer-Generated Essay Grades

The advent of computerized essay grading brings with it significant questions about the interaction between the grader and the student. An experiment is presented that compares two forms of human essay grading with computerized essay grading. Twenty human graders using either holistic or analytic grading methods marked 15 student essays. These essays were subsequently marked by computer using a grading implementation of latent semantic analysis. There were significant correlations between both holistic and analytic graders and the computerized essay grader.

Can a collaborative network environment enhance essay-writing processes?

The aim of this study is to examine whether a computer-supported learning environment enhances essay writing by providing an opportunity to share drafts with fellow students and receive feedback from a draft version. Data for this study were provided by 25 law students who were enrolled in a course in legal history at the University of Helsinki in February 2001. Both the students and the teacher were interviewed. The interviews showed that the students' experiences of the essay-writing process were very positive. The teacher's experiences were in line with the students'. The results showed that the students seemed to divide into two groups concerning their experiences towards sharing written drafts with peers: those who were very enthusiastic and enjoyed the possibility to share drafts and those who, on the other hand, felt that the idea of sharing unfinished essays was too threatening for them and required too much openness. The results further showed that the active use of a computer-supported learning environment was related to good essay grades.

An Overview of Current Research on Automated Essay Grading

An international association advancing the multidisciplinary study of informing systems. Founded in 1998, the Informing Science Institute (ISI) is a global community of academics shaping the future of informing science.

Writes of Passage: Stylistic features of male and female undergraduate history essays

As a student enters an academic field and seeks to become a member of a discourse community, they must learn to write in ways that are appropriate to that discipline. Academic success depends on determining what the rules are and on learning to follow them. A study of undergraduate writing in history is reported. Essays by 87 students were analysed and sentences were classified according to their bold, tentative or evaluative features. Findings were analysed by gender and by essay grade. They showed that there was more similarity than difference overall in the writing styles of men and women but that, in addition, men tended to use more bold constructions than women. The writing contained a predominance of assertive statements, lending weight to the view that argument is a key feature of academic writing. However, there was a greater occurrence of very assertive or bold statements in work receiving lower grades, which suggests that students must not write too boldly. Similar findings are reported in relation to the use of tentative and evaluative statements, leading to the conclusions that 'good' writing in history is a very finely judged performance and that learning to write history may be a very complex process.

Learning to make things happen in different ways: Causality in the writing of middle-grade English language learners

This study addresses two issues: the similarity between L1 and L2 writing development and the nature of the developmental path. The frequency of two types of causality markers in 5th–8th grade essays written by 189 students in ESL and 546 students in regular language arts classes is analyzed. Students wrote on one of two informative “how-to” prompts. The regular language arts students were found to differ in their usage of causality markers between the two topics, whereas the ESL students used the markers similarly across both topics. There were no differences between students at different grade levels in either group. In addition, the ESL students were found to have higher usage of causality markers in general than the regular language arts students. It is suggested that the developmental path for both groups involves moving away from primarily narrative expansion of topic towards display of diverse language forms and discourse strategies.

Simulating a full-length psychiatric interview with a complex patient: an OSCE for medical students.

Brief encounters with standardized psychiatric patients have been reported. In contrast, a videotaped objective structured clinical examination designed to simulate a comprehensive psychiatric evaluation was used to evaluate all students assigned to three successive psychiatry clerkship rotations (N=52). Performance assessment was based on standardized patients' completion of a 36-item content checklist and a patient perception scale, and on students' completion of a written examination component. Checked independently, all three assessments correlated with the students' score on the NBME psychiatry examination, their ward grade, and their essay examination grade. Students' ratings were favorable for time allotted, usefulness, and believability.

Grading essays in computer ethics

Computer ethics courses differ from technical courses in the manner in which they are taught and assessed. A common assignment in a computer ethics course is to write an essay that addresses a technical dilemma. Computer science faculty typically do not have training or experience in grading essays. The purpose of this paper is to present a scoring rubric that has been successfully used to grade and track students' knowledge development as they progress through a computer ethics course. Although this paper focuses upon a specific rubric, general principles will be emphasized to show how scoring rubrics can be used across different courses.

Trait Ratings for Automated Essay Grading

This study employed an automated grader to evaluate essays, both holistically and with the rating of traits (content, organization, style, mechanics, and creativity) for Webbased student essays serving as placement tests at a large Midwestern university. The authors report the results of two combined experiments, based on random selection from 1,193 essays. In the first experiment, the essays of 807 students were used to create statistical predictions for the essay-grading software. In the second experiment, the ratings from a separate, random sample of 386 essays were used to compare the ratings of six human judges against those generated by the computer. The interjudge correlation of the human raters alone was r = .71. But the interrater reliability of all six judges in combination with computer scoring reached .83. The essay-grading software was an efficient means for evaluating the essays, with a capacity for grading approximately six documents every second. Other potential feedback measures for use in writing courses are also discussed.

Trait Ratings for Automated Essay Grading

This study employed an automated grader to evaluate essays, both holistically and with the rating of traits (content, organization, style, mechanics, and creativity) for Webbased student essays serving as placement tests at a large Midwestern university. The authors report the results of two combined experiments, based on random selection from 1,193 essays. In the first experiment, the essays of 807 students were used to create statistical predictions for the essay-grading software. In the second experiment, the ratings from a separate, random sample of 386 essays were used to compare the ratings of six human judges against those generated by the computer. The interjudge correlation of the human raters alone was r = .71. But the interrater reliability of all six judges in combination with computer scoring reached .83. The essay-grading software was an efficient means for evaluating the essays, with a capacity for grading approximately six documents every second. Other potential feedback measures for use in writing courses are also discussed.

On-line Grading of Student Essays: PEG goes on the World Wide Web

This study examined the feasibility of employing Project Essay Grade (PEG) software to evaluate web-based student essays that serve as placement tests at a large Mid-Western university. The results of two experiments are reported. In the first experiment, the essays of 1293 high school and college students were used to create a statistical model for the PEG software. PEG identified 30 proxes (observed variables) that could be incorporated into an evaluation of the written work. In the second experiment, the ratings from a separate sample of 617 essays were used to compare the ratings of six human judges against those generated by the computer. The inter-judge correlation of the human raters was r = 0.62 and was r = 0.71 for the computer. Finally, the PEG software was an efficient means for grading the essays with a capacity for approximately three documents graded every second. Cycle time from the websubmission of the document to producing a report score was about 2 minutes. Although PEG would appear to be a cost-effective means of grading written work of this type, several cautionary notes are included.

Using soft computing to build real world intelligent decision support systems in uncertain domains

Whilst the builders of traditional decision support systems have regularly used game theory and operations research, they have rarely used statistical techniques to build intelligent support systems for fields that have weak domain models. Further, the principle tools in the artificial intelligence arsenal were centred on symbol manipulation and predicate logic, while the use of numerical techniques were looked upon with disfavour. We claim that soft computing techniques (such as fuzzy reasoning and neural networks) can be integrated with symbolic techniques to provide for efficient decision making in knowledge-based systems. We illustrate our claim through the discussion of two decision support systems that have been constructed using soft computing techniques. Split-Up uses rules and neural networks to advise on property distribution following divorce in Australia, whilst IFDSSEA uses fuzzy reasoning to assists teachers in New York State to grade essays. We focus on how both systems reason and how they have been evaluated.