Essay Grading Research Articles

THE IMPACT OF SAMPLING APPROACH ON POPULATION INVARIANCE IN AUTOMATED SCORING OF ESSAYS

ABSTRACTMany testing programs use automated scoring to grade essays. One issue in automated essay scoring that has not been examined adequately is population invariance and its causes. The primary purpose of this study was to investigate the impact of sampling in model calibration on population invariance of automated scores. This study analyzed scores produced by the e‐rater® scoring engine using a GRE® assessment data set. Results suggested that equal allocation stratification by language sampling approach performed optimally in maximizing population invariance using either human/e‐rater agreement or their correlation pattern differences with external variables as evaluation criteria. Guidelines were given to assist practitioners in choosing a sampling design for model calibration. Potential causes for lack of population invariance, study limitations, and future research are discussed.

An HMM-Based Algorithm for Content Ranking and Coherence-Feature Extraction

In this paper, we propose an algorithm called coherence hidden Markov model (HMM) to extract coherence features and rank content. Coherence HMM is a variant of HMM and is used to model the stochastic process of essay writing and identify topics as hidden states, given sequenced clauses as observations. This study uses probabilistic latent semantic analysis for parameter estimation of coherence HMM. In coherence-feature extraction, support vector regression (SVR) with surface features and coherence features is used for essay grading. The experimental results indicate that SVR can benefit from coherence features. The adjacent agreement rate and the exact agreement rate are 95.24% and 59.80%, respectively. Moreover, this study submits high-scoring essays to the same experiment and finds that the adjacent agreement rate and exact agreement rate are 98.33% and 64.50%, respectively. In content ranking, we design and implement an intelligent assisted blog writing system based on the coherence-HMM ranking model. Several corpora are employed to help users efficiently compose blog articles. When users finish composing a clause or sentence, the system provides candidate texts for their reference based on current clause or sentence content. The experimental results demonstrate that all participants can benefit from the system and save considerable time on writing articles.

An innovative approach for automatically grading spelling in essays using rubric-based scoring

Automated Essay Grading (AEG) is defined as a computer technology that evaluates and scores written prose. A number of AEG systems have been developed since the 1960s and in most of them, an ad-hoc or generalized approach is used to grade spelling even though it is an important element of an essay-scoring rubric. Existing approaches do not therefore give an accurate representation or measure of spelling in essays. According to the rubric-based scoring method used in the National Assessment Program “Literacy and Numeracy (NAPLAN) in Australia, spelling is marked in three steps” first, by identifying the correct and incorrect words in the essay; second, by categorizing each word based on the difficulty level into one of four classes: Simple, Common, Difficult or Challenging, and counting the number of correct and incorrect words in each category; finally, by using the pre-defined NAPLAN rubric scale to assign the mark. Only a small number of existing AEG systems can be used for rubric-based scoring, and none can be used to grade spelling according to the NAPLAN rubric. In this paper, we address this shortcoming in the existing literature and present an innovative approach to automatically mark spelling using rubric based scoring. We develop two algorithms based on the rules and heuristics of the English language to formulize the rubric for spelling and then implement these algorithms in Java language and perform a series of evaluations of our system using an essay dataset. Our results are very promising, even though it is the first system of this kind.

Soft computing auto essay grading

Soft computing techniques try to mimic the humans in reasoning, learning and prediction. We have implemented these techniques to address the auto essay grading, particularly for short-answer types. A knowledge extraction approach is proposed and then assessed using samples of short-answer essays. It is based on identifying keywords and their synonyms and providing the sub-grades that are summed to get the total grade. The soft computing modelling techniques have been used to enhance the KE method. These modelling techniques include the fuzzy logic – Mamdani and Sugeno models, the neural networks models with both the subtractive clustering and back propagation learning rules and the neuro fuzzy model. The same samples have been used to implement the modelling techniques. The overall obtained correlation coefficients that reached 0.996 demonstrate the feasibility, suitability and adequacy of these techniques to address the auto essay grading problem. We believe that further investigations to other samples of datasets in different fields are required before claiming that.

The Consistency Between Human Raters and an Automated Essay Scoring System in Grading High School Students' English Writing

This study investigates the consistency between human raters and an automated essay scoring system in grading high school students' English compositions. A total of 923 essays from 23 classes of 12 senior high schools in Taiwan (Republic of China) were obtained and scored manually and electronically. The results show that the consistency between human raters is significantly higher than the consistency between human raters and the automated essay scoring system. To discover whether the students were confident with the automated essay grading system, a questionnaire was also distributed. The results indicate the participants realize their vocabulary level was inadequate, and they wanted to know the scores the automated essay grading system gave them along with the generated comments regarding their compositions.

Discrimination in grading: experimental evidence from primary school teachers

This paper studies the effect of teacher expectations on essay grades in an experimental setting. For this purpose, we randomly assign Turkish or German first names to a set of essays so that some teachers believe a given essay was written by a German native pupil, whereas others believe it was written by a pupil of Turkish origin. We find that the same essays obtain significantly worse grades and lower secondary school recommendations when bearing a Turkish sounding name.

Automated Essay Scoring Using Generalized Latent Semantic Analysis

Automated Essay Grading (AEG) is a very important research area in educational technology. Latent Semantic Analysis (LSA) is an Information Retrieval (IR) technique used for automated essay grading. LSA forms a word by document matrix and then the matrix is decomposed using Singular Value Decomposition (SVD) technique. Existing AEG systems based on LSA cannot achieve higher level of performance to be a replica of human grader. We have developed an AEG system using Generalized Latent Semantic Analysis (GLSA) which makes n-gram by document matrix instead of word by document matrix. The system has been evaluated using details representation. Experimental results show that the proposed AEG system achieves higher level of accuracy as compared to human grader.

The value of peers and support from scaffolding: Applying constructivist principles to the teaching of psychology

The legacy and sustainability of a university education requires student independence and ownership of learning. Adopting a student–centred constructivist approach to teaching and learning allows students to develop a web of self–constructed, interconnected understanding, and supports their development into lifelong learners. The efficacy of this approach is illustrated with a case study relating to a series of academic skills tutorials for first–year psychology students. The tutorial materials, activities and teaching techniques were rated as more useful by students when delivered using constructivist principles. The use of constructivist techniques also enabled students to make larger gains in their essay grades over the course of the academic year. The implications of using such teaching methods in higher education are discussed.

Complaining about a compliment: Evaluating attributions of positive outcomes to discrimination

People evaluate stigmatized group members negatively when they attribute negative personal outcomes to discrimination, even when discrimination is the likely cause. The present studies examined whether similar interpersonal costs exist for attributing a positive personal outcome to discrimination in the context of positive stereotypes. In Study 1, 129 participants read an excerpt written by an Asian student who attributed an unexpectedly high essay grade to discrimination, writing quality, or easy grading. Participants evaluated the student least favourably when he attributed the positive outcome to discrimination. In Study 2, 140 participants evaluated a successful Asian job applicant less favourably than a successful, white applicant, although both applicants attributed the positive outcome to group membership. The implications for how stigmatized group members perceive and respond to prejudice and discrimination are discussed.

New technology, new pedagogy? Employing video podcasts in learning and teaching about exotic ecosystems

This paper examines the experiences of undergraduate university students in response to the employment of video podcasts to support learning and teaching about exotic ecosystems. Six, 15–20‐minute podcasts were made accessible to students through a virtual learning environment, either online or to download to mobile technology. The students were free to watch the podcasts whenever and wherever they chose to. The perceived and actual effectiveness of the technology was assessed by written questionnaire, focus groups and summative assessment results. Students agreed that the podcasts were effective in supporting learning and teaching on the course, largely by offering a flexible and visual learning experience. The podcasts were also perceived as a useful resource for revision and assessment, providing visual images that stimulated factual recall and highlighted knowledge gaps. There were no significant differences, however, in examination essay grades comparing cohorts prior to and post adoption of podcasts. The key to improving the student learning experience appears to lie not in adopting new pedagogy, but in reflexively developing the existing pedagogic strategies employed by both teachers and learners. Of primary importance is uniting the individual learning experience of podcasts with group exploration and critical discussion in a collaborative learning framework.

Six Key Topics for Automated Assessment Utilisation and Acceptance

Automated assessment technologies have been used in education for decades (e.g., com- puterised multiple choice tests). In contrast, Automated Essay Grading (AEG) technologies: have existed for decades; are 'good in theory' (e.g., as accurate as humans, temporally and financially efficient, and can enhance formative feedback), and yet; are ostensibly used comparatively infre- quently in Australian universities. To empirically examine these experiential observations we con- ducted a national survey to explore the use of automated assessment in Australian universities and examine why adoption of AEG is limited. Quantitative and qualitative data were collected in an online survey from a sample of 265 staff and students from 5 Australian universities. The type of assessment used by the greatest proportion of respondents was essays/reports (82.6%), however very few respondents had used AEG (3.8%). Recommendations are made regarding methods to promote technology utilisation, including the use of innovative dissemination channels such as 3D Virtual Worlds.

Investigating Factors Affecting the Uptake of Automated Assessment Technology

Automated assessment is an emerging innovation in educational praxis, however its pedagogical potential is not fully utilised in Australia, particularly regarding automated essay grading. The rationale for this research is that the usage of automated assessment currently lags behind the capacity that the technology provides, thus restricting the pedagogical benefits for students, job satisfaction for staff, and quality assurance and financial benefits for universities. The exploration of the different perspectives of stakeholders regarding their needs and expectations of automated assessment shows the high-stake game of surviving and thriving. Inter alia, students value education as a means to gain employment, educators’ value education as their means of employment, and universities value education as their means of existence. The various facets of this interplay are described, including the value of knowledge and learning (pedagogical considerations); society’s system for measuring worth and exchanging value (economic considerations); sociotechnological evolution and emerging innovations in educational assessment, and the adoption and resistance thereof.

The Validity of Examination Essays in Higher Education: Issues and Responses

AbstractThe use of timed, essay examinations is a well‐established means of evaluating student learning in higher education. The reliability of essay scoring is highly problematic and it appears that essay examination grades are highly dependent on language and organisational components of writing. Computer‐assisted scoring of essays makes use of language features and has demonstrated strong similarity to human ratings. Studies of examiner behaviour show that attention to content and language features contributes to grading decisions. However, given the time constraints on essay examinations, an overemphasis on language aspects may weaken the validity of essay examination grades. This article suggests alternative approaches to the standard essay prompt which should raise the validity of essay tasks and scoring in higher education. Suggested options include redesigning tasks so that organisational and language features are less influential in scoring and the use of content maps.

Design of an Automated Essay Grading (AEG) System in Indian Context

Automated Essay Grading (AEG) or scoring systems are not more a myth they are reality. These AI systems are blessings to the educational community where teachers normally face lots of problem while correcting studentspsila essays. Valuation of huge amount of student essays within stipulated time frame, with feed back is a real challenge. But today, the human written (not hand written) essays easily evaluated by AEG systems easily. The TOEFL exam is one of the best examples of this application. The studentspsila essays are evaluated both by human and automated essay grading system. Then the average is taken. AEG might provide precisely the platform we need to explicate many of the features those characterize good and bad writing and many of the linguistic, cognitive and other skills those underline the human capability for both reading and writing. They can also provide time-to-time feedback to the writers/students by using that the people can improve their writing skill. A meticulous research of last couple of years has helped us to understand the existing systems which are based on AI & Machine Learning techniques and finding the loopholes and at the end to propose a system, which will work under Indian context, presently for English language influenced by local languages. Currently most of the essay grading systems is used for grading pure English essays or essays written in pure European languages. In India we have almost 21 recognized languages and influence of these local languages, in English, is very much here. Due to the influence of local languages and English written by nonnative English speakers (ie. Indians) the result of TOEFL exams has shown lower scores against Indian students (also Asian students). This paper focuses on the existing automated essay grading systems, basic technologies behind them and proposes a new framework to over come the problems of influence of local Indian languages in English essays while correcting and by providing proper feedback to the writers.

STUDIES OF A LATENT‐CLASS SIGNAL‐DETECTION MODEL FOR CONSTRUCTED‐RESPONSE SCORING

ABSTRACTRater behavior in essay grading can be viewed as a signal‐detection task, in that raters attempt to discriminate between latent classes of essays, with the latent classes being defined by a scoring rubric. The present report examines basic aspects of an approach to constructed‐response (CR) scoring via a latent‐class signal‐detection model. The model provides a psychological framework for CR scoring and includes rater parameters with a clear cognitive basis. Simulations are used to examine how well rater parameters and latent‐class sizes are recovered as well as the accuracy of classification. The relation of rater parameters to agreement statistics and classification accuracy is examined. The effects of using a balanced, incomplete block design are compared to those for a fully crossed design. The model is applied to several ETS datasets.

The Relationship Between Grades and the Lexical Richness of Student Essays

The purpose of this study is to examine the relationship between lexical richness and the grades on essays produced by Swedish university students of English in order to shed light on the extent to which lexical richness is a predictor of overall essay quality. To this end, essays produced by 37 advanced learners of English were analyzed using a lexical richness measure that calculates the proportion of advanced vocabulary. The lexical richness score of the student essays were related to the following three variables: essay grade, course grade and vocabulary knowledge as measured by three discrete-item tests. In addition, a 14-item questionnaire administered to the teachers at the English department eliciting information about their essay assessment procedures was analyzed, in order to shed light on the relationship between the weight teachers put on lexical richness and the grade they award essays with different lexical richness profiles. The results show that there is a relationship between use of advanced vocabulary in student essays and the overall course grade. However, no relationship was found between lexical richness and overall essay quality as reflected by faculty teachers’ ratings. A possible explanation is that a majority of the surveyed faculty teachers state that their assessment of essay quality is primarily based on content and grammar features rather than lexical features.

In the News

On the face of it, scoring student essays would seem to push AI capabilities to their limits. After all, students express themselves through their writing in vastly different ways. Furthermore, they might misunderstand the essay questions they've been asked to write about, or drift off the topic in the course of writing. Even so, for decades now, researchers have known ways to automatically evaluate student writing. What's been lacking, explains Tom Landauer, executive vice president at the Knowledge Technologies group ol Pearson, is the computer processing power to make grading practical. Processing power has now become a nonissue, of course. LSA is one of many techniques that researchers are employing to advance the state of automated essay grading.

Design Of An Automated Essay Grading (AEG) System In Indian Context

Automated Essay Grading (AEG) or scoring systems are not more a myth they are reality. These AI systems are blessings to the educational community where teachers normally face lots of problem while correcting studentspsila essays. Valuation of huge amount of student essays within stipulated time frame, with feed back is a real challenge. But today, the human written (not hand written) essays easily evaluated by AEG systems easily. The TOEFL exam is one of the best examples of this application. The studentspsila essays are evaluated both by human and automated essay grading system. Then the average is taken. AEG might provide precisely the platform we need to explicate many of the features those characterize good and bad writing and many of the linguistic, cognitive and other skills those underline the human capability for both reading and writing. They can also provide time-to-time feedback to the writers/students by using that the people can improve their writing skill. A meticulous research of last couple of years has helped us to understand the existing systems which are based on AI & Machine Learning techniques and finding the loopholes and at the end to propose a system, which will work under Indian context, presently for English language influenced by local languages. Currently most of the essay grading systems is used for grading pure English essays or essays written in pure European languages. In India we have almost 21 recognized languages and influence of these local languages, in English, is very much here. Due to the influence of local languages and English written by nonnative English speakers (ie. Indians) the result of TOEFL exams has shown lower scores against Indian students (also Asian students). This paper focuses on the existing automated essay grading systems, basic technologies behind them and proposes a new framework to over come the problems of influence of local Indian languages in English essays while correcting and by providing proper feedback to the writers.

When the A is for Agreement: Factors That Affect Educators' Evaluations of Student Essays

Research has established that a variety of factors predict educators' perceptions of students' performance; however, no studies have looked at the impact of educators' political views. The present experiment investigated the effect of educators' political biases on their grading of student essays. Participants included 122 volunteers who were randomly assigned to read an essay that supported or opposed the reelection of President Bush. Participants then evaluated the essay by assigning a holistic grade using a 4.0 grading scale and by assessing the essay's components using a rubric. Essays that matched educators' self-reported political views received higher holistic grades than those that did not. This same relationship, where essays that matched educators' views received higher grades, was found when educators used a rubric, indicating that a rubric was not an effective tool in preventing grader bias. The experiment suggests that educators' political views affect evaluations of student performance; the implications of these findings are discussed.

Covariates of the Rating Process in Hierarchical Models for Multiple Ratings of Test Items

When constructed response test items are scored by more than one rater, the repeated ratings allow for the consideration of individual rater bias and variability in estimating student proficiency. Several hierarchical models based on item response theory have been introduced to model such effects. In this article, the authors demonstrate how these models may be extended to include covariates of the rating process. For example, how do features of an essay grader’s training affect his or her performance? The authors show how to include covariates by embedding a linear model at appropriate levels of the model hierarchy. Depending on the level, such covariates may be thought of as determining fixed effects or random effects on the rating process. The authors also discuss the appropriate design matrix for such covariates, discuss how to incorporate needed identifiability constraints, and illustrate the methods using data from a rating study of a student assessment.