What Is Criterion and e-rater, and How Can They Be Used in a Classroom?

Abstract

It is not uncommon for teachers to worry about delivering effective instruction and employing good pedagogical methods to maximize student learning. However, this becomes more problematic with the growing number of students per class and by the shortage of teachers who can accommodate the students. An immediate concern that will be discussed in this short piece is assessment, feedback, and scoring on writing from ESL/EFL students. A teacher becoming fatigued by an overwhelming amount of work is one concern, but it is often followed by unreliable and inaccurate assessment and grading of the students’ work. With little control over the shortage of teachers and the continuous enrollment of students, there must be a solution to alleviate the burden of teachers. This is where automated essay scoring (AES) tools such as Criterion by ETS may prove to be useful for teachers to effectively and efficiently provide meaningful feedback and grades. To elaborate, Criterion is an online platform providing convenience and practicality for its users, but it is the technology (also referred to as engine) of e-rater embedded in Criterion that provides automatic feedback on surface features of text such as grammar and scores Criterion is an online tool that provides instructors a platform for their students to submit writing assignments. When submitted, Criterion will generate automatic feedback on mechanical errors, and will also provide a score with great reliability and accuracy. For students, they will be able to submit their writing assignments online through Criterion and receive immediate grades, within a few seconds and feedback on mechanical errors in addition to in-depth instructor feedback (Criterion provides a separate section for this). When research in AES became popular in the 1990’s with rapid advancement in computer technology, e-rater was developed by the Educational Testing Service (ETS) in the 1990’s (Burstein, Brade-Harder, Chodorow, Hua, Kaplan, Kukich, Lu, Nolan, Rock & Wolff, 1998; Burstein, Leacock, & Swartz, 2001) and was first commercially used in 1999. e-rater uses statistics and natural language processing (NLP) to extract and analyze linguistic features from essays. Not only can it provide descriptive feedback of the writer’s mistakes and style, albeit formulaic and limited, but also a score (6 point scale) based on the writer’s grammar, mechanics, word use, style, lexical complexity, length, organization and more. Ultimately, e-rater attempts to look at certain features and provide feedback, along with rating as a human rater would. As wonderful as e-rater sounds, it still has some shortcomings when it comes to discourse, topic content, linguistic idiosyncrasies, and insufficient input such as two to three sentences (Higgins, Burstein, & Attali, 2006). In fact, some researchers claim that e-rater is neither reliable nor valid as ETS claims it to be (McCurry, 2010; Powers, Burstein, Chodorow, Fowles, & Kukich, 2002). Despite mixed findings on e-rater, teachers and students can still benefit from using Criterion. To reap the benefits, it is important to note that AES tools cannot replace human feedback and rating, and therefore, Criterion should be used as an augmenting tool at the discretion of an instructor.With objectivity, speed, and automaticity, Criterion can lessen a teacher’s workload by reducing repetitive and trivial error corrections, and allow both students