Holistic Ratings Research Articles

Patterns of Writing Skills Derived From Cluster Analysis of Direct-Writing Assessments

Current interest in performance assessment in education raises many measurement questions, one of which is concern for the validity and usefulness of holistic ratings versus multiple-dimension ratings of student work. A study of more than 10,000 students in Grades 3 and 8 was conducted to explore the advantages of analytical scoring in writing assessment. Trained readers scored student essays on a 6-trait analytical scoring system. Cluster analyses were employed to identify subgroups of students who showed similar patterns of trait scores. Eleven patterns of student scores, reflecting different patterns of strengths and weaknesses in writing, were replicated across five modes of writing (e.g., descriptive and persuasive). Limitations of the study and implications for performance assessment are discussed.

The Effects of Learning Strategy Instruction on the Completion of Job Applications by Students with Learning Disabilities

The purpose of this study was to assess the effects of learning strategy instruction on the completion of job applications by students identified as learning disabled. Thirty-three students (average age 15 years 6 months) were randomly assigned by grade and gender to one of two experimental conditions: learning strategy instruction or traditional instruction. The result was 16 students (10 boys and 6 girls) being placed under the learning strategy instruction condition and 17 students (10 boys and 7 girls) being placed under the traditional instruction condition. Results indicated that in addition to statistically significant lower numbers of information omissions and information location errors, holistic ratings of the overall neatness of the job applications were significantly higher for those students under the learning strategy instruction condition. In addition to these positive changes in the performance measures, social validity data suggest that students under the learning strategy condition would be more likely to receive an invitation for a job interview. The findings and future research needs are discussed.

The conversational skills of school‐aged children

AbstractDifferences associated with age and with language background were assessed in performance on a task thin taps conversational skill, a ‘TV talk show’ interview task administered to 98 second through fifth graders, including many nonnative speakers of English. Surprisingly, holistic ratings of the childrenapos;s skills as conversationalists were not related at all to gender, and only moderately to age or status as a native speaker of English. A set of 29 specific interactive and language behaviors were analyzed, to determine which of these were related to holistic ratings of conversational skill. Variables reflected strategies for eliciting information and for maintaining topic, indicators of disfluency, ami indicators of how the adult was contributing to the conversation. Children who had received higher ratings on conversational skill produced more topic continuations, a higher proportion of more sophisticated noncontingent responses, and fewer simple yes/no topic initiations and continuations. Age, gender, and language status related only minimally to the specific interactive or language behaviors associated with success as a conversationalist, although children from homes where English was spoken did tend to receive higher holistic ratings of conversational skill.

Progress Monitoring with Objective Measures of Writing Performance for Students with Mild Disabilities

To adequately monitor progress in writing, a test must show stability and demonstrate a performance profile over time that parallels those of accepted criterion measures. This study investigated the technical adequacy of seven objective indexes of writing quality in monitoring the progress of 36 middle school (Grades 6-8) students with mild disabilities over a 6-month period. The stability of each index was assessed. Holistic ratings of the same writing samples and the Test of Written Language served as validation criteria. Three indexes were moderately correlated with holistic ratings, but were not sufficiently stable over time. Direct, objective writing assessment must be used with caution for progress monitoring, given our present lack of precision in measuring the complex task of writing.

Partial Credit Analysis of Writing Ability

The direct assessment of the writing ability of 2000 randomly selected secondary school students was performed using the partial credit model. The effects on item parameter estimates of rating scale and type of writing sample were investigated. The rating scales used were a sum and an averaging of the individual raters' ratings and the types of items were expository and narrative. Results showed that expository items tended to provide more information for high ability examinees than did narrative items and that the sum holistic rating method yielded greater information than did the average holistic rating scale. Additional analyses of the rating scales used as well as implications for test construction are discussed.

What influences raters' judgment of student writing

This article compares the results of a linguistic analysis of 30 students' compositions with the holistic ratings of nine independent judges. The compositions were the examination scripts of 17-year-old high school students drawn from the two levels of the Caribbean Examinations Council English Language examinations: General Proficiency and Basic Proficiency. The General Proficiency compositions, according to the examination syllabus, were expected to display a greater “maturity of expression” than compositions at the Basic Proficiency level. The linguistic analysis was based on four potential indicators of maturity: morphological (past tense morpheme only), syntactic, semantic, and discourse. The judges' holistic ratings, the method used to divide the compositions into two groups, “more mature” and “less mature,” were based on the judges' definition of the criterion “maturity of expression.” A comparison of the two sets of results on an implicational scale revealed that the factor that weighed most heavily with the judges was correct past tense usage.

Cohesiveness as a basis for evaluating compositions

This study explores the relationship between holistic rating and scoring compositions for cohesive features. Thirty-eight compositions written by EFL learners of different language proficiency were analyzed for various cohesive ties and cohesive types. Four ESL/EFL instructors rated the essays for writing quality. The results of the comparison revealed that the evaluation of compositions by cohesive elements is feasible with writings of more advanced subjects.

Cognitive and Metacognitive Aspects of Learning Disabled Adolescents' Composing Problems

This study investigated the cognitive and metacognitive aspects of writing problems in learning disabled adolescents. Twenty-one learning disabled eighth and eleventh graders constituted the target population. The comparison groups included 15 normally achieving eighth graders and a reading-age control group of 23 normally achieving sixth graders. The participants wrote two reportive essays and one argument essay, and answered a questionnaire designed to probe their metacognition about the writing process. Results indicated that normally achieving eighth graders consistently and clearly surpassed their learning disabled counterparts in both quality and quantity of essay writing. Adult judges rated normally achieving eighth graders' essays to be substantially more interesting, much clearer in communicating the goals, and to contain a substantially more felicitous word choice than those of learning disabled eighth and eleventh graders. Moreover, they wrote longer essays with much fewer spelling errors than learning disabled eighth and eleventh graders. Finally, they possessed more mature and articulate conceptions of the writing task, and were more aware of the relevance of planning and audience in writing. In striking contrast, the learning disabled eighth and eleventh graders were comparable to the normally achieving sixth graders in their performances across a broad front. This comparability occurred in the three categories of holistic ratings of essays (interestingness, clarity in communication of goals, and word choice), structural ratings of paragraphs, and metacognition about the writing process. Interestingly, learning disabled eighth and eleventh graders were superior to normally achieving sixth graders in length of essays. However, the consistent, broad picture was that the learning disabled adolescents performed at a very similar level to that of younger normally achieving children. These findings permit the conclusion that the learning disabled adolescents in this study showed a developmental delay in the cognitive and metacognitive aspects of writing.

Assessment of Written Expression for Students in Compensatory and Special Education Programs

This study used eight methods to objectively score 6-minute writing samples from students in compensatory and specialized educational programs. Each sample was also evaluated subjectively by four trained judges, using a holistic judgment of communicative effectiveness. Direct counts were made of the number of words: (a) written, (b) written legibly, (c) spelled correctly, (d) written in correct sequence, and (e) written in continuous correct sequences (averaged over sequences). Secondary calculations produced the percentage of words: (a) written legibly, (b) spelled correctly, and (c) written in correct sequence. In both a cluster analysis and factor analysis of the eight measures, two factors were identified, with the first four measures clustering together (representing a production-dependent factor) and the latter four measures clustering together (reflecting a production-independent factor). Regression of holistic ratings on objective scores produced moderately strong results for two production-independent indices—percentage of words correctly spelled and percentage of words correctly sequenced. The production-independent factor scores were much stronger predictors of holistic ratings than were production factor scores, but were weaker than each of the two strongest individual predictors.

A COMPARISON OF HOLISTIC VERSUS DECOMPOSED RATING OF POSITION ANALYSIS QUESTIONNAIRE WORK DIMENSIONS

Recent studies have attempted to reduce the cost and intrusiveness of the Position Analysis Questionnaire (PAQ) by limiting the amount of information provided to the analyst, with consistently negative results. We examined an alternative technique for improving the cost‐effectiveness of the PAQ that avoids the need to rate the hundreds of items that constitute the instrument. Three groups of raters (professional job analysts, graduate students in industrial psychology who were familiar with the PAQ, and PAQ‐unfamiliar undergraduates) made direct holistic ratings of the PAQ dimensions for four familiar jobs. The holistic ratings were compared with decomposed PAQ dimension profiles obtained from the item‐level ratings of the professional analysts. Cronbach accuracy analyses indicated near‐zero convergence between the holistic and decomposed dimension ratings, even for the professional PAQ job analysts. We conclude that holistic rating of dimensions is not an effective means of reducing the cost of a PAQ job analyses and that it is likely to be similarly ineffective with task‐ or ability‐based instruments.

Dark Shadows: The Fate of Writers at the Bottom

The number of categories in holistic rating scales varies a good deal, from as few as three to as many as fifteen. What remains constant is the uniform way scorers conceive of the bottom-most cubicle. Essays settled there almost always have the highest concordance of scores assigned independently by raters, higher even than have the topmost essays. Apparently those performing holistic ratings-usually teachers of writing-agree better on who are bad writers than on who are good. This strikes me as a curiosity worth a closer look.

Writing Persuasive Documents

AbstractTwo experiments examined the role that college students’ verbal aptitude, a cognitive variable, and evaluation anxiety, an affective variable, play in the production of persuasive letters. In both experiments, the students with higher verbal aptitude tended to obtain higher holistic ratings of letter quality, with fewer punctuation and spelling errors, than the students with lower aptitude. In addition, the students’ verbal aptitude and anxiety were inversely related. In Experiment 1, where time limits were imposed to pressure students, there was a weak relationship between anxiety and letter quality. There was also some support for the notion that poor verbal aptitude leads to anxiety. In Experiment 2, where time was unlimited, the relationship between anxiety and letter quality was no longer significant. Taken together, these findings suggest that students’ evaluation anxiety has little, if any, influence on the overall quality of their writing. In general, the findings were interpreted as provi...

The Reliability, Validity, and Utility of Three Handwriting Measurement Procedures

This study investigated the reliability, validity, and utility of the following three measures of letter-formation quality: (a) a holistic rating system, in which examiners rated letters on a five-...

The Effects of Modifying the Formality Level of ESL Composition Questions

I We know of no investigations of the effect of topic on compositions written by ESL populations under closed-book, essay-examination conditions. In L1 contexts, several studies have been concerned with varying the amount and kind of information in the topic, or stimulus (e.g., Smith et al., 1985). In the most directly relevant of these studies, Brossell and Hoetker Ash (1984) investigated the effects of personal (you) as opposed to neutral phrasing and of a question versus an imperative format. They found no significant results and concluded that their study produced no evidence to support the contention that small changes in the wording of essay examinations of otherwise similar construction affect writers or the holistic ratings given their essays (p. 425). The study reported here examined the extent to which the short and general topics used in the Composition section of the Michigan English Language Assessment Battery (MELAB) (English Language Institute, 1984) generate rhetorically and linguistically unambitious answers that fail to indicate fully the communicative abilities of candidates. Since this investigation was part of a series of studies aimed at developing a new test specifically designed for the evaluation of nonnative-speaking (NNS) graduate and transfer students, it also examined the extent to which more ambitious and more academically appropriate writing could be elicited by simply increasing the level of formality of the topic. An example of simple and academic topic variants is given below.

A study of cohesion and coherence in English as a second language students’ writing

Abstract This article examines cohesion and coherence in ESL learners’ writing compared with the writing of native English speakers. Altogether six essays on argumentative tasks were analyzed using Halliday and Hasan's (1976) cohesion theory. The coherence in the essays was measured during holistic ratings and an interactive text analysis based on speech act theory. The results of the study show that to be cohesive, an ESL essay did not need to be coherent. Furthermore, general cohesion density was not found to be a discriminating factor between the native speakers and ESL writers. The ESL writers were found, however, to lack the variety of lexical cohesive devices used by the native speakers. With respect to coherence, ESL learners’ writing lacked: 1) adequate justification for claim statements, and 2) sufficient linking of concluding statements to preceding subtopics of the problem.

How characteristics of student essays influence teachers' evaluations.

This article explores the question of why competent evaluators award the ratings they do to college students' expository essays. Essays were rewritten to be stronger or weaker in four categories: content, organization, sentence structure, and mechanics. Twelve evaluators first used a 4-point holistic rating scale to judge the essays' quality. Then they rated whether each of the four rewriting categories in each rewritten essay was strong or weak (perceptions). Analyses of variance revealed content and organization to affect ratings most (p < .001). Mechanics and sentence structure had smaller effects, which differed when measured by the actual rewriting versus by the perceptions. Mechanics and sentence structure were significant in their interaction with organization (p < .001 andp < .01, respectively).

La evaluación de la competencia oral en segundas lenguas. El caso del inglés en el mundo profesional

La evaluación de la competencia oral es una de las más subjetivas en la enseñanza de una segunda lengua y a veces la que más conflictos origina en cuanto la evaluación justa e injusta dependiendo de los objetivos de dicha evaluación. Además del problema de la subjetividad hay que añadir la complejidad del consumo de tiempo que requiere si se quiere realizar de forma individual, la preparación de la actividad, la rúbrica específica que se quiere utilizar y las exigencias del proceso o protocolo que se debería de seguir para una correcta evaluación.En este documento realizaremos un planteamiento teórico de la evaluación de la competencia oral en segundas lenguas, el proceso de evaluación y la importancia de las escalas de valoración. Veremos las que son utilizadas por las instituciones más reconocidas en la enseñanza y evaluación del inglés además de sus descriptores, poniendo ejemplos de dicha evaluación en el mundo laboral y además propondremos una rúbrica para evaluar de forma holística que puede ser utilizada en un proceso de selección de candidatos a un trabajo.