Holistic Scales Research Articles

Assessing second language oral proficiency development with holistic and analytic scales

The present study discusses how a group of experienced raters use different types of scales to assess the development of oral proficiency in English as a second language (L2). Raters assigned rates to speech samples first using a holistic scale (CEFR, 2018) and then assigning rates for pronunciation, vocabulary, grammar and fluency performance using individual scales. The speech samples were recorded by five Brazilians. There were two data collection sessions, with the second one occurring 7-8 months after the first one. The results indicate high levels of agreement among raters for all scales. Furthermore, the raters detected changes in speakers’ performance in four out of five scales: L2 oral proficiency, vocabulary, grammar, and fluency, and these differences in rates across time were significant for oral proficiency, vocabulary, and fluency. Thus, the different types of scale allow detecting L2 oral proficiency development.

Integrated writing scores based on holistic and multi-trait scales: A generalizability analysis

Abstract Scoring reading-to-write (RTW) tasks is known to be more challenging than independent tasks given that raters should attend to features of source use, in addition to other writing skills. Holistic scales have been traditionally used with this task type; however, analytic scales have recently received increasing attention. While research has looked into score generalizability of RTW tasks, few studies have addressed the impact of rating scales on RTW score reliability. For this purpose, the current study compares score reliability from both holistic and multi-trait rating scales. Following a generalizability theory approach, five raters scored 60 essays written by EFL university students using both holistic and multi-trait scoring methods. The results indicated that scores obtained based on the multi-trait rubric were found to be more reliable than those obtained from a holistic rubric. The results also showed that source use was the most reliable feature among all dimensions of the multi-trait rubric. The study results provide sufficient evidence for the use of multi-trait rating scales in the context of scoring RTW tasks. This outcome could encourage more writing instructors to use these scales in their assessment practices given the detailed information they provide about writers’ performance on RTW tasks.

Regression‐Based Landmark Detection on Dynamic Human Models

AbstractDetecting anatomical landmarks on various human models with dynamic poses remains an important and challenging problem in computer graphics research. We present a novel framework that consists of two‐level regressors for finding correlations between human shapes and landmark positions in both body part and holistic scales. To this end, we first develop pose invariant coordinates of landmarks that represent both local and global shape features by using the pose invariant local shape descriptors and their spatial relationships. Our body part‐level regression deals with the shape features from only those body parts that correspond to a certain landmark. In order to do this, we develop a method that identifies such body parts per landmark, by using geometric shape dictionary obtained through the bag of features method. Our method is nearly automatic, as it requires human assistance only once to differentiate the left and right sides. The method also shows the prediction accuracy comparable to or better than those of existing methods, with a test data set containing a large variation of human shapes and poses.

The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE

ObjectiveAlthough previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills. MethodsThe University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training. ResultsFor analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales. ConclusionA combined analytic-holistic rating scale increased score reliability and was sensitive to level of training. Practice implicationsGiven increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.

Surface Heat Island in Shanghai and Its Relationship with Urban Development from 1989 to 2013

The continuous expansion of impervious artificial surfaces in cities has significantly influenced the urban thermal environment. This paper examines the spatiotemporal variation of the diurnal surface urban heat island (SUHI) in Shanghai from 1989 to 2013, a period during which the city has experienced drastic development changes. A remote sensing approach was taken to derive the spatial patterns of Shanghai’s land surface temperature (LST) from Landsat Thematic Mapper (TM) images and Operational Land Imager (OLI) data. The LST pattern was further classified into five LST classes to look at the relative SUHI intensity level across the whole city. Spatial analyses, namely, spatial association and centroid movement analysis, were conducted to reveal the trends of LST changes at both local and holistic scales. To understand the potential drivers for the present spatiotemporal variation of SUHI, different indicators including land use change, population density, night light data, and vegetation were analyzed and compared with LST changes. Based on the quantitative analysis and the socioeconomic context of Shanghai, “heating up” regions were identified, possible reasons for such SUHI variation were summarized, and districts that are most vulnerable to extreme heat conditions were projected. In terms of implication for urban development, planning and design recommendations were suggested to improve the urban thermal environment in Shanghai.

Development of a Spanish generic writing skills scale for the Colombian Graduate Skills Assessment (Saber Pro)

While many higher education institutions list the generic skills their graduates are intended to acquire during a course of study (Barrie, 2006), the relevant skills are rarely directly assessed at graduation. In Colombia, exit assessment is compulsory for all post-secondary training and education. To this end, a Spanish-language version of the Australian Graduate Skills Assessment (GSA) was developed for the Colombian context. However, problems were identified with the reliability of the Spanish version of the GSA writing scale. This paper describes the process of replacing the original version of the Spanish-language version of the GSA scale (an intuitively based writing scale) with an empirically based scale developed using a question tree method. Forty raters constructed two holistic (combined trait) and three analytic (individual trait) writing scales using benchmarked scripts from a previous test administration. The five scales were then trialled. Comparison of the scales showed the eight-level holistic scale provided the widest distribution of scores. This research provides insights into generic writing skill testing for higher education graduates in Colombia. In addition, the study uniquely provides a detailed description of the development of empirically-based analytic and holistic scales for assessing the writing of Spanish-L1 speaking graduates in Colombia.

Assessing learners’ writing skills in a SLA study: Validating the rating process across tasks, scales and languages

There is still relatively little research on how well the CEFR and similar holistic scales work when they are used to rate L2 texts. Using both multifaceted Rasch analyses and qualitative data from rater comments and interviews, the ratings obtained by using a CEFR-based writing scale and the Finnish National Core Curriculum scale for L2 writing were examined to validate the rating process used in the study of the linguistic basis of the CEFR in L2 Finnish and English. More specifically, we explored the quality of the ratings and the rating scales across different tasks and across the two languages. As the task is an integral part of the data-gathering procedure, the relationship of task peformance across the scales and languages was also examined. We believe the kinds of analyses reported here are also relevant to other SLA studies that use rating scales in their data-gathering process.

Measuring nursing competencies in the operating theatre: Instrument development and psychometric analysis using Item Response Theory

Concern about the process of identifying underlying competencies that contribute to effective nursing performance has been debated with a lack of consensus surrounding an approved measurement instrument for assessing clinical performance. Although a number of methodologies are noted in the development of competency-based assessment measures, these studies are not without criticism. The primary aim of the study was to develop and validate a Performance Based Scoring Rubric, which included both analytical and holistic scales. The aim included examining the validity and reliability of the rubric, which was designed to measure clinical competencies in the operating theatre. The fieldwork observations of 32 nurse educators and preceptors assessing the performance of 95 instrument nurses in the operating theatre were used in the calibration of the rubric. The Rasch model, a particular model among Item Response Models, was used in the calibration of each item in the rubric in an attempt at improving the measurement properties of the scale. This is done by establishing the 'fit' of the data to the conditions demanded by the Rasch model. Acceptable reliability estimates, specifically a high Cronbach's alpha reliability coefficient (0.940), as well as empirical support for construct and criterion validity for the rubric were achieved. Calibration of the Performance Based Scoring Rubric using Rasch model revealed that the fit statistics for most items were acceptable. The use of the Rasch model offers a number of features in developing and refining healthcare competency-based assessments, improving confidence in measuring clinical performance. The Rasch model was shown to be useful in developing and validating a competency-based assessment for measuring the competence of the instrument nurse in the operating theatre with implications for use in other areas of nursing practice.

The psychology of face construction: Giving evolution a helping hand

Face construction by selecting individual facial features rarely produces recognisable images. We have been developing a system called EvoFIT that works by the repeated selection and breeding of complete faces. Here, we explored two techniques. The first blurred the external parts of the face, to help users focus on the important central facial region. The second, manipulated an evolved face using psychologically-useful ‘holistic’ scales: age, masculinity, honesty, etc. Using face construction procedures that mirrored police work, a large benefit emerged for the holistic scales; the benefit of blurring accumulated over the construction process. Performance was best using both techniques: EvoFITs were correctly named 24.5% on average compared to 4.2% for faces constructed using a typical ‘feature’ system. It is now possible, therefore, to evolve a fairly recognisable composite from a 2 day memory of a face, the norm for real witnesses. A plausible model to account for the findings is introduced. Copyright © 2010 John Wiley & Sons, Ltd.

Measuring argumentative reasoning: What's behind the numbers?

The aim of this paper is to develop a more thorough, empirically-based understanding of the differences in measurement of written argumentation when alternative scoring frameworks are employed. Reflective compositions of 127 elementary school children were analyzed using analytic and holistic scales. The scales were derived from Argument Schema Theory, an explicit model of argumentation development. We investigated the relationships among the different scales, as well as their relative reliability and efficiency. The scores derived using analytic and holistic methods have adequate reliability. Although less efficient, analytic scoring allows for gathering more sensitive and detailed information about the differences in student performance. The results suggest that the choice of an analytic framework for measuring argumentation should not be arbitrary, as each scoring method taps into distinct facets of the construct.

Language testing

04–83Akiyama, Tomoyasu (U. Melbourne, Australia). Assessing speaking: issues in school-based assessment and the introduction of speaking tests into the Japanese senior high school entrance examination. JALT Journal (Tokyo, Japan), 25, 2 (2003), 117–141.04–84Chiang, Steve (Yuan Ze University, Taiwan). The importance of cohesive conditions to perceptions of writing quality at the early stages of foreign language learning. System (Oxford, UK), 31 (2003), 471–484.04–85Escamilla, Kathy, Mahon, Elizabeth, Riley-Bernal, Heather and Rutledge, David (U. of Colorado, Boulder, USA). High-stakes testing, Latinos, and English language learners: lessons from Colorado. Bilingual Research Journal (Arizona, USA), 27, 1 (2003), 25–49.04–86Gorsuch, Greta (Texas Tech U., USA; Email: greta.gorsuch@ttu.edu). Test takers' experiences with computer-administered listening comprehension tests: interviewing for qualitative explorations of test validity. Calico Journal (Texas, USA), 21, 2 (2004), 339–371.04–87Hardcastle, Peter.How to not test language (Part 2). Language Testing Update (Lancaster, UK), 33 (2003), 28–35.04–88Hemard, D. and Cushion, S. (London Metropolitan, University, UK; Email: hemard@lgu.ac.uk). Design and evaluation of an online test: assessment conceived as a complementary CALL tool. Computer Assisted Language Learning (Lisse, The Netherlands), 16, 2–3 (2003), 119–139.04–89Ishii, David N. and Baba, Kyoko (U. of Toronto, Canada; Email: dishii@aoise.utoronto.ca). Locally developed oral skills evaluation in ESL/EFL classrooms: a checklist for developing meaningful assessment procedures. TESL Canada Journal/Revue TESL du Canada (Burnaby, Canada), 21, 1 (2003), 79–96.04–90Iwashita, Noriko and Grove, Elizabeth (University of Melbourne, Australia). A comparison of analytic and holistic scales in the context of a specific-purpose speaking test. Prospect (Sydney, Australia), 18, 3 (2003), 25–35.04–91Lee, Yong-Won (Educational Testing Service, Princeton, NJ, US; Email: ylee@ets.org). Examining passage-related local item dependence (LID) and measurement construct using Q3statistics in an EFL reading comprehension test. Language Testing (London, UK), 21, 1 (2004), 74–100.04–92Qian, David D. (Hong Kong Polytechnic U., Hong Kong; Email: David.Qian@polyu.edu.hk) and Schedl, Mary (Educational Testing Service, Princeton, NJ, US). Evaluation of an in-depth vocabulary knowledge measure for assessing reading performance. Language Testing (London, UK), 21, 1 (2004), 28–52.04–93Rea-Dickins, Pauline (University of Bristol, UK). Classroom assessment of English as an additonal language: Key stage 1 contexts – summary of research findings. Language Testing Update (Lancaster, UK), 33 (2003), 48–53.04–94Rodgers, Catherine, Meara, Paul and Jacobs, Gabriel (U. of Wales Swansea, UK). Factors affecting the standardisation of translation examinations. Language Learning Journal (London, UK), 28 (Winter 2003), 49–54.

Examinee abilities and task difficulty in task-based second language performance assessment

This article summarizes findings from investigations into the development and use of a prototype English language task-based performance test. Data included performances by 90 examinees on 13 complex and skills-integrativetasks, a priori estimations of examinee proficiency differences, a priori estimations of task difficulty based on cognitive processing demands, performance ratings according to task-specific as well as holistic scales and criteria, and examinee self-ratings. Findings indicated that the task-based test could inform intended inferences about examinees’ abilities to accomplish specific tasks as well as inferences about examinees’ likely abilities with a domain of tasks. Although a relationship between task difficulty estimates and examinee performances was observed, these estimates were not found to provide a trustworthy basis for inferring examinees’ likely abilities with other tasks. These findings, as well as study limitations, are further discussed in light of the intended uses for performance assessment within language education, and recommendations are made for needed research into the interaction between task features, cognitive processing and language performance.

SECOND LANGUAGE DEVELOPMENT IN WRITING: MEASURES OF FLUENCY, ACCURACY, AND COMPLEXITY. Kate Wolfe-Quintero, Shunji Inagaki, and Hae-Young Kim. Honolulu: University of Hawai‘i Press, 1998. Pp. viii + 187. $20.00 paper.

Second language development in writing: Measures of fluency, accuracy, and complexity is a technical report that reviews 39 studies examining measures of writing development. It is a comprehensive, lucid, and carefully researched piece of work analyzing an untidy body of literature. This volume is essential reading for anyone doing quantitative research in L2 writing. The studies included in this review are those that have attempted to correlate potential linguistic measures of writing development with writing proficiency, not simply studies that have used the various measures as dependent variables to examine the effects of some instructional treatment. Complicating the picture is the fact that indicators of writing proficiency are varied, as is the case with indicators of oral proficiency (Thomas, 1994). Each of these indicators, including standardized tests, levels in various language programs, and holistic measures, has its own set of problems. For example, of the 16 studies using holistic scales, only half reported interrater or intrarater reliability.

Establishing Validity for Performance-Based Assessments: An Illustration for Collections of Student Writing

Abstract Techniques for establishing the reliability and validity of assessments of student writing are presented. Raters scored collections of elementary students' narrative writing with the holistic scales of two rubrics—a new rubric designed for classroom use and known to enhance teacher practice, and an established rubric for large-scale writing assessment. Comparisons of score reliabilities were based on three methods: percentage agreement, correlations between rater pairs, and generalizability studies. Comparisons of the evidence for validity of scores were based on (a) correlations of scores with results from two other methods of writing assessment, (b) developmental patterns across grade levels, and (c) consistency of decisions made across methods of assessment. Results were mixed; good evidence was provided for the reliability and developmental validity of the new rubric, but correlational patterns were not clear. The importance of establishing performance-based assessments of writing that are bo...