Assessment Of Oral Proficiency Research Articles

Oral proficiency assessment: the use of automatic speech recognition systems

The development and assessment of oral proficiency and listening comprehension is one of the most problematic aspects in language teaching, especially when the majority of test-takers are non-standard users of English. The main problems concern the feasibility of such testing and the need for reliable scoring. As far as feasibility is concerned, oral proficiency tests of communicative proficiency require much time as well as repeated assessment in the course of a semester or a year. To obtain a reliable score it is necessary to have more than one examiner, preferably also users of English as an additional language, to assess the same task. This paper will describe an attempt to use automatic speech recognition systems to obtain an objective score for oral proficiency. The process of test development and the subsequent digitalisation of speech, trialling and evaluation will be discussed with specific reference to a course that leads up to a language endorsement required by teacher trainees in South Africa.

A Study of web-based oral activities enhanced by Automatic Speech Recognition for EFL college learning

Recently, a promising topic in computer-assisted language learning is the application of Automatic Speech Recognition (ASR) technology for assisting learners to engage in meaningful speech interactions. Simulated real-life conversation supported by the application of ASR has been suggested as helpful for speaking. In this study, a web-based conversation environment called CandleTalk, which allows learners to seemingly talk with the computer, was developed to help EFL learners receive explicit speech acts training that leads to better oral competence. CandleTalk is equipped with an ASR engine that judges whether learners provide appropriate input. Six speech acts are presented as the foci of the materials with local cultural information incorporated as the content of the dialogues to enhance student motivation. The materials were put to use on 29 English major and 20 non-English major students in order to investigate their learning outcome and perception in an EFL context. Oral proficiency assessment using the format of the Discourse Completion Test (DCT) given before and after the use of CandleTalk and an evaluation questionnaire were two instruments used for data collection. The results of the study showed that the application of ASR was helpful for the college freshmen in the teaching of speech acts, particularly for the non-English major students. Most learners perceived positively toward the instruction supported with speech recognition.

World Englishes in language testing: a call for research

Native speakers of English are a minority; there are far more non-native speakers in the world (cf. Kachru 1997, Pennycook 2001). In addition, native speakers' standard or ‘correct’ English, in terms of its grammar and phonology, is not always useful or even appropriate in international contexts (cf. Gisborne 2000, Newbrook 1998, Shim 1999). However, despite global changes in the use of the language, the norms for ENL (English as a Native Language) remain dominant, most notably for the assessment of oral proficiency. Yet it is a major deficiency in the use of international oral tests that the proficiency of non-native speakers is measured against unrealistic and irrelevant standards (cf. Jenkins, 1996). The present paper focuses on the need to revisit the testing of English oral proficiency for non-native speakers, bearing in mind that English is used for world-wide communication and that being able to understand one another (cf. McKay, 2002) is the most important goal.

Bilingualism/Second-Language Research and the Assessment of Oral Proficiency in Minority Bilingual Children

This article discusses some of the challenges that researchers working in the fields of bilingualism and second-language acquisition (SLA) and in the field of language testing face in developing comparable and culturally and cognitively appropriate data collection and language assessment tools for bilingual children from rural minority-language communities with low levels of literacy. These challenges include defining language proficiency in a bilingual continuum, avoiding cultural biases, and attaining cross-linguistic validity. Using evidence from oral proficiency and aural comprehension tasks developed for bilingualism and SLA research and for the assessment of language proficiency in Quechua–Spanish and Aymara–Spanish bilingual students in the public schools of Peru, I illustrate recent attempts to develop appropriate tests to probe into bilingual students' linguistic competence.

ELICITED SPEECH FROM GRAPH ITEMS ON THE TEST OF SPOKEN ENGLISH™

ABSTRACTThis research applied a cognitive model to identify item features that lead to irrelevant variance on the Test of Spoken English™ (TSE®). The TSE is an assessment of English oral proficiency and includes an item that elicits a description of a statistical graph. This item type sometimes appears to tap graph‐reading skills—an irrelevant construct; TSE raters report that many examinees perform worse on this item type than they do on the other 11 items in the test. We adapted a cognitive theory of graph comprehension to predict the degree to which TSE graph items tap irrelevant skills such as graph reading. Through analyses of existing TSE data as well as an experiment, we show how the theory provides specific, empirically justified recommendations on the construction of graph items that minimize the influence of extraneous skills.

Estimating the difficulty of oral proficiency tasks: what does the test-taker have to offer?

This study investigates the impact of performance conditions on perceptions of task difficulty in a test of spoken language, in light of the cognitive complexity framework proposed by Skehan (1998). Candidates performed a series of narrative tasks whose characteristics, and the conditions under which they were performed, were manipulated, and the impact of these on task performance was analysed. Test-takers recorded their perceptions of the relative difficulty of each task and their attitudes to them. Results offered little support for Skehan’s framework in the context of oral proficiency assessment, and also raise doubts about post hoc estimates of task difficulty by test-takers.

12. DISCOURSE AND ASSESSMENT

Many contemporary currents in applied linguistics have favored discourse studies within assessment; there have been calls for cross-fertilization with other areas within applied linguistics, critiques of the positivist tradition within language testing research, and the growing impact of Conversation Analysis (CA) and sociocultural theory. This chapter focuses on the resulting increase in discourse-based studies of oral proficiency assessment techniques. These studies initially focused on the traditional oral proficiency interview but have since been extended to new test formats, including paired and group interaction. We discuss the research carried out on a number of factors in the assessment setting, including the roles of interlocutor, candidate, and rater, and the impact of tasks, task performance conditions, and rating criteria. Recent research has also concentrated more specifically on the assessment of pragmatic competence and on the applications of technology within the assessment of spoken language, including the comparability of semidirect and direct methods for such assessment and the use of computer corpora.

TALKING AND TESTING: DISCOURSE APPROACHES TO THE ASSESSMENT OF ORAL PROFICIENCY.Richard Young and Agnes Weiyun He (Eds.). Amsterdam: Benjamins, 1998. Pp. x + 395. NLG 138 cloth.

The goal of this volume is to provide a rich, interdisciplinary foundation of empirical studies describing the discourse of Language Proficiency Interviews (LPIs), focusing on “interactional competence.” The studies not only show how difficult it is to validly assess oral proficiency; they also show that discourse analysis of LPIs—as the editors point out—is very much in its infancy. Most of these studies face research design challenges (e.g., small sample size, sample comparability, coding reliability, or a combination of these), yet they illustrate the potential contributions of various (mostly qualitative) discourse analytic approaches to an understanding and assessment of oral proficiency.

Simulating conversations in oral-proficiency assessment: a conversation analysis of role plays and non-scripted interviews in language exams

Several recent studies have investigated the nature of interaction in oral proficiency exams and have concluded that the interview format obscures differences in the conversational competence of the candidates. The present paper examines what opportunities test takers have to display their knowledge of managing conversations in the L2 in two types of tasks: non-scripted interviews and guided role-play activities. The data for the study consists of 30 interviews and 30 role-play activities between near-native examiners and intermediate learners used in language exams in Hungary. The interviews and role-plays have been analysed for the number of topics introduced and ratified by the examiner and the candidate respectively, as well as for the number of interruptions, openings and closings produced by the examiner and the candidate. The findings show that the conversational interaction is more symmetrical in the guided role-play activity with the candidates introducing and ratifying approximately the same number of topics as the examiners. In addition, the examinees have the opportunity to interrupt and hold the floor more effectively during the role-play activity and can demonstrate their knowledge of how to open and close a conversation. These findings suggest that guided role-play activities used in the study exhibit several characteristics of real-life conversations and therefore can be used for assessing the candidates’ conversational competence.

Simulating conversations in oral-proficiency assessment: a conversation analysis of role plays and non-scripted interviews in language exams

Simulating conversations in oral-proficiency assessment: a conversation analysis of role plays and non-scripted interviews in language exams

The ‘free conversation’ and the assessment of oral proficiency

The ‘free conversation’ and the assessment of oral proficiency

Second Language Acquisition/Foreign Language Learning

Part 1 Some perspectives on SLA-FLL: theory and research in SLA and FLL - on producers and consumers what is foreign language research second and foreign language learning - same, different or none of the above models, processes, principles and strategies - second language acquisition in and out of the classroom. Part 2 Frameworks and approaches to research issues: Terence Odlin - word-order transfer, metalinguistics awareness and constraints on foreign language learning the acquisition of clitic pronouns in Spanish - two case studies, co-existing discourse worlds - the development of pragmatic competence in and outside the classroom foreign language learning - a social interaction and perspective on topic choice in oral proficiency assessment. Part 3 From research and theory to practice: second language acquisiton/foreign language learning, nothing is more practical than a good theory prefabricated speech for language learning a role for communicative competence and the acqusition/learning distinction in translator training can foreign language learning be like second language acquisition? - the curious case of immersion conservation of language resources.

A Criterion-Based Approach to Software Evaluation

AbstractThe overall poor quality of educational software on the market suggests that educators must continue efforts to evaluate available packages and to disseminate their findings. In this paper, weaknesses in published evaluation procedures are identified, and an alternative, criterion-based model is described. The rationale for the model is drawn from the fields of the assessment of student writing, criterion-referenced testing, and the assessment of second language oral proficiency. Data are presented on the mean ratings of software evaluated with the model, scale intercorrelations, and indicators of its validity and reliability.

Speech as Process

ABSTRACT The emphasis on assessment of oral proficiency that is inherent in the proficiency movement has favored a view of speech as a product, to which a certain rating is attached. However, with proficiency's growing impact on language learning and teaching it is imperative that proficiency testing take a stronger learner stance. This article advocates a “process” viewpoint of language use and explores its potential for greater insights from five perspectives: text linguistics and discourse analysis, interlanguage, expectancy, negotiation and interaction, and communication strategies.

Pragmatic and strategic competence in the assessment of oral proficiency

The present study, which is related to English as a foreign language in Danish schools, was designed to improve the construct validity and interscorer reliability of the oral exam after the completion of compulsory education. Focus in the article is on the description of expected pupil performance with regard to strategic and pragmatic competence. Strategic competence (Canale and Swain, Færch and Kasper), i.e. the pupils' ability to use communication strategies, is seen as a means of overcoming communication problems as well as an important teaching objective in its own right. With a view to pragmatic competence the main aim was to create a situation that would challenge the pupils to employ a wider range of speech acts than has traditionally been the case. An important aspect of the study was to test whether or not language teachers were able to use the descriptions of performance levels. ∗ ∗ The study was carried out by a research team and a full report is available in Brick-Hansen, A., Engel, E., Gregersen, J., Haastrup, K. and Møller, P.: Mundtlige prøver i engelsk, Brøndby kommune—Danmarks Lærerhøjskole, 1983.

THE STABILITY OF ORAL PROFICIENCY ASSESSMENT ON THE ORAL INTERVIEW TESTING PROCEDURES

The paper first discusses the complexity of measuring oral proficiency in communicative situations. The difficulty is due to the large number of variables, linguistic and social, which interact with one another. It then reports on a study which examined the stability of the assessment of oral proficiency on the oral interview test. Students of Hebrew as a foreign language underwent four administrations of different versions of that test. The administrations differed from one another by the occasion, the interviewer, the speech style, and the topic. Results from the analysis indicated that the different speech style and topic significantly affected students' scores on these tests while the occasion and the interviewer did not. The correlational analysis between pairs of tests pointed to low reliability and lack of stability of the tests, especially when two variables (i.e., occasion and tester) interacted. The results call for use of caution when decisions about individuals are made based on administration of communicative tests, for a need to identify sources of error in communicative tests, and for drawing stringent guidelines for the use of such tests.

Deep dyslexia Ed. by Max Coltheart, Karalyn Patterson, and John C. Marshall (review)

490 LANGUAGE, VOLUME 58, NUMBER 2 (1982) is a detailed report of the construction, development , and validation of an EFL placement test for German university students. The DELTA is a norm-referenced mix of discrete point and integrative measures, which is shown to be very efficient in predicting later success in learning English, as well as being useful in languageacquisition research. 'Measurement or global assessment of oral foreign language proficiency?', by W. Knibbeler , compares the global and analytic measurement of the French of 108 Dutch adult learners . K attempts to ascertain—via the creation of indices of correctness, fluency, and lexical variety—whether speech acts can or should be described by counting words and errors. The results show a high correlation between the global ratings of proficiency given by trained judges and the ratings derived from quantitative analysis oftranscription ofrecorded speech, the major difference being that global assessment is much less time-consuming. The last three articles are more directly concerned with the quantitative analysis of learner errors. T. Slama-Cazacu and T. Dutescu-Caliban present 'The first hierarchical system of errors (HSE) in the acquisition of English by native speakers ofRomanian' , a statistical study from the Romanian-English Contrastive Project of Bucharest. A total of 1183 errors produced by 252 speakers are analysed hierarchically by grammatical structure, taking into account frequency of occurrence and communicative importance. The authors scrutinize the different tasks involved, learner strategies such as 'disguising', the effect of time on learning and errors, and other variables; they conclude that the classical contrastive hypothesis is not valid, since most errors result not from the base language, but rather from influences in the material already acquired in the target language. They outline routes for further contrastive analysis and error therapy. U. O. H. Jung compares the written English of dyslexic and non-dyslexic German school children in 'Dyslexia and the foreign-language teacher'. J first reviews the literature on dyslexia , and then examines the performance of388 subjects on a dictation-cloze test. The results indicate that the classic definition of dyslexia, as characterized by reversal or inversion ofgraphemes in reading and writing, needs to be adjusted (at least in second-language learning situations ), since such a phenomenon represents only 3% of the performance errors; indeed, the 'normal' children tended to reverse and invert letters more often than the dyslexic children. R. Grotjahn closes the volume with a critical assessment of the methodological issues inherent in the quantification of articulatory difficulties —'Zur Quantifizierung der Schwierigkeit des Sprechbewegungsablaufs'. His specific focus is the quantitative definition of the problems which learners encounter in the articulation of German consonant sequences. This volume, which shows substantial awareness of the state of language research in the United States, seems significant for several reasons , not the least of which is the presentation ofthe meticulous research ofEuropean scholars who are not well-known in America. Particularly because ofits critical perspective and dedication to empirical substantiation of learning and teaching theory, the volume will be of considerable value to researchers in language acquisition and proficiency—whether linguists, psychologists, or psychometricians. [Alicia Pousada, Centro de Estudios Puertorriqueños, CUNY.] Deep dyslexia. Ed. by Max Coltheart , Karalyn Patterson, and John C. Marshall. London & Boston: Routledge & Kegan Paul, 1980. Pp. xi, 444. $45.00. In 1966, John Marshall and Freda Newcombe published a case study concerning 'selective dysphasic impairment', with specific emphasis on 'the linguistic aspect of paralexia'. When their patient was asked to read aloud single, unrelated words, he showed an interesting error pattern; instead of saying the given words, he gave semantically-related substitutions, e.g. 'uncle-nephew', 'liberty-freedom', 'little-short' etc. In a subsequent paper (1973) Marshall & Newcombe suggested that these substitution errors indicated the impairment of the spelling -sound route to the lexicon. Within half a decade, the dysphasic profile indicating nonfluent speech, fairly good object-naming, reduced short-term memory span, and predominantly semantic substitutions in oral reading became known as 'deep dyslexia' or 'phonemic dyslexia' (T. Shallice & E. Warrington, 1975). In 1978 a symposium was held at Jesus College, Cambridge, to discuss the 22 cases (16 English and 6 Japanese) reported in the neuropsychological literature. The present book is a result of this meeting. BOOK...

Some Guidelines for the Assessment of Oral Language Proficiency and Dominance

According to the Survey of Languages Supplement to the July 1975 Current Population Survey, more than 7.6 million school age students in the United States live in households where languages other than English are spoken. Given these language environments, one's degree of bilingualism may theoretically range from a point approximating monolingualism to a state of balanced bilingualism. With such a range of possible degrees of bilingualism, there is a pressing need for some general guidelines on its measurement, especially in light of recent federal and state legislation requiring school districts to assess the language dominance and proficiency of their bilingual children in each of the languages. We therefore begin by defining four dimensions of bilingual measurement in a manner that is both theoretically sound and educationally useful. Next, we address three major topics in the assessment of language proficiency and language dominance: (1) selection of the language components to be assessed; (2) appropriateness of certain elicitation tasks used; and (3) general checkpoints that can be used to evaluate language proficiency dominance instruments. We hope this survey will bring into focus some of the important issues and questions facing those attempting to assess the linguistic proficiency and dominance of linguistic minority students enrolled in the United States public schools.