Language Testing Research Research Articles

A New Decade of Language Testing Research: Selected Papers from the 1990 Language Testing Research Colloquium

A New Decade of Language Testing Research: Selected Papers from the 1990 Language Testing Research Colloquium

Test dimensionality: IRT analysis of an ESP listening test1

The use of Item Response Theory (IRT) in the examination of the qualities of language tests is a comparatively recent development, and one that has proved controversial. Oller (1983) contains no reference to IRT in a wide-ranging col lection. By contrast, IRT, particularly involving the family of models deriving from the work of Rasch (1960 [1980]), has featured in a number of studies since the early 1980s. Although Rasch IRT has sometimes been discussed primarily as a tool for improved investigation of the reliability of tests (Skehan, 1989), its potential for investigating aspects of the validity of language tests has also been demonstrated (McNamara, 1990a). The application of Rasch IRT in this latter role has in some cases met with objections based on what are claimed to be its unsatisfactory theoretical assumptions, in particular the so-called 'unidimensionality' assumption (Hamp-Lyons, 1989). In this paper, this issue will be discussed in the context of the analysis of data from an ESP listening test for health professionals, part of a larger test, the Occupational English Test (OET), developed on behalf of the Australian Government (McNamara, 1989b). The paper is in three sections. First, there is a brief description of the listening subtest of the OET. Second, the appropriateness of the use of Rasch IRT in language testing research is discussed. Third, the use of Rasch IRT in the validation of the listening subtest of the OET is reported. In this part of the paper, the issue of unidimensionality is considered in the context of analysis of data from the two parts of this test.

Fundamental Considerations in Language Testing

Preface 1. Introduction The aims of the book The climate for language testing Research and development: needs and problems Research and development: an agenda Overview of the book Notes 2. Measurement Introduction Definition of terms: measurement, test, evaluation Essential measurement qualities Properties of measurement scales Characteristics that limit measurement Steps in measurement Summary Notes Further reading Discussion questions 3. Uses of Language Tests Introduction Uses of language tests in educational programs Research uses of language tests Features for classifying different types of language test Summary Further reading Discussion questions 4. Communicative Language Ability Introduction Language proficiency and communicative competence A theoretical framework of communicative language ability Summary Notes Further reading Discussion questions 5. Test Methods Introduction A framework of test method facets Applications of this framework to language testing Summary Notes Further reading Discussion questions 6. Reliability Introduction Factors that affect language test scores Classical true score measurement theory Generalizability theory Standard error of measurement: Interpreting individual test scores within classical true score and generalizability theory Item response theory Reliability of criterion-referenced test scores Factors that affect reliability estimates Systematic measurement error Summary Notes Further reading Discussion questions 7. Validation Introduction Reliability and validity revisited Validity as a unitary concept The evidential basis of validity Test bias The consequential or ethical basis of validity Post mortem: face validity Summary Notes Further reading Discussion questions 8. Some Persistent Problems and Future Directions Introduction Authentic language tests Some future directions A general model for explaining performance on language tests Apologia et prolegomenon Summary Notes Further reading Discussion questions Bibliography Author index Subject index

Newbury House TOEFL Preparation Kit: Preparing for the TOEFL by Daniel B, Kennedy, Dorry Mann Kenyon, and Steven J. Matthiesen and Newbury House TOEFL Preparation Kit: Preparing for the Test of Written English by Liz Hamp-Lyons

Issues in Applied Linguistics Vol.1 No. 2 Newbury House TOEFL Preparation Kit: Preparing for the TOEFL by Daniel B. Kennedy, Dorry Mann Kenyon, and Steven J. Matthiesen. New York: Newbury House, 1989. 262 pp. Newbury House TOEFL Preparation Newbury House, Preparing for the Test of Written English by Liz Hamp-Lyons. New York: Kit: 134 pp. Reviewed by Sara T. Gushing University of California, Los Angeles Recent research on the Educational Testing Service's Test of English as a Foreign Language (TOEFL) suggests that, while the TOEFL is a valid test of language proficiency, it tests only a limited range of communication skills and ignores many skills required by students in American university settings (Duran et al., 1985). There is also no clear evidence of any relationship between high TOEFL scores and academic achievement (Graham, 1987). Such findings raise questions about the appropriacy of using TOEFL preparation books in courses for university-bound ESL students, especially given the current orientation in academic-purpose language teaching to promote communicative competence rather than the sort of discrete language knowledge which is tested in many sections of the TOEFL. Indeed, conscientious teachers may wonder whether they are doing their students a disservice by spending class time on exercises specifically designed to raise TOEFL scores, at the expense of activities which promote the general academic skills needed at the university. At the same time, those language testing researchers concerned with the TOEFL's limitations might ask whether TOEFL preparation books do more harm than good since they perpetuate the status quo rather than encourage improvement of the TOEFL along the lines that some experts have suggested (Savignon, 1986; Larsen- Freeman, 1986) so as to increase its ability to measure communicative skills. The Newbury House TOEFL Preparation Kit is by no means radical in its attitude towards the TOEFL, but it does attempt to provide a partial answer to the first of these dilemmas. While the authors take a pragmatic approach toward the discrete-point sections of the TOEFL, they also emphasize those areas of the TOEFL which

Recent Publications on Statistics, Language Testing, and Quantitative Research Methods: II

Fundamental Considerations in Language Testing Lyle Bachman. Assessing Achievement in the Learner-Centred Curriculum Geoff Brindley. Testing for Language Teachers Arthur Hughes. Cambridge

Assessment and Evaluation

Research and development in the assessment of language abilities in the past decade have been concerned both with achieving a better understanding of the nature of language abilities and other factors that affect performance on language tests and with developing methods of assessment that are consistent with the way applied linguists view language use. The way language testers conceptualize language abilities has been strongly influenced by the broadened view of language proficiency as communicative competence that has emerged in applied linguistics. And while this view of language proficiency provides a much richer conceptual basis for characterizing the language abilities to be measured, it has presented language testers with a major challenge in defining these abilities and the interactions among them with sufficient precision to permit their measurement. Language testing researchers have also been influenced by developments in second language acquisition, investigating the effects on test performance of other factors such as background knowledge, cognitive style, native language, ethnicity, and sex. Finally, language testing research and practice have been influenced by advances in psychometrics, in that more sophisticated analytic tools are being used both to unravel the tangled web of language abilities and to assure thhat the measures of these abilities are reliable, valid, efficient, and appropriate for the uses for which they are intended.

Recent Publications on Statistics, Language Testing, and Quantitative Research Methods: I

Recent Publications on Statistics, Language Testing, and Quantitative Research Methods: I

Field independence : a source of language test variance?

Recent language testing research investigates factors other than language profi ciency that may be responsible for variance in language test performance. There is some evidence indicating field independence may be one variable responsible for introducing systematic error into language test scores. This paper reports research investigating the relationship between field independence and lan guage measures. Results indicate differential relationships of field indepen dence with cloze, dictation, and multiple-choice language tests. The relative strengths of these relationships also differ for native speakers in regular English classes, native speakers in remedial English classes, and nonnative speakers. Directions for further research are suggested.

Preliminary Evidence for the Reliability and Validity of a Foreign Language Anxiety Scale

Dickerson, W.B. (1984a, October). A new look at an old problem: The Z and D morphemes. Paper presented at the Fourth Annual Midwest Regional TESOL Conference, Cincinnati. Dickerson, W.B. (1984b). The role of formal rules in pronunciation instruction. In J. Handscombe, R. Orem, & B. Taylor (Eds.), On TESOL '83 (pp. 135148). Washington, DC: TESOL. Farhady, H. (1982). Measures of language proficiency from the learner's perspective. TESOL Quarterly, 16, 43-59. Harris, T. (1981). Practical English, Book 1. New York: Harcourt Brace Jovanovich. Hisama, K. (1980). An analysis of various ESL proficiency tests. In J.W. Oller, Jr., & K. Perkins (Eds.), Research in language testing (pp. 47-53). Rowley, MA: Newbury House. Pike, L.W. (1979). An evaluation of alternative item formats for testing English as a foreign language (TOEFL Research Rep. No. 2). Princeton, NJ: Educational Testing Service. Suter, R.W. (1976). Predictors of pronunciation accuracy in second language learning. Language Learning, 26, 233-253.

Issues in the testing of English for specific purposes

This paper examines recent approaches to communicative language testing, and, in particular, the testing of English for specific purposes (ESP). The first three sections of the paper are concerned with validation procedures as a means of verifying the claims made for new tests. These three sections con trast two approaches to language testing research — that related to Oller's unitary competence hypothesis, and that connected to the development of the ELTS (Carroll, 1978). Although the focus is on the latter, discussion of the Oller hypothesis is offered because it often provides a revealing contrast in methodology. Finally, the fourth section of the paper examines problems deriving from scoring procedures associated with ESP tests.

AN INTRODUCTION TO GENERALIZABILITY THEORY IN SECOND LANGUAGE RESEARCH1

During the last decade researchers in the area of second language testing have adopted increasingly more sophisticated evaluation techniques in an attempt to better understand the process of learning a second language. Analysis of variance, regression analysis, and factor analysis are among the statistical procedures that have allowed for insightful evaluation of language testing data. This paper explores the feasibility of applying another procedure, generalizability theory analysis, to second language testing research. An earlier paper (Hinofotis, Bailey, and Stern 1981) reported on the development of a rating scale for assessing the oral proficiency of nonnative‐speaking teaching assistants and the actual use of the scale by a panel of raters in evaluating pilot data. The data from that study are reanalyzed here using generalizability theory, which provides simultaneous estimates of consistency along several dimensions of the research, e.g., raters, subjects, and occasions. The paper includes a comparison of generalizability theory and classical test theory, an analysis of the oral proficiency data, and a brief discussion involving the interpretation of generalizability study results.

John W. OllerJr, and Kyle Perkins, Research in language testing. Rowley, Mass.: Newbury House, 1980. Pp. xi + 316.

An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.

Oller, John W. Jr. and Kyle Perkins. Research in Language Testing. Rowley, Mass.: Newbury House, 1980.

Oller, John W. Jr. and Kyle Perkins. <i>Research in Language Testing</i>. Rowley, Mass.: Newbury House, 1980.

: Research in Language Testing . John W. Oller, Jr., Kyle Perkins.

: Research in Language Testing . John W. Oller, Jr., Kyle Perkins.

: Kiswahili kwa Kitendo: An Introductory Course . Sharifa Zawawi.

American AnthropologistVolume 83, Issue 1 p. 213-213 Free Access Linguistics: Research in Language Testing. John W. Oller. Jr. and Kyle Perkins. Laurel A. Briscoe, Laurel A. Briscoe University of Texas, AustinSearch for more papers by this author Laurel A. Briscoe, Laurel A. Briscoe University of Texas, AustinSearch for more papers by this author First published: March 1981 https://doi.org/10.1525/aa.1981.83.1.02a00640AboutPDF ToolsExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL No abstract is available for this article. Volume83, Issue1March 1981Pages 213-213 RelatedInformation