TOEFL Test Research Articles

Dependability of scores for a new ESL speaking assessment consisting of integrated and independent tasks

A multitask speaking measure consisting of both integrated and independent tasks is expected to be an important component of a new version of the TOEFL test. This study considered two critical issues concerning score dependability of the new speaking measure: How much would the score dependability be impacted by (1) combining scores on different task types into a composite score and (2) rating each task only once? To answer these questions, generalizability theory (G-theory) procedures were used to examine the impact of the numbers of tasks and raters per speech sample and of subsection lengths on the dependability of speaking scores. Univariate and multivariate G-theory analyses were conducted on rating data collected for 261 examinees for the study. The finding in the univariate analyses was that it would be more efficient to increase the number of tasks rather than the number of ratings per speech sample in maximizing the score dependability. The multivariate G-theory analyses also revealed that (1) the universe (or true) scores among the task-type subsections were very highly correlated and that (2) slightly larger gains in composite score reliability would result from increasing the number of listening - speaking tasks for the fixed section lengths.

Achievement Goal Theory: A Perspective on Foreign-Language-Learners' Motivation

It has long been recognized that varying achievement goals elicit varying motivational patterns with varying behavioral consequences. Several sets of contrasting goal orientations have been proposed to explain differences in language students' achievement behaviors. A total of 135 third-year (n = 54 male; n = 81 female) students in a preservice English teacher education program participated in this study of goal orientation. The proposed goal orientations were measured by adapting the Goal Orientation Scale developed by Skaaalvik (1997) for the L2 learning domain, and students' language achievement was measured by a TOEFL test. Results indicated that students placed most emphasis on task mastery goals and that this was related to language achievement. Task mastery goals are negatively correlated with work-avoidance. Self-defeating ego orientation is positively correlated with both work-avoidance and self-enhancing ego orientation. The analysis also suggested that male students had a stronger tendency to avoid work.

The performance of native speakers of English and ESL speakers on the computer-based TOEFL and GRE General Test

The purpose of this study was to replicate previous research on the construct validity of the paper-based version of the TOEFL and extend it to the computer-based TOEFL. Two samples of Graduate Record Examination (GRE) General Test-takers were used: native speakers of English specially recruited to take the computer-based TOEFL, and ESL test-takers who routinely took the computer-based TOEFL recently. Native speakers performed well on the TOEFL, relative to ESL test-takers and to the maximum possible scores on the test, and varied less in their test performance than did ESL test-takers. TOEFL scores were moderately or highly correlated with the computer-based GRE General Test scores for both groups of test-takers. Regressions of the GRE General Test Verbal scores on the TOEFL scores for ESL test-takers were nonlinear, and the regressions of the other GRE General Test scores were linear. And trends in the variances of the GRE General Test Verbal scores associated with TOEFL scores were also nonlinear, and the trends were either unsystematic or negatively linear for the other GRE General Test scores. All of the ndings are consistent with previous results with the paper-based TOEFL, support the construct validity of the computer-based TOEFL, and underscore its usefulness in interpreting the performances of ESL test-takers on admissions tests for English-speaking test-takers.

Language testing

04–218Barrette, Catherine (Wayne State U., USA). An analysis of foreign language achievement test drafts. Foreign Language Annals (New York, USA), 37, 1 (2004), 58–70.04–219Cho, Yeonsuk (Ballard & Tighe, California, USA; Email: ycho@ballard-tighe.com) Assessing writing: are we bound by only one method?Assessing Writing (New York, USA), 8, 3 (2003), 165–91.04–220Cumming, Alister (U. of Toronto, Canada; Email: acumming@oise.utoronto.ca). Grant, Leslie, Mulcahy-Ernt, Patricia and Powers, Donald E. A teacher-verification study of speaking and writing prototype tasks for a new TOEFL. Language Testing (London, UK), 21, 2 (2004), 107–45.04–221Pae, Tae-Il (Yeungnam U., Republic of Korea; Email: paet@gwm.sc.edu). Gender effect on reading comprehension with Korean EFL learners. System (Oxford, UK), 32 (2004), 265–81.04–222Penny, James A. (Castle Worldwide Inc., North Carolina, USA; Email: jpenny@castleworldwide.com). Reading high stakes writing samples: my life as a reader. Assessing Writing (New York, USA), 8, 3 (2003), 192–215.04–223Snellings, Patrick and Van Gelderen, Amos (U. of Amsterdam, Holland) and de Glopper, Kees. Validating a test of second language written lexical retrieval: a new measure of fluency in written language production. Language Testing (London, UK), 21, 2 (2004), 174–201.04–224Stricker, J. Lawrence (Educational Testing Service, USA). The performance of native speakers of English and ESL speakers on the computer-based TOEFL and GRE general test. Language Testing (London, UK), 21, 2 (2004), 146–73.

Evaluation of an in-depth vocabulary knowledge measure for assessing reading performance

The central purpose of this study was to empirically evaluate an in-depth vocabulary knowledge measure in the context of developing the new TOEFL test. The study was carried out with a sample of 207 international students attending an intensive English as a second language (ESL) program in a major Canadian university, in order to determine whether this vocabulary knowledge measure could provide a basis for developing appropriate and useful item types for assessing test-takers’ reading comprehension. The results indicate that, compared with existing TOEFL vocabulary measures, the new measure has a similar difficulty level and provides a similar amount of prediction of test-takers’ reading performance.

TOEFL and FCE Tests as Predictors of Academic Success for Advanced Students at the University of Bahrain

TOEFL and FCE Tests as Predictors of Academic Success for Advanced Students at the University of Bahrain

DESIGN AND EVALUATION OF A COMPUTER‐BASED TOEFL TUTORIAL

ABSTRACTIn order to train examinees whose native language is not English to take a computerized Test of English as a Foreign Language (TOEFL®), a special set of tutorials was designed and developed. The TOEFL tutorials were field tested as part of a computer familiarity study in 1996. This report describes the development of the tutorials. Also, the experiences of the 1,169 individuals who participated in the computer familiarity study are characterized in terms of timing and performance data, as well as self‐reported attitudes. These analyses took into account computer familiarity and English ability, which both proved to be important in explaining some differences in time to complete the tutorials and perception of the tutorials' usefulness. Most examinees were successful in completing the practice items in the tutorials and thought that the tutorials were helpful. Some changes were subsequently made before operational implementation of the computerized TOEFL test in order to reduce the time needed to complete the TOEFL tutorials.

The relative importance of persons, items, subtests and languages to TOEFL test variance

The purpose of this project was to explore the relative contributions to TOEFL score dependability (which is analogous to classical theory reliability) of various numbers of persons, items, subtests, languages and their various interactions. To these ends, three research questions were formulated: (1) What are the characteristics of the distributions, and how high are the classical theory reliability estimates for the whole test and its subtests? (2) For each of the 15 languages, what are the relative contributions to test variance of persons, items, subtests and their interactions? (3) Across all 15 languages, what are the relative contributions to test variance of persons, items, subtests and languages, as well as their various interactions? The study sampled 15 000 test takers, 1000 each from 15 different language backgrounds, from the total of 24 500 participants in the TOEFL generic data set which itself was a sample from the May 1991 worldwide administration of the TOEFL. The test was administered under normal operational conditions and included all three subtests: (1) Listening Comprehension, (2) Structure and Written Expression, and (3) Vocabulary and Reading Comprehension. The analyses included descriptive statistics, classical theory reliability estimates, and a series of generalizability studies conducted to isolate the variance components due to persons, items, subtests and languages, and their effects on the dependability of the test. Unlike previous research, the results here indicate that, when considered in concert with other important sources of variance (persons, items and subtests), language differences alone account for only a very small proportion of TOEFL test variance. These results should prove useful to test developers and researchers interested in the relative effects of such factors on test design.

The relative importance of persons, items, subtests and languages to TOEFL test variance

The relative importance of persons, items, subtests and languages to TOEFL test variance

Facility with the English language and problem-based learning group interaction: findings from an Arabic setting.

The Faculty of Medicine and Health Sciences (FMHS), United Arab Emirates (UAE) University is in a unique position to explore issues related to English language proficiency and medical student performance. All students entering the FMHS have English as a second language. This study focused on the issues of students' proficiency in English as measured by the TOEFL test, student background factors and interaction in problem-based learning (PBL) groups. Using a modification of Bales Interaction Process Analysis, four problem-based learning groups were observed over four thematic units, to measure the degree of student interaction within PBL groups and to compare this to individual TOEFL scores and key background variables. The students' contributions correlated highly with TOEFL test results in the giving of information (range r = 0.67-0.74). The female students adhered to interacting in English during group sessions, whereas the male students were more likely to revert to using Arabic in elaborating unclear phenomena (p < 0.01). The educational level of the student's mother was highly predictive of TOEFL scores for the male students, but not for female students. Multivariate analysis was undertaken to analyse the relative contribution of the TOEFL, parental education and years of studying in English. The best predictor of students' contributions in PBL groups was identified as TOEFL scores. The study demonstrates the importance of facilitating a locally acceptable level of English proficiency prior to admission to the FMHS. However, it also highlights the importance of not focusing only on English proficiency but paying attention to additional factors in facilitating medical students in maximizing benefits from interactions in PBL settings.

COMPUTER FAMILIARITY AMONG TOEFL EXAMINEES

ABSTRACTThe growing use of computer‐based testing raises concerns about access and equity. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 may introduce bias by confounding the measurement of English‐language proficiency with computer familiarity. As the first study in a series of research efforts to address this question, a 23‐item questionnaire focusing on examinees' access to, attitude toward, and experience using computers as well as related technologies was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration.The examinee response rate for the survey was over 90%, and the sample of respondents was determined to be highly similar to the annual TOEFL examinee population in terms of gender, native language, test center region, TOEFL test score ranges, and reason for taking the test.Factor analysis procedures (specifically, a principal factors approach with varimax rotation) were used to determine the dimensionality of the survey questionnaire. The first factor from a two‐factor common factor solution consisted of 11 items and had a reliability of .91. This factor was used to classify TOEFL examinees into one of three groups according to their level of familiarity with computers: low, moderate, or high. Overall, some 16% of the TOEFL population was judged to have low computer familiarity, another 34% to have moderate familiarity, and approximately 50% to have high familiarity. In terms of background characteristics, computer familiarity was found to be unrelated to age, but was related to gender, native language, region of the world where the examinee was born, and test‐center region. Computer familiarity was also shown to be related to individuals' TOEFL test scores and their reason for taking the test but unrelated to whether or not they had taken the test previously.Analyses of variance were conducted to investigate the relationships among proficiency as measured by the paper‐and‐pencil TOEFL test, computer familiarity, and selected background characteristics, including gender, age, native language, and reason for taking the test. Effect sizes for each variable as well as for the interactions among the variables are discussed herein. The study results are also considered in light of existing research relating to various aspects of computer familiarity as well as in terms of their implications for professionals in universities and intensive language programs who will need to serve the needs of this diverse set of learners.

THE RELATIONSHIP BETWEEN COMPUTER FAMILIARITY AND PERFORMANCE ON COMPUTER‐BASED TOEFL TEST TASKS

ABSTRACTThe increasing use of computer‐based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 will confound language proficiency with computer proficiency and thus bring construct‐irrelevant variance to the measurement of examinees' English‐language abilities.In a Phase I study (Kirsch, Jamieson, Taylor, &amp; Eignor, 1998), TOEFL examinees were surveyed regarding their computer familiarity and classified into one of three computer familiarity groups: low, moderate, and high. In this study, Phase II, more than 1,100 “low‐computer‐familiar” and “high‐computer‐familiar” examinees from 12 international sites were identified from the Phase I survey and administered a computer tutorial and a set of 60 computer‐based TOEFL test items. The relationship between level of computer familiarity and performance on the computer‐based items was then examined. The examinees in Phase II were largely representative of those in Phase I, who were representative of the general TOEFL test‐taking population. Thus, results from this phase of the study are considered generalizable to the current TOEFL examinee population.The effect of computer familiarity after adjustments for language ability was examined by performing a series of analyses of covariance (ANCOVAs), using TOEFL paper‐and‐pencil test score as the covariate. These analyses were followed by a series of ANCOVAs involving the computer familiarity variable and a number of other variables: gender, reason for taking the TOEFL test, times the TOEFL test had been taken, and location where the TOEFL test was taken. In a final set of analyses, the TOEFL paper‐and‐pencil test scores of the low‐ and high‐computer‐familiar examinees were weighted such that the groups had identical distributions on the covariate.After controlling for language ability, the researchers found no meaningful relationship between level of computer familiarity and level of performance on computerized language tasks among TOEFL examinees who had completed the computer tutorial. This finding was consistent for all but one of the subgroups considered. A small but practically significant interaction between computer familiarity and reason for taking the test was found on the set of computerized reading items. Researchers concluded that there was no evidence of adverse effects on the computer‐based TOEFL performance due to lack of prior computer experience.

DEVELOPMENT OF A SCALE FOR ASSESSING THE LEVEL OF COMPUTER FAMILIARITY OF TOEFL EXAMINEES

ABSTRACTThe increasing use of computer‐based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 may introduce bias by confounding the measurement of English‐language proficiency with level of computer familiarity. As the first step in a series of research efforts to address this question, a 23‐item questionnaire focusing on individuals' access to, attitude toward, and experience with using computers as well as related technology was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration. Results of this survey administration are documented in a paper by Kirsch, Jamieson, Taylor, and Eignor (1998).This paper has two purposes. The first is to explain decisions as to which questionnaire items would subsequently be used together as a score to separate TOEFL examinees into levels of computer familiarity. The second purpose is to provide details on the procedures used to assess the underlying factor structure of the complete questionnaire. Factor analysis procedures (specifically, a principal factors approach with both varimax orthogonal and promax oblique rotations) were used for both purposes.Eleven items, which loaded heavily on the first factor of a two‐factor common factor solution with promax oblique rotation, were selected to create a computer familiarity score. This score was then used to classify TOEFL examinees into one of three familiarity groups: low, moderate, and high. A four‐factor common factor solution with promax oblique rotation was seen as providing the best accounting of the underlying factor structure of the complete questionnaire.

SURVEY OF STANDARDS FOR FOREIGN STUDENT APPLICANTS

ABSTRACTThis project consisted of a survey of practices and standards in the use of the Test of English as a Foreign Language (TOEFL®) and attempted to learn the processes by which these standards were developed. By “standards” we mean the information and requirements established by institutions to evaluate applicants' English proficiency. Information, including other measures that were used with the TOEFL test, was obtained toward that end.Several findings emerged. The most common practice for using the TOEFL test in admissions was that students scoring above a specified score were judged adequate from the standpoint of English proficiency, but students who scored below it had to provide other evidence of English proficiency that could justify admission. Only a few institutions used a rigid cut score. The measures most commonly allowed as substitutes for the TOEFL test in admissions and placement decisions were the Scholastic Assessment Test (SAT®) (The College Board, 1995), the American College Testing Assessment (ACT™) (ACT, 1988) score, and the Michigan Test of English Language Proficiency (MTELP) (Buros, 1965). It was found that most changes in standards consisted of increases in the minimum test scores, and that they were initiated in response to problems noted in admitted foreign students' performance. In addition to performance, practices of similar institutions were commonly used to set minimum scores. Participants most often mentioned in revised standard setting were administrators (policy), ESL staff (language trainers), and faculty (teachers of academic content).This survey provides information about the standards for different institutions and how those standards have been set and maintained. Users had the opportunity to tell us about their use of the TOEFL test. The following suggestions emerged:• When score user surveys are conducted, relevant institutional characteristics could be collected and used to categorize the results. Such information would facilitate the process of adjusting qualifying scores.• Data that connects TOEFL scores with speaking and essay scores could be made available. Such data could be useful when essay or speaking measures are used to supplement processing of applicants who achieve marginal scores on TOEFL.• A system of behaviorally anchored language proficiency scales based on incidents that have stimulated changes in cut scores could be developed. Availability of such scales would facilitate cross‐institutional communication about language proficiency levels and could add a new dimension to test validation.

CONCURRENT CALIBRATION OF DICHOTOMOUSLY AND POLYTOMOUSLY SCORED TOEFL ITEMS USING IRT MODELS

ABSTRACTIn order to meet the needs of the Test of English as a Foreign Language (TOEFL®) constituencies, the TOEFL program is sponsoring a development project known as TOEFL 2000. Drawing from current linguistic theory and models of communicative competence, it is anticipated that the new test or test battery developed by the TOEFL 2000 project will likely be designed to test all four language skills — reading, writing, listening, and speaking — in an integrated fashion. However, one compromise level or position on integration of skills is one in which reading and writing would be tested together, and listening and speaking also tested together. It is also assumed that the test will largely be performance‐based, meaning a substantial portion of the items on the test will likely be constructed‐response items, and an examinee's score on such items will be in one of multiple ordered categories.Two groups of item response theory (IRT) models have been developed to calibrate items with multiple ordered categories (i.e., polytomously scored items): (a) the partial credit model (Masters, 1982) and the generalized partial credit model (Muraki, 1992); and (b) the graded response model (Samejima, 1969,1972). These models have been used jointly with the dichotomous three parameter logistic (3PL) IRT model to concurrently calibrate dichotomously and polytomously scored items for the National Assessment of Educational Progress (NAEP). However, the performance of these polytomous IRT models and the concurrent calibration of dichotomous and polytomous scored items have not been investigated with data from the TOEFL examinee population.The purpose of this study was to obtain a good understanding of the performance of a combination of dichotomous and polytomous IRT models with TOEFL data. TOEFL Vocabulary and Reading Comprehension and Test of Written English (TWE®) items, and TOEFL Listening Comprehension and Test of Spoken English (TSE®) items were concurrently calibrated using a combination of the generalized partial credit model and the 3PL IRT model. The two sets of combined items were also concurrently calibrated using a combination of the graded response model and the 3PL IRT model.The results of this study indicate that data from a reading/writing combination made up of the TOEFL Vocabulary and Reading Comprehension section and the TWE were reasonably well fit by a combination of the 3PL and generalized partial credit models or 3PL and graded response models. In a similar fashion, data for a listening/speaking combination made up of the TOEFL Listening Comprehension section and selected tasks from the TSE were also reasonably well fit by the 3PL/generalized partial credit and 3PL/graded response model combinations.A variety of comparisons across the generalized partial credit and graded response models seem to indicate some preference for using the generalized partial credit model when PARSCALE is used as the calibration program. The results of this study provide useful information about test construction and item calibration procedures that might later be used for the TOEFL 2000 project.

HOW RELIABLE IS THE TOEFL TEST?

ABSTRACTWe estimated the reliability of scores on four forms of the Test of English as a Foreign Language (TOEFL®) using a hybrid IRT model. Very little difference between their overall reliability was found when the testlet items were assumed to be independent and their dependence was modeled. A larger difference in reliability was found when test sections were analyzed individually. Then we found as much as a 40 percent overestimate in reading comprehension testlets, with the longer testlets of the newest form of the TOEFL test showing the most local dependence. The listening comprehension testlets exhibited much less local dependence. We also found that the test was unidimensional enough for the use of univariate IRT to be efficacious, and that the reading comprehension testlets showed essentially no differential functioning by sex.

The University of Michigan Press: Intermediate TOEFL test practices

The University of Michigan Press: Intermediate TOEFL test practices

AN INVESTIGATION OF PROPOSED REVISIONS TO SECTION 3 OF THE TOEFL TEST

ABSTRACTThe TOEFL testing program is currently exploring a change in Section 3 of the TOEFL® test that would replace the vocabulary subpart with additional reading comprehension questions. This change has been proposed by internal test development specialists and is supported by external experts in the field of English as a second language. The purpose of this study was to investigate the proposed revision to Section 3 in terms of the length and timing that would be necessary to address concerns of test speededness of the section. The study was carried out using an experimental design with test length and testing time defined as independent variables, and examinee test performance defined as the dependent variable. In addition, several psychometric issues relating to the proposed revision to Section 3 were investigated as part of the study.The results of the study supported the implementation of a revised TOEFL Section 3 consisting of five reading passages with a total of 50 items. The results of the study also suggested that a total testing time of no less than 55 minutes should be allowed for the revised TOEFL Section 3. Additional psychometric analyses indicated that the current TOEFL score scale can be maintained with the revised Section 3, and that the proposed revisions will not appreciably affect the reliability and validity of Section 3 of the TOEFL test.

AN EXPLORATORY STUDY OF CHARACTERISTICS RELATED TO IRT PARAMETER INVARIANCE WITH THE TEST OF ENGLISH AS A FOREIGN LANGUAGE

IRT equating methods have been used successfully with the TOEFL® test for many years, and for the most part the observed properties of items have been consistent with model predictions. However, items that do not appear to hold their IRT pretest estimates do exist. If relationships can be found between features of TOEFL items in pretest calibrations and subsequent lack of model-data fit when these items are used in final forms, steps to eliminate the use of such items in TOEFL final forms can be taken. The purpose of this study was to provide an exploratory investigation of item features that may contribute to a lack of invariance of TOEFL item parameters. The results of the study indicated the following: (1) subjective and quantitative measures developed for the study provided consistent information related to the model-data fit of TOEFL test items, (2) for Sections 1 and 2, items that were pretested before 1986 exhibited poorer model-data fit than items that were pretested after 1986, and (3) for Section 3 reading comprehension, model-data fit appeared to be related to changes in the relative position of items within the sections from the pretest to the final form administrations. Based on the results of the study, it was recommended that (1) the TOEFL program investigate the feasibility of not using pretest IRT statistics for items pretested before 1986 for Sections 1 and 2 and (2) that guidelines be developed for test developers to use with reading comprehension items to limit the change in relative positions of items in the test from pretest to final form administrations.

AN INVESTIGATION OF THE APPROPRIATENESS OF THE TOEFL TEST AS A MATCHING VARIABLE TO EQUATE TWE TOPICS

ABSTRACTThe feasibility of using linear and equipercentile equating methods (Angoff, 1984) to equate forms of the Test of Written English (TWE) by using the TOEFL test as an anchor was explored. These two equating methods assume that either the TOEFL test and TWE test measure the same skills or that the examinee groups across TWE administrations are equivalent in skills. The differences between equated and observed scores (equating residuals) and differences among the mean equated scores for examinee groups were further examined in terms of characteristics of the TWE topics.An evaluation of the assumptions underlying the equating methods suggests that the TOEFL and TWE tests do not measure the same skills and that examinee groups are often dissimilar in skills. Therefore, use of the TOEFL test as an anchor to equate TWE tests does not appear appropriate. An alternative equating model based on expert judgment during pretest evaluation of potential essay prompts is recommended for future investigation.