TOEFL iBT Research Articles

Washback: Exploring what constitutes “good” teaching practices

Realising the social consequences involved in language testing, many researchers have investigated “washback”, or the influence of testing on teaching and learning. While it is widely acknowledged that the nature of washback is dependent on context (Cheng, 2005; Tsagari, 2011), the very definition of washback is problematic due to its reliance on what constitutes “good” teaching and learning practices, which can differ from one educational context to another. This article explores the attitudes, beliefs and teaching practices of four teachers teaching both TOEFL iBT preparation courses and general English classes in Vietnam with the aim to better understand how washback (positive or negative) is realised in these contexts. In this qualitative case study, the participating teachers were observed and interviewed in both their TOEFL iBT preparation and general English class. The study revealed that teachers’ beliefs about what constitutes “good” teaching practices differed depending on the teaching context, or more specifically, the role and purpose of the course (test preparation versus general English).

Student and instructor perceptions of writing tasks and performance on TOEFL iBT versus university writing courses

Abstract This study examined student and instructor perceptions of writing tasks and performance on TOEFL iBT versus university writing courses. Participants included 103 international, nonnative-English-speaking undergraduate students enrolled in required university writing courses and their writing instructors (n = 18). Students completed a background questionnaire, two TOEFL iBT writing tasks (one Integrated and one Independent), and a questionnaire about their perceptions of the writing tasks. The 18 instructors also completed a questionnaire and sx participated in an extended interview. Students and instructors reported that neither the Independent nor the Integrated task alone was representative of the types of writing they do in their writing course. However, the Independent and Integrated task together represented many of the characteristics of course assignments. Additionally, instructors perceived the criteria in the TOEFL iBT writing rubrics to be very similar to the criteria that they use in class to assess student writing suggesting that TOEFL iBT tasks and course assignments are based on a similar operationalization of the writing construct. Finally, students and instructors generally perceived the quality of the writing produced for the TOEFL iBT writing tasks to be comparable to the quality of writing produced in course assignments.

Investigating the Predictive Validity ofTOEFL iBT®Test Scores and Their Use in Informing Policy in a United Kingdom University Setting

The project examined the predictive validity of theTOEFL iBT®test with a focus on the relationship between TOEFL iBT scores and students' subsequent academic success in postgraduate studies in one leading university in the United Kingdom, paying specific attention to the role of linguistic preparedness as perceived by students and tutors. We employed a mixed‐methods approach to enrich traditionally quantitatively oriented studies with a qualitative perspective. For the sample of 504 students who entered the university for postgraduate studies in the years 2011–2013 on the basis of a TOEFL iBT score, we analyzed the relation between TOEFL iBT scores and final academic award by correlation and regression analyses, taking into consideration discipline, nationality, and additional language support. For the qualitative strand, students entering the university in 2013 on the basis of a TOEFL iBT score were invited to complete questionnaires and interviews, as were their English for academic purposes and academic tutors. A total 48 students and 58 tutors participated, with 25 students and 36 tutors being interviewed at 3 points over the course of the year. Our findings show that students entering the university on the basis of TOEFL iBT scores feel well prepared and generally regard the test as an effective means of preparation for their academic studies in a U.K. setting. They cope well with linguistic demands, and a vast majority graduate successfully. Our findings support the appropriateness of the university's entrance policy with regard to setting minimum test score requirements, thus underpinning the predictive validity of TOEFL iBT in a U.K. setting.

Use of Automated Scoring in Spoken Language Assessments for Test Takers With Speech Impairments

This study is an evaluation of the performance of automated speech scoring for speakers with documented or suspected speech impairments. Given that the use of automated scoring of open‐ended spoken responses is relatively nascent and there is little research to date that includes test takers with disabilities, this small exploratory study focuses on one type of scoring technology, automatic speech scoring (the SpeechRaterSM automated scoring engine); one type of assessment, spontaneous spoken English by nonnative adults (six TOEFL iBT® test speaking items per test taker); and one category of disability, speech impairments. The results show discrepancies between human and SpeechRater scores for speakers with documented speech or hearing impairments who receive accommodations and for speakers whose responses were deferred to the scoring leader by human raters because the responses exhibited signs of a speech impairment. SpeechRater scores for these studied groups tended to be higher than the human scores. Based on a smaller subsample, the word error rate was higher for these groups relative to the control group, suggesting that the automatic speech recognition system contributed to the discrepancies between SpeechRater and human scores.

Game-Theoretic Question Selection for Tests

Conventionally, the questions on a test are assumed to be kept secret from test takers until the test. However, for tests that are taken on a large scale, particularly asynchronously, this is very hard to achieve. For example, TOEFL iBT and driver's license test questions are easily found online. This also appears likely to become an issue for Massive Open Online Courses (MOOCs, as offered for example by Coursera, Udacity, and edX). Specifically, the test result may not reflect the true ability of a test taker if questions are leaked beforehand. In this paper, we take the loss of confidentiality as a fact. Even so, not all hope is lost as the test taker can memorize only a limited set of questions' answers, and the tester can randomize which questions to let appear on the test. We model this as a Stackelberg game, where the tester commits to a mixed strategy and the follower responds. Informally, the goal of the tester is to best reveal the true ability of a test taker, while the test taker tries to maximize the test result (pass probability or score). We provide an exponential-size linear program formulation that computes the optimal test strategy, prove several NP-hardness results on computing optimal test strategies in general, and give efficient algorithms for special cases (scored tests and single-question tests). Experiments are also provided for those proposed algorithms to show their scalability and the increase of the tester's utility relative to that of the uniform-at-random strategy. The increase is quite significant when questions have some correlation---for example, when a test taker who can solve a harder question can always solve easier questions.

The Effects of Different Levels of Performance Feedback onTOEFL iBT®Reading Practice Test Performance

The purpose of the present study is to examine whether performance on the TOEFL iBT® Reading practice test is affected by 3 different levels of feedback provided to learners upon completion of reading exercises: (a) correctness of learner response (the knowledge of correct results [KCR] feedback), (b) KCR feedback and rationales for correct/incorrect answers, and (c) KCR feedback and the rationales along with on-demand video lectures. Japanese learners of English completed 18 reading exercises and reviewed the results by using the type of feedback provided to the group to which each learner had been assigned. They also completed 2 TOEFL iBT Reading test forms: 1 as a pretest and 1 as a posttest. A multilevel modeling analysis of data from 193 participants who completed the study showed that despite a gain of over 3 scaled score points on the TOEFL iBT Reading test from the pretest to the posttest, the level of feedback was not found to affect the test score. A learner background variable, previous knowledge about the TOEFL iBT test, was identified as a predictor positively related to the test score. Moreover, a series of ANOVAs conducted on learner survey data suggested that the level of feedback affected the degree to which study participants perceived the usefulness of the study materials in understanding different item types that appear on the TOEFL iBT Reading section and in developing strategies for understanding the gist of the text within a short time.

Do the TOEFL iBT® section scores provide value-added information to stakeholders?

The present study examined the reliability of the reading, listening, speaking, and writing section scores for the TOEFL iBT® test and their interrelationship in order to collect empirical evidence to support, respectively, the generalization inference and the explanation inference in the TOEFL iBT validity argument (Chapelle, Enright, & Jamieson, 2008). By combining Haberman’s (2008) subscore analysis and confirmatory factor analysis (CFA), data from four operational TOEFL iBT test administrations were analyzed for all examinees and three major native language (L1) groups (Arabic, Korean, and Spanish). Key results were consistent across the forms and samples. First, Haberman’s (2008) subscore analysis suggested that the reliabilities of the section scores were generally satisfactory but for the writing section the reliability was relatively low. Second, Haberman’s subscore analysis and CFA offered different degrees of support for the distinctness of the TOEFL iBT section scores. A subsequent multiple-group CFA based on a correlated four-factor model generally supported the measurement invariance across the L1 groups in terms of factor loadings as well as indicator residuals and intercepts, despite the population heterogeneity indicated by the partial invariance of the latent factor variances and differences in the latent factor means across the groups. In addition, Haberman’s subscore analysis suggested that the speaking section score offered value-added information owing to its generally high level of reliability and relative distinctness from the other three section scores, which is relevant to the utilization inference in the validity argument from a perspective of psychometric quality of the TOEFL iBT section scores.

Helping Students Select Appropriately Challenging Text: Application to a Test of Second Language Reading Ability

A model-based approach for matching language learners to texts of appropriate difficulty is described. Results are communicated to test takers via a targeted reading range expressed on the reporting scale of an automated text complexity measurement tool (ATCMT). Test takers can use this feedback to select reading materials that are well matched to their abilities, that is, materials that are expected to be challenging, yet not so challenging as to cause frustration or reduce motivation. An application to the problem of helping students prepare to take the TOEFL iBT® test is presented.

The relationships between test performance and students\u2019 perceptions of learning motivation, test value, and test anxiety in the context of the English benchmark requirement for graduation in Taiwan\u2019s universities

BackgoundHaving been influenced by the trend of internationalization of higher education, most universities in Taiwan have implemented an English benchmark requirement for graduation, which requires students to demonstrate their English ability at a specified Common European Framework of Reference for Languages (CEFR) level through taking a standardized English language test (e.g., GEPT, IELTS, TOEFL iBT). This practice has been increasingly criticized for failing to achieve its intended goals of enhancing students’ English language proficiency and increasing students’ career mobility. Therefore, there is an urgent need to explore the consequences of using standardized tests in support of the policy.MethodsTo this end, this study investigated students’ views of the graduation policy in three universities where students are required to take the General English Proficiency Test (GEPT) prior to graduation. Structural equation modeling was employed to find the best fitting model that illustrates the complex interrelationships among test performance, students’ perceptions of the requirement, test value, test anxiety, and learning motivation.ResultsThe findings show that university students, regardless of English proficiency, generally hold a positive attitude towards the English graduation benchmark policy. Results further reveal that the Intermediate group shows more positive attitudes towards the graduation requirement than the High-Intermediate group. SEM results show that the attitudes of university students towards the English graduation requirement positively impact their perceived test value and their learning motivation. However, there is no significant relationship between the attitudes towards the policy and test performance.ConclusionsThe findings contribute to our understanding of university students as the major stakeholders who defined the context of test use.

Interpreting the relationships between TOEFL iBT scores and GPA: Language proficiency, policy, and profiles

This study examines the predictive validity of the TOEFL iBT with respect to academic achievement as measured by the first-year grade point average (GPA) of Chinese students at Purdue University, a large, public, Research I institution in Indiana, USA. Correlations between GPA, TOEFL iBT total and subsection scores were examined on 1990 mainland Chinese students enrolled across three academic years (N2011 = 740, N2012 = 554, N2013 = 696). Subsequently, cluster analyses on the three cohorts’ TOEFL subsection scores were conducted to determine whether different score profiles might help explain the correlational patterns found between TOEFL subscale scores and GPA across the three student cohorts. For the 2011 and 2012 cohorts, speaking and writing subscale scores were positively correlated with GPA; however, negative correlations were observed for listening and reading. In contrast, for the 2013 cohort, the writing, reading, and total subscale scores were positively correlated with GPA, and the negative correlations disappeared. Results of cluster analyses suggest that the negative correlations in the 2011 and 2012 cohorts were associated with a distinctive Reading/Listening versus Speaking/Writing discrepant score profile of a single Chinese subgroup. In 2013, this subgroup disappeared in the incoming class because of changes made to the University’s international undergraduate admissions policy. The uneven score profile has important implications for admissions policy, the provision of English language support, and broader effects on academic achievement.

Preparing for the Speaking Tasks of the TOEFL iBT® Test: An Investigation of the Journeys of Chinese Test Takers

AbstractLanguage test preparation has often been studied within the consequential validity framework in relation to ethics, equity, fairness, and washback of assessment. The use of independent and integrated speaking tasks in the TOEFL iBT® test represents a significant development and innovation in assessing speaking ability in academic contexts. Integrated tasks that involve synthesizing and summarizing information presented in reading and listening materials have the potential to generate new test preparation strategies. This study investigated the experiences of over 1,500 Chinese test takers and 23 teachers who were preparing for the TOEFL iBT speaking tasks. It examined the frequency of use of a number of different test preparation activities and materials, reasons, and expectations for taking preparation courses and the features of preparation courses. In addition, we examined the usefulness of test preparation from two perspectives: students' and teachers' perceptions as well as the relationship between test preparation and performance. Data were collected via questionnaires, focus group discussions, interviews with test takers and teachers, and classroom observations. The data showed that (a) test preparation was a hugely complex, multiple‐components construct, and teaching and learning test‐taking strategies compose the most prominent feature of intensive preparation courses; (b) there were significant age‐related differences in students' preparation activities and focuses, although with small effect sizes; (c) there was a high agreement between teachers and students in their views on the usefulness of test preparation activities; and (d) there existed only a weak relationship between test preparation and performance. The only significant predictor of students' test performance was the frequency of their use of the TOEFL Practice Online TPO® practice tests. The findings of the study can enhance our understanding of the pedagogical practices that characterize test preparation programs and contribute to the ongoing validity argument for the TOEFL iBT Speaking test. The implications of the findings for test publishers, test takers, teachers, and test preparation schools are discussed with reference to the instructional, learning, and affective aspects of the multifaceted construct of test preparation.

The TOEFL iBT writing: Korean students’ perceptions of the TOEFL iBT writing test

Abstract The TOEFL is one of the most widely recognized language proficiency tests developed to measure international students’ level of readiness for degree study. Whereas there exist a number of correlational studies conducted by various affiliates of ETS based on large-scale quantitative data, there is a dearth of studies that explore test-takers’ perceptions and experiences concerning the TOEFL iBT. Writing skills have paramount importance for academic success, and high-stakes tests such as the TOEFL have a tendency to influence test-takers’ perceptions on what defines good academic writing. To date, no research has specifically focused on test-takers’ perceptions on the writing section of the TOEFL iBT. To fill this gap, this study explores Korean students’ perceptions of effective strategies for preparing for the TOEFL iBT writing test, challenges they face in the test-taking and test-preparation processes, and implications such findings have for various stakeholders, by analyzing online forum data. Findings indicate that the scores for the writing section of the TOEFL iBT, albeit helpful for the initial benchmarking tool, may conceal more than it reveals about Korean students’ academic writing ability. The study suggests that the format, questions, and the scoring of the TOEFL iBT writing test be critically examined from test-takers’ perspectives.

The Effect of Computer Literacy on the Participants’ Writing Ability in TOEFL iBT

One major concern in language testing relates to computer based and Internet based tests of second language. Performance on these tests is influenced by many different factors other than language ability. One of these factors relates to the participants’ speed to use the Internet. This study investigated the effect of computer literacy on TOEFL-iBT scores by comparing a paper-based and internet based test. The comparison of groups’ performance on both TOEFL-iBT and pBT revealed that computer literacy significantly affects writing performance on the exam. Writing skill of candidates with higher computer literacy was significantly better than those with lower computer literacy. Both groups’ performance was equally the same when they took the paper based type of test. Participants who lack proficiency in computer skills often might use most of their mental power and cognitive resources on working with the computer rather than focusing on what they are actually being tested on. This would lead to inability to perform fully on the language test and consequently to lower scores.

Is Writing Performance Related to Keyboard Type? An Investigation From Examinees’ Perspectives on the TOEFL iBT

ABSTRACTTo investigate whether the type of keyboard used in exams introduces any construct-irrelevant variance to the TOEFL iBT Writing scores, we surveyed 17,040 TOEFL iBT examinees from 24 countries on their keyboard-related perceptions and preferences and analyzed the survey responses together with their test scores. Results suggest that controlling for their TOELF iBT Reading, Listening, and Speaking scores, examinees’ Writing scores were statistically significantly associated with examinees’ prior awareness of the TOEFL iBT keyboard policy, the type of keyboard they used during the test, or examines’ perceptions of the type of keyboard used. However, the score differences associated with these factors were negligible, supporting the argument that the current practice of mandatorily using the U.S. standard keyboard poses limited threat to the validity of test scores. Survey results also revealed that although most examinees considered the keyboard used during the test convenient and efficient, two-fifths of them reportedly preferred a more familiar local keyboard, which suggests room for improvement in the keyboard-related test-taking experience.

TOEFL iBT Speaking, Goal – orientation and self-assessment a study of Iranian EFL learners

Owing to the growing attention to learner's independence and autonomy, self-assessment has received considerable attention in recent years. Nevertheless, this notion is not well-known to most English language teachers and students in Iran where traditional assessment is still dominant. This study explored the effect of self-assessment on EFL learners’ TOEFL iBT speaking score, goal orientation as well as scrutinizing the relationship between Iranian EFL learners' goal-orientation and their speaking scores in TOEFL iBT test. To do so, 72 advanced EFL learners who were both male and female studying in Jahan Elm institute of higher education in Mashhad, Iran participated in the current study. Experimental group consisted of 36 EFL learners while control group encompassed 36 EFL learners. All participants took TOEFL iBT test, goal orientation questionnaire during the first and final week of the semester (i.e. pretest, posttest) and being trained through the TOEFL iBT software. Independent sample t test and Pearson correlation coefficient were used to find the difference between the two groups in terms of their speaking scores and goal orientated behavior and to find the relationship between goal orientation and speaking score of TOEFL iBT test. Results from the two post measurements indicated that students in the experimental group showed statistically significant gains on speaking scores and their goal orientated behavior as compared with the control group. Moreover, outcomes illustrated the existence of significant positive correlation between the learning goal orientation and TOEFL iBT speaking scores of EFL learners. In sum, the results of this study suggest that self-assessment should be integrated into regular TOEFL iBT preparation courses in order to help TOEFL iBT candidates become independent learners and ultimately achieve more satisfactory results in English proficiency tests, particularly TOEFL iBT test.

Can a Two‐Question Test Be Reliable and Valid for Predicting Academic Outcomes?

Scores on essay‐based assessments that are part of standardized admissions tests are typically given relatively little weight in admissions decisions compared to the weight given to scores from multiple‐choice assessments. Evidence is presented to suggest that more weight should be given to these assessments. The reliability of the writing scores from two of the large volume admissions tests, the GRE General Test (GRE) and the Test of English as a Foreign Language Internet‐based test (TOEFL iBT), based on retesting with a parallel form, is comparable to the reliability of the multiple‐choice Verbal or Reading scores from those tests. Furthermore, and even more important, the writing scores from both tests are as effective as the multiple‐choice scores in predicting academic success and could contribute to fairer admissions decisions.

Washback and CLT

Realizing the social consequences involved in language testing, many researchers have attempted to investigate ‘washback’, or the influence of testing on teaching and learning. While it is widely acknowledged that the nature of washback is dependent on context (Burrows, 2004; Shih, 2010; Shohamy, Donitsa-Schmidt, & Ferman, 1996), the very definition of washback is problematic due to its reliance on what constitutes ‘good’ teaching and learning practices which can differ from one educational context to another. To determine whether a test has had a positive washback effect on teaching and learning, one must identify the characteristics that constitute positive washback, which is likely to espouse a preferred teaching paradigm. This article argues that the majority of washback studies to date suggest that positive and negative washback are defined by the presence or absence of Communicative Language Teaching (CLT). The findings from an empirical investigation into the washback of the Test of English as a Foreign Language Internet-Based Test (TOEFL iBT) in a Vietnamese context raises the need to rethink how washback is defined.

Topical knowledge in L2 speaking assessment: Comparing independent and integrated speaking test tasks

Integrated speaking test tasks (integrated tasks) provide reading and/or listening input to serve as the basis for test-takers to formulate their oral responses. This study examined the influence of topical knowledge on integrated speaking test performance and compared independent speaking test performance and integrated speaking test performance in terms of how each was related to topical knowledge. The researchers derived four integrated tasks from TOEFL iBT preparation materials, developed four independent speaking test tasks (independent tasks), and validated four topical knowledge tests (TKTs) on a group of 421 EFL learners. For the main study, they invited another 352 students to respond to the TKTs and to perform two independent tasks and two integrated tasks. Half of the test takers took the independent tasks and integrated tasks on one topic combination while the other half took tasks on another topic combination. Data analysis, drawing on a series of path analyses, led to two major findings. First, topical knowledge significantly impacted integrated speaking test performance in both topic combinations. Second, the impact of topical knowledge on the two types of speaking test performances was topic dependent. Implications are proposed in light of these findings.

The effects of Iranian EFL learners' locus of control (external vs. internal) on their test anxiety and test scores

The present study aimed at investigating the effect of Iranian EFL learners' locus of control (external vs. internal) on their test anxiety and test scores. To this end, 130 freshman students of English translation from Sobhe Sadegh Institute of Higher Education were selected. Three samples of TOFEL reading comprehension texts, selected randomly from TOEFL iBT, were run to homogenize participants in terms of their English reading comprehension proficiency. Then the selected participants (those in intermediate level who are 100 in number) were provided with Trice Academic Locus of Control Scale (1985) for determining their internal or external orientation of locus of control. Also, Cognitive Test Anxiety Scale (CTAS) (Cassady & Jonson, 2002) was administered for measuring their level of text anxiety. In the last phase of data collection, three other samples of TOEFL reading comprehension texts were run for collecting the participants test scores. This test was considered as their final achievement test in order to be a real test. The results revealed that external locus of control could negatively influence participants' test anxiety and test scores. On the other hand, the participants with internal orientation of locus of control outperformed the participants with external orientation of locus of control.

Voice in timed L2 argumentative essay writing

The concept of voice is included in various writing textbooks, learning standards, and assessment rubrics, indicating the importance of this element in writing instruction and assessment at both secondary and postsecondary levels. Researchers in second language (L2) writing, however, often debate the importance of voice in L2 writing. Due to the elusiveness of this concept, much of such debate is still at the theoretical level; few empirical studies exist that provide solid evidence to either support or refute the proposition that voice is an important concept to teach in L2 writing classrooms. To fill this gap, the present study empirically investigated the relationship between voice salience, as captured by an analytic rubric, and official TOEFL iBT argumentative essay scores in 200 timed L2 essays. Results showed that voice was a significant predictor of TOEFL essay scores, explaining about 25% of the score variances. Moreover, while each individual voice dimension was found to be strongly or moderately correlated with essay scores when examined in isolation, only the ideational dimension became a significant predictor of text quality, when the effect of other dimensions was controlled for. Implications of such results for L2 writing instruction are discussed.