Michigan English Language Assessment Battery Research Articles

Reflective Reciprocal Teaching: A Technique for Improving Iranian EFL Learners’ Reading Comprehension Ability

The primary aim of this study was to shed light on the impact of a new instruction model, reflective reciprocal teaching (RRT), on English as a foreign language learners' (EFL) reading comprehension ability. Its mode of inquiry was a mixed-method, and it took on a quasi-experimental design, including a pretest, treatment, and posttest paradigm. The sampling techniques were both convenience and random sampling by which 100 EFL freshman learners were selected and assigned into three groups of reflective reciprocal teaching (RRT), reciprocal teaching (RT), and control. Two tests, namely Oxford Quick Placement Test and Michigan English Language Assessment Battery, were used to measure EFL learners' proficiency level and reading comprehension ability in its quantitative phase. ANOVA was utilized to analyze the collected data. Also, during the next step, which aimed to explore the learners' perceptions of RRT instruction, semi-structured interviews were used to collect qualitative data. The results indicated that the RRT group outperformed the control group regarding their reading comprehension ability; however, the RT instruction did not significantly impact this issue. Accordingly, the qualitative data analysis findings indicated that self-regulated learning, perceived competence, metacognitive awareness, confidence, and intrinsic motivation were the significant results of the RRT instruction model, contributing to the learners' reading comprehension ability.

The Impact of Genre-Based Instruction on Iranian Intermediate EFL Learners’ Writing Skills

The teaching of writing skills includes different methods and approaches. English language learners should be equipped with genre-based pedagogy to be able to write for specific settings. However, the idea of using Genre-based instruction of writing in non-native English-speaking countries has empirically received inadequate attention. This study aimed to investigate the impact of GBI on writing skills. Genres are Argumentative, Discussion, Explanation, and Exposition. This study used a quasi-experimental design. Fifty-one male students majoring in aviation courses in civil aviation technology of Tehran with an intermediate level of proficiency were assigned to experimental and control groups. The author examined their proficiency by Michigan English language assessment Battery (MELAB test) and tested their writing skills by administering a writing test of TOEFL iBT 2008. The control group took part in the traditional teaching of writing, and the experimental group was taught explicitly through the GBI method based on a lesson plan proposal. Four pre-and post-tests were taken by two groups to check the writing GBI progress. Independent sample t-tests revealed that the writing GBI could significantly improve the coherence, cohesion, and organization of writing. This study can benefit EFL practitioners by shedding light on how we can implement GBI of writing in an authentic classroom situation.

Predicting Second Language Writing Proficiency in Learner Texts Using Computational Tools

This study explores whether linguistic features can predict second language writing proficiency in the Michigan English Language Assessment Battery (MELAB) writing tasks. Advanced computational tools were used to automatically assess linguistic features related to lexical sophistication, syntactic complexity, cohesion, and text structure of writing samples graded by expert raters. The findings of this study show that an analysis of linguistic features can be used to significantly predict human judgments of the essays for the MELAB writing tasks. Furthermore, the findings indicate the relative contribution of a range of linguistic features in MELAB essays to overall second language (L2) writing proficiency scores. For instance, linguistic features associated with text length and lexical sophistication were found to be more predictive of writing quality in MELAB than those associated with cohesion and syntactic complexity. This study has important implications for defining writing proficiency at different levels of achievement in L2 academic writing as well as improving the current MELAB rating scale and rater training practices. Directions for future research are also discussed.

Developing a reading comprehension test for cognitive diagnostic assessment: A RUM analysis

A critical issue in cognitive diagnostic assessment (CDA) lies in the dearth of research in developing diagnostic tests for cognitive diagnostic purposes. Most research thus far has been mainly carried out on large-scale tests, e.g., Test of English as a Foreign Language (TOEFL), Michigan English Language Assessment Battery (MELAB), International English Language Testing System (IELTS), etc. In particular, CDA of formative language assessment that aims to inform instruction and to discover strengths and weaknesses of students to guide instruction has not been conducted in a foreign (i.e., second) language-learning context. This study explored how developing a reading comprehension test based on a cognitive framework could be used for such diagnostic purposes. To achieve this, initially, a list of 9 reading attributes was prepared by experts based on the literature, and then the targeted attributes were used to construct a 20-item reading comprehension test. Second, a tentative “Q-matrix” that specified the relationships between test items and the target attributes required by each item was developed. Third, the test was administered to seven language-testing experts who were asked to identify which of the 9 attributes were required by each item of the test. Fourth, on the basis of the overall agreement of the experts’ judgments concerning the choices of attributes, review of the literature and results of student think-aloud protocols, the tentative Q-Matrix was refined and used for statistical analyses. Finally, the test was administered to 1986 students of a General English Language Course at the University of Tehran, Iran. To examine the CDA of the test, the Reparameterized Unified Model (RUM) (also known as the Fusion Model), a type of cognitive diagnostic measurement model (CDM), was used for further refining the Q-Matrix for future data analyses and, most importantly, for diagnosing the participants' strengths and weaknesses. Data analysis results confirmed that the nine proposed reading attributes are involved in the reading comprehension test items. Such diagnostic information could be helpful for teachers and practitioners to prepare instructional materials that target specific weaknesses and inform them of the more problematic areas that need to be emphasized in class in order to plan for better L2 reading instruction. Further, such information could inform individualized student instruction and produce improved diagnostic tests for future use.

Using corpus linguistics to examine the extrapolation inference in the validity argument for a high-stakes speaking assessment

Investigations of the validity of a number of high-stakes language assessments are conducted using an argument-based approach, which requires evidence for inferences that are critical to score interpretation (Chapelle, Enright, & Jamieson, 2008b; Kane, 2013). The current study investigates the extrapolation inference for a high-stakes test of spoken English, the Michigan English Language Assessment Battery (MELAB) speaking task. This inference requires evidence that supports the inferential step from observations of what test takers can do on an assessment to what they can do in the target domain (Chapelle et al., 2008b; Kane, 2013). Typically, the extrapolation inference has been supported by evidence from a criterion measure of language ability. This study proposes an additional empirical method, namely corpus-based register analysis (Biber & Conrad, 2009), which provides a quantitative framework for examining the linguistic relationship between performance assessments and the domains to which their scores are extrapolated. This approach extends Bachman and Palmer’s (2010) focus on the target language use (TLU) domain analysis in their study of assessment use arguments by providing a quantitative approach for the study of language. We first explain the connections between corpus-based register analysis and TLU analysis. Second, an investigation of the MELAB speaking task compares the language of test-taker responses to the language of academic, professional, and conversational spoken registers, or TLU domains. Additionally, the language features at different performance levels within the MELAB speaking task are investigated to determine the relationship between test takers’ scores and their language use in the task. Following previous studies using corpus-based register analysis, we conduct a multi-dimensional (MD) analysis for our investigation. The comparison of the language features from the MELAB with the language of TLU domains revealed that support for the extrapolation inference varies across dimensions of language use.

Comparing Language Use in Oral Proficiency Interviews to Target Domains: Conversational, Academic, and Professional Discourse

Oral Proficiency Interviews (OPIs) are widely used to measure speaking ability in a second or foreign language. The Michigan English Language Assessment Battery (MELAB) Speaking Test is an OPI used for academic and professional purposes around the world. However, little research on this or other OPIs has quantitatively compared test takers’ speech with the target domains of the test. Such a comparison could be used as evidence for the validity argument (Kane, ) for the MELAB. In this study we use corpus‐based register analysis and Multi‐Dimensional (MD) analysis, investigating a large number of linguistic features to determine the extent to which the language of the MELAB is similar to conversational, academic, and professional spoken discourse, specifically nurse–patient interactions, since many of the test takers are preparing for nursing licensure. The results show that the MELAB has similarities with conversation in its use of stance, and is closely aligned with academic registers and nurse–patient interactions in the use of language for informational exchange, which provides support for the validity argument of the MELAB. However, the use of narrative features and discussion of future possibilities and suggestions are important aspects of both conversation and academic and professional registers but may be harder to evaluate through the MELAB and other similar OPIs.

Cohesive Devices in Written Discourse: A Discourse Analysis of a Student’s Essay Writing

<h1>Discourse is essential in communicating thoughts and ideas. People around the world communicate their ideas through stretches of language. In order to understand any discourse, it must achieve cohesion. The purpose of this paper is to define and describe the cohesive devices based on the work of Halliday and Hasan (1976). It also aims to emphasize the necessity of using these devices by analyzing a Michigan English Language Assessment Battery (MELAB) sample examination of a student’s essay writing. The student’s writing shows clear evidence of cohesion and demonstrates the use of grammatical and lexical devices. It is noticed that the most grammatical devices used are reference and conjunction. On the contrary, there is little evidence of using lexical devices. Although the essay is somewhat cohesive, some areas still need improvement.</h1>

The selection of cognitive diagnostic models for a reading comprehension test

Cognitive diagnostic models (CDMs) have great promise for providing diagnostic information to aid learning and instruction, and a large number of CDMs have been proposed. However, the assumptions and performances of different CDMs and their applications in regard to reading comprehension tests are not fully understood. In the present study, we compared the performance of a saturated model (G-DINA), two compensatory models (DINO, ACDM), and two non-compensatory models (DINA, RRUM) with the Michigan English Language Assessment Battery (MELAB) reading test. Compared to the saturated G-DINA model, the ACDM showed comparable model fit and similar skill classification results. The RRUM was slightly worse than the ACDM and G-DINA in terms of model fit and classification results, whereas the more restrictive DINA and DINO performed much worse than the other three models. The findings of this study highlighted the process and considerations pertinent to model selection in applications of CDMs with reading tests.

How Do Chinese ESL Learners Recognize English Words During a Reading Test? A Comparison With Romance-Language-Speaking ESL Learners

This study examines how Chinese ESL learners recognize English words while responding to a multiple-choice reading test as compared to Romance-language-speaking ESL learners. Four adult Chinese ESL learners and three adult Romance-language-speaking ESL learners participated in a think-aloud study with the Michigan English Language Assessment Battery (MELAB) reading test. As indicated by the think-aloud verbal reports, the Chinese ESL learners generally had more difficulty with English vocabulary, probably due to the vast difference between the writing system of Chinese and that of English. Rather, they were found to compensate for their deficiencies in vocabulary knowledge by extensively relying on test-taking strategies. The findings of this study are well supported by the cross-linguistic transfer theory and the compensatory nature of reading comprehension. The implications for teaching English vocabulary skills to Chinese ESL learners are also discussed.

Examining the Notion of Listening Subskill Divisibility and its Implications for Second Language Listening

The testing and teaching of listening has been partially guided by the notion of subskills, or a set of listening abilities that are needed for achieving successful comprehension and utilization of the information from listening texts. Although this notion came about mainly through applications of theoretical perspectives from psychology and communication studies, the actual divisibility of the subskills has rarely been examined. This article reports an attempt to do so by using data from the answers of 916 test takers of a retired version of the Michigan English Language Assessment Battery listening test. First, an iterative content analysis of items was carried out, identifying five key subskills. Next, the discriminability of subskills was examined through confirmatory factor analysis (CFA). Five independent measurement models representing the subskills were evaluated. The overall CFA model comprising the measurement models showed excessively high correlations among factors. Further tests through CFA resolved the inadmissible correlations, though the high correlations persisted. Finally, we made 23 aggregate-level items which were used in a higher-order model, which induced best fit indices and resolved the inadmissible estimates. The results show that the subskills in the test were empirically divisible, lending support to scholarly attempts in discussing components in the listening construct for the purpose of teaching and assessment.

Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analyses of a Reading Test

Cognitive diagnostic analyses have been advocated as methods that allow an assessment to function as a formative assessment to inform instruction. To use this approach, it is necessary to first identify the skills required for each item in the test, known as a Q-matrix. However, because the construct being tested and the underlying cognitive processes associated with it are usually not fully understood, establishing a Q-matrix, especially for an existing test, is a challenging task. This study reports the process of constructing and validating a Q-matrix for the reading comprehension section of the Michigan English Language Assessment Battery (MELAB). An initial Q-matrix was first generated based on evidence gathered from related literature, students' think-aloud protocols, and expert ratings. This initial Q-matrix was then validated empirically by applying the fusion model to a large MELAB data set. A well-supported Q-matrix was produced for potential future diagnostic applications.

Detecting native language group differences at the subskills level of reading: A differential skill functioning approach

Differential skill functioning (DSF) exists when examinees from different groups have different probabilities of successful performance in a certain subskill underlying the measured construct, given that they have the same ability on the overall construct. Using a DSF approach, this study examined the differences between two native language groups – a group with an East Asian language background and one with a Romance language background – in regard to reading subskills as represented in the Michigan English Language Assessment Battery (MELAB) reading test. Based on a combination of literature review and think-aloud reports from a sample of ESL students, hypotheses on reading subskill differences between the two groups were generated. These hypotheses were tested by first identifying the subskill profile of each examinee in a large MELAB database via the application of a previously determined item-skill Q-matrix to a Fusion Model of cognitive diagnostic modeling. The subskill profiles of the East Asian examinees were then compared against those of examinees with a Romance language background through logistic regression techniques. Some important DSFs were found between the two groups. Based on results of this study, instructional strategies were suggested to address some specific weaknesses in ESL learners’ reading subskills.

An Investigation of Differential Item Functioning in the MELAB Listening Test

Differential item functioning (DIF) analysis is a way of determining whether test items function differently across subgroups of test takers after controlling for ability level. DIF results are used to evaluate tests' validity arguments. This study uses Rasch measurement to examine the Michigan English Language Assessment Battery listening test for DIF across gender subgroups. After establishing the unidimensionality and local independence of the data, the authors used two methods to test for DIF: (a) a t-test uniform DIF analysis, which showed that two test items displayed substantive DIF, and favored different gender subgroups; and (b) nonuniform DIF analysis, which revealed several test items with significant DIF, many of which favored low-ability male test takers. A possible explanation for gender-ability DIF is that lower ability male test takers are more likely to attempt lucky guesses, particularly on multiple-choice items with unattractive distracters, and that having only two distracters makes this strategy likely to succeed.

Relationship between Iranian EFL Learners’ Beliefs about Language Learning, their Language Learning Strategy Use and their Language Proficiency

This study aimed at collecting data from a group of 203 Iranian undergraduate learners of English concerning their beliefs about language learning through Horwitz's (1987) “Beliefs About Language Learning Inventory”, concerning their language learning strategy use through Oxford's (1990) “Strategy Inventory for Language Learning” and concerning their general language proficiency through “Michigan English Language Assessment Battery”. Some open-ended questions were added to the both questionnaires. Data analyses indicated that EFL learners with more positive and reasonable beliefs, generally, use the strategies more and also have higher level language proficiency. Besides, some findings were reported as the result of doing factor analyses on the both questionnaires.

Use of tree-based regression in the analyses of L2 reading test items

The purpose of this study was to explore whether the results of Tree Based Regression (TBR) analyses, informed by a validated cognitive model, would enhance the interpretation of item difficulties in terms of the cognitive processes involved in answering the reading items included in two forms of the Michigan English Language Assessment Battery (MELAB). A cognitive model was first generated to explain the performance of the MELAB reading items, and then validated by expert judgment and student verbal protocols. Next, the validated model was used in the TBR analyses to obtain the final trees for each form. Finally, the cognitive processes (i.e., reading processes and testing strategies) measured by each item were traced back for each item in the terminal nodes of each tree. The results revealed that TBR, informed by a supportable cognitive theory, appears to be a promising addition to statistical item analysis that can be effectively used to enhance the interpretation of item analyses results.

The influence of rater language background on writing performance assessment

Language performance assessments typically require human raters, introducing possible error. In international examinations of English proficiency, rater language background is an especially salient factor that needs to be considered. The existence of rater language background-related bias in writing performance assessment is the object of this study. Data for this study are ratings assigned by Michigan English Language Assessment Battery (MELAB) raters to compositions written by examinees of various language backgrounds. While most of the raters are native speakers of English, four have first languages other than English: two Spanish, one Korean, and one bilingual speaker of Filipino and Chinese (Amoy). Examinees were divided into 21 language groups. The IRT application FACETS was used to estimate and control for rater severity when calculating the amount of bias reflected by each rater’s set of ratings for each language/language group. Results show that the magnitude of bias terms for all raters for all language groups was minimal, thus having little effect on examinee scores, and that there is no pattern of language-related bias in the ratings.

To change or not to change: investigating the value of MCQ answer changing for Gulf Arab students

This study investigates whether the practice of answer changing on multiple-choice questions (MCQs) is beneficial to Gulf Arab students’ overall test performance. The proficiency exam used in this study is the Michigan English Language Institute College English Test - Grammar, Cloze, Vocabulary, Reading (MELICET-GCVR), which was developed using retired forms of the Michigan English Language Assessment Battery (MELAB). This exam was administered to 286 students at Kuwait University and Dubai Men’s College, Higher Colleges of Technology. Findings suggest that we should encourage students to change answers judiciously after they have scrutinized their original answers for more plausible alternatives. Most important, students should not be influenced by traditional perceptions that incorrectly malign answer changing. The effect of item difficulty as well as its impact on answer changing is also reported. Finally, information on the efficacy of one specific objective test strategy, that of changing answers, is disseminated.

The Effects of Modifying the Formality Level of ESL Composition Questions

I We know of no investigations of the effect of topic on compositions written by ESL populations under closed-book, essay-examination conditions. In L1 contexts, several studies have been concerned with varying the amount and kind of information in the topic, or stimulus (e.g., Smith et al., 1985). In the most directly relevant of these studies, Brossell and Hoetker Ash (1984) investigated the effects of personal (you) as opposed to neutral phrasing and of a question versus an imperative format. They found no significant results and concluded that their study produced no evidence to support the contention that small changes in the wording of essay examinations of otherwise similar construction affect writers or the holistic ratings given their essays (p. 425). The study reported here examined the extent to which the short and general topics used in the Composition section of the Michigan English Language Assessment Battery (MELAB) (English Language Institute, 1984) generate rhetorically and linguistically unambitious answers that fail to indicate fully the communicative abilities of candidates. Since this investigation was part of a series of studies aimed at developing a new test specifically designed for the evaluation of nonnative-speaking (NNS) graduate and transfer students, it also examined the extent to which more ambitious and more academically appropriate writing could be elicited by simply increasing the level of formality of the topic. An example of simple and academic topic variants is given below.