Common European Framework Of Reference Levels Research Articles

Automatic profiling of L2-simplified texts: Identifying discriminate features of linguistic proficiency

Abstract We describe the first wide results of the linguistic profiling of the Common European Framework of Reference (CEFR)-levelled English Corpus (CLEC), a corpus built up for Natural Language Processing purposes. The CLEC is a proficiency-levelled English corpus that covers A1, A2, B1, B2, and C1 CEFR levels and that has been built up to train statistic models for automatic proficiency assessment. We describe not only the main aspects of the corpus development but also display the linguistic features and the statistic results for levels A2, B1, and B2 written examples, carried out automatically. We show how raw text, lexical, morphosyntactic, or syntactic statistic outcomes can help to identify levels of proficiency, to test teaching materials accurate proficiency classification, to provide computable support to new text proficiency validation, and to specify level boundaries. In fact, upper levels strengthen proficiency by showing higher outcomes of lexical and syntactic complexity. This analysis validates the use of automatic tools for proficiency level identification based on lexical and syntactic data, whereas morphosyntactic features strengthen competence-level distinctions. Finally, we suggest that these results are a first step onto the CEFR-levelled automatic assessment of new texts.

The CEFR Rating Scale Functioning: An Empirical Study on Self- and Peer Assessments

One of the criticisms on the Common European Framework of Reference (CEFR) rating scales pertains to the lack of reference to the performance of learners in the construction process of the scales. Therefore, this study attempted to delve into rating scale functioning by English as a Second Language (ESL) learners during self-assessment and peer assessment of their oral proficiency practice. Two objectives guided the study: 1) to gauge the overall rating scale functioning and 2) to measure each criterion scaling structure. Three self- and peer assessments’ cycles were conducted in three months. In each cycle, eleven learners recorded their own speech, uploaded their video clips to a private YouTube channel and assessed their own videos as well as selected peers based on five CEFR oral assessment criteria with six levels of ratings (A1-C2). Findings revealed that four of the CEFR levels were utilised (B1-C2). Categories A1 and A2 (basic user level) however, were not observed during the practice. Analysis from the Many-Facet Rasch Measurement (MFRM) indicated that utilised categories seemed to function usefully since each category observed was advancing by more than 1.4 logits. Category B2 dominated four criteria of ratings awarded while B1 dominated the rating distribution for fluency. The implications of this study will be discussed in relation to rating scale development, specifically on matching learners’ proficiency to the psychometrically developed rating criteria as well as illustrating assessment as learning approach in the ESL classroom where learners become the key assessors for their own performance.

Empirical Learner Language and the Levels of the Common European Framework of Reference

The Common European Framework of Reference (CEFR) is the most widespread reference tool for linking language tests, curricula, and national educational standards to levels of foreign language proficiency in Europe. In spite of this, little is known about how the CEFR levels (A1–C2) relate to empirical learner language(s). This article sums up recent trends to meet the need of empirical CEFR level research, where learner corpus‐based analyses play an increasing role. A first focus of the article is on studies that aim at illustrating CEFR levels by analyzing rated learner texts (“criterial features”). Furthermore, research that tries to disentangle the empirical validity of the CEFR scales by operationalizing its descriptors is presented. Before concluding with an outline of the most urgent research needs, potentials, and boundaries of interdisciplinary work between the fields of language testing and assessment, second language acquisition, and learner corpus research are discussed.

“We All Make Mistakes!”. Analysing an Error-coded Corpus of Spanish University Students’ Written English

The present study analyses the errors identified in the written argumentative texts of 304 Spanish university students of English taken from two different corpora –one from a technical university context and the other from learners enrolled in the Humanities. Considered an important design criterion for computer learner corpora studies, the metadata of the students’ was recorded and their competence levels were measured using the Oxford Quick Placement Test. The scores obtained (0 to 60) were then related to the CEFR (Common European Framework of Reference for Languages) levels ranging from A1 to C2. Within the field of applied linguistics and language teaching/learning, many studies have been carried out over the years designed to address the phenomenon of interlanguage errors made by learners of English (Dusková 1969, Green & Hecht 1985, Lennon 1991, Olsen 1999, among many others). These studies involved analyzing a small number of texts with a limited number of tags, based on either linguistic taxonomies or surface structure categories of errors (Dulay, Burt, & Krashen 1982). However, in the last three decades, technological advances have been made which have facilitated the analysis of much larger amounts of data using computers for both the development of learner corpora and programs for a more detailed analysis of the learner data. The error coding system used in the present research work has been designed to address all the possible levels of error (with as many sub-categories as required) since learners writing in a foreign language not only make errors related to grammar and vocabulary, but also with regard to their competence in the use of syntax, discourse relations and pragmatics, among others. The aim of the present study is two-fold. Firstly, we explore the nature of the errors coded in the corpus i.e. which errors are most frequent, including not only the main categories but also the most delicate levels of errors. Secondly, we address the question of the relationship, if any, of the learners’ competence levels and the type and frequency of the errors they make. The results show that grammar errors are the most frequent, and that the linguistic competence of the learners has a lower than expected influence on the most frequent types of errors coded in the corpus.

Word derivational knowledge and writing proficiency: How do they link?

Abstract Although word derivational (WD) knowledge, i.e., how new words are formed from existing words with help of derivational affixes, is considered important for learners of second or foreign languages (L2), there is still no clear answer as to what aspects comprise the construct of L2 English word derivational knowledge and how it develops. The present study adds to our knowledge on how the ability to derive English words develops among L2 English learners. More specifically, it sheds light on how word derivational knowledge relates to communicatively defined Common European Framework of Reference (CEFR) language proficiency levels regarding learners' writing skills. In the study, 117 10th grade learners of English in Estonia and Finland were administered two writing tasks as well as nine measures which were hypothesised to tap learners' word derivational knowledge. The findings indicated that the learners' performance on almost all WD measures were significantly and fairly strongly (at .4–0.6 level) correlated with their writing proficiency. The findings also suggest that some aspects of WD ability develop rather steadily between CEFR levels, but others may increase more rapidly after level A2 or B1. These findings thus demonstrate a relationship between word derivational knowledge and language proficiency.

Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels

This study contributes to ongoing discussions on how measures of lexical diversity (LD) can help discriminate between essays from second language learners of English, whose work has been assessed as belonging to levels B1 to C2 of the Common European Framework of Reference (CEFR). The focus is, in particular, on how different operationalizations of what constitutes a ‘different word’ (type) impact on the LD measures themselves and on their ability to discriminate between CEFR levels. The results show that basic measures of LD, such as the number of different words, the TTR (Templin 1957), and the Index of Guiraud (Guiraud 1954) explain more variance in the CEFR levels than sophisticated measures, such as D (Malvern et al. 2004), HD-D (McCarthy and Jarvis 2007), and MTLD (McCarthy 2005) provided text length is kept constant across texts. A simple count of different words (defined as lemma’s and not as word families) was the best predictor of CEFR levels and explained 22 per cent of the variance in overall scores on the Pearson Test of English Academic in essays written by 176 test takers.

What Are We Aligning Tests to When We Report Test Alignment to the CEFR?

The study reported here investigates the validity of judgments made when aligning tests to the Common European Framework of Reference (CEFR). Listening tests operationalizing pre-defined difficulty-determining characteristics were to be aligned to CEFR levels. We employed a modified version of the item-descriptor-matching-method. Ten judges stated the CEFR descriptors they thought each item operationalizes and assigned a global CEFR level per item. We compared agreement on CEFR level judgments and CEFR descriptors quoted. Analyzing the relationship between CEFR level judgments and item ratings of difficulty-determining characteristics shed light on further criteria the judges employed. Follow-up interviews helped to triangulate the findings by examining judges’ perceptions of the alignment procedure.We found that judges relied on different criteria and CEFR descriptors to a varying degree, interpreting CEFR levels differently. There seemed little comparability in what aspects judges used to form their global CEFR judgments. Therefore, if an alignment does not take into account the meaning of the CEFR levels as expressed by existing CEFR descriptors, this raises issues with alignment validity, and hence the validity of test-score interpretation and usage. Given the impact of using CEFR aligned tests for high-stakes purposes, this article aims to shed more light on what assigning a CEFR level to a test actually means.

Testing Writing in EFL Exams: The Learners’ Viewpoint as Valuable Feedback for Improvement

In various English as Foreign Language exams, writing is one of the four skills tested, along with reading, listening and speaking. The combination of competences that writing requires often results in student confusion and anxiety. Teaching experience implied an increased difficulty in learners’ writing. In order to gain valid and detailed information on the learners’ views and add to this underrepresented area in the literature, it was decided to conduct a survey based on questionnaires addressing issues such as the time limitation factor, feedback, and main causes of difficulty and teaching methods. Students’ answers are analysed quantitatively according to age and CEFR (Common European Framework of Reference) level variables and presented in graphs. Drawing on existing literature, we discuss these findings in relation to the Greek EFL context and its particular characteristics and show where there is room for improvement regarding teaching and testing practices.

Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR

Selecting appropriate texts for L2 (second/foreign language) learners is an important approach to enhancing motivation and, by extension, learning. There is currently no tool for classifying foreign language texts according to a language proficiency framework, which makes it difficult for students and educators to determine the precise difficulty/complexity levels of an unclassified text. Taking the Chinese language as an example, this study aimed to create a readability assessment system, called the Chinese Readability Index Explorer for Chinese as a Foreign Language (CRIE–CFL), in order to level—that is, to sort by proficiency level—texts that will be used for instructional purposes. The framework of choice in this project is the Common European Framework of Reference (CEFR). A team of expert CFL teachers first classified 1,578 CFL texts into their appropriate CEFR levels. A set of 30 CFL readability features was then developed or drawn from previous research, and sorted according to importance using F‐scores. In addition, a support vector machine model was trained by sequentially integrating the features into the model to optimize accuracy. The empirical evaluation of CRIE–CFL revealed average exact‐ and adjacent‐level accuracies of 74.97% and 99.62%, respectively, for predicting the expert classification of a text. The functionalities of CRIE–CFL are introduced and discussed.

The development of a CEFR-based scale for assessing young foreign language learners’ writing skills

The Common European Framework of Reference (CEFR) was designed with adults in mind, which is clearly reflected in the six levels encompassing a range of proficiency that represents lifelong learning. Therefore, any use of the CEFR levels as a basis for describing the ability of young learners requires adapting the content of each level, as well as identifying which levels on the scale are appropriate for children.The present article examines the contribution that feedback, in the form of an assessment scale, can make to valid classroom assessment of the writing of young learners, in the age group of 9–13 years. It shows that a scale of descriptors adapted from the CEFR can play a central role in this assessment. The article presents the AYLLIT (Assessment of Young Learner Literacy) research project, which developed a CEFR-based writing scale and guidelines for teachers, enabling them to provide their students with feedback, and to gain a clearer insight into their students’ progress. After describing the procedures followed in the project, the article examines the extent to which its outcomes may enable teachers to give feedback that could contribute to valid classroom assessment.

Britain’s new language testing powerhouse: An interview with Professor Anthony Green, University of Bedfordshire

In this interview Professor Green explains the work of CRELLA (the Centre for Research in English Language Learning and Assessment at the University of Bedfordshire), and its role in the improvement of language testing. The institute contributes to this effort in many ways. For example, in the field of language education they are partners in English Profile (EP: www.englishprofile.org), a collaborative research programme directed towards a graded guide to learner language at different CEFR (Common European Framework of Reference) levels, based on the 50 million word Cambridge Learner Corpus. Among other things, the EP has helped to inform the development of the CEFR-J in Japan. In this interview, Professor Green also outlines his own work, especially in the areas of washback and assessment literacy.

Completing the English Vocabulary Profile: C1 and C2 vocabulary

AbstractThe English Vocabulary Profile is an online vocabulary resource for teachers, teacher trainers, exam setters, materials writers and syllabus designers. It offers extensive information about the Common European Framework of Reference (CEFR) levels of words, phrases, phrasal verbs and idioms, and currently includes just under 7,000 headwords. This article reports on the trialling and validation phase of the A1−B2 levels of the resource, as well as outlining the research and completion of the C1 and C2 levels. The project has followed a ‘can-do’ rationale, focusing on what learners actually know rather than prescribing what they should know, and is underpinned by up-to-date corpus evidence, including the 50-million word Cambridge Learner Corpus and the 1.2-billion word Cambridge English Corpus of first language use. At C1 and C2 levels, the English Vocabulary Profile describes both General and Academic English, and the additional sources used to research this area of language learning are described in the article. Polysemous words are treated in depth and the project has sought to determine which meanings of these important words appear to be acquired first; new, less frequent meanings often continue to be learned across all six CEFR levels. Phrases form another substantial part of the resource and this aspect has been guided by expert research (see Martínez 2011).

Using CEFR level descriptors to raise university students’ awareness of their speaking skills

The Common European Framework of Reference (CEFR) aims to encourage language learners to think about what they do when communicating, and to think about what they can do to help themselves and others to learn a language better. This paper considers to what extent the CEFR supports language learning by looking at how a group of learners, in the course of a term, used parts of the Common Reference Levels (CRLs) of the CEFR for self-assessment. The paper discusses the extent to which the CRL helped to raise the learners’ awareness of their language use and learning, and looks in detail at how the learners described their speaking skills and how their use of CRL statements and their perceptions of self-assessment changed during the term. Findings show that the students used CRL statements to write longer, more relevant, and more detailed and critical descriptions of their speaking skills.

LINKING ENGLISH‐LANGUAGE TEST SCORES ONTO THE COMMON EUROPEAN FRAMEWORK OF REFERENCE: AN APPLICATION OF STANDARD‐SETTING METHODOLOGY

ABSTRACTThe Common European Framework of Reference (CEFR) describes language proficiency in reading, writing, speaking, and listening on a 6‐level scale. In this study, English‐language experts from across Europe linked CEFR levels to scores on three tests: the TOEFL® iBT test, the TOEIC® assessment, and the TOEIC Bridge™ test. Standard‐setting methodology (a modified Angoff approach and a modified examinee paper selection approach) was used to construct the linkages. Linkages were established for TOEFL iBT at levels B1, B2, and C1. Linkages were established for TOEIC at levels A1 through C1, with the exception of Reading at the C1 level. The TOEIC Bridge test was linked to its three targeted levels of the CEFR. The report details the methods, procedures, and results of the study.

The Common European Framework of Reference (CEFR) and the design of language tests: A matter of effect

Language test development proceeds best when the test's effect is borne in mind, throughout the test development process. The authors discuss the flexible language of the Common European Framework of Reference for Languages (CEFR) and explore the pragmatic utility of such language to guide language test development. They select service encounters (e.g. airline ticket sales, open-air markets) as a sample language use domain to illustrate demonstrable weaknesses in the Framework. Using the CEFR Level A1 service encounter descriptor, suggested testing materials are shown in a versioned evolution of a proposed test specification. Provided that effect is kept in mind, the authors argue, the CEFR is actually a valuable – even an optimistic – starting point for language test development.

Analysing Tests of Reading and Listening in Relation to the Common European Framework of Reference: The Experience of The Dutch CEFR Construct Project

The Common European Framework of Reference (CEFR) is intended as a reference document for language education including assessment. This article describes a project that investigated whether the CEFR can help test developers construct reading and listening tests based on CEFR levels. If the CEFR scales together with the detailed description of language use contained in the CEFR are not sufficient to guide test development at these various levels, then what is needed to develop such an instrument? The project methodology involved gathering expert judgments on the usability of the CEFR for test construction, identifying what might be missing from the CEFR, developing a frame for analysis of tests and specifications, and examining a range of existing test specifications and guidelines to item writers and sample test tasks for different languages at the 6 levels of the CEFR. Outcomes included a critical review of the CEFR, a set of compilations of CEFR scales and of test specifications at the different CEFR levels, and a series of frameworks or classification systems, which led to a Web-mounted instrument known as the Dutch CEFR Grid. Interanalyst agreement in using the Grid for analyzing test tasks was quite promising, but the Grids need to be improved by training and discussion before decisions on test task levels are made. The article concludes, however, that identifying separate CEFR levels is at least as much an empirical matter as it is a question of test content, either determined by test specifications or identified by any content classification system or grid.