Second Language Assessment Research Articles

Evaluating Iranian L2 Teachers’ Assessment Literacy for L2 Pragmatics by Applying the CEFR’s Pragmatic Competence Model: Possible Sociocultural-Informed Solutions

Nearly all multidimensional models of communication competence have pragmatic competence at their core. Proper assessment of second language (L2) pragmatics makes many demands on L2 teachers, both in terms of understanding the construct and in language test development. Therefore, being assessment literate helps teachers in developing effective tests geared toward educational outcomes. Despite the importance of assessment literacy (AL) for pragmatics, the field is still under-researched. Mindful of this fact and the significance of pragmatics in L2 learning and assessment, this study adopted the Common European Framework of Reference (CEFR) model of pragmatic competence to examine Iranian L2 teachers’ AL for L2 pragmatics and linked the results to the Sociocultural Theory (SCT) for formulating theoryinformed answers to the problems in the teachers’ assessment of L2 pragmatics. Group interviews were run with 67 participants and then qualitative and quantitative content analyses were performed. The paper discusses the identified problems in the assessment of L2 pragmatics and offers recommendations for raising L2 teachers’ AL for L2 pragmatics.

Visual Analysis of Dynamic Assessment Research Based on CiteSpace (1990–2022)

Abstract This study uses CiteSpace to map the scientometric analysis of dynamic assessment (DA) research in the Web of Science core collection from 1990 to 2022 in the areas of psychology, education and language learning. Firstly, the study found that DA was initially employed in the fields of psychology and special education, and has been gaining attention from second language acquisition researchers in recent years. And Vygotsky, the representative figure of sociolcultural theory, has the highest influence, and Budoff has the greatest degree as a bridge. Secondly, five research topics are found through keyword clustering, which are early identification, group dynamic assessment, short-term progress monitoring, computerized dynamic assessment and teacher education. At the same time, the analysis of burst reference shows that the computerized dynamic assessment will be the trend in the future, and dynamic assessment of second language tends to use mixed research method. Through a review of previous studies, this paper explores the paradigms and methods of empirical research in DA, hoping to provide new alternatives for future research.

Trialing alternative listening assessment tasks: interactions between text authenticity, item focus and item presentation condition

ABSTRACT Purpose: The current study applies an innovative approach to the assessment of second language listening comprehension skills. This is an important focus in need of innovation because scores generated through language assessment tasks should reflect variation in the target skill and the literature broadly suggests that conventional methods of assessing listening may fall short of achieving this principle (Field [2019]. Rethinking the Second Language Listening Test from Theory to Practice. Equinox). Design: The study investigated interactions between different methods of presenting listening comprehension questions, the focus of the comprehension questions and the relative authenticity of the sound file in an English-medium university entrance listening test. In a balanced design, 61 participants completed a listening test featuring both scripted and unscripted sound files by answering explicit and implicit information comprehension questions under five counterbalanced question preview and presentation conditions. Test scores were analysed using ANOVA and examined for interactions. Findings: The results revealed interesting relationships between text authenticity and item focus whereby item responses were most frequently correct overall on the explicit items on scripted tasks. However, the reverse was observed on the implicit items, which were more frequently correct on the unscripted tasks. Value: The research findings have important implications for construct definition and highlight possible directions for the development of pedagogical listening tasks and assessments for English-medium academic study.

Second‐language writing in university‐level basic language programs: A survey of student and instructor beliefs

AbstractThe majority of US university students studying foreign languages are enrolled in introductory courses that are typically part of a coordinated curriculum. Such courses conventionally include the assessment of second language (L2) writing skills. However, given that these assessments can be broadly conceived and vary by program, their design and implementation are subject to differing opinions and beliefs from the stakeholders involved. In an effort to better understand how their views of L2 writing assessments overlap and/or diverge, the present study examined instructor (N = 28) and student (N = 183) beliefs in Spanish language programs at three public US universities using an online Likert‐scale and ranked‐choice questionnaire. Results revealed that although there was misalignment regarding the pedagogical purpose of such assignments, in general there was broad agreement among the two groups, including the use of a writing‐to‐learn approach to develop both specific and broad linguistic skills.

Exploring shared and individual assessment of paired oral interactions

Studies concerning the assessment of second language (L2) paired oral interaction to date have investigated interactional patterns that emerge from paired oral tests and identified the factors that create variability in an individual’s test scores. However, less work has addressed concerns about whether scores should be shared or individual when assessing L2 speakers’ oral interactions. Therefore, the present study compared shared and individual assessment of L2 English paired oral task performances. Paired oral interaction episodes were sampled from a larger corpus of university-level L2 speakers engaged in paired speaking tasks and were assessed by 60 raters who were randomly assigned to rate Speaker A, Speaker B, or both speakers. To avoid any possible rating effects due to rating stimuli, half the raters evaluated audio recordings while the other half assessed video recordings. The raters used an analytic rubric with four domains: discourse management, collaborative communication, content development, and language accuracy and complexity. Comparison of raters’ scores revealed that individual discourse management ratings were significantly higher than shared ratings for both members of the pair regardless of the rating modality (audio vs. video). Implications for assessing pair interactions are discussed.

Assessing Second Language Listening Over the Past Twenty Years: A Review Within the Socio-Cognitive Framework.

The assessment of second language (L2) listening has received much attention. To understand the state-of-the-art research on L2 listening assessment, a total of 87 studies published in 14 peer-reviewed journals and two research report series between 2001 and 2020 were reviewed, using the socio-cognitive framework for developing and validating listening tests proposed by Weir (2005). Thirteen research themes were identified in relation to the six components of the framework, including test-taker characteristics, cognitive validity, context validity, scoring validity, consequential validity, and criterion-related validity. Context validity was the most investigated component, covering three research themes, that is, task setting, linguistic demands (input and output), and speakers. Based on a detailed analysis of the 13 research themes, recommendations for future research in L2 listening assessment were given.

Rater Effects on L2 Oral Assessment: Focusing on Accent Familiarity of L2 Teachers

ABSTRACT In the present study, I examined the effects of rater characteristics, in particular, raters’ familiarity with a foreign accent, on the assessment of second language (L2) pronunciation. Forty-three native English-speaking teachers were divided into three groups according to their reported types of familiarity with Korean accents: heritage, familiar, and unfamiliar. The teachers assessed global proficiency and accentedness of speech samples provided by 40 Koreans speaking L2 English. The ratings made on 1–9 numerical scales were analyzed for interrater reliability and rater severity. While the three teacher groups overall exhibited high interrater reliability, the findings suggest that prior exposure to, or regular contact with, foreign-accented speech may affect teachers’ consistency in ratings. A comparison of the three groups’ severity measures revealed that the heritage teachers were more lenient than the non-heritage (i.e., familiar and unfamiliar) teachers in evaluating global proficiency and accentedness. The difference in the level of severity between the familiar and unfamiliar teacher groups was not significant.

Direct assessment of second language writing: Holistic and analytic scoring

Direct assessment of second language (L2) writing skill, in which a student is usually assessed by means of an essay on a topic during a limited time period, has been used as a valid measurement of ability to write in comparison to indirect assessment of writing through multiple-choices. Although both direct and indirect assessments have a risk of reliability, it is effectively argued that direct assessment is more representative of the integrative aspects of writing (Coffman,1966; Morris-Friehe & Leuenberger, 1992). In this era of globalization, a large number of culturally and linguistically diverse students seek higher education in North America. Direct assessment of L2 writing skill is widely used to assess these newly-arrived students’ English proficiency for placement decisions at the beginning of their university programs in the target language. As such, understanding direct assessment of L2 writing is paramount to improve assessment procedures for test validity and fairness. This article reviews direct assessment of L2 writing in the existent research literature with an attention to its two major scoring rubrics: holistic and analytic. The literature review concludes that the purpose of the writing task is significant in deciding which scoring method to use in specific contexts.

Beyond single-word measures: L2 writing assessment, lexical richness and formulaic competence

Formulaic competence, the native-like use of ready-made sequences of words, is a key aspect in the development of L2 writing proficiency. Becoming increasingly important in foreign language teaching, it is largely neglected when assessing learner writing. Recently, several (semi-)automatic methods have been proposed to analyse the formulaic sequences in learner texts. After describing these methods and discussing their strengths and limitations, this study aims at determining the usefulness for assessing L2 text quality of collgram, a technique that assigns to each pair of contiguous words in a learner text two association scores (mutual information and t-score) computed on the basis of a large native speaker reference corpus. Correlation and hierarchical regression analyses, conducted on two datasets of English-as-a-foreign-language texts, showed that formulaic measures were the best predictors of text quality and provided a much higher specific contribution to the prediction than single-word lexical measures of diversity and sophistication. This study also confirms the need, as is being increasingly emphasized in applied linguistics, to replicate analyses on several corpora. The conclusion addresses the limits of the study and points to some potential implications for the teaching and assessment of second language writing.

Exploring the relationship of organization and connection with scores in integrated writing assessment

Abstract Traditionally, second language writing assessment has employed writing tasks that require only a single skill; however, in many academic contexts, writing requires the integration of several abilities, including reading and listening. To improve authenticity, integrated tasks are increasingly used in the research and assessment of second language writing. Scholars have proposed discourse synthesis as an underlying construct for these tasks. This study investigated performances on integrated reading-listening-writing tasks to consider how organization and connection, subprocesses in discourse synthesis, are reflected in scores. Four hundred eighty responses to an integrated writing prompt were analyzed for organizational patterns, coherence, and cohesion in relation to test scores. Raters coded essays for type and appropriateness of organization and coherence quality, while computational analysis was used to look at cohesion features. The results indicate that organization and coherence were related to writing score, with quality improving as score increased. However, the cohesion markers analyzed in this study yielded no statistical differences across the score levels.

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

An increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggest growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses. A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level English as a Second Language (ESL) writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilisation inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students’ abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers’ approach to accuracy, and instructors’ assumptions about the underlying purposes of AWE-based writing activities.

An Experiment in the Ability of Raters to Evaluate Lexis in Writing

Abstract For many years, people have considered lexis and grammar separately in the context of teaching and learning English. In the assessment of second language writing, lexis and grammar continue to be considered separately. However, recent corpus studies have questioned this approach and argued that lexis and grammar are fundamentally inseparable. While the assessment of lexis and grammar as two distinct qualities lends face validity to assessment criteria, the corpus literature suggests that raters may not be able to accurately distinguish the two. The current study examines the ability of raters to separate lexis and grammar when using an analytic rating scale to assess timed essays. In this experiment, the lexical content of 27 essays was manipulated before rating in order to determine the effect of lexical accuracy, lexical variation and lexical richness on lexis and grammar scores. From the results, it seems that raters are sensitive to lexical accuracy, but not lexical variation nor lexical richness. In addition, the manipulation of lexical qualities had a significant effect on grammar scores but not on lexis scores, supporting the idea that raters find it challenging to distinguish lexis from grammar

Approaching pedagogical language knowledge through student teachers: assessment of second language writing

The article examines student teachers' pedagogical language knowledge. The analysis is based on data from an applied task in which Finnish student teachers (n = 221) of 16 school subjects assessed second language (SL) learners' writing skills. First, we briefly discuss subject teachers' role in language and literacy teaching in the multilingual and multicultural classroom. Our findings indicate that the student teachers use a range of criteria but focus mainly on word-level assessment when assessing writing samples, and that their assessment orientation varies from technical to analytical. Finally, we discuss the challenges of developing teacher education to promote pedagogical language knowledge across the curriculum.

Assessing second language speaking

While the viva voce (oral) examination has always been used in content-based educational assessment (Latham 1877: 132), the assessment of second language (L2) speaking in performance tests is relatively recent. The impetus for the growth in testing speaking during the 19th and 20th centuries is twofold. Firstly, in educational settings the development of rating scales was driven by the need to improve achievement in public schools, and to communicate that improvement to the outside world. Chadwick (1864, see timeline) implies that the rating scales first devised in the 1830s served two purposes: providing information to the classroom teacher on learner progress for formative use, and generating data for school accountability. From the earliest days, such data was used for parents to select schools for their children in order to ‘maximize the benefit of their investment’ (Chadwick 1858). Secondly, in military settings it was imperative to be able to predict which soldiers were able to undertake tasks in the field without risk to themselves or other personnel (Kaulfers 1944, see timeline). Many of the key developments in speaking test design and rating scales are linked to military needs.

Quantifying the development of phraseological competence in L2 English writing: An automated approach

Based on the large body of research that shows phraseology to be pervasive in language, this study aims to assess the role played by phraseological competence in the development of L2 writing proficiency and text quality assessment. We propose to use CollGram, a technique that assigns to each pair of contiguous words (bigrams) in a learner text two association scores (mutual information and t-score) computed on the basis of a large reference corpus, the Corpus of Contemporary American English. Applied to the Michigan State University Corpus of second language writing, CollGram shows a longitudinal decrease in the use of collocations made up of high-frequency words that are less typical of native writers. It also shows that the mean MI scores of the bigrams used by L2 writers are positively correlated with the quality of the essays, while there is a negative correlation between the quality of the texts and the proportion of bigrams that were absent in the reference corpus, most of which were shown to be erroneous. The conclusion discusses the marked differences in the effects revealed by the longitudinal and pseudolongitudinal analyses, the limitations of the study, and some potential implications for the teaching and assessment of second language writing.

Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study

This study explores whether linguistic features can predict second language writing proficiency in the Test of English as a Foreign Language (TOEFL iBT) integrated and independent writing tasks and, if so, whether there are differences and similarities in the two sets of predictive linguistic features. Linguistic features related to lexical sophistication, syntactic complexity, cohesion, and basic text information were investigated in relation to the writing scores for both integrated and independent samples. The results of this study show that linguistic features can be used to significantly predict essay scores in the integrated as well as the independent writing. When comparing across the two writing tasks, there are both similarities and differences in the two sets of predictive features. For instance, lexical sophistication was found to be a significant predictor for both tasks while features such as verbs in 3rd person singular form and semantic similarity were only significant predictors for the integrated task. These findings demonstrate that evaluation of the two writing tasks rely on similar and distinct features, and are at least partially assessed using different linguistic criteria. Implications of these findings for the assessment of second language (L2) writing are also discussed.

Promoting metapragmatic development through assessment in the zone of proximal development

The assessment of second language (L2) pragmatic competence typically involves questionnaires, such as discourse completion tasks. This article describes a novel approach to using questionnaires to assess L2 metapragmatic capacities while simultaneously promoting their development: engaging learners in cooperative interaction as they complete the task. Following dynamic assessment principles, cooperative interaction reveals both fully formed and emerging competencies (i.e. zone of proximal development) while at the same time furthering their continued growth. This study draws on data collected during a concept-based pedagogical enrichment program in which US university students of French were learning the concepts of social distance and power hierarchies as illustrated by the second-person pronouns tu and vous ‘you’. We present a case study of one learner, Nikki, to show how support provided by a tutor around one questionnaire item (i.e. choosing tu or vous in an ambiguous situation) both assessed and promoted her developing conceptual knowledge about the dynamics of interpersonal relationships and how these are indexed through language.

Language Proficiency in Native and Nonnative Speakers: An Agenda for Research and Suggestions for Second-Language Assessment

This article addresses the question of what language proficiency (LP) is, both theoretically and empirically. It does so by making a distinction, on one hand, between basic and higher language cognition and, on the other hand, between core and peripheral components of LP. The article furthermore critically examines the notion of level in most second language (L2) assessment scales, showing that it is confounded with people's intellectual functioning because higher levels of LP cannot be attained by people with lower intellectual, educational, occupational, or leisure-time profiles. It is probably for this reason that the Common European Framework of Reference for Languages (Council of Europe, 2001) fails to consistently distinguish between L2 development and L2 proficiency. The LP construct presented in this article can account for the fact that L2 learners with higher intellectual, educational, occupational, or leisure-time profiles may perform, at a given point in time, both better (i.e., in the domain of higher language cognition) and more poorly (i.e., in the domain of basic language cognition) than native speakers with lower profiles. While offering a research agenda for investigating individual differences in first language and L2 acquisition, the article also presents several implications for L2 assessment.

What Needs to Be Developed to Facilitate Classroom‐Based Assessment?

When invited to contribute to this Forum asked myself, What issues have seemed most fundamental?yet perplexing because they are not well or widely understood?for the students who usually take my masters level course called Second-Language Assessment? These master's stu dents are experienced teachers, mostly of English, but also of other languages such as French or Japanese, who come from Ontario and many other parts of the world to improve their professional qualifications and knowledge. They teach in schools, businesses, colleges, or universities in Canada, Asia, Europe, or the Middle East. What have they, while reflect ing during assignments or class discussions, considered crucial to under stand and develop about classroom-based A primary issue is these teachers' own knowledge and professional abili ties. A common remark is, Why wasn't there a full course on assessment during my initial degree (i.e., bachelor of education or TESL certificate)? I've needed to know these things for years now. Or 'Why don't my employ ers (or principals or department heads) give us workshops or organize projects on assessment? A second issue concerns relating assessment to professional or curriculum standards in their teaching. Students in this master's course might remark, Now can see how to use the criteria in this policy to help my students evaluate their own progress. Or they may ask, with astonishment, Why didn't anyone else ever tell us about the Common European Framework of Reference for Languages? A third issue involves relating assessment to their students' learning. They might ponder, for instance, I am never sure whether my responses to students' writing really help them improve. These are three issues about classroom-based assess ment in need of further development, not only conceptually but also in

Pair versus individual writing: Effects on fluency, complexity and accuracy

The assessment of oral language is now quite commonly done in pairs or groups, and there is a growing body of research which investigates the related issues (e.g. May, 2007). Writing generally tends to be thought of as an individual activity, although a small number of studies have documented the advantages of collaboration in writing in the second language classroom (e.g. DiCamilla & Anton, 1997; Storch, 2005; Swain & Lapkin, 1998). Particularly in university contexts, group or pair assignments are widely used in many disciplines. In addition, collaborative writing could be used in second language classroom assessment contexts as formative assessment. However, research which compares texts produced by learners collaboratively to texts produced individually, and the implications of this for assessment practices, is rare. This study is a first step in the investigation of using collaborative writing in second language contexts and comparing the performance of two groups of second language learners: one group worked individually, and the other group worked in pairs. When writing in pairs, each pair produced a single text. All participants completed one writing task: an argumentative essay. The performances of the individuals (N = 48) and the pairs (N = 48) were compared on detailed discourse analytic measures of fluency, complexity and accuracy. This comparison revealed that collaboration impacted positively on accuracy, but did not affect fluency and complexity. A detailed analysis of the pair transcripts recorded during the writing activity provides insights into the ways in which pairs work together, and the foci of their endeavour. The implications of these findings for in-class assessment of second language writing are discussed.