Assessment Of Oral Proficiency Research Articles

Effectiveness of Mobile-assisted Language Learning in Developing Oral English in Higher Education: A Comparative Systematic Review

English is a globally prominent language, and oral English proficiency is both crucial and challenging. Mobile technology offers a promising avenue for language enhancement, but research on the role of Mobile-Assisted Language Learning (MALL) in English-speaking skills is relatively scarce. Literature reviews on this topic are even rarer, particularly those that provide comparative analyses between China and other nations. This study addresses this gap through a comparative systematic literature review of 30 relevant studies from 2019 to 2023. The findings reveal similarities between Chinese and global studies, with only slight differences in sample size and oral English proficiency assessment methods. The preferences for mixed research methods, tests, questionnaires, and interviews were found. Additionally, this review identifies limitations in previous research, including a lack of theoretical frameworks, limited large-scale studies, and a need for deeper exploration of mobile app utilization. This comparative analysis provides valuable insights that can guide future studies and foster a more comprehensive understanding of MALL’s effectiveness in enhancing oral English proficiency, both in China and globally.

Becoming a French Second Language Teacher

This study explores the development of linguistic competence and confidence of pre-service French as a second language (FSL) teachers enrolled in a teacher education program in New Brunswick, Canada. In addition to their regular program requirements, participants in the study engaged in a series of language support sessions. We collected French oral proficiency assessment results pre- and post-session, as well as focus group interview data. The analysis shows that participants’ language proficiency increased over the course of the sessions and the program. Interview data reveal professional identity-related themes such as confidence, and the importance of safe spaces for interaction and mediation. Implications for FSL teacher recruitment and retention are given, specifically with regard to teacher candidates’ need for certain types of support.

An Efficient Probabilistic Deep Learning Model for the Oral Proficiency Assessment of Student Speech Recognition and Classification

Natural Language Processing is a branch of artificial intelligence (AI) that focuses on the interaction between computers and human language. Speech recognition systems utilize machine learning algorithms and statistical models to analyze acoustic features of speech, such as pitch, duration, and frequency, to convert spoken words into written text. The Student English Oral Proficiency Assessment and Feedback System provides students with a comprehensive evaluation of their spoken English skills and offers tailored feedback to help them improve. It can be used in language learning institutions, universities, or online platforms to support language education and enhance oral communication abilities. In this paper constructed a framework stated as Latent Dirichlet Integrated Deep Learning (LDiDL) for the assessment of student English proficiency assessment. The system begins by collecting a comprehensive dataset of spoken English samples, encompassing various proficiency levels. Relevant features are extracted from the samples, including acoustic characteristics and linguistic attributes. Leveraging Latent Dirichlet Allocation (LDA), the system uncovers latent topics within the data, enabling a deeper understanding of the underlying themes present in the spoken English. To further enhance the analysis, a deep learning model is developed, integrating the LDA topics with the extracted features. This model is trained using appropriate techniques and evaluated using performance metrics. Utilizing the predictions made by the model, the system generates personalized feedback for each student, focusing on areas of improvement such as vocabulary, grammar, fluency, and pronunciation. Simulation mode uses the native English speech audio for the LDiDL training and classification. The experimental analysis stated that the proposed LDiDL model achieves an accuracy of 99% for the assessment of English Proficiency.

PRAAT scripts to measure speed fluency and breakdown fluency in speech automatically

ABSTRACT Fluency in terms of speed of speech and (lack of) hesitations such as silent and filled pauses (‘uhm’s) is part of oral proficiency. Language assessment rubrics therefore include aspects of fluency. Measuring fluency, however, is highly time-consuming because of the manual labour involved. The current paper aims to automatically measure aspects of L2 fluency, including filled pauses, in both Dutch and English. A revised existing script and a new script for filled pauses are tested on accuracy. We also gauged whether the outcomes of the new script could be used for language assessment purposes by relating the outcomes to human judgements. Without further investigations, the current script should not (yet) be used for the purpose of assessing fluency automatically in (high-stakes) oral proficiency assessment. However, the performance of the scripts for measuring aspects of fluency globally and quickly are promising, especially given their stability in accuracy on new corpora.

Roles of collocation in L2 oral proficiency revisited: Different tasks, L1 vs. L2 raters, and cross-sectional vs. longitudinal analyses

There is emerging evidence that collocation use plays a primary role in determining various dimensions of L2 oral proficiency assessment and development. The current study presents the results of three experiments which examined the relationship between the degree of association in collocation use (operationalized as t scores and mutual information scores) and the intuitive judgements of L2 comprehensibility (i.e. ease of understanding). The topic was approached from the angles of different task conditions (Study 1), rater background (first language or L1 vs. second language or L2) (Study 2) and cross-sectional vs. longitudinal analyses (Study 3). The findings showed that: (1) collocation emerged as a medium-to-strong determinant of L2 comprehensibility in structured (picture description) compared to free (oral interview) oral production tasks; (2) with sufficient immersion experience, L2 raters can demonstrate as much sensitivity to collocation as L1 raters; and (3) conversational experience is associated with more coherent and mutually-exclusive combinations of words in L2 speech, resulting in greater L2 comprehensibility development.

Proficiência oral em Português Brasileiro: cenário, contextos de avaliação e de instrução e questionamentos

Os movimentos migratórios de grandes proporções, que ocorreram no passado e continuam ocorrendo no presente, tornam o Brasil um país multilíngue com demanda para o investimento em instrução formal em Português como Língua Estrangeira. Neste cenário, importa considerar questões relacionadas à formulação de políticas educacionais, à formação de profissionais aptos para o ensino de Português como Língua Estrangeira, à elaboração e aplicação de exames de proficiência linguística e à facilitação de acesso de estrangeiros à instrução formal em língua portuguesa. No presente artigo, abordamos questões relacionadas ao ensino de Português como Língua Estrangeira e aspectos da avaliação da Parte Oral do exame de proficiência em língua portuguesa aplicado no contexto brasileiro, o Celpe-Bras. Também descrevemos algumas das dificuldades relativas à percepção e à produção de sons do Português Brasileiro por falantes de outras línguas maternas. Nosso objetivo é apontar as contribuições que uma perspectiva foneticamente embasada, que considera o vínculo direto que se estabelece entre a percepção das distinções entre sons e a formação de categorias novas em L2, pode trazer para o desenvolvimento e a avaliação da proficiência oral em Português como Língua Estrangeira diante do impacto que um forte sotaque estrangeiro pode causar na inteligibilidade, na compreensibilidade e na avaliação atitudinal do falante. Em relação à avaliação da proficiência oral, defendemos a necessidade de formulação de critérios foneticamente embasados que balizem os avaliadores a considerar o que é uma pronúncia com características que prejudicam a interação.

Effects of Rater Training on the Assessment of L2 English Oral Proficiency


 
 
 The main objective of this study was to examine whether a Rater Identity Development (RID) program would increase interrater reliability and improve calibration of scores against benchmarks in the assessment of second/foreign language English oral proficiency. Eleven primary school teachers-as-raters participated. A pretest–intervention/RID–posttest design was employed and data included 220 assessments of student performances. Two types of rater-reliability analyses were conducted: first, estimates of the intraclass correlation coefficient two-way random effects model, in order to indicate the extent to which raters were consistent in their rankings, and second, a many-facet Rasch measurement analysis, extended through FACETS®, to explore variation regarding systematic differences of rater severity/leniency. Results showed improvement in terms of consistency, presumably as a result of training; simultaneously, the differences in severity became greater. Results suggest that future rater training may draw on central components of RID, such as core concepts in language assessment, individual feedback, and social moderation work.
 
 

“I start learning English through speaking”: Social agency demand and inter-school readiness for Indonesian young English learners

Preserving Indonesian as a national identity and institutional readiness to perform the national curriculum 2013 instruction has become a critical factor in the exclusion of English as a compulsory subject at the elementary school level. This leaves rooms for teachers’ confusion and creativity, leading to various practices at different school clusters, depending on parental demands and school readiness. This study thus tries to cast light on the social agency demand and inter-school readiness for young learners in the Indonesian EFL pedagogical contexts. This instrumental case study portrays the practices of English language teaching at three clusters of Indonesian primary schools; local, national, and independent, as well as responses of parents and students to these practices. Involving 4-6 grade students and English teachers of six primary schools, the data were garnered through classroom observation, interviews, and document analysis. The findings revealed that although teaching English to young learners is somehow communicative and fun, it more emphasises literacy skills (reading, writing, and grammar), particularly at local schools as the majority cluster across the country. Since English is a non-phonetic language, this literacy-focused practice is rather contradictory to what parents as social agencies expect, that is, to develop their children’s English oral proficiency. The study also offers ways by which teachers and schools can accommodate this social agency’s needs for speaking skills by focusing more on student-centred and oral English proficiency activities and assessment.

Holding up one’s end of the conversation in spoken English

AbstractThis study investigates the use of lexical backchannels in the discourse of L2 English users sitting Trinity College London’s Graded Examinations in Spoken English (GESE). It is based on theTrinity Lancaster Corpus Sampleand explores the language produced during the Discussion, Conversation and Interactive tasks of the language examinations by L2 English users from Chinese, Indian and Italian linguistic backgrounds, whose proficiency ranges from the B2 to C2 levels (i.e. high intermediate, advanced, expert) of the CEFR. The findings suggest that the L2 users with an Italian background and to a lesser extent those with a Chinese background often supported their examiners’ turns with items conveying uncertainty, while those with an Indian background with items of certainty. Furthermore, the L1 Chinese speakers used lexical backchannels the most, especially those expressing surprise or request for confirmation, while the speakers from India used them the least. Implications for the assessment of oral proficiency are discussed.

Exploring the Use of Drama-Based Activities as a Learning Tool in English Oral Proficiency Development among Students from Non-Malay-Medium Nation Schools: A Preliminary Study

Transitioning from Non-Malay-medium national schools to Malay-medium or English-medium secondary schools proved to be a struggle for many students due to the negligence in developing audio-lingual skills of the second language (L2) throughout the six-years of primary education. This study aims to explore the effectiveness of drama-based activities as a language learning strategy on L2 oral proficiency among the students aged between 12 to 13 years old from Non-Malay-Medium National School. Over a period of five weeks, pre-intervention interviews (on the first week), intervention (over the span of three weeks with two hours per week), and post-intervention interviews (on the fifth week) were carried out. The content of the intervention was adapted from Trinity College London's Grade 4 Communication Skills syllabus. The instruments used for data collection included: (1) pre-determined pre- and post-intervention speaking assessments, (2) classroom observation speaking assessments and (3) oral proficiency assessment rubric adapted from Cambridge English speaking assessment of Common European Framework of Reference Level B1 and a published drama assessment tool focusing on elements such as Grammar and Vocabulary, Pronunciation, Discourse Management, and Task Completion. The results of the post-intervention oral proficiency assessment showed that the there was an overall improvement especially in discourse management and task completion.

The effect of training and rater differences on oral proficiency assessment

As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. In the present study we seek first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, we examine the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, we examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

The Assessment of Oral Proficiency through Holistic and Analytic Techniques of Scoring: A Comparative Study

The Assessment of Oral Proficiency through Holistic and Analytic Techniques of Scoring: A Comparative Study

Evaluating the effectiveness of the training program on direct and semi-direct oral proficiency assessment: A case of multifaceted Rasch analysis

AbstractAn Oral Proficiency Interview (OPI) may be evaluated either during the interview procedure (direct-method) or from a tape-made oral interaction (semi-direct method). Such variety in methods...

Pronunciation’s role in English speaking-proficiency ratings

Abstract Although pronunciation is an integral part of speaking, the role pronunciation plays in determining speaking-proficiency levels is unclear (Higgs & Clifford, 1982; Kang, 2013). To contribute to our understanding of this area, the research reported here investigated the relationship between English as a Second Language (ESL) learners’ pronunciation ability and their speaking-proficiency ratings. At an intensive English program (IEP) in the United States, a speaking test was administered to 223 ESL students. Their speaking proficiency was rated using an oral proficiency assessment based on standardized guidelines. In addition, their pronunciation was rated in six categories (vowels, consonants, word stress, sentence stress, intonation, and rhythm) by 11 raters using a rubric specifically developed and validated for this study. Many-Facet Rasch Measurement (MFRM) was used to estimate the students’ pronunciation ability, which was then compared to their speaking ability. The study found that sentence stress, rhythm, and intonation accounted for 41% of the variance in the speaking-proficiency test scores with sentence stress being the most powerful factor.

The form and functions of newcomer EAL students’ speech in English: patterns of progression and communication in semi-structured interview dialogue

ABSTRACTAnalysis of progression in spoken English by newcomer migrant-background learners has traditionally oscillated between formal assessment of oral proficiency and ethnographic description of naturally occurring peer discourse. This paper reports on data gathered from a longitudinal study of newly arrived students with English as an additional language (EAL) in secondary schools in England. The corpus examined consists of semi-structured interviews with 22 students of different L1 backgrounds, each interviewed twice within a 12-month interval. Findings reveal marked improvement in comprehension with greatly reduced reliance on interpreters and clarification requests; limited range in use of verb tense, with heavy reliance on the present tense; areas of improvement in communicative function of speech mainly limited to conjecture, likes and dislikes, and expressions of feeling. Progression in the use of reported speech was limited for the group as a whole but where it was used it shed light on the individual's language development and into their views on their social experiences in their new environment. We argue that there is an urgent need for EAL practitioners and researchers to measure language development along similar lines in order to ground future policy-making in evidential knowledge of patterns of development in oral expression in English.

Should We Use Sentence- or Text-Level Tasks to Measure Oral Language Proficiency in Year-One Students following Whole-Class Intervention?

Aims: To compare students’ oral language proficiency on sentence- versus text-level tasks at school entry and following tier 1 intervention in their first year of formal schooling. Methods: 104 students participated in this study. Participants were part of a broader longitudinal study and were enrolled at 3 low socioeconomic, linguistically diverse Australian primary schools. Tasks were administered to all students at the beginning and end of the school year. Performance on the sentence-level task, the Renfrew Action Picture Test (RAPT), was analysed for information and grammar as per the test manual. Performance on the text-level task, the Profile of Oral Narrative Ability, was analysed for measures of story length, mean length of utterance, grammatical accuracy, number of different words, and story quality. Results: Results showed that both tasks are sensitive to measure progress following tier 1 intervention. However, RAPT concern status was not related to oral narrative concern status. Furthermore, if only the RAPT task had been used, between 11 and 21% of students performing below expectations in oral narrative would not have been identified. Conclusion: It is recommended that the assessment of oral language proficiency of students from culturally diverse, low socioeconomic backgrounds goes beyond the sentence level and includes an oral narrative retell task.

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters’ severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters’ biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.

The Impact of Raters’ and Test Takers’ Gender on Oral Proficiency Assessment: A Case of Multifaceted Rasch Analysis

The application of Multifaceted Rasch Measurement (MFRM) in rating test takers’ oral language proficiency has been investigated in some previous studies (e.g., Winke, Gass, & Myford, 2012). However, little research so far has ever documented the effect of test takers’ genders on their oral performances and few studies have investigated the relationship between the impact of raters’ gender on the awarded scores to male and female test takers. Thus, this study aimed to address the above-mentioned issue. Twenty English as a Foreign Language (EFL) teachers rated the oral performances of 300 test takers. The outcomes demonstrated that test takers’ gender differences did not have any significant role in their performance differences when they were rated by the raters of the same or opposite gender. The findings also reiterated that raters of different genders did not demonstrate bias in rating test takers of the opposite or same gender. Moreover, no significant difference was observed regarding male and female raters’ biases towards the rating scale categories. The outcomes of the study showed that both male and female raters assign fairly similar scores to test takers. This suggests no evidence based on which either male or female raters must be excluded from the rating process. The findings imply that there is no need to worry about the impact of gender for a more valid and reliable assessment.

DLLs and the development of self-regulation in early childhood

Current literature and research demonstrates that learning multiple languages allows for young learners to develop higher levels of executive functioning skills. Research also suggests that Dual Language Learners (DLLs) can surpass monolinguals in these executive functioning skills. Yet, there is a dearth of literature that explicitly discusses DLLs in the early childhood setting and the development of self-regulatory skills. Self-regulation skills have been linked to better indicators of academic achievement than numeracy and literacy. This research describes the differences between DLLs’ behavioral and emotional regulation, which are categorized as impulse control and cognitive regulation. This analysis examines the development of cognitive regulation and impulse control in both DLLs and non-DLLs. Results from an ANOVA of a convenience sample of 63 participants, 32 DLL (English and Spanish) and 31 non-DLL (English) preschool students, were assessed using the Preschool Self-Regulation Assessment (PSRA) and several measures of oral language proficiency. Participants were drawn from a Head Start and Universal Pre-Kindergarten program located in a low SES and culturally diverse district. The result yielded a statistically significant effect, (F(1, 61) = 8.56, p = .005; partial Eta squared = .123) for non-DLLs. ANOVA results suggest differences in cognitive regulation between the two groups. Implications relating to self-regulation, DLLs, culture and classroom practice, as well as policy are further discussed.

Equity in L2 English oral assessment: Criterion-based facts or works of fiction?

For assessment to be equitable, it is central that teachers/raters perceive and apply grade criteria similarly. However, in assessing L2 oral proficiency in paired tests, raters must grade test-takers individually on a joint interaction performance. With a conversation analytic approach, we examine closely one recording from a 9th-grade national test of L2 English with an aim to uncover some aspects that underpin vastly divergent assessments (as assigned by three raters) of one test-taker. Findings pointing to issues such as moral stance, rater experiences, and the interlocutor effect are discussed in light of equity in L2 oral proficiency testing and assessment.