Oral Communication Assessment Research Articles

Exploring the construct of interactional competence in different types of oral communication assessment

Abstract Research on interaction in speaking assessment suggests that both verbal and nonverbal interaction are integral parts of the construct of interactional competence (Galaczi & Taylor, 2018; Plough et al., 2018; Young, 2011). However, little has been done to investigate which features significantly contribute to interactional competence scores. This study, therefore, examined which interaction features that raters noticed in individual scripted interview and paired discussion tasks to gain an insight into the interactional competence construct, providing validity evidence for an inclusion of interactional competence in speaking assessment. Sixty-eight student performances were rated based on interaction rating scales. Exploratory factor analysis revealed four factors: nonverbal communication, topic management, interactional management, and interactive listening. Logistic regressions showed that while raters attended to more topic management features in the individual scripted interview task, they noticed more interactional management features in the paired discussion task. Simple regressions showed that nonverbal communication and topic management features predicted interactional competence scores in the individual scripted interview task, whereas nonverbal communication, topic management, interactional management, and interactive listening features were predictors of scores in the paired discussion task. The findings suggest that both nonverbal and verbal interaction features are important in the interactional competence construct with the paired task providing test-takers with more opportunities to demonstrate their interactional ability.

Exploring the Potential of a Spoken Dialog System-Delivered Paired Discussion Task for Assessing Interactional Competence

ABSTRACT While several studies support the appropriateness of tasks delivered through Spoken Dialog Systems (SDSs) for L2 oral communication assessment, few studies have explored the construct of interactional competence (IC) as measured by such tasks. This study aims to address this gap in the literature by investigating the use of IC features by high- and low-ability test-takers in performance on an SDS-delivered paired discussion task, along with the perceived appropriateness of the SDS as a conversation partner. Data collected from 30 test-takers and three trained raters included test-taker task responses, task scores, and semi-structured interviews. The analysis of the task responses for IC features based on a coding scheme adapted from Galaczi (2014) showed that the majority of the features included in the construct definition in the study were consistently observable in the task responses, and the use of observed features generally varied with score levels as expected. The qualitative analysis of the interviews revealed that test-takers generally perceived the system as an appropriate partner for oral communication assessment, although they noted several technical limitations that could affect their task performance. This study helps us better understand the construct of IC underlying SDS-delivered tasks and offers implications for designing and validating these tasks.

Strategy use in a spoken dialog system–delivered paired discussion task: A stimulated recall study

With recent technological advances, researchers have begun to explore the potential use of spoken dialog systems (SDSs) for L2 oral communication assessment. While several studies support the feasibility of building these systems for various types of oral tasks, research on the construct validity of SDS-delivered tasks is still limited. Thus, this study examines the cognitive processes engaged by an SDS-delivered paired discussion task, focusing on strategic competence, an essential component of L2 oral communication ability. Thirty adult test-takers completed a paired discussion task with an SDS acting as an interlocutor and provided stimulated recalls about their strategy use in the task. Three trained raters independently evaluated the test-takers’ oral task responses using a holistic rating scale devised for the task. Findings revealed the use of six categories of construct-relevant strategies during task performance. While no statistically significant differences were found in the use of these categories between high- and low-ability test-takers, marked differences were observed in the use of individual strategies within the categories between the test-takers at the two levels. These findings provide insight into how test-takers at different ability levels cognitively interact with SDS-delivered paired discussion tasks and offer implications for the design and validation of such tasks.

Investigating the effect of multimodality and sentiments on speaking assessments: a facial emotional analysis.

This quasi-experimental study aimed to determine the relationship between (i) oral language ability and emotions represented by facial emotions, and (ii) modality of assessment (audios versus videos) and sentiments embedded in each modality. Sixty university students watched and/or listened to four selected audio-visual stimuli and orally answered follow-up comprehension questions. One stimulus was designed to evoke happiness while the other, sadness. Participants' facial emotions during the answering were measured using the FaceReader technology. In addition, four trained raters assessed the responses of the participants. An analysis of the FaceReader data showed that there were significant main and interaction effects of sentiment and modality on participants' facial emotional expression. Notably, there was a significant difference in the amount of facial emotions evoked by (i) the happy vs. sad sentiment videos and (ii) video vs. audio modalities. In contrast, sentiments embedded in the stimuli and modalities had no significant effect on the measured speaking performance of the participants. Nevertheless, we found a number of significant correlations between the participants' test scores and some of their facial emotions evoked by the stimuli. Implications of these findings for the assessment of oral communication are discussed.

Evaluating technology-mediated second language oral communication assessment delivery models

ABSTRACT As our understanding of the construct of oral communication (OC) has evolved, so have the possibilities of computer technology undertaking the delivery of tests that measure this ability. It is paramount to understand to what extent such developments lead to accurate, comprehensive, and useful assessment of OC. In this paper, we discuss five models of technology-delivered OC assessment that have appeared in the past three decades. We evaluate these models in terms of how well their respective methods aid in assessing OC. To achieve this aim, we use a framework which takes into account a contemporary view of OC ability, including the call for incorporating English as a lingua franca (ELF) considerations into English language assessment. The evaluation of the five models suggests strengths and weaknesses of each that should be considered when determining which is used for a particular purpose.

Connecting Assessment with Teaching through Faculty Capacity Building: An Example of an Oral Communication Assessment Project

Connecting assessment with teaching through faculty capacity building can be a design principle in any assessment project aimed at improving student learning. This design principle is supported by the evaluation capacity-building literature, backward design curriculum approach, research on learning, and inquiry-driven assessment-for-learning framework. This paper provides an example of how this design principle guided the implementation of an institutional oral communication assessment project at a large public research-intensive university throughout an assessment cycle, from developing learning outcomes to using assessment results. Steered by five operating principles (be transparent, provide teaching support, foster shared understanding, form collaborations and value faculty expertise, and offer technical support), our center and campus groups carried out major strategies such as building and restructuring a project website, providing pedagogy workshops and panels for faculty, compiling and publicizing teaching and assessment resources, organizing faculty study groups on assessment, and collaborating with and motivating stakeholders and faculty to use assessment results. We advocate that connecting assessment with teaching should be intentional in the design and implementation of an assessment project to maximize the meaning and usefulness of assessment, ideally, through capacity-building activities.

The Development and Evaluation of Interactional Competence Elicitor for Oral Language Assessments

This paper describes the development and evaluation of Interaction Competence Elicitor (ICE), a spoken dialog system (SDS) for the delivery of a paired oral discussion task in the context of language assessment. The purpose of ICE is to sustain a topic‐specific conversation with a test taker in order to elicit discourse that can be later judged to assess the test taker's oral language ability, including interactional competence. The development of ICE is reported in detail to provide guidance for future developers of similar systems. The performance of ICE is evaluated on two aspects: (a) by analyzing system errors that occur at different stages in the natural language processing (NLP) pipeline in terms of both their preventability and their impact on the downstream stages of the pipeline, and (b) by analyzing questionnaire and semistructured interview data to establish the test takers' experience with the system. Findings suggest that ICE was robust in 90% of the dialog turns it produced, and test takers noted both positive and negative aspects of communicating with the system as opposed to a human interlocutor. We conclude that this prototype system lays important groundwork for the development and use of specialized SDSs in the assessment of oral communication, which includes interactional competence.

CLASSROOM: VIDEO FOR HOMEWORK

<p>This action research was conducted in order to investigate the students experience and perception towards using the video making assessment approach when presenting their oral communication assessment. This was to see the impact on the students compared to presenting the traditional way in-class. Forty-five (45) respondents undertaking DUE 3012 - Communicative English 2 in Polytechnic Balik Pulau, Information Technology Department, participated in this study. The respondents had to make a video based on Chapter 1- Product and Services whereby it will be evaluated as their oral presentation assessment. A Likert Scale questionnaire was distributed and the data were tabulated using descriptive statistics. The outcome of the study was that the respondents had a positive perception towards this approach and it helped them to build their confidence level to use the English Language. Besides that, the respondents felt that this approach of assessment was successful and beneficial in helping them to understand the chapter better. <strong></strong></p>

New and not so new methods for assessing oral communication

Author/s
 Gary J. OckeyZhi LiIowa State University, USA
 
 ABSTRACT
 The assessment of oral communication has continued to evolve over the past few decades. The construct being assessed has broadened to include interactional competence, and technology has played a role in the types of tasks that are currently popular. In this paper, we discuss the factors that affect the process of oral communication assessment, current conceptualizations of the construct to be assessed, and five tasks that are used to assess this construct. These tasks include oral proficiency interviews, paired/group oral discussion tasks, simulated tasks, integrated oral communication tasks, and elicited imitation tasks. We evaluate these tasks based on current conceptualizations of the construct of oral communication, and conclude that they do not assess a broad construct of oral communication equally. Based on our evaluation, we advise test developers to consider the aspects of oral communication that they aim to include or exclude in their assessment when they select one of these task types.

English Language Anxiety and the Delivery in Oral Communications

As a developing country, the Philippines encourages its human resources to learn the English language to become globally competitive. As much as it is not technically considered a second or foreign language, many Filipinos, especially in rural areas still find English a new and difficult language. This research is anchored on the theories of Krashen (1987) on Second Language Acquisition and Horwitz et al. (1986) on Foreign Language Anxiety. The study aimed to assess the English Language Anxiety and Delivery in Oral Communications in relation to Academic Performance among Grade 8 students at Union National High School in Ubay, Bohol. Using a purposive complete enumeration, the study employed a questionnaire patterned after the “Foreign Language Classroom Anxiety Scale (FLCAS)” and a rubric patterned from the Rubric for the Assessment of Oral Communication for Delivery” used by Valencia Community College in Orlando, Florida. Using documentary analysis, the raw scores from a periodical test administered during the course of the study were utilized. Data were treated to Chi-square and Pearson r moment of correlation to test relationships among variables. Results show that there was no significant degree of correlation between English Language Anxiety and the students’ Oral Communication Delivery; however, there was a significant degree of correlation between Oral Communication Delivery and Academic Performance.

Student experience of oral communication assessment tasks online from a multi-disciplinary trial

Purpose – The purpose of this paper is to evaluate the experiences of tertiary students learning oral presentation skills in a range of online and blended learning contexts across diverse disciplines. Design/methodology/approach – The research was designed as a “federation” of trials of diverse online oral communications assessment tasks (OOCATs). Tasks were set in ten courses offered across all five faculties at University of Newcastle, Australia. The authors collected and analysed data about students’ experiences of tasks they completed through an anonymous online survey. Findings – Students’ engagement with the task was extremely positive but also highly varied. This diversity of student experience can inform teaching, and in doing so, can support student equity. By understanding what students think hinders or facilitates their learning, and which students have these experiences, instructors are able to make adjustments to their teaching which address both real and perceived issues. Student experience in this study highlighted five very clear themes in relation to the student experience of undertaking online oral communications tasks which all benefit from nuanced responses by the instructor: relevance; capacity; technology; time; and support. Practical implications – Using well-designed OOCATs that diverge from more traditional written assessments can help students successfully engage with course content and develop oral communication skills. The student experience can be used to inform teaching by catering for different student learning styles and experience. Student centred approaches such as this allows instructors to reflect upon the assumptions they hold about their students and how they learn. This understanding can help inform adjustments to teaching approaches to support improved student experience of learning oral communications tasks. Originality/value – The importance of learning oral communication skills in tertiary education is widely acknowledged internationally, however, there is limited research on how to teach these skills online in a way that is student centred. This research makes a contribution toward addressing that gap.

Assessment of Oral Proficiency in Online Language Courses: Beyond Reinventing the Wheel

The article looks at assessment in online foreign language courses, discussing how the Internet offers opportunities for forms of assessment and associated learning experiences that are different from, and maybe superior to, those available in regular classroom settings. It discusses the washback effect, or the effect of the implementation of high-stakes testing on teaching and curriculum, suggesting it can be a positive effect. Assessment of oral communication in online courses is discussed.

Analysis of Observation Lesson based on Oral-Communication Skills

As the education system increasingly focuses on teachers and teaching, educators, policy-makers, and researchers need valid and reliable measures that can be used to evaluate individual teachers, provide guidance for improving teaching performance, and support research in ways that advance instruction and classroom dialog and practice. A new generation of classroom evaluation tools has recently been developed to support evaluation of teaching. Live observations tend to be the standard for studies of teaching and teacher evaluations in practice. They have the benefit of the observer being in the teacher`s physical classroom. This is valuable for teacher evaluations because it gives observation scores credibility among teachers. My observation was focused on oral-communication skills and oral-communication assessment. So I observed a professor at International Burch university while having oral-communication class with the upper-intermediate level students, and took some notes. Two hours of observation. It was focused observation method; lesson was particularly focused on speaking skills and oral-communication assessment.

Oral communication skills assessment in a synchronous hybrid MBA programme: does attending face-to-face matter for US and international students?

The ability to communicate effectively is an essential skill for graduates of Masters of Business Administration (MBA) programmes; however, as synchronous hybrid learning becomes more common, business schools may find it challenging to assess students’ proficiency in this core area. An additional layer of complexity is added by the burgeoning enrolment of international students in this technology-enriched learning environment. The objective of this exploratory study was to examine the relationships among attendance mode (online vs. on-campus), student nationality (domestic vs. international) and oral communication assessment scores in a synchronous hybrid MBA programme. Quantitative data from 202 hybrid MBA students collected over 12 semesters revealed domestic students were rated as more proficient orators than international students when assessed face-to-face. These results, however, were not replicated when students were assessed in a synchronous online environment. Furthermore, the results indicated that international students received higher scores when their oral communication skills were assessed online compared to face-to-face. Conversely, domestic students who were assessed face-to-face had higher mean scores than the domestic students who were assessed online. The findings are discussed with respect to teaching and learning in hybrid environments, technology-mediated assessment design and accreditation practices in non-traditional programme delivery modes.

Editorial: language testing in the Philippines

© L p Assessment reform is one of the keys to educational reform. In the Philippines, assessment reform is part of the K to 12 reform agenda implemented in June of 2012. As in any other reform, teachers are pressured to implement the changes in standards and student evaluation. One way to help teachers is by providing research reports that look into actual classroom assessment tools and models. School leaders also need to see how other educational institutions are able to successfully align national and school reforms. It is for this reason that this special issue brings together research reports that look into classroom-based assessment of writing and speaking as well as a report documenting the process of aligning standards and assessment in the school and department level. The first article will be of interest to those who have a writing process and writing assessment research agenda. The report provides a holistic scoring guide, a pilot-tested writing background questionnaire, and an idea generation process survey. Dr. Gustilo, the author of this paper, also reports findings that will be useful to those who want to find out if topic familiarity affects writing performance and if writing process affects written product. The second article, on the other hand, will be of interest to school leaders and education policymakers because it documents the process of aligning a school’s vision, its expected learning outcomes for all students, and assessment. It also describes the process that a department followed to integrate the school’s reform agenda in its English 1 syllabus. The third article is relevant to those who are exploring the topic of oral communication assessment. It reports how students used rubrics to understand their teacher’s expectations as well as how they used the scoring guide to assess their own performance. The fourth and final article, on the other hand, is relevant to those who are interested in the role of feedback. This article takes a different approach because it does not merely describe the comments given by language teachers, but it also compares the comments of language and non-language teachers. I hope that will issue will increase the interest of researchers not only in the Philippines but also in Asia as a whole on assessment reform. The collaborative efforts of researchers will help teachers and educational institutions to increase their assessment literacy and eventually improve teaching and learning. Finally, I would like to

General Education Oral Communication Assessment and Student Preferences for Learning: E-textbook versus Paper Textbook

As part of a yearly university mandated assessment of a large basic communication course that fulfills the oral communication general education requirement, this study examined student preferences for textbooks, reading, and learning. Specifically, basic course students (N=321) at a large state university in the Midwest were asked to complete a survey regarding paper textbook and e-textbook usage in their classes. The results indicated that neither reading the paper textbook nor the e-textbook was a predictor of grade, but comfort in accessing the e-textbook through the online portal was associated with grade. Many students reported strong preferences for paper textbook usage, and indicated several reasons for not liking or not using the e-textbook. The results of this assessment study could help communication course instructors make decisions about adopting e-textbooks versus paper textbooks, as well as alert them to the challenges students may experience with textbook reading.

Bravery, technological literacy and political philosophy: replacing oral presentations with student-created video presentations

This paper overviews and analyses a project to replace the oral presentation element of a political theory module with a video presentation created by students. Its aim is not to provide a practical guide for the educator but to discuss how video presentations address a number of concerns that students raise about oral presentations. In particular, this paper will focus on the role that bravery has on a student’s journey in education and how video presentations can help by moving the assessment of oral communication from a public to a private act. It will also look at how creating video presentations might stimulate digital literacy and suggest that it is the educator’s rather than the student’s literacy that is challenged by this process.

Results of a feasibility study to assess the health literacy environment: navigation, written, and oral communication in 10 hospitals in Catalonia, Spain

Adult literacy skills studies indicate that large percentage of adults in most industrialized nations have difficulty using print material to accomplish everyday tasks with accuracy and consistency. Words in signs, postings, and forms within health care settings establish a physical literacy environment and reflect the demands and expectations of the institutions. We report on an assessment of the literacy environment of 10 hospitals in Catalonia, Spain, to indentify factors that hinder or support the ability of people to make their way to and within, a hospital. The authors used standardized rating tools for the evaluation of navigation, specific readability tools for the assessment of written communication, and a patient survey for the evaluation of patients' perception of written and oral communication. Thematic analysis of results indicates insecurity and confusion in finding one's way throughout health care facilities. Readability assessment of selected health education materials and informed consent documents shows extensive use of scientific language that is inappropriate for the general public. Patients' assessment of oral communication was rated high although statistically significant differences exist between hospitals. This assessment offers insight into the literacy demands placed on patients and visitors.

Using verbal reports to explore rater perceptual processes in scoring: a mixed methods application to oral communication assessment

In recent years, there has been a growth in the use of rater cognitive data to inform test development and validity arguments. In this study, we examined differences in feature attention and categorisation between experienced and inexperienced raters for a college-level assessment of oral communication. The focus was two-fold: (a) rater cognition as it informs rubric development; and (b) impact of verbal protocols on data collection in a speech scoring context. The findings of the study indicate that the complexity of the scoring rubric may be as much of a threat to scoring consistency as raters’ pre-existing cognitive frameworks. A next step in the development of this tool is to simplify the rubric by identifying the critical features that most clearly define the construct, and modify training to address the cognitive influences raters bring to scoring. The results did not indicate a verbal report main effect on scoring.

Assessment of Oral Communication: A Major Review of the Historical Development and Trends in the Movement from 1975 to 2009

This comprehensive review of the assessment of oral communication in the communication discipline is both descriptive and empirical in nature. First, some background on the topic of communication assessment is provided. Following the descriptive background, we present an empirical analysis of academic papers, research studies, and books about assessing communication, all of which were presented or published from 1975 to 2009. The results are outlined of content and thematic analyses of a database of 558 citations from that time period, including 434 national convention presentations, 89 journal articles, and 35 other extant books and publications. Three main themes and eight subthemes are identified in the database, and trends evident in the resulting data are considered. The study concludes with a discussion of the trends and overarching themes gleaned from the research efforts, and the authors’ recommendations of best practices for how to conduct oral communication assessment.