Multimodal Conversation Research Articles

The Interactional Organization of Video‐Mediated Collaborative Writing: Focus on Repair Practices

AbstractComputer‐supported collaborative writing in online interactional settings has long been a central research topic in TESOL at the interface of social interaction and writing. Although some studies focused on side‐by‐side participants’ social interactions for the co‐production of texts, geographically dispersed participants’ video‐mediated interactional practices for collaborative writing in digital spaces have largely remained a research gap. Using multimodal conversation analysis for the examination of screen recordings data, the study sets out to explore the interactional organization of collaborative writing in Google Docs via Skype. More specifically, the study limits its focus to the co‐construction of a single sentence within a second language telecollaborative writing task. The close analysis of the procedural unfolding of the writing activity shows that repair organization is central to the practice of collaborative writing and the participants move the text forward largely in an environment of repair sequences. This also brings further insights into the complex ecology of video‐mediated interactions for writing activities in digital spaces. All in all, this study describes the discursive aspect of computer‐supported collaborative writing in English as a foreign language and provides implications for future practices.

Mediated learning materials: visibility checks in telepresence robot mediated classroom interaction

ABSTRACT Videoconferencing is increasingly used in education as a way to support distance learning. This article contributes to the emerging interactional literature on video-mediated educational interaction by exploring how a telepresence robot is used to facilitate remote participation in university-level foreign language teaching. A telepresence robot differs from commonly used videoconferencing set-ups in that it allows mobility and remote camera control. A remote student can thus move a classroom-based robot from a distance in order to shift attention between people, objects and environmental structures during classroom activities. Using multimodal conversation analysis, we focus on how participants manage telepresent remote students’ visual access to classroom learning materials. In particular, we show how visibility checks are accomplished as a sequential and embodied practice in interaction between physically dispersed participants. Moreover, we demonstrate how participants conduct interactional work to make learning materials visible to the remote student by showing them and guiding the ‘seeing’ of materials. The findings portray some ways in which participants in video-mediated interaction display sensitivity to the possibility of intersubjective trouble and the recipient’s visual perspective. Besides increasing understanding of visual and interactional practices in technology-rich learning environments, the findings can be applied in the pedagogical design of such environments.

‘There are two gaps, so’: teaching materials as resources for correction in pre-service teachers’ EFL classes

ABSTRACT This study investigates how pre-service teachers of English as a foreign language do correction during their initial teaching practice at lower-secondary schools. The study employs multimodal conversation analysis on a dataset of 16 lessons taught by three pre-service teachers in Czechia. The analysis focuses specifically on how the teachers orient to the task format when doing correction in answer-check sequences. It demonstrates how the teachers employ talk, gesture and gaze to make the task format (such as fill-in-the-blanks, matching) relevant and thereby use the materials as points of reference as well as resources for structuring classroom discourse. This use of materials contributes to the completion of the correction sequences. It is concluded that (over-)reliance on teaching materials and limited flexibility, which can be observed in not accepting alternative answers produced by learners, seem to be manifestations of some of the traits of pre-service teachers’ performance.

“Laughing moments”: the complex negotiation of laughing acts among students and teachers in an English as a second language classroom

ABSTRACT Using multimodal conversation analysis, this study closely examines moments when an instructor’s embodied explanations elicit laughter from his students – which we refer to as laughing moments – in an English as a second language classroom. Such laughing moments can exhibit students’ attention to the teacher’s explanation and also illuminate learner agency in deciding what is laughable and/or humorous. In particular, the negotiated nature of laughter and humour – how students orient to and select what is laughable and/or humorous – has been under-researched. Based on the findings, we discuss implications of a multimodal orientation for second language (L2) classroom research on laughter and humour, and implications for L2 teachers concerning negotiating laughter and humour with students.

The Smartphone as a Personal Cognitive Artifact Supporting Participation in Interaction

AbstractThis article uses multimodal conversation analysis to investigate how the smartphone as a personal cognitive artifact features in second language (L2) use and learning. The data come from a pedagogical intervention that was organized as part of an integration learning course for adult L2 students with emerging literacy. The purpose of the intervention was to guide the students to participate in everyday interactions outside the language classroom and to learn from them. The analysis concentrates on a focal student's smartphone use during different phases of the intervention and offers a detailed account of how the smartphone provides affordances for the student to formulate recognizable social actions and participate in different phases of the pedagogical activity. The analysis adds to our current understanding of the role of mobile technology in L2 learning and illustrates how experiential pedagogy supports language learning as social activity. The findings can be used in designing pedagogical practices that support L2 students to develop their interactional competences on the basis of their own needs and goals.

Phygital highlighting: Achieving joint visual attention when physically co-editing a digital text

In this paper, we introduce the concept of phygital highlighting. Phygital actions, by their very nature, are intertwined, simultaneously produced both physically and digitally, and may be used for practices of highlighting. The aim of this paper is therefore to revisit Goodwin's (1994) concept of highlighting and expand it to include phygital actions. We show how phygital highlighting is a participant method for achieving joint attention while engaged in face-to-face, computer-supported cooperative activity. Phygital highlighting is shown to be an observable and socially recognisable practice, composed of multimodal resources, specifically related to pointing practices and indexical terms in combination with moving the mouse and cursor, all of which are used to achieve a shared reference to specific parts of the displayed content. The paper is based on video ethnography and multimodal conversation analysis of a single case from an open office environment, which is considered a perspicuous setting for research into phygital highlighting.

Young students’ treatment of synthetic voicing as an interactional resource in digital writing

ABSTRACT This multimodal conversation analysis study is part of a larger video ethnographic project that explores the media literacy practices that children develop as they use digital and mobile technologies. The study investigates how Swedish students in grades 3–4 make use of text to speech (TTS) technology as an interactional resource during collaborative writing on iPads in the classroom. The results show that students routinely make use of synthetic voicings to display and claim knowledge about the voiced written units and negotiate writing roles with differing epistemic rights and obligations to assess voicings and practice repair.

Foreign language acquisition via artificial intelligence and extended reality: design and evaluation

Artificial Intelligence (AI) and Extended Reality (XR) have been employed in several foreign language education applications to increase the availability of experiential learning methods akin to international immersion programs. However, research in multi-modal spoken dialogue in L2 combined with immersive technologies and collaborative learning is thin, limiting students’ experiences to solo interactions focused mostly on vocabulary and grammar in such settings. We intend to fill this gap as we present the Cognitive Immersive Language Learning Environment (CILLE). The AI in CILLE can hear, see, and understand its users and can engage with them in non-dyadic multimodal conversations. The XR offers students a feeling of being somewhere else without the use of intrusive devices and supports multi-party, multi-modal interactions. Together, AI and XR create naturalistic conversational interactions targeted towards comprehensive foreign language acquisition. We evaluate CILLE as a Chinese-as-a-foreign-language (CFL) education tool through a seven-week, mixed-methods study with university students (N = 10). Results display statistical significance and retained improvement in CFL vocabulary, comprehension, and conversation skills. Coupled with an analysis of student feedback and researcher observations, we show how CILLE is designed and experienced by students to learn CFL.

Many-to-Many Unsupervised Speech Conversion From Nonparallel Corpora

We address a nonparallel data-driven many-to-many speech modeling and multimodal style conversion method. In this work, we train a speech conversion model for multiple domains rather than a specific source and target domain pair, and we generate diverse output speech signals from a given source domain speech by transferring some speech style-related characteristics while preserving its linguistic content information. The proposed method comprises a variational autoencoder (VAE)-based many-to-many speech conversion network with a Wasserstein generative adversarial network (WGAN) and a skip-connected autoencoder-based self-supervised learning network. The proposed conversion network trains the models by decomposing the spectral features of the input speech signal into a content factor that represents domain-invariant information and a style factor that represents domain-related information to automatically estimate the various speech styles of each domain, and the network converts the input speech signal to another domain using the computed content factor with the target style factor we want to change. Diverse and multimodal outputs can be generated by sampling different style factors. We also train models in a stable manner and improve the quality of generated outputs by sharing the discriminator of the VAE-based speech conversion network and that of the self-supervised learning network. We apply the proposed method to speaker conversion and perform the perceptual evaluations. Experimental results revealed that the proposed method obtained high accuracy of converted spectra, significantly improved the sound quality and speaker similarity of the converted speech, and contributed to stable model training.

The face of communication breakdown: Multimodal repair in L2 oral proficiency interviews

All communication entails moments of breakdown and repair. Research has shown that test-takers in oral proficiency interviews use both verbal and nonverbal behaviors to navigate these breakdowns, which may occur more often in online speaking tests than face-to-face settings. While individuals are often idiosyncratic in their baseline nonverbal behavior, it is unknown to what extent speakers resemble or differ from each other during moments of repair. To date, a research agenda including a broader range of embodied behavior in the context of language tests is lacking. In this study, four Zoom-based video recordings of second language tests were used to investigate seven categories of behavior and their alignment with speech during moments of other-initiated repair., This contribution uses multimodal conversation analysis to analyze differences in behavior amongst the participants and how those patterns align with the turn-by-turn nature of repair. The analysis shows that participants varied substantially in both the frequency and duration of each type of behavior during repair, and furthermore differed in how they combined these during their interaction with the examiner. The study highlights the need for more research in how individuals demonstrate their interactional competence both verbally and nonverbally for greater construct representation in second language speaking tests.

Learning Potentials Afforded by a Film in Task‐Based Language Classroom Interactions

AbstractThe use of films as teaching and learning materials can provide a variety of opportunities for interaction in second language classrooms. Research on the usage of films in language‐learning tasks to provide opportunities for learning and interaction, however, is scarce. Drawing on a database of video‐recorded interactions in an upper‐secondary English‐as‐a‐foreign‐language classroom in Sweden and using multimodal conversation analysis, this study examines affordances of student interactions during a film‐based discussion task. Taking a sociomaterial perspective, we focus on students’ co‐narrations of the film in a group task and show how the emergent discussions about the film facilitate collaborative attention work (CAW). Our findings reveal that the CAW in this film‐based discussion task unfolds when students (a) correct each other, or (b) collaboratively search for words while discussing the scenes in the film. Our analysis of these sequences reveals the learning potentials that emerge in film‐based discussions. The findings have direct implications for the use of audio‐visual materials—in particular, films and movies—in language‐learning tasks.

Rewilding Language Education: Emergent Assemblages and Entangled Actions

AbstractIntegrating concepts and techniques from ethnomethodology and sociomaterialism, this article investigates the observable material processes involving human action and place‐based contexts of language use enabled by locative media. The focal pedagogical intervention utilized mobile augmented reality (AR) activities, the development of which was inspired by research on learning ‘in the wild.’ Applying the principle of reverse engineering, we introduce a pedagogical approach termed ‘rewilding’ for its emphasis on designing supportive conditions for goal‐directed interaction outside of classrooms. Three instances of AR materials use are presented from an out‐of‐class activity associated with university‐level language courses involving a quest‐type AR game calledChronoOps. Video data of 3‐player groups were transcribed using conventions from multimodal conversation analysis. The empirical investigation illustrates meaning making through visible embodied displays, the performance of new actions through incorporation of public semiotic resources, and the contributions of the material surround as actants in the flow of interaction. Analysis illustrates that mobile AR activities enable languaging events among assemblages of environments, mobile devices, and embodied experience. We conclude by outlining the affordances of mobile AR activities as one example of rewilding approaches to creating material conditions for language use and learning.

Co-Learning in Hong Kong English medium instruction mathematics secondary classrooms: a translanguaging perspective

Recent studies on classroom discourse have challenged the traditional classroom role set and emphasised equal contributions from the participants and the emergence of knowledge through active participation. Co-learning emphasises the process in which teachers and students attempt to adapt to one another's behaviour and learn from each other in order to produce desirable learning outcomes. Current research has paid little attention to the ways in which content teachers and students jointly negotiate new knowledge in bi/multilingual classrooms. Based on data collected from a linguistic ethnography in Hong Kong English-Medium-Instruction secondary mathematics classrooms, this paper uses translanguaging as an analytical perspective to analyse how the EMI teacher and students co-learn in the classroom. The data are analysed using Multimodal Conversation Analysis and triangulated with the video-stimulated-recall-interviews which are analysed using Interpretative Phenomenological Analysis. The paper argues that translanguaging creates a safe space for co-learning that emphasises equity in knowledge construction and challenges the hierarchical relationship between the teacher and the learner.

The Analysis of The Elimination of Communication Barriers From the Perspective of Multimodal Synergy Theory

Multimodal conversation analysis has received more and more attention in recent years. This paper uses two videos as analysis materials, focusing on how the conversationalists in the video use multiple modal communication resources to achieve intrapersonal coordination and inter-personal coordination in the case of language communication barriers, thereby solving the obstacles in communication. There is currently no analysis of focusing on the elimination of communication barriers from a multimodal perspective in China, so this article hopes to provide some ideas for later researchers.

Embodiment of Values

Relational, multi-modal conversations between the authors’ experiences of a damaged environmental site occur through different knowledge systems including life sciences, art, agriculture and environmental science. The authors respond to the risks of the dramatic impact of the loss of water flow in the Barwon River, Victoria, Australia. This is a river that flows through the Indigenous lands of Wathaurong, Gulidjan, and Gadubanud country from the Otway Ranges and near to one of Deakin University’s campuses. Early in this century, groundwater extraction dried a swamp wetland, generating toxic levels of acid and heavy metals which generated a major fish kill in 2016. Loss of water led to the aquifer site at Yeodene Swamp revealing great depths of peat that, when burning, follows underground peat layers (an unknown river path) and emerges to ignite new above-ground fires. These issues and experience of dwelling in this part of Victoria inspire our embodied thinking, conversations, and art. They have prompted us to be ambitious in our actions—even provoking us to develop campaigns. Our value and respect for this place in the most holistic sense—geographic, experiential, spiritual, historical and biophilic—inspires us to come together to contextualise and apply responsibility, accountability, ethics, morality, justice and integrity. We respond to the question: What does embodiment of values look like in this context? Having brought this story into the 2019 Body of Knowledge Conference through walks and conversations by Gardiners Creek at Deakin University’s campus in Burwood, we have explored it further in this co-authored article.

Embodiment of Values

Relational, multi-modal conversations between the authors’ experiences of a damaged environmental site occur through different knowledge systems including life sciences, art, agriculture and environmental science. The authors respond to the risks of the dramatic impact of the loss of water flow in the Barwon River, Victoria, Australia. This is a river that flows through the Indigenous lands of Wathaurong, Gulidjan, and Gadubanud country from the Otway Ranges and near to one of Deakin University’s campuses. Early in this century, groundwater extraction dried a swamp wetland, generating toxic levels of acid and heavy metals which generated a major fish kill in 2016. Loss of water led to the aquifer site at Yeodene Swamp revealing great depths of peat that, when burning, follows underground peat layers (an unknown river path) and emerges to ignite new above-ground fires. These issues and experience of dwelling in this part of Victoria inspire our embodied thinking, conversations, and art. They have prompted us to be ambitious in our actions—even provoking us to develop campaigns. Our value and respect for this place in the most holistic sense—geographic, experiential, spiritual, historical and biophilic—inspires us to come together to contextualise and apply responsibility, accountability, ethics, morality, justice and integrity. We respond to the question: What does embodiment of values look like in this context? Having brought this story into the 2019 Body of Knowledge Conference through walks and conversations by Gardiners Creek at Deakin University’s campus in Burwood, we have explored it further in this co-authored article.

Respecifying the information sheet

This paper considers how information sheets are used as a resource for joint attention and decision-making in a situated service encounter. The optician shop is a perspicuous setting for addressing decision-making as an interactional accomplishment informed by information sheets, because they are routinely used and made relevant by sellers. The paper builds on ethnomethodological multimodal conversation analysis and a large corpus of more than 700 hours of video recordings from eleven different Danish optician shops. Based on one single analysis from this corpus, the paper shows how the sheet is used cooperatively and oriented to in situ as a shared resource in the process of selling/buying glasses. A key finding is that the information sheet is not just a resource for the seller but a shared resource for established joint attention and decision-making. More generally, the paper contributes to studies of service and sales encounters by highlighting the importance of inscribed objects to decision-making processes in social interaction.

RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot

Abstract Face-to-face interaction is a primordial site for human activity and intersubjectivity. Empirical studies have shown how people reflexively exhibit a face orientation and work to establish a formation in which everyone is facing each other in local participation frameworks. The Face has also been described by, e.g., Levinas as the basis for a first ethical philosophy. Humans have established these Face-formations when interacting since time immemorial, but what happens when one of the participants is present through a telepresence robot? Based on ethnomethodology, Peircean/Goodwinian semiotics, multimodal conversation analysis and video data from a Danish residential rehabilitation center, the article shows the ways in which participants manage to interactively, cooperatively, and moment by moment achieve an F-formation in situ. The article contributes a detailed analysis and discussion of the kind of participant a telepresence robot is, in and through situated interactions: I propose that we term this participant the RoboDoc, given that it is an assemblage of a doctor who controls a robot. By focusing on the affordances of mobility, the article contributes to a renewed understanding of the importance and relevance of establishing Face-orientations in an increasingly technofied telepresence world.

From appearings to disengagements: Openings and closings in video-mediated tele-homecare encounters

In this article, we examine openings and closings in video-mediated tele-homecare for older adults in Finland, using multimodal conversation analysis. We demonstrate how participants organise these boundaries sequentially and multimodally, how visual appearing and disengaging are of key importance in these processes, and how openings and closings mirror each other in this institutional setting. In the openings, the participants orient to sequential structures that resemble those from mundane telephone conversations and Skype interactions: summons−answer, appearing−noticing, greeting−greeting and the “how are you” question−answer. The participants treat appearing as an accountable part of the opening, and delay advancing to the “how are you” question until a proper visual appearing is produced. Closings are managed through stepwise transition practices that result in a terminal exchange and both participants disengaging from the encounter: the clients, by walking away; the nurses, by closing down the connection. In addition to managing visuality, time-oriented talk is present in both openings and closings. A comparison of our results with findings from other technology-mediated encounters emphasises the importance of visuality in managing closings, and shows that tele-homecare is an interesting hybrid of institutionality and informality.

Pragmatic socialization through gameplay directives: Multimodal conversation analysis of avatar-embodied interactions

This article explores the pragmatic socialization of peers into the virtual world using gameplay directives. The researcher integrates pragmatics and multimodal conversation analysis in the study of online gameplay discourse between experienced and novice players. The analyses highlight the participants’ indexical reframing of the gameplay activities, shifting forcefully compelled verbal imperatives into playfully fashioned directive trajectories of avatar-embodied interactions. The results suggest a broadening of directive strategies through contingent instructions and multiple forms of play for the development of gameplay skills and communicative actions.