Multimodal Communication Research Articles

Vocal gestures in early multimodal communication: A commentary on Karadöller, Sümer and Özyürek

Vocal gestures in early multimodal communication: A commentary on Karadöller, Sümer and Özyürek

Body language and words: a multimodal analysis of elite athletes' communication

Introduction: Competitive sports, especially elite badminton, are rich in complex communication interactions. Beyond technical and physical skills, communication dynamics play a crucial role in the outcome of a match. This study analyzes multimodal communication, namely the interaction of various communication modes such as verbal language, facial expressions, and body movements in a badminton match. Objective: To examine multimodal communication in an elite badminton match. Method: This qualitative study with a multimodal approach explores the meaning of the interaction of linguistic and kinesic cues at crucial moments in the men's doubles final of the 2024 PON in Indonesia. An in-depth case study analysis reveals the contribution of speech, intonation, facial expressions, and body movements to communication dynamics in competitive sports. Data were taken from a 2024 Youtube video recording (59:40 minutes, 5:40:10-6:39:50). Systematic multimodal analysis went through three stages: (1) synchronization of linguistic and kinesic transcripts with the video, (2) development of a cue coding framework, and (3) thematic analysis of main themes. Results: Nonverbal communication (facial expressions, body movements) is more dominant than verbal, reflecting the complexity of athlete interaction. Nonverbal functions to convey emotions, build team spirit, and set strategies. Differences in communication patterns between teams affect team performance and dynamics. Conclusion: Analysis of nonverbal communication is important for understanding athlete performance and team interaction. This study contributes to the development of multimodal linguistic theory and practice in athlete training, improving the understanding of team dynamics, and developing effective game strategies.

Face to face: The eyes as an anchor in multimodal communication.

Face to face: The eyes as an anchor in multimodal communication.

Unimodal speech perception predicts stable individual differences in audiovisual benefit for phonemes, words and sentencesa).

There are substantial individual differences in the benefit that can be obtained from visual cues during speech perception. Here, 113 normally hearing participants between the ages of 18 and 60 years old completed a three-part experiment investigating the reliability and predictors of individual audiovisual benefit for acoustically degraded speech. Audiovisual benefit was calculated as the relative intelligibility (at the individual-level) of approximately matched (at the group-level) auditory-only and audiovisual speech for materials at three levels of linguistic structure: meaningful sentences, monosyllabic words, and consonants in minimal syllables. This measure of audiovisual benefit was stable across sessions and materials, suggesting that a shared mechanism of audiovisual integration operates across levels of linguistic structure. Information transmission analyses suggested that this may be related to simple phonetic cue extraction: sentence-level audiovisual benefit was reliably predicted by the relative ability to discriminate place of articulation at the consonant-level. Finally, whereas unimodal speech perception was related to cognitive measures (matrix reasoning and vocabulary) and demographics (age and gender), audiovisual benefit was predicted only by unimodal speech perceptual abilities: Better lipreading ability and subclinically poorer hearing (speech reception thresholds) independently predicted enhanced audiovisual benefit. This work has implications for practices in quantifying audiovisual benefit and research identifying strategies to enhance multimodal communication in hearing loss.

Indonesian Netizens’ Impoliteness Toward Government Loudspeaker Policies: A Cyberpragmatic Approach

The issue of politeness on social media has become increasingly important and requires attention from various stakeholders. This is crucial due to the rise in impolite behavior by netizens when commenting online. This study aims to describe the forms of impoliteness in comments on the guidelines for the use of loudspeakers in mosques, a policy issued by the Indonesian Ministry of Religious Affairs. A qualitative approach was employed, with data consisting of impolite comments responding to the policy. The data were drawn from netizens' reactions to the Ministry’s announcement. Data analysis utilized the referential and pragmatic matching methods, supported by Leech’s (2014) theory of impoliteness and Yus’ (2011) cyberpragmatics framework. The findings revealed that impolite comments regarding the loudspeaker policy were manifested in four main forms: 42.5% as deviations from the maxim of wisdom, 27.5% as deviations from the maxim of politeness, 15% as deviations from the maxim of agreement, and 15% as deviations from the maxim of sympathy. These findings serve as a basis for evaluating government policies and could also be incorporated into educational programs on politeness, particularly at the elementary and secondary school levels, in the context of today’s multimodal communication environment.

Quantifying Facial Gestures Using Deep Learning in a New World Monkey.

Facial gestures are a crucial component of primate multimodal communication. However, current methodologies for extracting facial data from video recordings are labor-intensive and prone to human subjectivity. Although automatic tools for this task are still in their infancy, deep learning techniques are revolutionizing animal behavior research. This study explores the distinctiveness of facial gestures in cotton-top tamarins, quantified using markerless pose estimation algorithms. From footage of captive individuals, we extracted and manually labeled frames to develop a model that can recognize a custom set of landmarks positioned on the face of the target species. The trained model predicted landmark positions and subsequently transformed them into distance matrices representing landmarks' spatial distributions within each frame. We employed three competitive machine learning classifiers to assess the ability to automatically discriminate facial configurations that cooccur with vocal emissions and are associated with different behavioral contexts. Initial analysis showed correct classification rates exceeding 80%, suggesting that voiced facial configurations are highly distinctive from unvoiced ones. Our findings also demonstrated varying context specificity of facial gestures, with the highest classification accuracy observed during yawning, social activity, and resting. This study highlights the potential of markerless pose estimation for advancing the study of primate multimodal communication, even in challenging species such as cotton-top tamarins. The ability to automatically distinguish facial gestures in different behavioral contexts represents a critical step in developing automated tools for extracting behavioral cues from raw video data.

Vlogs as Korean College Students’ Translanguaging Space

This study investigates translanguaging practices in vlogs created by 28 Korean EFL preservice teachers. The rise of digital platforms has reshaped traditional literacy, extending it to include multimodal formats that integrate visual, auditory, and textual elements. In this context, vlogs provide an ideal space for translanguaging meaning-making and creative language use in EFL education. Through systematic content analysis of their English-language vlogs, including coding of multimodal elements and translingual features across multiple viewing sessions, this research uncovers key strategies that reflect creativity in multimodal communication. Drawing on Canagarajah’s (2013) translingual meaning negotiation framework and transmodality concepts (Im et al., 2022), the analysis identifies five major translanguaging strategies: translingual words (varied spellings of Korean terms), code-meshing of Korean and English, genre blending, hybrid pronunciation, and transmodal resource integration. These practices demonstrate how preservice teachers negotiate their cultural and linguistic identities while developing pedagogical awareness of translingual communication. The findings suggest that vlog creation can serve as a powerful tool in EFL teacher education, fostering metalinguistic awareness and multimodal literacy skills. This research informs the design of teacher preparation programs by highlighting how digital projects can develop both linguistic flexibility and a pedagogical understanding of translanguaging in contemporary EFL contexts.

Virtual Social Interaction and Empathy Development in South Korea

Purpose: The purpose of this article was to analyze virtual social interaction and empathy development in South Korea. Methodology: This study adopted a desk methodology. A desk study research design is commonly known as secondary data collection. This is basically collecting data from existing resources preferably because of its low cost advantage as compared to a field research. Our current study looked into already published studies and reports as the data was easily accessed through online journals and libraries. Findings: Studies in South Korea show that virtual social interactions, especially through immersive VR experiences, enhance empathy development. Programs like Our Neighbor Hero increased prosocial behavior, while high immersion and perspective-taking in VR strengthened empathetic responses. These findings highlight VR’s potential in empathy education. Unique Contribution to Theory, Practice and Policy: Social presence theory, media richness theory & social learning theory may be used to anchor future studies on the virtual social interaction and empathy development in South Korea. Educators, mental health professionals, and organizational leaders are encouraged to adopt multi-modal digital communication strategies that incorporate video conferencing, instant messaging, social media engagement, and online collaborative tools. Funding should be allocated toward research and the development of technologies that not only support mental health initiatives but also foster inclusive, empathetic digital communities

Integrating collaborative digital multimodal tasks in Spanish as a second language course

Abstract The emphasis in L2 learning has mainly focused on individual writers and monomodal academic genres (e.g. narration, argumentation), neglecting the potential of collaborative composing and the use of digital genres that introduce additional semiotic sources, for fear of having to deal with “a messy transition to digital multimodal communication” (Lotherington, 2021: 220). Yet, because Web 2.0 technological upgrades have enabled interactivity, literacy has morphed from discretely reading and writing a static page to dynamically reading and writing a multimodal one, which underpins collaborative authorship and (local and global) audience awareness. Considering the inclusion of working collaboratively with multimodal tasks in the L2 classroom, the question of how to help students effectively incorporate multimodal with academic monomodal texts remains unanswered. In response to this challenge, this study examines the design and implementation of an online task to foster multiliteracies. Thirty-seven international students of diverse disciplines (e.g. economics, engineering, history), enrolled in a Spanish as a second language course, worked collaboratively to create multimodal texts based on previously created monomodal texts. Informed by a student questionnaire and a teacher focus group, we analyzed both students’ and teachers’ perceptions to ascertain the effectiveness of the intervention and the possibilities these kinds of tasks bring to the foreign language classroom. Both sets of participants reported positive results concerning linguistic advancement, motivation, and multiliteracies development. Pedagogical recommendations related to the inclusion of this pedagogical practice are provided.

Integrating Multimedia Tools to Enrich Interactions in Live Streaming for Language Learning

Purpose: According to the study, multimodal engagement may improve language learning in live-streamed contexts. These include text, speech, video, images, and stickers. The main goal is to examine how new technologies enable more interactive, real-time language instructor-student interactions by overcoming text-only communication in typical online classrooms. Materials and Methods: One language instructor and nine students participated in a two-week field research project. In four live-streamed lessons, the teacher tested three English teaching methods using text, audio, video, photographs, and stickers. These multimodal channels allow students to engage in class discussions and provide constructive critique actively. Findings: The study found that multimodal communication engaged students and provided quick feedback. The tools' effectiveness depended on group size, learning context, time of day, and instructors' and students' online identities. Implications to Theory, Practice and Policy: Online language education may benefit from real-time feedback and involvement via audio, video, text, etc. When creating courses, teachers must consider group size class time, and how students utilize resources.

Digital well-being begins with inclusion: A systematic review of videoconferencing guidelines for equitable learning

As higher education institutions increasingly adopt videoconferencing technologies to broaden access to learning, the need for evidence-based, inclusive practices to support digital well-being becomes paramount. Integrating these technologies into the curriculum necessitates careful design considerations to prevent unintended consequences and uphold learners' privacy, safety, equity and humanity. Our systematic review, based on eight dimensions of digital wellness, has identified key inclusive design decisions for videoconference-enabled formal learning experiences. Drawing from data analysed from 36 empirical studies, we organised six inclusive design considerations for digital wellness in videoconferencing learning environments. These considerations – accessibility, active learning strategies, multimodal communication, readiness, social presence and sociocultural sensitivity – offer course designers a practical framework to create evidence-based practices that foster digital wellness and inclusion in videoconferencing learning spaces. Implications for practice or policy: Academic institutions should recognise digital wellness as a shared responsibility among institutional stakeholders, including faculty, learners, and administrative professionals. Institutional policies should prioritise learner choice and equitable access for co-creating knowledge and fostering safe communication. Stakeholders should be empowered to make informed choices about digital habits to mitigate unintended consequences and encourage mindful technology use. Accessibility barriers must be addressed through intentional learning design, ensuring meaningful participation and interaction for all.

A bibliometric analysis of metonymy in SSCI-indexed research (2000-2023): retrospect and prospect.

Metonymy has gained increasing attention for its role in shaping language, thought, and communication. Despite its prominence, the thematic evolution and future directions of metonymy research remain underexplored. This study seeks to address this gap by analyzing metonymy research published between 2000 and 2023, providing a comprehensive overview of its key trends and emerging themes. A bibliometric analysis was conducted using data sourced from the Social Science Citation Index (SSCI) within the Web of Science Core Collection. Co-citation and co-word analysis were employed alongside k-means clustering techniques to identify research themes. Predictive modeling, including ARIMA and LSTM approaches, was used to forecast future research topics based on keyword trends. The analysis identified 11 key research clusters, highlighting the central role of cognitive and conceptual linguistics in metonymy research, along with its applications in semantics, pragmatics, and multimodal contexts. Predictive modeling suggested the emergence of seven new research themes for 2024-2028, including the interaction between metonymy and discourse, its role in multimodal communication, and its application in social and cultural narratives. This study underscores the interdisciplinary nature of metonymy research, bridging linguistic, cognitive, and social dimensions. The findings highlight promising areas for future exploration, namely, its integration into digital communication and its impact on cultural identity construction. The methodological approach offers a robust framework for analyzing and predicting research trends, paving the way for innovative contributions to the field.

Envisaging Human-Nature Conflict in the Trailer for Oppenheimer (2023): A Multimodal Ecolinguistic Study

Ecolinguistics is widely acknowledged as a field of study that cares about the relationship between language and ecology. Since the conflict between humans and nature has been intensifying, ecolinguists have concentrated on ecological studies adopting discourse analysis approaches to demonstrate how discourse constructs the relationship between humans and the ecosystem. This study aims to conduct a multimodal analysis of ecological dynamic discourse in an attempt to integrate multimodality with ecolinguistic studies, an area which is still under-researched. The trailer for Oppenheimer (2023) comprises the data of the study. The movie is a dramatization of the life story of Oppenheimer, the leader of the Manhattan Project that produced the first nuclear weapons. The research focuses on exploring how the human-nature conflict is rendered via multimodal resources. The theoretical framework of the study dwells upon Visual Environmental Communication Research (Hansen 2018) supported by the updated version of the Relevance Theory (RT) as extended to multimodal communication (Forceville 2020). By utilizing ecolinguistics as an analytical lens and applying RT, this study uncovers how movie trailers significantly underscore the urgency and cruciality of the human-nature interconnection amid tragedy and destruction. The findings of the study indicate that multimodal affordances in Oppenheimer trailer, both verbal and non-verbal, contribute to portraying the complexities of nature-human interface.

Dancing With children or dancing for children? Measuring the effects of a dance intervention in children’s confidence and agency

ABSTRACT Dance has benefits in children’s development, including increased self-confidence, physical and mental growth. While much research explores preschool children’s experiences in dance courses, most studies involve pre-designed choreographies led by adults. Some argue that young children have limited ability to create their own choreography. This research challenges that by giving children the opportunity to co-create choreographies, with practitioners, using music of their choice. A new dance framework, Dancing with Dr E, was implemented over 5 months in five preschool classrooms in London. Data were collected through semi-structured interviews with practitioners. The findings revealed numerous instances where children’s participation increased, showing excitement and multimodal communication. There was a noticeable increase in children’s confidence, by becoming more vocal about their needs and ideas. Due to the limited number of participants and the absence of a control group, further research on the effects of dance education in children’s confidence and agency is recommended.

Multimodal Interaction, Interfaces, and Communication: A Survey

Multimodal interaction is a transformative human-computer interaction (HCI) approach that allows users to interact with systems through various communication channels such as speech, gesture, touch, and gaze. With advancements in sensor technology and machine learning (ML), multimodal systems are becoming increasingly important in various applications, including virtual assistants, intelligent environments, healthcare, and accessibility technologies. This survey concisely overviews recent advancements in multimodal interaction, interfaces, and communication. It delves into integrating different input and output modalities, focusing on critical technologies and essential considerations in multimodal fusion, including temporal synchronization and decision-level integration. Furthermore, the survey explores the challenges of developing context-aware, adaptive systems that provide seamless and intuitive user experiences. Lastly, by examining current methodologies and trends, this study underscores the potential of multimodal systems and sheds light on future research directions.

“Age of risk” shapes simpler multimodal communication in the juvenile plains zebra (Equus quagga)

Juveniles occupy a different social niche than adults, engaging in a smaller diversity of social contexts and perceiving greater social risks. Either or both of these factors may influence the form communication takes in immaturity and its developmental trajectory. We investigated the relative influence of these social forces on the development of multimodal communication in plains zebras (Equus quagga). Juveniles possessed smaller repertoires than adults, with lower combinatorial flexibility and greater stereotypy, particularly for signals used in submission. When interacting with adults, juveniles used a larger fraction of their repertoire, but with reduced combinatorial flexibility. The usage of a contextually flexible signal, “snapping”, also shifted across development, beginning as a stereotyped, submissive signal before diversifying into the full range of adult usage. Taken together, the lower complexity of juvenile communication may reduce signal ambiguity and the risk of miscommunication when interacting with social partners perceived as higher risk, like adults.

Translanguaging in AACtion: An Expansive Approach to Multimodal Communication With Augmentative and Alternative Communication Users

Purpose: The purpose of this framework is to challenge the deficit perspectives toward multimodal communication and to encourage speech therapists and classroom teachers to apply educational justice and equity practices with modally marginalized individuals in education and clinical practice. Method: We summarize contemporary challenges in augmentative and alternative communication (AAC) as a practice area and integrate translanguaging and crip linguistics literature to propose a novel framework: Translanguaging in AACtion. Results: The authors propose the Translanguaging in AACtion framework to provide a praxis for engaging with and supporting all language users in and beyond the classroom by rethinking service delivery, supporting reciprocal engagement, and letting students language. The Translanguaging in AACtion framework guides speech therapists toward supporting disabled nonspeaking individuals to use their full-bodied semiotics (modes, languages, and dialects) in every setting, with every person, in every situation. Conclusions: This work highlights the languaging practices of (dis)abled, nonspeaking language users. Next steps are suggested to support speech therapists and classroom teachers with rethinking their praxis, with a focus on AACtion.

Sloyd Education for Newly Arrived Students

Swedish compulsory schools face challenges in organizing education for newly arrived students and often use practical-aesthetic subjects such as sloyd to mitigate their transition into the school system. This article presents two examples of newly arrived students in the sloyd classroom and explores the interaction, communication, and learning environment that they encounter in this context. This is done through analyzing video recorded material of sloyd lessons with newly arrived students in Years 8–9. The findings highlight that the sloyd classroom environment offers newly arrived students rich multimodal interaction and communication opportunities. Furthermore, the findings show that these students encounter complex and abstract subject-specific content and concepts in situated learning situations. The use of multimodal pedagogy enables newly arrived students to demonstrate their knowledge beyond verbal actions.

Integration of Natural Language Processing (NLP) in MongoDB NoSQL: A New Era of Efficient Text Data Management

Advancements in computing power enable efficient management of large and complex datasets, leading to growing importance in big data analytics. This is evidenced in the progress demonstrated by word embedding’s in creating extensive index files of interconnected entities. Digital collections, which represent layered textual structures and multimodal communication annotations—ranging from linguistic to gestural data—point out the complexity of modern datasets. Evolving data forms call for new techniques for effective storage and management of data forms to be used in NLP applications. This paper analyzes six distinct Database Management Systems (DBMS) that will help in understanding optimum methods for dealing with the complex data types. Specifically, the paper looks into the areas of tokenization, semantic analysis, named entity recognition, sentiment analysis, text classification, and information retrieval. These were found to not be dominant in all tasks. We thus propose a web-based multi-database management system MDBMS that integrates specialized databases in various paradigms; this is scalable and adaptive. This MDBMS model integrates the strengths of different systems to meet the needs of heterogeneous datasets in NLP applications.

Plains Zebras Prioritize Foraging Without Sacrificing Social Bonds During a Severe Drought.

Anthropogenically induced climate change has significantly increased the frequency of acute weather events, such as drought. As human activities amplify environmental stresses, animals may be forced to prioritize survival over behaviors less crucial to immediate fitness, such as socializing. Yet, social bonds may also enable individuals to weather the deleterious effects of environmental conditions. We investigated how the highly social plains zebra (Equus quagga) modify their activity budgets, social networks, and multimodal communication during a drought. Although animals prioritized feeding and the number of social interactions dramatically decreased in the late drought period, social associations remained robust. We observed age/sex class-specific changes in social behavior, reflecting the nutritional needs and social niche of each individual. Stallions devoted more time to greeting behaviors, which could mitigate harassment by bachelor males and facilitate grazing time for the females of the harem. Juveniles significantly increased time spent active socializing, despite mothers showing the greatest decrease in the number of social interactions. Instead, unrelated, nonlactating females served as social partners, accommodating both juveniles' social needs and lactating mothers' nutritive requirements. Using a network-based representation of multimodal communication, we observed a decrease in the number of signals used during the drought. Individuals used less diverse multimodal combinations, particularly in the costly context of aggression. These findings illustrate how social roles and differential responses to acute environmental stress within stable social groups may contribute to species resilience, and how communication flexibly responds to facilitate both survival and sociality under harsh environmental conditions.