Multimodal Communication Research Articles

Large AI Model Empowered Multimodal Semantic Communications

Large AI Model Empowered Multimodal Semantic Communications

Research on the Multimodal Integration of Visual Communication Design and Public Art in Digital Perspective

Abstract With the rise of digital media, the visual design of public art has gradually added multimodal interactive experience from purely conforming to information communication. Starting from the multimodal integration of public art visual communication design, this study takes several groups of public art landscape projects in a certain region as research objects to explore the relationship between the multimodal visual design of public art landscape and its public satisfaction. Firstly, the evaluation standard for public art visual communication design is constructed. The sample visual communication design is analyzed using the semantic difference method and then combined with the factor analysis method to provide a comprehensive evaluation. Based on determining the satisfaction effect indexes, the AHP method is used to assign weights to each index. Finally, the satisfaction effect results of the sample visual communication designs are obtained. After utilizing the result equation, the relationship between public art visual communication design and landscape satisfaction effect is explored. The overall appearance of public art visual communication design is presented better, and the impact of multimodal fusion visual communication design is especially prominent in samples 3, 2, and 10. The satisfaction effect of the sample visual communication design is also more satisfactory, with 70% of the sample landscapes scoring above 1.5. Meanwhile, the study verified the positive and significant influence of public art visual communication design on the satisfaction effect of landscapes. The theoretical basis and methodological exploration for the multimodal integration research of public art can be provided by this study.

Language and cultural communication —— Smart media has become a bridge for cross-cultural communication

Language and culture are two indispensable elements of human society. They interweave and influence each other, and jointly constitute the diversity and uniqueness of human society. As a tool of communication, language not only transmits information, but also is a carrier of culture. This paper aims to explore the importance and mutual influence of language and culture on cross-cultural communication in the era of intellectual media. Through the analysis of related literature review, this paper puts forward in the era of intellectual media, can to a certain extent, break the barriers between cultural communication and cross-cultural language communication, “cross language and cross-cultural” the humanities social important proposition, make the communication activities of “language”, “culture”, “communication” three elements maximum transmission efficiency, make based on intellectual media multimodal culture communication and language communication more affinity and story.

Application and Optimization of Multimodal Information Communication in Cultural and Creative Products in the New Media Era

Abstract As one of the most important parts of today’s cultural industry products, the design and presentation form of cultural and creative products have been optimized with the development of new media technology. Based on the multimodal theory, the study constructs a system of cultural and creative products using graph neural networks, attention mechanisms, knowledge maps, and other cultural and creative product design methods. The cultural representation evaluation system of cultural and creative products is constructed using the hierarchical analysis method, and the cultural and creative products designed by the model of this paper are evaluated. Analyze the color of cultural and creative products to examine the performance of this paper’s model on color. By analyzing the user experience score of cultural and creative products, the design effect of this paper’s model is explored. In the evaluation index system of cultural representation of cultural and creative products, the largest proportion of the weight of the criterion layer is the external level (54.68%), and the evaluation index with the largest comprehensive weight is the pattern (27.08%). The cultural and creative products designed by the model of this paper have the largest weight proportion of the value to be modified in pattern and material, which are 36.63% and 24.15%, respectively. The mean value of each of the chromaticity analyses is less than 2, and the mean value of the total color difference is less than 3, indicating that the color performance of the cultural and creative products designed by the model of this paper is excellent. The overall scores of the four types of cultural and creative products designed by the model of this paper are between 3.92 and 4.04 in terms of user experience, and the user experience is better.

Toward Interdisciplinary Integration in the Study of Comparative Cognition: Insights from Studying the Evolution of Multimodal Communication

Toward Interdisciplinary Integration in the Study of Comparative Cognition: Insights from Studying the Evolution of Multimodal Communication

Conferência online entre pares: uma janela de oportunidade a favor do feedback para a escrita

Abstract The pandemic has had quite an impact on the way we teach and learn. During that time, Brazilian universities implemented remote learning for more than 15 months. While the initial move was a reaction to an emergency and saw the transposition of many classes to the virtual mode, the following months saw the flourishing of initiatives that enhanced learning by streamlining the use of digital technology. In this context, this descriptive qualitative study employs an exploratory multiple case approach to investigate the affordances in the digital context tertiary students use to facilitate engagement with computer-mediated peer feedback during a course of academic literacies in English at a university in Brazil. Data come from individual semi-structured interviews, online peer-to-peer feedback meeting recordings, and retrospective individual interviews with three pairs of students. The participants are tertiary learners who had two online meetings, using Google Meet, to discuss their peer feedback on two texts they wrote for the course over one semester. Results indicate that online peer conferences offer a multimodal communication channel where different semiotic aspects play a role in mediating written input. In addition, WhatsApp was extensively used for informal peer feedback. We conclude with pedagogical implications and suggest actions so that learners can harness the full potential offered by these tools.

Design and construction of a social media corpus: Influencers’ speech in vlogs

This article outlines the creation of a social media corpus of Turkish vlogs on YouTube, aimed at analyzing the translanguaging practices and multimodal communication of Turkish social media influencers. It firstly describes the process of constructing the corpus, including transcription conventions and ad hoc annotation. The article then analyzes the phenomenon of translanguaging, with an emphasis on its prevalent forms and modes. Given the challenges associated with compiling a multimodally rich social media corpus, this paper provides strategies for manually transcribing and annotating linguistic and semiotic features in ELAN software, as well as strategies for managing tier-based annotations for vlog datasets. Additionally, the study presents approaches for handling non-standard linguistic codes and marked occurrences in language contact zones, illustrated through examples drawn from the vlog corpus where Turkish serves as the standard code.

Task-Oriented Multimodal Communication Based on Cloud–Edge–UAV Collaboration

Task-Oriented Multimodal Communication Based on Cloud–Edge–UAV Collaboration

Genç ve Yaşlı Yetişkinlerde Sözdizimsel Karmaşıklık ve Jest Kullanımı

Age-related effects are observed in both speech and gesture production. Older adults produce grammatically fewer complex sentences and use fewer iconic gestures than younger adults. This study investigated whether gesture use, especially iconic gesture production, was associated with the syntactic complexity within and across younger and older age groups. We elicited language samples from these groups, using a picture description task (N=60). Results suggested shorter and less complex speech for older than younger adults. Although the two age groups were similar in overall gesture frequency, older adults produced fewer iconic gestures. Overall gesture frequency, along with participants’ ages, negatively predicted grammatical complexity. However, iconic gesture frequency was not a significant predictor of complex syntax. We conclude that each gesture might carry a function in a coordinated multimodal system, which might, in turn, influence speech quality. Focusing on individual differences, rather than age groups, might unravel the nature of multimodal communication.

A Multimodal Study of How Pronunciation-Induced Communication Breakdowns are Managed During Tandem Interactions

This paper offers quantitative and qualitative findings from the exploration of communication breakdowns in English tandem interactions, by adopting a multimodal perspective. It focuses on the ways in which pronunciation-induced CBs are managed by language peers in a tandem setting. This study shows cases where it was the non-native participant’s output that was the main communicative stumbling block, with a view to reporting on pronunciation-induced breakdowns. More specifically, our analyses target the ways in which CBs are signaled to the interlocutor with different multimodal cues (verbal / vocal / visual). Those pronunciation issues are dealt with in a highly collaborative manner, through multimodal communication strategies, revealing recurrent visual patterns involving different visible body articulators (i.e., the face, the trunk, and the hands) which differ according to participants’ status (native versus non-native).

Snapshots of learning: exploring the meaning making potential of everyday visual literacy practices

ABSTRACT This article explores the meaning making potential of everyday multimodal text production from a group of students’ communication in Snapchat. By in-depth analysis of three examples, the characteristics of the snaps are explored, and the students are interviewed to find out what they thought about the texts’ function, content, and use. We find this vernacular communication being strongly shaped by the applications’ scripts and affordances, by usage patterns and by the context in which they are created. We also find that the students’ vernacular multimodal communication is time-effective, easy and intuitive, and mainly based on visuals. The vernacular communication thus differs clearly from the typical, verbal-based communication in school, but the article still argues there is some learning value in everyday digital communication. Being able to communicate adequately in visuals is found to be an important digital skill, developed informally, which can be significant for meaning making in school.

Retracted: Research on Spectrum Feature Identification of Indoor Multimodal Communication Signal

Retracted: Research on Spectrum Feature Identification of Indoor Multimodal Communication Signal

Methods of vehicle selection in the multimodal passenger transportation system

The article presents the results of the analysis of the methods of justification and choice of vehicles in the organization of long-distance multimodal passenger transportation involving water and related modes of transport. The analysis of the factors determining the need to use a combined approach in the organization of passenger transportation is carried out. Analytical and mathematical methods of selecting modes of transport are considered, including methods of multi-criteria optimization in the problem of selecting vehicles in the combined passenger transport system. An algorithm for selecting vehicles in the multimodal passenger transportation system has been compiled, which takes into account all of the above stages. The calculation of quantitative and analysis of qualitative indicators of the choice of the mode of transport with the participation of water transport and specific vehicles is carried out on the example of the organization of the passenger line "Nizhny Novgorod – Kazan – Samara" using various schemes of combining modes of transport. The proposed approach and algorithm for the complex solution of the problem ensure the coordination of the economic interests of the interacting modes of transport and increase the attractiveness of long-distance multimodal communication for passengers

The place of language in multimodal communication in humans and other primates

Much attention has been paid to ways in which different categories of individual combine different modalities to communicate meanings to others. One major challenge that remains is to gain a deeper understanding of the cognitive processing responsible for the simultaneous deployment and integration of various different resources during multimodal communication. In response to this challenge, a Modular Cognition Framework analysis will be applied to the question of how people communicate using all the resources at their disposal. The discussion will include relevant features of this framework. Then research into similarities between humans and chimpanzee communication will be discussed followed by a comparison between verbal and non- verbal communication and a consideration of a distinct but similar approach. Emphasis will be placed on the role of meaning and on synergies between the conceptual system and the two systems responsible for linguistic structure.

Comparative histology of the vocal sac in three species of hylid frogs with comments on its functional correlates


 
 
 
 The vocal sacs of frogs are elastic structures responsible for the circulation of air during vocalization, amplifying the sounds produced by these animals during multimodal communication. Vocal sacs present a wide array of morphologies among species and may be single, paired, or absent in adult males. Most studies on vocal sacs in Anura deal with their external morphology, and not with their internal structure, which has been explored in only a handful of species. The aim of this study was to assess vocal sac structure in three hylid species, Dendropsophus haddadi, D. elegans, and Scinax fuscovarius, using histological techniques. These species differ greatly in the degree of development and histological properties of the gular skin and submandibular musculature. In particular, elastic fibers are abundant in the thick m. interhyoideus and the relatively tight gular skin of S. fuscovarius. In contrast, in both species of Dendropsophus (although more evident in D. elegans), the m. interhyoideus is extremely thin and expanded, with a negligible number of elastic fibers that appear as a loose, pleated sheet when deflated. We analyzed videos of calling males of the three species and their close relatives, which show two different patterns of inflation/deflation. These patterns are strongly correlated with the histological properties of the vocal sac wall. The three species have different vocal sac shapes and rely differently on elasticity for vocal sac function.
 
 
 

A Multimodal Analysis on Rick Singer from Operation Varsity Blues: The College Admissions Scandal

Multimodal communication employs multiple semiotic modes to convey meaning. Operation Varsity Blues: The College Admissions Scandal is a documentary about Rick Singer, an education entrepreneur who committed the bribes. This study analyzed his representation through the oral and visual modes and was aimed at showing how multimodality could help reveal particular character representations. This study applies multimodal theory (Kress & Leeuwen, 2006) and semiotic modes (Chia & Chan, 2017). The oral monologues and dialogues, and the visuals of people, objects, and places were scrutinized to observe how Rick Singer was depicted. As a result, he was seen being portrayed as a wealthy and successful entrepreneur with many high-ranking connections, who strategically manipulated his clients and his university partners in a clever scheming maneuver. The interplay of both the oral and visual modes has resulted in a relatively complete picture of the main actor of this documentary.

Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey

Human–robot interaction (HRI) has escalated in notability in recent years, and multimodal communication and control strategies are necessitated to guarantee a secure, efficient, and intelligent HRI experience. In spite of the considerable focus on multimodal HRI, comprehensive disquisitions delineating various modalities and intricately analyzing their combinations remain elusive, consequently limiting holistic understanding and future advancements. This article aspires to bridge this inadequacy by conducting a profound exploration of multimodal HRI, predominantly concentrating on four principal modalities: vision, auditory and language, haptics, and physiological sensing. An extensive review encapsulating algorithmic dissection, interface devices, and applicative dimensions forms part of this discourse. This manuscript distinctively combines multimodal HRI with cognitive science, deeply probing into the three dimensions, perception, cognition, and action, thereby demystifying algorithms intrinsic to multimodal HRI. Finally, it accentuates the empirical challenges and contours preemptive trajectories for multimodal HRI in human‐centric smart manufacturing.

Multimodal analysis of disinformation and misinformation.

The use of disinformation and misinformation campaigns in the media has attracted much attention from academics and policy-makers. Multimodal analysis or the analysis of two or more semiotic systems-language, gestures, images, sounds, among others-in their interrelation and interaction is essential to understanding dis-/misinformation efforts because most human communication goes beyond just words. There is a confluence of many disciplines (e.g. computer science, linguistics, political science, communication studies) that are developing methods and analytical models of multimodal communication. This literature review brings research strands from these disciplines together, providing a map of the multi- and interdisciplinary landscape for multimodal analysis of dis-/misinformation. It records the substantial growth starting from the second quarter of 2020-the start of the COVID-19 epidemic in Western Europe-in the number of studies on multimodal dis-/misinformation coming from the field of computer science. The review examines that category of studies in more detail. Finally, the review identifies gaps in multimodal research on dis-/misinformation and suggests ways to bridge these gaps including future cross-disciplinary research directions. Our review provides scholars from different disciplines working on dis-/misinformation with a much needed bird's-eye view of the rapidly emerging research of multimodal dis-/misinformation.

Impact of Multimodal Digital Media Communication on Generation Z’s Language Use and Literacy Practices

The impact of increased use of digital communication mediated through social media was observed during COVID-19, in particular, on students’ language and its use in the classroom and their literacy practices. This led to an investigation of the use of multimodal digital communication in the language of Generation Z users at the focal university. Social-semiotic theory of multimodality in digital communication provided the theoretical framework for the study. A quantitative survey was done with 394 respondents on the frequency of use of different apps for different purposes, as well as students’ perceptions of the impact of social media on their literacy practices. Subsequently, qualitative interviews were done to gain a more in-depth understanding of the survey results. The results of this mixed-methods study indicate that Generation Z users are well aware of the affordances and constraints of different social media platforms and apps and use this knowledge judiciously for varied purposes and audiences in their digital communication. This has also impacted their crafting and interpreting of digital multimodal messages. The study findings have implications for teaching English (and other languages); similarly, other disciplines also need to take into account students’ changing literacy practices to enhance their learning outcomes.

Revealing spatial multimodal heterogeneity in tissues with SpaTrio

Capturing and depicting the multimodal tissue information of tissues at the spatial scale remains a significant challenge owing to technical limitations in single-cell multi-omics and spatial transcriptomics sequencing. Here, we developed a computational method called SpaTrio that can build spatial multi-omics data by integrating these two datasets through probabilistic alignment and enabling further analysis of gene regulation and cellular interactions. We benchmarked SpaTrio using simulation datasets and demonstrated its accuracy and robustness. Next, we evaluated SpaTrio on biological datasets and showed that it could detect topological patterns of cells and modalities. SpaTrio has also been applied to multiple sets of actual data to uncover spatially multimodal heterogeneity, understand the spatiotemporal regulation of gene expression, and resolve multimodal communication among cells. Our data demonstrated that SpaTrio could accurately map single cells and reconstruct the spatial distribution of various biomolecules, providing valuable multimodal insights into spatial biology.