Multimodal Language Research Articles

Meme virality and humour style: Exploring internet memes as a multimodal language in social media

The study only includes the randomly chosen students from one of the state universities in Tacloban City, Philippines, in its scope. This study takes into account every aspect of the students' individual personal lives, including their understanding and prior experience with internet memes. For this study, the researchers used qualitative research, especially the survey/interview method. This design will be used by the researchers to examine the influence that an internet meme has on an individual across all social media platforms. Studies that use randomization and the evaluation of interventions are known as qualitative research. The following factors contributed to the researchers' decision to randomize the respondents: (1) easy access to study participants, (2) large/universal sample size, and (3) to be able to identify trustworthy participants who, based on their own experiences, have sufficient knowledge of internet memes. Memes can be used to spread a particular message, serve as a platform for mass communication, and serve as a form of entertainment, according to the study's analysis of data from 26 carefully chosen participants. According to the research, it would be preferable for the reader to understand the meme's meaning because it would make it possible for them to more clearly understand the message or even just enjoy the meme. The reader will be able to understand the content as a result.

A multidimensional alignment sustainability model for language development: Evidence from L1 and L2 semio-semantic and semio-pragmatic markers

Language development is subject to its interaction and alignment with environments. However, how it interacts and aligns with environments necessitates further research, given current incompatible views and findings on language–context relations, particularly marker–context relations in multimodal or second language's (L2) sustainable development. Thus, this article proposes from the Ecolinguistic Continuum (Xiao, 2021) perspective a multidimensional alignment sustainability model (MASM) verified via instantiating a semio-semantic and semio-pragmatic marker continuum in the first language (L1, Chinese) and examining L1 (English) and L2 (English) written and spoken corpora-driven data. Results showed (1) a semio-semantic and semio-pragmatic marker continuum in both languages, ranging/evolving from the conceptually rich/explicit/formal to the partly conceptual/neutral, finally to the conceptually empty/implicit/informal, a process of uni-bi-multi-functions or grammaticalization-semio-pragmaticalization, and (2) the dis/similarities between L1 and L2 marker use distributions, pinpointing the multidimensional niches for languages' sustainable alignment evolvement/development. The findings corroborate the Ecolinguistic Continuum Paradigm, particularly the MASM, indicating that ideational/referential, structural, interpersonal, cognitive, and psychological functions/meanings of markers emerge dynamically depending on the extent to which they align with their corresponding environments. This view extends previous one-dimensional linguistic and context-related studies and helps unravel the problems in L1 and L2 sustainability development.

Intact speech-gesture integration in narrative recall by adults with moderate-severe traumatic brain injury

PurposeReal-world communication is situated in rich multimodal contexts, containing speech and gesture. Speakers often convey unique information in gesture that is not present in the speech signal (e.g., saying “He searched for a new recipe” while making a typing gesture). We examine the narrative retellings of participants with and without moderate-severe traumatic brain injury across three timepoints over two online Zoom sessions to investigate whether people with TBI can integrate information from co-occurring speech and gesture and if information from gesture persists across delays. Methods60 participants with TBI and 60 non-injured peers watched videos of a narrator telling four short stories. On key details, the narrator produced complementary gestures that conveyed unique information. Participants retold the stories at three timepoints: immediately after, 20-min later, and one-week later. We examined the words participants used when retelling these key details, coding them as a Speech Match (e.g., “He searched for a new recipe”), a Gesture Match (e.g., “He searched for a new recipe online), or Other (“He looked for a new recipe”). We also examined whether participants produced representative gestures themselves when retelling these details. ResultsDespite recalling fewer story details, participants with TBI were as likely as non-injured peers to report information from gesture in their narrative retellings. All participants were more likely to report information from gesture and produce representative gestures themselves one-week later compared to immediately after hearing the story. ConclusionWe demonstrated that speech-gesture integration is intact after TBI in narrative retellings. This finding has exciting implications for the utility of gesture to support comprehension and memory after TBI and expands our understanding of naturalistic multimodal language processing in this population.

Introducing “Trans~Resistance”: Translingual Literacies as Resistance to Epistemic Racism and Raciolinguistic Discourses in Schools

Translingual students’ identities transcend multiple languages and cultural allegiances. Sociolinguistics widely discusses the linguistic and racial oppressions these students face in schools due to epistemic racism, which is often observed in the tension between their multilingual and multimodal communicative styles and language perspectives rooted in monolingual and monocultural ideologies. This paper expands on the literature that denounces epistemic racism, uses Raciolinguistics and New Literacy Studies as theoretical frameworks, and reports on the following inquiries: What are the characteristics of delegitimizing school stakeholders who become agents of epistemic racism in their interactions with translingual students? How do translingual students reject these agents’ marginalization? Critical focus groups, semi-structured and arts-based interviews, and emplaced observations were used to collect data, centering the identities and voices of participants. Two key findings emerged. First, school stakeholders with various roles, social power, and degree of impact epitomize epistemic racism through ideological discourses. Second, “Translinguals” resist through novel concepts for which I have coined the terms "Covert and Overt Transresistance,” enacted by the means of resisting transliteracies. The theoretical, research, and practical implications of these findings, along with recommendations for future research, are discussed.

A Transition to Multimodal Multilingual Practice: From SimCom to Translanguaging

Historically, the field of deaf education has revolved around language planning discourse, but little research has been conducted on Deaf and Hard of Hearing (DHH) students with additional disabilities as dynamic multilingual and multimodal language users. The current study focuses on the language planning process at a school serving DHH and Deaf–Blind students with varied additional disabilities. A previous Total Communication philosophy at the school was implemented in practice as Simultaneous Communication (SimCom) and later revised as a multimodal-multilingual approach with the goal of separating American Sign Language (ASL) and English and using multimodal communication such as tactile ASL and Augmentative and Alternative Communication (AAC). To implement this philosophy without reverting back to SimCom, the school employed a language planning process using action research to reflect on cycles of improvement. A grounded theory approach was used to identify and analyze themes over a three-year period of language planning and professional development in multimodal communication. Triangulated data includes language planning artifacts and an online survey of staff perceptions—analyzed by coding concepts and categories, relating concepts to define translanguaging mechanisms and attitudes, and developing an overarching theory on how a school values translanguaging after 3 years of valuing complete access to language. In the context of a multilingual, multimodal language planning cycle, developing a shared language ideology guided by how Deaf, DeafBlind, and Deaf-Disabled (DDBDD) people use language emerged as an overarching theme that promoted dynamic languaging and understanding of strategies for effective communication.

Sensorimotor foundations of speech perception in infancy

The perceptual system for speech is highly organized from early infancy. This organization bootstraps young human learners' ability to acquire their native speech and language from speech input. Here, we review behavioral and neuroimaging evidence that perceptual systems beyond the auditory modality are also specialized for speech in infancy, and that motor and sensorimotor systems can influence speech perception even in infants too young to produce speech-like vocalizations. These investigations complement existing literature on infant vocal development and on the interplay between speech perception and production systems in adults. We conclude that a multimodal speech and language network is present before speech-like vocalizations emerge.

Multimodal language in child-directed versus adult-directed speech.

Speakers design their multimodal communication according to the needs and knowledge of their interlocutors, phenomenon known as audience design. We use more sophisticated language (e.g., longer sentences with complex grammatical forms) when communicating with adults compared with children. This study investigates how speech and co-speech gestures change in adult-directed speech (ADS) versus child-directed speech (CDS) for three different tasks. Overall, 66 adult participants (Mage = 21.05, 60 female) completed three different tasks (story-reading, storytelling and address description) and they were instructed to pretend to communicate with a child (CDS) or an adult (ADS). We hypothesised that participants would use more complex language, more beat gestures, and less iconic gestures in the ADS compared with the CDS. Results showed that, for CDS, participants used more iconic gestures in the story-reading task and storytelling task compared with ADS. However, participants used more beat gestures in the storytelling task for ADS than CDS. In addition, language complexity did not differ across conditions. Our findings indicate that how speakers employ different types of gestures (iconic vs beat) according to the addressee's needs and across different tasks. Speakers might prefer to use more iconic gestures with children than adults. Results are discussed according to audience design theory.

PORTRAYING HOW IMAGES ARE USED BY EFL TEACHERS IN THE TEACHING OF TEXTS

Multimodal language literacy generates social and contextual ties and refers to the study of language combining two or more modes of meaning. This study aims to investigate how EFL teachers utilized pictures as a meaningful resource. Data collection was carried out face-to-face between teachers and students at one of Junior High Schools in Garut. In this study, an analysis was carried out to determine whether the teacher could explain an image in a text that produced meaning. The results of this analysis found that teachers only mastered limited multimodal competencies. This shows that teachers use pictures only to help teach texts, but teachers do not have enough teaching materials to use for teaching texts and only focus on textbooks when using pictures to make meaning. Then, based on these findings, it is suggested that teachers increase their competency in explaining multimodal meanings in texts so that images are used not only to create understandable learning materials but also to generate more meaning from texts. Based on these findings, it is recommended that teachers improve their ability to explain multimodal meanings in texts so that images are employed not just to make understandable learning materials, but also to generate additional meaning from texts.

Cucumber disease recognition with small samples using image-text-label-based multi-modal language model

Few-shot learning methods only need a small size of samples to train a good model. Moreover, most of these methods consider a single modality, ignoring the correlation between multi-modal data. Therefore, using multi-modal methods to solve the small-sample-size problem has become the development trend of artificial intelligence. In recent years, a multi-model method called Vision-Language Pre-training (VLP) has emerged. The semantic relation between multiple modalities can be learned through pre-training, thus obtaining better performance on downstream tasks. Accordingly, this paper took cucumber disease recognition with small samples as an example and proposed a recognition method of a multi-modal language model based on image-text-label information. First, image-text multi-modal contrastive learning, image self-supervised contrastive learning, and label information were combined to measure the distance of samples in the common image-text-label space. Second, the classification methods and optimization of large-scale vision-language pre-training on small sample cucumber datasets were studied. The proposed model achieved a recognition accuracy rate of 94.84% on a small multi-modal cucumber disease dataset. Finally, some experiments on the public dataset demonstrated that our method has good generalization.

Digital reading of literary works in comics format during the COVID-19 pandemic

As social distancing was officially recognized worldwide due to the COVID-19 pandemic, the physical presence of teachers and students in the traditional classroom has been abruptly replaced by a digital environment in an online class. Visual literacy is directly related to understanding that visual information is an ornament of verbal information and a semiotic element that adds meaning to the text. Due to the multimodal language of comics, which provides a fruitful situation for constructing meaning during remote emergency learning, the relevance of the topic has been verified. Thus, this paper aims to present insights concerning the use of classic literature in comic book format. In this sense, the theoretical basis was built with the contributions of authors who corroborate the discussions presented here, anchored in the proposal of a didactic sequence carried out with elementary school students in the city of Barra do Garças - MT (Brazil), amid the pandemic context. The results revealed a change in the student's perception, even though the verbal language is predominant over the image. Regarding the classic literary works, it was noticed that the language change facilitated the understanding of the text read, making the reading enjoyable and exciting and making a bridge between past and present.

Contrastive Adversarial Training for Multi-Modal Machine Translation

The multi-modal machine translation task is to improve translation quality with the help of additional visual input. It is expected to disambiguate or complement semantics while there are ambiguous words or incomplete expressions in the sentences. Existing methods have tried many ways to fuse visual information into text representations. However, only a minority of sentences need extra visual information as complementary. Without guidance, models tend to learn text-only translation from the major well-aligned translation pairs. In this article, we propose a contrastive adversarial training approach to enhance visual participation in semantic representation learning. By contrasting multi-modal input with the adversarial samples, the model learns to identify the most informed sample that is coupled with a congruent image and several visual objects extracted from it. This approach can prevent the visual information from being ignored and further fuse cross-modal information. We examine our method in three multi-modal language pairs. Experimental results show that our model is capable of improving translation accuracy. Further analysis shows that our model is more sensitive to visual information.

Design and Construction of Intelligent Digital English Teaching Mode Based on Multimodal Low-Resource Language Information Processing

With the development of information technology, intelligent digital education is gradually emerging. At the same time, with the closer economic and trade relations between regions, the demand for English talents is becoming more and more urgent. Applying the concept and method of intelligent digital education to English Teaching (ET) is of great significance for improving students’ learning efficiency and academic level, and enhancing students’ English application ability and cross-cultural communication ability. However, due to the limitations of the traditional teaching model, such as the single and outdated model, the low interest of students in learning, and the lack of emphasis on cultivating students’ English practical ability, it can not effectively meet the requirements of improving students' learning effect.Therefore, this paper studies the intelligent digital ET mode based on multimodal low-resource language information processing, proposes the construction method of intelligent digital ET mode, and conducts experimental research on it. The research showed that the positive emotional input level index of Group X students was 8.47% higher than that of Group T students; the efficiency index of Group X students was 5.02% higher than that of Group T students; the autonomous learning ability and English learning level of Group X students were higher than those of Group T; the students in Group X had higher recognition of ET mode than those in Group T. The intelligent digital ET mode can effectively improve the learning effect.

Interactive attention and improved GCN for continuous sign language recognition

RGB sign videos are easily influenced by light, perspective, and clothing, and skeleton data might be an effective complement. The challenge of multi-modal sign language recognition lies in the fusion of different modalities and the feature extraction of each modality. Therefore, we propose a continuous sign language recognition method based on interactive attention and improved graph convolutional networks (GCN). For the skeleton stream, a hand shift decoupling GCN is proposed to improve the graph modeling capabilities by capturing the interaction between the two hand nodes through the shift of hand features and decoupling operation. Then a cascaded attention module progressively in space, time, and channel dimensions effectively extracts high-level semantic features. To make the best of the two streams’ features, an interactive attention mechanism is designed which takes the semantic features of the skeletons as queries and RGB features as keys and values, enhancing the spatial–temporal correlation mining ability and recognition performance of the network. The proposed method achieves competitive results on two public sign language datasets CSL and RWTH-PHOENIX-Weather-2014.

A multimodal human-robot sign language interaction framework applied in social robots.

Deaf-mutes face many difficulties in daily interactions with hearing people through spoken language. Sign language is an important way of expression and communication for deaf-mutes. Therefore, breaking the communication barrier between the deaf-mute and hearing communities is significant for facilitating their integration into society. To help them integrate into social life better, we propose a multimodal Chinese sign language (CSL) gesture interaction framework based on social robots. The CSL gesture information including both static and dynamic gestures is captured from two different modal sensors. A wearable Myo armband and a Leap Motion sensor are used to collect human arm surface electromyography (sEMG) signals and hand 3D vectors, respectively. Two modalities of gesture datasets are preprocessed and fused to improve the recognition accuracy and to reduce the processing time cost of the network before sending it to the classifier. Since the input datasets of the proposed framework are temporal sequence gestures, the long-short term memory recurrent neural network is used to classify these input sequences. Comparative experiments are performed on an NAO robot to test our method. Moreover, our method can effectively improve CSL gesture recognition accuracy, which has potential applications in a variety of gesture interaction scenarios not only in social robots.

Visual Communication: Exploring Representations of Vietnamese Culture in Advertising

Visual communication has received much attention from studies in the field of foreign language education. In the Vietnamese context, however, little research has been conducted to explore the meanings of Vietnamese cultural aspects in visual advertising. To fill such a gap, this article investigated visual meanings of Vietnamese cultural values represented in visual elements of common advertising in the Vietnamese market. The data were firstly selected from various logos and images of popular brands in the Vietnamese market and then interpreted based on the Iceberg Model for Interpreting Vietnamese Images (IMIVI framework) (Huynh, 2022). The results revealed the selected advertisements carried core Vietnamese cultural values in terms of visible and invisible elements. Specifically, culturally specific images such as flowers, animals, and traditional clothing were identified as visible factors. Buddhist, Confucian, and Taoist values were found as invisible factors in selected images of popular advertisements in the Vietnamese context. Findings reported this article might be helpful for research in visual communication, multimodality, and foreign language education. The outcomes of this article are also interesting for cultural studies, especially for the roles of culture in commercial advertising, intercultural communication, and foreign language education.

Study of Vocabulary Teaching Model in Senior Middle School from the Perspective of Multi-modal

Vocabulary is crucial to successful English language learning, which enables learners to develop their language skills and promote the comprehensive ability of language. Presently, with the rapid development of modern educational technology, multi-modal language teaching is emphasized and is used worldwide for media-assisted language learning. Multi-modal teaching advocates the proper use of multi-modalities,such as text typesetting, pictures, sound, video,gesture and body language ,etc. So as to stimulate students’ multiple sensory learning reactions. The study aimed to develop, implement multi-modal vocabulary teaching mode, used for enhancing English vocabulary acquisition. And it is expected to help the students understand English vocabulary better, remember vocabulary more efficient and to improve their comprehensive understanding ability of vocabulary.

Multimodal language in bilingual and monolingual children: Gesture production and speech disfluency

AbstractBilingual and monolingual children might have different styles of using multimodal language. This study investigates speech disfluency and gesture production of 5- and 7-year-old Turkish monolingual (N = 61) and Turkish–English bilingual children (N = 51). We examined monolinguals’ Turkish narratives and bilinguals’ Turkish and English narratives. Results indicated that bilinguals were more disfluent than monolinguals, particularly for silent and filled (e.g., umm) pauses. Bilinguals used silent pauses and repetitions (e.g., cat cat) more frequently in English than in Turkish. Gesture use was comparable across language and age groups, except for iconic gestures. Monolinguals produced more iconic gestures than bilinguals. Children's overall gesture frequency predicted disfluency rates only in Turkish. Different gesture types might be orchestrated in the multimodal system, contributing to narrative fluency. The use of disfluency and gesture types might provide insight into bilingual and monolingual children's language development and communication strategies.

Using multimodal language learning histories to understand learning experiences and beliefs of second language learners in Japan

AbstractThis study explores the possibilities of using digitally produced multimodal language learning histories (MLLHs) to understand the experiences of learning a second language (L2) of a group of university students in Japan (N = 21). The study observes that learners’ MLLHs are texts in which the visual elements of language, place, person, learning resource, and self‐analysis of learning process are represented visually. It also identifies 4 patterns in the ways the MLLHs are constructed by focusing on a certain type of visual element. Person‐oriented MLLHs focus on L2 learning as emotional experience. In resource‐oriented MLLHs, learning is perceived to occur because of engaging with language via favorite media. In analysis‐oriented MLLHs, learning is regarded as a matter of going up and down the scale of linguistic measures. Place‐oriented MLLHs emphasize being on site at an L2‐speaking region as being the key in the era of globalization and mobility. These patterns reflect learners’ beliefs on L2 learning as viewed from a contextual perspective. This study further argues that a multimodal approach can have a significant influence on making visible learners’ subjective perspectives and beliefs as lenses through which they frame their learning experiences, as well as visualizing the dynamics and individualities of learning.

Trends in Teaching Critical Reading in the Thai Context

This study identifies and reviews recent research on trends in teaching methods and teaching levels for teaching critical reading skills in Thailand. Thirteen relevant studies were analyzed, including academic documents and research published between 2011 and 2022. The data were organized into two categories including the teaching methods and teaching levels. The results revealed eight popular teaching methods for teaching critical reading skills, including literature-based instruction, the group reading strategy (GRS), the QUEST analysis method, the multimodal tasks method, the content and language integrated learning (CLIL) model, the program for international student assessment (PISA) reading literacy assessment method, the applied linguistic-based method, and the survey, question, read, recite, record, react, and review (SQ5R) method. Moreover, critical reading is taught at three levels in Thailand: the university level, secondary school level, and elementary school level. This study is for those interested in finding an appropriate method for teaching critical reading in the Thai context.

Detection of Dyslexic Children Using Machine Learning and Multimodal Hindi Language Eye-Gaze-Assisted Learning System

Children with dyslexia need specific instructions for spelling and word analysis from an early age. It is important to provide appropriate tools using technology for writing aids to such children that can help them to input text, while providing multiple feedback. However, it is unclear how children with dyslexia can efficiently use a gaze-based virtual keyboard (VK). In this study, we propose to use the typing performance of a multimodal Hindi language eye-gaze-assisted learning system based on a VK to help in the reduction of tracking errors for people with writing and reading deficiencies and to detect children with dyslexia. Performance was assessed at three levels: eye tracker, eye tracker with soft switch, and touchscreen as a baseline modality using a predefined copy-typing task. The system was validated through a series of experiments with 32 children (16 dyslexic and 16 control). The results show that the workload and the usability of the system are substantially different for children with dyslexia. Children with dyslexia have a lower typing performance when using the touchscreen modality or the eye tracker only. The detection of children with dyslexia from others was assessed with seven different types of classifiers using the typing speed on different words (AUC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 0.9). These results highlight the need to have fully inclusive VKs. This work demonstrates the superior use of a multimodal system with participants having unique neuropsychological conditions and that the proposed system can be used to detect children with dyslexia.