Multimodal Texts Research Articles

The need to strengthen literacy in Indonesian language learning requires teachers to be innovative in designing assessment instruments that are relevant to current developments. One approach that can be used is the application of multimodal texts, namely texts that combine various communication modes such as writing, images, sound, and symbols to convey meaning more comprehensively. This community service program was carried out with the aim of improving the competence of Indonesian language teachers at the junior high school level throughout Sragen Regency in developing literacy questions based on multimodal texts that align with the demands of the Independent Curriculum. The activity was carried out through two months of intensive training and mentoring, which included a theoretical understanding of the concept of multimodal texts, analysis of sample questions, and practice in developing literacy-based assessment instruments. The results of the activity showed a significant increase in teachers' understanding of the principles of multimodal texts and their ability to design questions that challenge students' critical and creative thinking skills. Through this program, teachers not only gained new knowledge but also were motivated to be more adaptive to developments in digital media and innovation in learning. Thus, this activity contributed to improving the quality of Indonesian language learning that is contextual, interactive, and meaningful for students.

UNESCO (2023) has called for educators to use pedagogical approaches that prioritize human agency and a responsible interaction between humans and generative artificial intelligence (GenAI). The present study, therefore, sought to design and implement an innovative 5-stage GenAI-based project with Sports Science English for Specific Purposes (ESP) students. Specifically, the objective of the project was for students to create multimodal texts for promotional purposes while developing both critical thinking skills and vocabulary range. Upon completing the project, 42 students participated in a survey, analyzed with both quantitative and qualitative methods, to gauge the impact of the project. Regarding critical thinking skills, 95.2% of students revealed they learned it was important to use these skills when using chatbots, emphasizing the need to verify GenAI-generated content and external sources a chatbot uses. Concerning vocabulary learned, 85.7% of the students, irrespective of their self-perceived level, reported learning useful topic-based and general vocabulary, thus strongly implying generalized intentional and incidental vocabulary learning. The study’s main conclusion is that GenAI-based projects, when designed to primarily assess student critical reasoning, can create optimal conditions for vocabulary acquisition and critical thinking skill development. This approach can also have significant implications for GenAI-related teaching practice and student assessment. Keywords: Critical thinking; Vocabulary learning; GenAI; Project-based learning; ESP; Multimodal text creation

The problem of the research. The main requirement for future professionals in many fields of activity is to be able to creatively solve assigned problems, often related to the perception and interpretation of a large amount of information in modern socio-economic situations. These qualities are especially necessary for a teacher who participates in the development of the students’personality. Identification of individual markers of thinking and development, based on them, of the universal model for the development of students’creativity in the system of higher foreign language pedagogical education provides the development of creative abilities and meta-subject skills of creative information processing. The study was aimed at developing and testing the implementation efficiency of the models of students’creativity development in the context of higher foreign language pedagogical education. Methods of investigation. In the study the following approaches, methods and techniques were used: bibliometric, ideographic, etymological and hermeneutic approaches; psychological tests by J. Bruner and by E.P.; methods and technologies for developing creativity by M. Michalko, M. Rowdy, T. Seelig, T. Brown, K. Sawyer, E. de Bono, W. Disney's method; determination method of the level of information and communication competence development by O.N. Ionova. KEYWORDS The results of the study. The post-experimental analysis demonstrated an increase in the level of development of specific markers of thinking and creativity among students in years 2-5 from 40.6% to 59.25, which is an increase of 19% (L = 155,1; p < 153). Conclusion. The objectives of the study were confirmed in the theoretical justification and practical application of the developed model for the development of designated specific markers of thinking and creativity in classes on a practical French language course for students of 2-5 years of training in the direction of training «Pedagogical education (with two profiles)», specialization «Foreign languages». The developed model was successfully tested during the academic year and is used in the education system, taking into account the need for in-depth development of the designated markers of thinking and creativity.

Abstract This paper explores the multimodal communication strategies of German and French Guerrilla gardening activists on social media. Guerrilla gardening is a phenomenon that originated in New York during the 1970s and has since spread worldwide. In the interests of (re)greening the urban space, the representatives of this Guerrilla gardening movement plant vegetation in different places without official approval. In addition to this, the activists present now their actions on social media in order to increase their reach and convince other people of their cause. By using social media, activists bridge the gap between the online sphere and the physical world, leveraging new possibilities for multimodal communication. The study analyzes a corpus of 304 posts (French/German, X and Instagram). Building on the principles of multimodality theory and contrastive discourse analysis, it investigates the features of social media posts as multimodal texts, focusing on the relationships between the different semiotic codes – particularly written text and static images – in relation to their pragmatic functions. The three-level intersemiotic analysis (micro, meso, macro) reveals limited language/culture-specific multimodal preferences, with variations mainly due to platform-specific communicative styles, suggesting a trend towards globalized design principles.

This study explores how former child soldiers in Colombia narrate identity repair following their experiences of conflict. Twenty-five adolescents and young people (aged 14-19) participated in participatory life-history workshops using a creative auto/biography (CAB) method. A narrative analysis of over 200 first-person multimodal texts identified four recurring plots- the Struggler, Learner, Advocate, and Survivor-each reflecting distinct, non-sequential pathways of self-reconstruction, emotional processing, and social re-engagement. These plots show how young people reimagine self and negotiate belonging amid stigma, rupture, and transition. The study proposes a Narrrative Identity Repair Compass comprising six interrelated domains: narrative coherence, self-concept (self-image and self-worth), self-efficacy (capacity to act on and shape circumstance), temporal orientation, relational positioning, and cognitive processing (narrative work integral to change). This multidimensional model conceptualises identity repair as fluid, non-linear, and relational. Post-traumatic growth (PTG) defined here as positive psychological change emerging from the struggle with major adversity is reframed as a contingent possibility shaped by reflection, relationships, and creativity. This framework offers a developmentally appropriate and context-sensitive approach to understanding how former child soldiers reconstruct meaning, repair identity, and move toward psychosocial integration following experiences of armed violence.

Abstract This paper argues for a multimodal ‘phonology’, a ‘phonology’ which returns to the original meaning of the word ‘phone’ as referring to ‘voice’ as well as ‘sound’ in general, and which can be applied to speech, music, as well as other sounds. Three areas are discussed. The first is rhythm. Systemic functional linguistics has reduced the role of rhythm to the ‘foot’, but others see rhythm as also playing a significant role at higher ranks and as the fundamental organising principle of all time-based multimodal texts. The second is voice quality or, more generally, sound quality, as a key resource for expressing identity as well as fleeting states and emotions. Based as it is on qualities common to all sounds, this too applies to all sound-based semiotic modes. The third is the meaning-making potential of pitch contours, which, the paper argues, primarily contribute to the communication of emotion – again in speech, music, as well as other sounds.

Stylistic analyses of negation have traditionally and predominantly focused on linguistic texts due to lack of a well-defined tool for investigating negation in multimodal texts. To fill this methodological gap, the present study integrates the critical stylistics tool of negation in written texts with Kress and van Leeuwen’s (2006) framework of visual analysis to develop a tool for the analysis of negation in multimodal texts. This tool is named the multimodal textual conceptual function of negation (MTCFN) and is used to explore how multimodal meanings of negation are constructed in Princess Diana Panorama interview, broadcasted in 1995. The analysis revealed that the co-occurrence of language and images in the same text creates a co-text that regulates and determines the meanings of negation produced by both semiotic systems. The combination of the visual affordances of gaze direction, head tilts, and different shot types and angles helps reinforce and make coherent the meanings initiated through the verbal medium, thus creating a coherent and impactful multimodal narrative. The study concludes that stylistics holds significant potential for informing approaches to the analysis of multimodal texts and recommends that further research is carried out on other multimodal text types to test the explanatory adequacy of the proposed MTCFN tool.

This study aims to describe the use of multimodal text media in folklore learning in grade V students of SD Negeri 88 Pinrang, including the application of media, student responses, and obstacles faced by teachers and students in learning. This study uses a type of qualitative method with a qualitative descriptive design. Data collection is carried out through observation or observation in the learning process, using media use checklists, interviews, and using student response questionnaires. The data analysis techniques used are data reduction, data presentation, and conclusion drawn. The results show that the use of multimodal text media, such as images, sounds, and videos, can attract students' attention, increase motivation, and help them understand the cultural values contained in folklore. Limited infrastructure, devices, and networks are some of the challenges encountered. However, it is evident that multimodal text media increases students' interest, active participation, and understanding of the lessons. This research is expected to contribute to the development of more innovative and contextual learning methods in elementary schools.

Technology has significantly transformed how texts are imagined and produced, leading to the emergence of multimodal texts such as films, TV talk shows, TV advertisements, social media platforms and books, among others. The communicative purposes of these texts highly depend on the integration of social semiotic modes. TV talk shows, as one form of such texts, showcase a complex interplay of these resources. This paper presents an analysis of the actions and interactions of semiotic modes in the Jeff Koinange’s talk show—JK Live, a prime-time televised program hosted by Jeff Koinange on Citizen TV, Kenya. Four episodes of the show were purposively selected and downloaded from Citizen TV’s YouTube channel for content analysis. Observation and focus group discussions were used as data collection instruments. Guided by Kress and van Leeuwen’s Multimodal Discourse Analysis, the study investigated how the host, guests and production team, as the participants in the show, strategically orchestrate space, speech and visual acts as multimodal resources to fulfil communicative functions within the show’s discourse. The findings of this study demonstrate how communication in the JK Live show is not merely verbal but depends on the interaction of other semiotic modes utilised by participants to shape the different. The findings contribute to the understanding of TV talk shows as highly interactive multimodal texts.

The purpose of this study was to compare and analyze the gaze movement patterns of sixthgrade elementary school students while reading multimodal texts—graphic novels and picturebooks—in order to investigate their reading strategies and meaning-making processes. A crossover experimental design was employed in which the same participants read both types of texts. Quantitative and qualitative data were collected using an eye-tracking device, focusing on gaze path, heatmap, and areas of interest (AOI), and were supplemented by retrospective think-aloud interviews. The analysis of gaze paths revealed that during graphic novel reading, students followed a structured pattern in line with the sequential flow between speech balloons and images, indicating a text-centered information processing tendency. In contrast, during picturebook reading, students exhibited more fluid gaze patterns led by image exploration, highlighting the construction of meaning through the interaction between visual and verbal information. Heatmap analysis showed strong fixation on speech balloon texts and characters’ faces in graphic novels, whereas in picturebooks, fixations were concentrated on text and images marking narrative transitions. In AOI analysis, the text area in graphic novels showed the highest dwell time and number of fixations, while in picturebooks, significant gaze distribution was also observed in image areas related to the text. Interpreting these results through Serafini’s (2012) reader resource theory for successful multimodal text reading, readers of graphic novels primarily activated roles as "navigators" and "designers," whereas readers of picturebooks more prominently exhibited the roles of "interpreters" and "navigators." This study provides an empirical, process-oriented approach to multimodal text reading and offers meaningful implications for reading education and research in multimodal literacy by identifying genre-specific differences in information integration strategies.

The proliferation of information-sharing platforms and the ease of access to diverse resources have led to an overwhelming volume of multimodal data that is increasingly difficult to process effectively. The integration of multiple data types, including text, images, video, and audio, highlights the growing importance of Multimodal Text Summarization (MMTS). Collecting and synthesizing existing research on this topic can provide a comprehensive foundation for advancing the field. Following a Systematic Literature Review (SLR) methodology, we addressed three pivotal research questions concerning methodologies, evaluation measures, and datasets in MMTS. Through a systematic analysis of 132 papers , we examined the strategies employed to address MMTS challenges, assessed the evaluation methods used to quantify performance, and compiled a detailed list of available datasets along with their limitations. This review offers critical insights and identifies future research directions, aiming to inform and guide continued innovation in this dynamic and evolving domain.

Pneumonia is a prevalent and serious respiratory disease, responsible for a significant number of cases globally. With advancements in deep learning, the automatic diagnosis of pneumonia has attracted significant research attention in medical image classification. However, current methods still face several challenges. First, since lesions are often visible in only a few slices, slice-based classification algorithms may overlook critical spatial contextual information in CT sequences, and slice-level annotations are labor-intensive. Moreover, chest CT sequence-based pneumonia classification algorithms that rely solely on sequence-level coarse-grained labels remain limited, especially in integrating multi-modal information. To address these challenges, we propose a Multi-modal Text-Guided Network (MTGNet) for pneumonia classification using chest CT sequences. In this model, we design a sequential graph pooling network to encode the CT sequences by gradually selecting important slice features to obtain a sequence-level representation. Additionally, a CT description encoder is developed to learn representations from textual reports. To simulate the clinical diagnostic process, we employ multi-modal training and single-modal testing. A modal transfer module is proposed to generate simulated textual features from CT sequences. Cross-modal attention is then employed to fuse the sequence-level and simulated textual representations, thereby enhancing feature learning within the CT sequences by incorporating semantic information from textual descriptions. Furthermore, contrastive learning is applied to learn discriminative features by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Extensive experiments on a self-constructed pneumonia CT sequences dataset demonstrate that the proposed model significantly improves classification performance.

This qualitative action research case study explored how a blended literacy learning intervention combining the flipped classroom model with youth-selected multimodal texts influenced sixth-grade Academic Intervention Services (AIS) students’ comprehension of figurative language. The study was conducted over four months in a New York State middle school and involved seven students identified as at-risk readers. Initially, students engaged with teacher-created instructional videos outside of class and completed analytical activities during class time. However, due to low engagement and limited comprehension gains, the intervention was revised to incorporate student autonomy through the selection of multimodal texts such as graphic novels, song lyrics, and YouTube videos. Data was collected through semi-structured interviews, journal entries, surveys, and classroom artifacts, and then analyzed using inductive coding and member checking. Findings indicate that students demonstrated increased the comprehension of figurative language when given choice in both texts and instructional videos. Participants reported increased motivation, deeper engagement, and enhanced meaning-making, particularly when reading texts that reflected their personal interests and experiences. The study concludes that a blended literacy model emphasizing autonomy and multimodality can support comprehension and bridge the gap between in-school and out-of-school literacy practices.

This study investigates how racialized youths in Hong Kong challenge the intertwined systems of linguistic racism and nationalized identities through the production of counternarratives and critical, multimodal texts. Drawing on data from ‘Antiracist Language and Literacy Workshops’, where participants engaged in collaborative dialogues and creative meaning-making through multilingual and multimodal texts, the study reveals how these participants disrupted dominant discourses that positioned them as deficient outsiders in a society shaped by colonial legacies and ethno-linguistic hierarchies. Their counternarratives expose the normalization of linguistic racism and entrenched social hierarchies that often marginalize racialized youths, engendering feelings of disempowerment and exclusion. However, through collaborative dialogues and creative counter-text production, participants cultivated critical consciousness, emotional resilience, and agency to challenge and disrupt these oppressive discourses. Their acts of resistance are not simply individual expressions of ­frustration—they are profoundly antiracist and decolonial interventions that expose and contest the systemic marginalization of racialized and linguistically diverse students. This research advocates for pedagogical approaches that center student voices, affirm diverse identities, and empower learners with the knowledge, skills, and critical awareness necessary to confront systemic inequities, ultimately contributing to more inclusive, equitable, and socially just educational environments.

Early recognition of autoaggressive tendencies in young people is essential for diagnostic screening and reducing suicidality risks. This can be achieved through psycholinguistic approaches such as corpus analysis and eye-tracking studies. Corpus research helps to develop generalized speech patterns of those at risk of suicide, while oculographic methods examine perceptual cues linked to suicidal tendencies. We formulated an algorithmic framework for constructing verbal, visual, and multimodal material to identify autoaggressive tendencies among youth. The stimuli material was created following the idiolect paradigm of forensic authorship attribution. The first stage involved analyzing corpus data including materials from social networks and social media, the Rusentiment database, and a text collection from the Privolzhsky Research Medical University. Python's NLTK and SpaCy libraries for automated text processing were used to extract corpus statistics, n-grams, keywords, and collocations for identifying linguistic markers of autoaggression. Keywords were statistically ranked using Log-likelihood, T-score, and mutual information, while collocations were derived via T-score analysis. Sentiment analysis for the Dostoevsky Python library and stylistic indices (lexical diversity, readability) were also applied. The total analyzed material comprised more than 100 million tokens. We next integrated, stimulus and filler materials into an eye-tracking application (developed by LLC Lad IT Group) using standard laptop video cameras. Oculographic data quantified gaze delay differences via a percentage excess formula to pinpoint the most diagnostically relevant stimuli. In two iterations of the pilot experiment, 66 youths from the control group and 29 from the target group participated in the oculographic experiments. In multimodal texts, most stimuli derived from corpus statistics were relevant, and all individuals in the target group showed a prolonged gaze delay; visual stimuli (pseudo-self-portraits, anime/game characters) elicited 26-36% longer gaze delay in the target group. Verbal stimuli analysis revealed prolonged gaze fixations on self-referential pronouns (12-25%) and metaphorical death expressions, although direct terms, like "suicide" showed the gaze avoidance (-11.9 to -129% deviation). We then developed a system of weighted coefficients for an automated diagnostic model. The algorithm showed 72 % accuracy in identifying autoaggression, presenting a promising tool for early diagnostic screening of this phenomenon. The present methodology focuses on creating and employing a novel selective dataset consisting of visual, linguistic, and multimodal text stimuli integrated into the oculographic examination protocol. The oculographic detection of eye movement perceptual cues in response to exposure to the stimuli dataset may identify objective markers for evidence-based diagnostics of mental disorders (e.g., depression) and fundamental psychopathological phenomena (e.g., suicidality), including at-risk states (e.g., autoaggression). Furthermore, this approach may contribute to the enhancement of suicide prevention programs, particularly targeted interventions for the vulnerable population of young people who experience autoaggressive tendencies (i.e., self-aggression).

Abstract Sound sculptures can serve as resonant mediums of remembrance, reawakening historical soundscapes and evoking memories through the use of sound. In this way, they enable a sensory connection to the past. This paper explores how sound functions as a semiotic resource, interacting with other modalities within sound sculptures and the contexts in which they are placed. It examines how memory can be accessed through the sense of hearing and investigates cases where sound memorializes historical events. I draw on a social semiotic approach, particularly Günter Kress’s theory of multimodality, considering sound sculptures as multimodal texts and multisensory objects. Complementing this, I employ sound theory and acoustic ecology studies, primarily the works of Raymond Murray Schafer, which provide a language for describing the sound modality within sound sculptures. To ground this exploration, I analyze sonic objects and sonic events in the sound sculptures by artists Bill Fontana, Markus Kison, and Nikita Kadan through the lenses of the acoustic ecology framework. These works exemplify how sound can operate as both a symbol and a mnemonic trigger, how its sensory and emotional dimensions contribute to memory-making processes.

This study explores the cognitive and discursive features of media texts created at the intersection of cinema and interactive game design in the digital age. The research focuses on hybrid media products—interactive films, visual novels, and story-driven video games that combine cinematic linearity with gamified non-linearity, engaging users in active narrative construction. The paper examines meaning-making mechanisms triggered by multimodal representation and the cognitive processes involved in interpretation, emotional identification, and audience engagement. Special attention is given to phenomena such as framing, scenario activation, mental modeling, performativity of utterances, the effect of presence, and moral reflection within user interaction with the media text. The analysis is situated within the broader context of media discourse transformation and the evolving role of language in digital storytelling. Methodologically, the study applies an interdisciplinary approach that integrates cognitive linguistics, narratology, critical discourse analysis, and semiotics. The novelty of the research lies in its conceptualization of media text as a dynamic cognitive-discursive system emerging from the interaction between the author, interface, media environment, and user. Unlike traditional models, the study views interactive media texts as co-authored structures, where users not only interpret but actively shape narrative meaning. It is shown that interactive formats activate complex mental models and decision-making trajectories, supported by moral evaluation, emotional engagement, and narrative polyphony. The findings offer new insights into the transformation of communicative strategies and meaning-construction mechanisms in digital culture and are valuable for future work in media linguistics, digital storytelling, and the cognitive analysis of multimodal texts.

Reading is a fundamental tool for gleaning information and engaging in many fields of literacy. However, poor reading outcomes among Indonesian students on the PISA test indicate a low literacy level. The reading ability referred to in this study is the ability to understand reading texts on various topics and forms. This study used a sample of 234 students from eight schools in North Kalimantan Province. The data were obtained from reading tests carried out by students via computers. The test consists of 40 multiple-choice questions, short answers, and essays. The data was processed in several stages, including data reduction and item calibration using the Rasch Model. Apart from reading tests, data were also obtained through questionnaires to gather information about literacy activities carried out by students at school and at home. The results showed that nationally the average reading literacy ability of students in North Kalimantan Province was at a low level, namely at score 349, while the highest reading literacy ability average score in Indonesia was 489. The results also showed that there was a relationship between students' reading literacy skills and their conditions and habits at home. As a recommendation, students need to be more accustomed to reading multimodal texts through multimedia devices. It is important to make them capable of understanding texts containing not only words, but also pictures, numbers, graphs, and tables as demands for modern literacy. Moreover, parents also need to be to encourage to help improve the students’ literacy capacity.

IntroductionScientific literacy in early education depends on both content knowledge and young learners' ability to organize and express scientific ideas across linguistic and visual-graphical modes. This integrative ability, known as multimodal competency, enables children to construct and communicate scientific understanding by coordinating text structure, language, and visual representation.MethodsIn this study, 1,705 science writing samples from 1,008 students in kindergarten through 2nd grade were analyzed using a two-stage quantitative approach. First, a content analysis was conducted in which each sample was systematically coded for text structure, linguistic and visual-graphical modes, and scientific content. These coded features were then subjected to latent class analysis (LCA) to identify distinct typologies of multimodal text production.ResultsLCA identified six typologies of multimodal composition, ranging from simple unlabeled visual texts to strategically integrated complementary compositions. These typologies capture meaningful variation in how students organize and communicate scientific ideas, reflecting differences in metacognitive strategy use, modality coordination, and sensitivity to the communicative functions of linguistic and visual modes.DiscussionFindings indicate that young children's multimodal texts act as both communicative and epistemic tools, supporting scientific meaning-making. Rather than requiring explicit instruction, just-in-time scaffolds may help children make strategic representational choices, reinforcing the role of multimodal composition in early science learning.

Abstract Climate change impacts and stresses young people and although their pro-environmental behaviours have been studied their perspectives have not been widely heard. This creative output is a lo-fi comic engaging with themes of imagined alternative futures in climate fiction. It was constructed to provide an example of a multimodal text with a low barrier to entry for use in the classroom, to complement the study of solar punk texts. The methodology of an autoethnographic art provides a tool for reflection and provides a suitably rebellious outlet for their perspectives, a departure from factual poster assignments on environmental issues. This particular perzine discusses the challenges faced by young people in addressing environmental issues and sustainable practice with limited personal agency.