Multimodal Text Analysis Research Articles

Revising Writing Assignments in Response to Generative AI

The author describes how she revised writing assessments in the university’s first-year writing sequence to emphasize rhetorical analysis of multimodal texts, prompts to which generative AI and ChatGPT struggle to respond.

Linguistic means of expressing communicative strategies in non-cooperative discourse (based on the material of English-language talk shows

Every year there is a rapid development of the media sphere, including its visual and discursive component, which contributes to the development of the "digital language" and the "language of television", the parallel development of YouTube journalism as a media phenomenon, which are evolving along with all the ongoing processes. Any language is susceptible to this, especially English, which is the most widespread and popular in the world. The study of linguistic issues of the language of television shows, the language of talk shows in line with the anthropological paradigm of modern linguistics in its broadest sense (issues of the functioning of language in various types of discourse, journalism and PR, media linguistics and political linguistics) and its central concepts such as the manipulative influence of language, linguistic personality, pragmalinguistics of language and text as part of it are an integral part of both the theory of language and the linguodidactic component of foreign language teaching (English as a foreign language). Taking into account the current conflicting political situation in the world, aggravated political and armed conflicts, the study of non-cooperative discourse in the media of English-language talk shows is certainly of scientific novelty. The main content of the research is concentrated around the analysis of the concepts of "non-cooperative discourse", "PR-discourse" from a linguistic point of view. Media linguistic analysis, discourse analysis, the method of multimodal text analysis, etc. are used as research methods. Results suggest that the concept of non-cooperative discourse is revealed (based on the material of English-language talk shows). Even so, the present study concludes that the approach is worthwhile and promising for further research.

Student Teachers as Co-Creators when Elaborating on a Model for Multimodal Text Analysis

In this paper we present our work process, methods and concerns in developing and implementing a model for analysis of multimodal texts. Using a design-based research approach and a multimodal social semiotic framework on communication, we investigated three different groups of students using video recordings and participatory observations. The results revealed the development of the model through three different design cycles into the current version and how students contributed to this process. In addition, we examined how the design process can be understood from the concept of student agency and how it contributes to the understanding of theory. The findings show that by applying a design process, students are offered student agency while creating an opportunity for a joint exploration between students and teachers, which is a rewarding experience that contributes to the development of all participants.

Meaning-Making in Ecology Education: Analysis of Students’ Multimodal Texts

Teaching and learning in ecology depend on multimodality, involving semiotic resources such as visual representations, subject-specific symbols, and written and spoken language. Furthermore, the ecology field involves complex processes and relationships, presenting student challenges. However, more research has yet to investigate how students design multimodal texts to represent complex biological processes. For a holistic understanding of ecology, it is crucial to understand different complex processes, such as the matter cycle, energy flow, decomposition, and their relations. Therefore, this study aims to, through multimodal text analysis based on systemic functional linguistics (SFL), identify how secondary students collectively present and combine such processes and how they position themselves through their textual choices. Results indicate that representing biological processes comprises several challenges for students. One way in which this is shown is the unclear use and meaning of arrows. Thereto, the students include various aspects uncommon in the field of ecology, for example, symbols inspired by comic books, values, and the role of humans, thereby relating ecosystems to their interests and everyday life. Implications for teaching are discussed, for instance, the importance of supporting students in terms of scientific content and how to represent it, which can be conducted through text discussions.

Integrated Learning-based Multimodal Text Analysis Algorithm for College English Textbooks

Integrated Learning-based Multimodal Text Analysis Algorithm for College English Textbooks

Мультимодальні тексти сугестивної спрямованості в інформаційному просторі України воєнної доби

The article presents an analysis of multimodal texts with a suggestive orientation in the information space of wartime Ukraine. These texts effectively influence society in a positive way and objectify the narratives that are emotionally rich in description. Multimodal patriotic texts are broadcast through classical and new media, as well as social networks, combining differently encoded information (visual, verbal, and musical text). It has been found that the basic and most important means of suggestive influence is the rhythm of the text, which is created by means of different codes. Rhythm permeates all levels of the text and ensures the result of the recipient's psycho-emotional perception of certain suggestively marked meanings. Rhythm as a proportional alternation of certain elements of the text is "comprehended" by the addressee primarily on the emotional level; the rhythmic structure of the text implies "text assembly", which begins with the unification of the relevant content parts and ends with the creation of a new content unit, i.e. the "text concept". It has been proved that multimodal texts based on a complex of suggestive techniques are associated with the so-called "iconic turn" in communication, i.e. focus on various kinds of visual components of the communication process. The interaction of codes creates a unique information product, the semantics of which cannot be conveyed by means of only one of the codes. Multimodal texts with a suggestive orientation have an extremely wide audience; they represent a quick response to certain wartime crises; they use artistic means to outline the prospects for the wartime crisis development; they embody the relevant emotions of a mentally traumatized society; they objectify the content of events in a vivid and imaginative way; they appeal to the cultural roots of the nation and make extensive use of folklore elements; they enhance the psycho-emotional state of the society and provide multiple reproductions, exercising a long-lasting suggestive influence.

National aeronautics and space administration on IGTV: Multimodal discourses of space

This study employed critical analysis of multimodal texts to examine how NASA utilizes multiple modes like text, visuals, font, and sound to communicate space on a modern multimodal platform—Instagram. Four main themes emerged in the analysis of 34 NASA IGTV videos: space as a scientific aim, space as a scientific means, space as a U.S. territorialization effort, and space within the general public’s reach. Multimodal discursive practices enabled NASA to produce institutional discourses and disseminate meanings of space that enacted and reinforced gendered and racial exceptionalism, technological and economic space elitism, and hegemonic territorialization operationalized through space appropriation and commodification. While visuals, text, and auditory mode in many cases worked together to create meanings, the use of visuals communicated that space-related accomplishments are still achieved in white and male environments. Thus, the unpacking of multimodal discursive practices calls for more holistic approaches that address multimodal arrangements and an inclusive approach to the roles of individual modes in meaning-making.

Multimodality and transmediality in Kamal Abdulla’s short fiction: a cognitive-emotive interface

This paper addresses the issues of in-built multimodality and transmediality as well as their interface employed in “Could You Teach Me to Fly…?”, a short story by Kamal Abdulla, a well-known Azerbaijani writer, scholar, and public figure. Relying upon the cognitive-emotive approach as the ground for multimodal text analysis, the research interprets the above concepts as interphenomena, which, along with iconicity, intermedial references, and manifestations of verbal holography as the interplay of planes and vectors, create the effect of literary text multidimensionality. The paper claims that the short story that belongs to intellectual prose foregrounds the metaphor of love as a magic gift that endows a person with capacity to fly. This metaphor is embodied in the iconic image of a white bird the woman in love turns into. The paper shows that the magic of imagery based on fairytale and mythopoetic motifs reveals itself through a set of visual, auditory, and kinesthetic manifestations of multimodality accompanied by the use of zoom-in/zoom-out cinematic techniques. The magic of paradoxical imagery, where a naked woman symbolizes an emotionally intense silence, is enhanced by discourse transmediality, due to which the key visual image of the woman-bird flying high into the sky as if evaporating transforms into an integrated kinesthetic poetry-dance-film image. Given all this, the paper suggests several techniques of cognitive-emotive multimodal analysis, which might further enrich the metamethod of literary text disambiguation as a way of its interpreting aimed to reconstruct a literary work’s conceptual structure while defining the factors of textual multidimensionality and deepness.

Developing multimodal communicative competence: adolescent English learners' multimodal composition in an after‐school programme

AbstractThis qualitative study aimed to investigate adolescent English learners' multimodal communicative competence. Grounded upon a social semiotic approach with the concepts of affordances and transmediation, this research was carried out in an after‐school programme at a secondary school in Korea. Three high school students participated in the multimodal composition project and produced narrative and argumentative multimodal texts. Multiple sources of data were gathered from interviews, video recordings with fieldnotes and student artefacts. The data were analysed using an inductive and deductive approach, along three orientations: modes as social action; modes as framing devices; and modes as agency representation. The students' multimodal texts were analysed based on multimodal text analysis. This study demonstrated that the participants personalised, avoided, modified and orchestrated semiotic modes to engage in multimodal composition. The study also revealed that the students' selection and use of semiotic modes were influenced by their audience awareness, genre knowledge and identities, and vice versa, all of which contributed to developing their multimodal communicative competence. The paper concludes with pedagogical implications for multimodal composition practices and suggestions for future multimodal composition studies.

Fostering students’ Multimodal Communicative Competence through genre-based multimodal text analysis

The multiplicity of semiotic resources employed in communication, the rapid advancement of information, communication, and technology (ICT), and burgeoning interdisciplinary research into multimodality have led to a paradigmatic shift from a mono-modal to the multimodal perspective of communication. Conversely, actualising multimodal concepts in teaching and learning practises remains underexplored, notably in developing the students’ multimodal communicative competence (MCC). For this reason, this study endeavoured to probe genre-based multimodal text analysis in fostering the students’ MCC. Grounded on Action Research (AR), the present study facilitated students to cultivate their MCC through the activities of Genre-based multimodal text analysis (hereafter, GBMTA). Practically speaking, students performed the analysing practises in the course at an English Education Department of a state university in Tasikmalaya, West Java, Indonesia, namely Grammar in Multimodal Discourse (GiMD. Four Indonesian EFL students were recruited as the participants. The data were collected through semi-structured interviews and analysed with thematic analysis. The findings showed that the students could: (1) build their knowledge on multimodality, (2) engage with theoretical and practical learning activities, (3) assign analytical and reflective task-based learning activities, and (4) provide constructive feedback about their learning performances, and (5) raise awareness of the contributions of multimodality to prospective English teachers’ competences. The main implication of this study is the promotion of increased awareness of deploying multimodal aspects to English language teaching, learning, and investigative practises to attain optimum MCC.

The Islamic State: constructing identity and articulating propagandistic discourse through intertextuality and recontextualisation

This paper examines the discourse of the Islamic State (IS) as a terrorist group claiming Islamic authenticity and the right to establish <em>khilafa</em>. The analysis of multimodal texts produced and posted online by this group shows how their discourse uses intertextuality and recontextualisation to authenticate their political practice in religious terms, and how they employ strategies of horizontal and vertical propaganda to maintain control and hegemony, and instil fear and terror in the lives of civilians and belligerents alike. The analysis shows that in order to survive, their discourse shifts positions in response to needs, desires and antagonistic social realities. The discussion of these shifting positions reveals the political foundation of their alleged religious discourse and shows how this will change in the course of history.

Compositional and interpersonal analysis of the picture book Stella brings the family: deconstructing affection

This study attempts to carry out an analysis of the compositional and interpersonal metafunctions of the picture book Stella brings the family (2015). The analytical tools employed in this study are based on the model for reading visual narratives proposed by Painter, Martin and Unsworth (2013) and on Kress and van Leeuwen’s Visual Social Semiotics (2021), which are useful models for the analysis of multimodal texts. The compositional and interpersonal analysis intends to deepen the deconstruction of meaning in the picture book and how the characters express affection. The compositional analysis reveals that the visual and the written text are complementary and that the different focus groups highlight the characters. The types of themes and the layout foreground the protagonist Stella. In addition, from an interpersonal perspective, the analysis reveals that Stella has more physical contact with one of her fathers, which suggests that Stella has a closer relationship with him. The compositional and interpersonal meanings contribute to establishing a close connection with readers so that they identify with Stella’s problem of not having a mother to celebrate Mother’s Day.

Modes of subjectivation, ethos and the “Pathway of passions”: a theoretical-methodological apparatus for rhetorical analysis

The text argues for the possibility of multimodal argumentative analyses. The main objective of this essay was to speculate on how an orator's modes of subjectivation, discursively materialized by ethos, provide evidence for the persuasion of audiences. In order to do that, we use the “Pathway of passions” theory as a theoretical-methodological analysis apparatus. The object of study was the official campaign video of the current state deputy for the State of São Paulo, Arthur Moledo do Val, published on the YouTube website, in 2018. The results of the work indicate that the analysis of multimodal texts is essential at the present time, since the most varied social practices, such as politics, have been using these materialities to persuade. It is also clear that ethos gains more argumentative relevance within a virtual communicative context. Furthermore, the rhetorical perspective of argumentation, transdisciplinary since its genesis, can provide the necessary means to carry out such an undertaking

Analisis Teks Multimodal pada Logo Hari Santri Nasional 2021 sebagai Alternatif Media Pembelajaran Poster

Along with the times, science and technology, teachers and students are required to be able to master technological literacy skills, data literacy and human literacy (multiliteracy). The most appropriate way to developing these skills is to apply multiliteracy learning. Therefore, teachers must be able to plan and prepare multiliterate learning media to support learning activities in the 21st century. One of the media that can be used to implement multiliteracy learning is Instagram. The purpose of this study is to describe the results of multimodal text analysis on the 2021 National Santri Day logo as an alternative poster learning media. In this study, the literature study research method was used. The results of this study indicate that developing multi-literacy learning by conducting multimodal text analysis of the 2021 National Santri Day logo on Instagram can increase students' competence to cultivate creative souls, critical thinking in educational activities and can carry out exploration so that they can gain their own knowledge to support the realization of the next generation. Indonesia is gold in the era of society 5.0&#x0D;

Analysing Multimodal Texts in Science—a Social Semiotic Perspective

AbstractTeaching and learning in science disciplines are dependent on multimodal communication. Earlier research implies that students may be challenged when trying to interpret and use different semiotic resources. There have been calls for extensive frameworks that enable analysis of multimodal texts in science education. In this study, we combine analytical tools deriving from social semiotics, including systemic functional linguistics (SFL), where the ideational, interpersonal, and textual metafunctions are central. In regard to other modes than writing—and to analyse how textual resources are combined—we build on aspects highlighted in research on multimodality. The aim of this study is to uncover how such a framework can provide researchers and teachers with insights into the ways in which various aspects of the content in multimodal texts are communicated through different semiotic resources. Furthermore, we aim to explore how different text resources interact and, finally, how the students, or authors of teaching resources, position themselves in relation to the subject. Data consist of one student text and one teaching resource text, both comprising drawn and written elements in combination with symbols. Our analyses of the student text suggest that the proposed framework can provide insights into students’ content knowledge and, hence, how construction of multimodal texts may be a useful tool for formative assessment. When it comes to teaching resources, the framework may be a useful tool for teachers when choosing resources, particularly in relation to students’ possibilities of meaning making when engaging with such texts, but also, as a basis for classroom discussions.

Constructing the pedagogy of multiliteracies. The role of focalisation in the development of critical analysis of multimodal narratives

This paper aims to contribute to the construction of multiliteracies pedagogy. We argue that the analysis of focalisation is an essential dimension in multiliteracies pedagogy aimed at promoting the development of critical analysis of multimodal narratives. Our point is sustained by a case study of focalisation in an animated narrative comprising a hybrid multimodal text offered for students learning about refugees in a public digital platform. We performed a critical analysis of the visual narrative supported by a grammar of focalisation in animated images and a framework for critical analysis of multimodal discourse. The analysis shows that viewers (and characters) are differently positioned throughout the text as uninvolved observers of the represented characters as ‘others’; as emphatic witnesses of the others’ suffering; and as committed social actors who see the refugees as ‘one of us’. We interpret these findings as conveying different ideological meanings associated with assimilatory or liberating ideologies. We discuss the major implications of our findings concerning the role of focalisation in the enactment of critical analysis of multimodal texts, the interdependence of critical analysis vis-à-vis the other learning processes as well as the need for teachers’ professional development as a pre-requisite to construct the pedagogy of multiliteracies.

A Wild Animal Inside My House: An Analysis of the Children’s Picturebook Svenn the Otter and the Magic Rock by Espen Villseth and Anita Sletten

Svenn, a wild otter, has become an Instagram celebrity with over 140,000 followers around the world. Svenn has his own YouTube channel and Facebook page. Moreover, in December 2020, NRK, an influential Norwegian TV channel, released a series entitled “Oteren Svenn” (Svenn, the Otter), consisting of ten episodes. In addition to all of this, Svenn has his own picturebook for children, which is entitled Svenn the Otter and the Magic Rock (Villseth &amp; Sletten, 2020). This article presents the picturebook and analyzes three of its significant spreads. The aim is to uncover aspects of the sociocultural context in which Svenn – along with other characters – is portrayed by images and verbal language in the book, and to provide a short discussion of how the book may be used in English Language Teaching (ELT). Our analysis builds on tools from multimodal text analysis (Kress &amp; van Leeuwen, 2005; van Leeuwen, 2005) and Narrative Intelligence (Mateas &amp; Senger, 2003). The analysis focuses on the significance of the picturebook to represent participants in actions, the relationship between the book’s participants and the reader, and the value of presenting images and verbal text together. Our analysis also touches on some implications for ELT, on the potential that visual literacy can have in language teaching.

The semiotics of visual and textual legitimacy in the 2014 Gaza war

ABSTRACT In this article I conduct a multimodal critical discourse analysis of news coverage of the 2014 Israeli assault on Gaza by select British and South African print news outlets. The representation of the actors’ legitimacy is the theoretical framework used for the study. Interviews were also conducted with journalists who reported on the war for insights into aspects of news production that impacted their reporting. Textually and visually Israeli military personnel were accorded legitimacy. The Palestinian perspective was marginalised in addition to being de-legitimated. A key facet of the war’s coverage is the highly-structured reporting environment in Israel and Gaza. I argue that this continues to exert a significant impact on the work of journalists there. The article enhances current research on the analysis of multimodal texts using qualitative approaches. Important insights are provided about the manner in which textual and visual semiotic resources in news texts function in unison to convey or rescind legitimacy. The second contribution concerns the impact of the news production process in the coverage of the Israeli-Palestinian conflict. The media’s role in legitimising Israeli violence is particularly pertinent.

Covid-19 and Media Representation: A Multimodal Study of Positive Discourse in Pakistani Television Commercials

The present study seeks to highlight the role of mainstream media in the dissemination of positive discourse, creating awareness among the masses with regard to the outbreak of COVID-19 in Pakistan. Media discourse plays a significant role in promoting ideology at a larger scale due to its vast viewership across the country. These ideological narratives are implicitly employed by the advertisers through several multiple resource systems. Electronic media has the power to (re)frame the ideology of a larger audience via significant visual/linguistic content and paves the way for social change through viable social institutions. The study follows the theoretical stance of Multimodal Text Analysis propounded by Kress and Leeuwan (2006). The data is comprised of some popular television commercials which are broadcast on the most viewed channels during the peak time viewership in the Pakistani context. The purposive sampling technique has been employed for the data collection process, including only those commercials that largely reflect COVID-19 patterns. The study finds that the commercials present layers of meanings via several orders of signification where the representation has been aimed at creating awareness to a greater extent among the general public so that they can take essential precautions and help to fight against the pandemic.

Adolescent English language learners’ digitally mediated multimodal compositions: multimodal enactment across different genres of writing in the EFL context

PurposeThis paper aims to investigate adolescent English as a foreign language (EFL) learners’ digitally mediated multimodal compositions across different genres of writing.Design/methodology/approachThree Korean high school students participated in the study and created multiple multimodal texts over the course of one academic semester. These texts and other materials were the basis for this study’s qualitative case studies. Multiple sources of data (e.g. class observations, demographic surveys, interviews, field notes and students’ artifacts) were collected. Drawing upon the inductive approach, a coding-oriented analysis was used for the collected data. In addition, a multimodal text analysis was conducted for the students’ multimodal texts and their storyboards.FindingsThe study participants’ perceptions of multimodal composing practices seemed to be positively reshaped as a result of them creating multimodal texts. Some participants created multimodal products in phases (e.g. selecting or changing a topic, constructing a storyboard and editing). Especially, although the students’ creative processes had a similarly fixed and linear flow from print-based writing to other modalities, their creative processes proved to be flexible, recursive and/or circular.Originality/valueThis study contributes to the understanding of adolescent English language learners’ multimodal composing practices in the EFL context, which has been underexplored in the literature. It also presents the students’ perspectives on these practices. In short, it provides theoretical and methodological grounds for future L2 literacy researchers to conduct empirical studies on multimodal composing practices.