Visual Semiotic Resources Research Articles

Gender in Fashion Advertising on Social Media: A Multimodal Discourse Analysis Approach

This study examines digital fashion advertising discourse on social media from a multimodal perspective. The focus is on Facebook clothing posts published by 'H&M' and the effect of gender on determining the semiotic modes used in such advertisements. The data consists of 8 clothing posts for the two genders published by the company on Facebook during (2020-2022). The data were analyzed using Kress and Van Leeuwen's (2006) Visual Grammar paradigm to explore the role of gender in the choice of visual modes employed in these posts. The findings show that digital fashion clothing ads widely involve visual semiotic modes as a tool that creates meaning and enables interaction with the target audiences. The results also indicate that gender is a primary factor in the selection of such visual semiotic resources. In particular, the construction of visual meaning, especially representational and interactive meanings, is different from that in male ads.

Audiovisual Translation, Multilingual Desire, and the Construction of the Intersectional Gay Male Body

This study focuses on the HBO series Looking, whose two seasons and film make up a critical telecinematic artifact that reveals how authorial vision integrates ideologies on class, race, and desire that are identifiable in visual modes and language use—particularly multilingual dialogues. The analysis begins with the assumption that Looking is a relevant case of complex television and centers on the narrative structure of the series and the way that language, translation, and visual semiotic resources interact in the construction of a gay Latino character in the source version of the series and two Spanish dubbed versions—one for Latin America and the other for Spain. The findings reveal that Looking, as a televisual and aesthetic artifact, proposes a post-gay discourse of homoerotic relationships while also constructing racialized objects of desire, particularly the Latinx (male) body. A comparative linguistic analysis shows that both the dubbed versions highlight the boundaries of the so-called globalized gay identity. The data gathered demonstrate that the representation of ethnic, racial, and erotic difference changes according to the language system used. Moreover, new interactions between dubbed dialogues and visual resources result in a greater degree of semiotic layering of ideological discourses throughout the series.

Matter, meaning and semiotics

We inhabit two worlds – the world of matter and the world of meaning (see Halliday, ‘On matter and meaning: The two realms of human experience, 2005). In this article, these two worlds and the physical, biological, social and semiotic systems that connect them are investigated. In this respect, semiotic systems are the most complex because they involve physical systems (the material sign), biological systems (human beings), social systems (society and culture) and meaning itself. Semiotic frameworks need to take into account these various dimensions as changes in one system reverberate across the meta-system as a whole. With this in mind, the interplay between material and semiotic worlds from a social semiotic perspective, are explored with a focus on meaning and its significance in relation to human existence. Using examples from various industrial ages, the article explores how semiotic resources (in this case, in mathematics, science and computer programming languages) are organized to structure reality in specific ways, and how semiotic combinations and the technologies arising from those constructions have changed the course of human history. In this discussion, attention is paid to the role of visual communication, both in terms of visual semiotic resources (e.g. graphs, digital images) and visual aspects of multimodal texts. It thus becomes evident that the functionalities of any one semiotic resource (including language) must be viewed in relation to its collective co-deployment with other semiotic resources. Lastly, the author examines semiosis in the digital age and considers the social implications of the current digital ecosystem. In doing so, she conceptualizes digital technologies as a one-way mirror where members of society use digital media for every facet of their lives while being watched, analysed and manipulated by those who have designed and own the digital platforms. It is apparent that semiotics has a major role to play in terms of design, policymaking and activism around future digital technologies.

Marvel’s Captain America Civil War poster: An analysis based on the Grammar of Visual Design

Movie posters, a multimodal genre used to attract viewers, are a complex form of communication that contains verbal and visual semiotic resources. The entertainment company Marvel has created a cinematic universe where movie posters play an essential role in marketing. Despite Marvel’s global influence in sociocultural practices, few studies investigate its movie posters. Hence, this study analyzes the visual structures used in the Captain America: Civil War theatrical movie poster, drawing on the Grammar of Visual Design, specifically the representational, interactive, and compositional metafunctions. Regarding the representational metafunction, results show a bidirectional transaction, highlighting the faceoff between the two teams of heroes. Their symmetry also hints at classificational processes. Concerning interactional meanings, the heroes gaze at each other, inviting viewers to observe the image from a social distance while keeping some detachment. Moreover, the camera angle creates an equal power relation between the main heroes and viewers. Finally, compositional meanings strengthen the separation between the two teams through different elements, but they simultaneously indicate the teams will reunite. Thus, these three kinds of visual meanings together provide an adequate summary of the movie plot and aptly contribute to the construal of a future conflict between the two teams.

The Image of Muslim in The Kingdom Trailer: A Multimodal Analysis

Based on multimodal analysis, the study at focus analyses The Kingdom trailer (2007) by adopting Kress and van Leeuwen’s Visual Grammar (1996, 2006), Chandler’s social categories of codes (2007) and Bernstein’s (2003) language codes. This kind of analysis is used to illustrate how the image of Muslim is portrayed in such a trailer. It aims to show how multimodal representational aspects are utilized in depicting such distorted image. Also, it aims at presenting how visual semiotic resources and speeches are constructing certain negative images. This research assumes that meaning construction is not monomodal but an outcome of multimodal means of communication.
 The prime conclusions of the current study can be presented as both, visual and verbal aspects which disseminate a negative image of Muslims. Through guns, fight, and extreme behaviour, Muslims are presented as a threat to the Western and prosperous way of life that necessitates force majeure, military actions and murdering. The study at focus also shows that this trailer makes use of a visual mode more than language and the kind of image presented is distorted, unfavourable and misleading. Also, (mis)representation of Muslims in this trailer fuels Islamophobia in a radical manner. Moreover, it has been found that behavioural codes are utilized to link Muslims and Islam to terrorism via the depiction of religious rituals such as prayer and call for prayer after terrorist acts.

Digital Synesthesia in Heritage and Second Language Writing during Collaborative and Individual Digital Storytelling

Using the social semiotic multimodal approach, this study compared the transformation and transduction moves of eight Spanish heritage language learners (SHLs) and six second-language learners (L2s) as they composed collaborative and individual digital stories (DSs) in an undergraduate advanced Spanish writing class. The research analyzed the learners’ composing processes regarding digital synesthesia when working collaboratively and individually as they integrated written, oral, aural, and visual semiotic resources. The data revealed that SHLs demonstrated more frequent and more complex digital synesthesia than did their L2 classmates during their quest for digital multimodality in Spanish, indicating improvement via task repetition in frequency and variety of integrated modes.

The semiotics of visual and textual legitimacy in the 2014 Gaza war

ABSTRACT In this article I conduct a multimodal critical discourse analysis of news coverage of the 2014 Israeli assault on Gaza by select British and South African print news outlets. The representation of the actors’ legitimacy is the theoretical framework used for the study. Interviews were also conducted with journalists who reported on the war for insights into aspects of news production that impacted their reporting. Textually and visually Israeli military personnel were accorded legitimacy. The Palestinian perspective was marginalised in addition to being de-legitimated. A key facet of the war’s coverage is the highly-structured reporting environment in Israel and Gaza. I argue that this continues to exert a significant impact on the work of journalists there. The article enhances current research on the analysis of multimodal texts using qualitative approaches. Important insights are provided about the manner in which textual and visual semiotic resources in news texts function in unison to convey or rescind legitimacy. The second contribution concerns the impact of the news production process in the coverage of the Israeli-Palestinian conflict. The media’s role in legitimising Israeli violence is particularly pertinent.

Addressing international students on Australian and Chinese university webpages: A comparative study

This article investigates the discursive construction of international students on Australian and Chinese university webpages that are directed at international students. The international student theme webpages of three Australian and three Chinese universities were examined in relation to how verbal and visual semiotic resources were co-deployed on the webpages to address international students in the climate of neoliberal thinking in higher education. The tools used for analysis and interpretation were informed by socio-semiotic approaches to multimodal discourse analysis, international education discourses and communication accommodation theories. It was found that the Australian and Chinese university webpages differed in several salient ways to the effect that international students were portrayed as agentive and informed individuals to explore a study abroad experience at the Australian universities versus being explicitly guided through their study abroad at the Chinese universities. These results were compared and interpreted as reflecting each country’s conception of transnational education. The article concludes with a summary of the impacts of cultural and intercultural factors and neoliberal thinking in higher education on multimodal representations of international students on university webpages and directions for further research.

Multimodal Language of Attitude in Digital Composition

Communication using popular digital media involves understanding multimodal systems of appraisal for expressing attitude, which traditionally deals with emotions, ethics, and aesthetics in language. The formulation and teaching of multimodal grammars for attitudinal meanings in popular texts and culture is currently underresearched. This article reports findings from multisite qualitative research that developed students’ ability to use semiotic resources for communicating attitude multimodally. The research participants were 68 students (ages 9–11 years) from two elementary schools. Students learned how to use attitudinal language—affect, judgment, and appreciation—and applied this knowledge to multimodal design. The findings advance a leading system of appraisal for discourse by adapting the system to the multimodal communication of attitude in digital comic making in schooling. The research is significant because it demonstrates the potentials for augmenting students’ linguistic and visual semiotic resources to convey multimodal attitudinal meanings in contemporary communication.

A multimodal model of analysis for the translation of songs from stage musicals

Song translation has only recently become an area of interest for research purposes, with the development of studies on opera, films, folk music, cover songs, and more. Not many scholars have researched stage musicals, even though songs play a fundamental role in this type of performance, conveying meaning via verbal, audio and visual semiotic resources.A few studies on song translation can be identified, such as Low’s (2003; 2005) Pentathlon Approach and Franzon’s (2005) functional approach. These models of translation offer valuable guidelines on how to treat the lyrics, but what is missing is a systematic and multimodal model of analysis that can be applied to the song in its entirety. Kaindl (2005; 2013) takes into consideration the multimodality of songs, but only focuses on popular music and opera. Acknowledging the lack of substantial research on the interaction between modes, which is typical of stage musicals, this paper focuses on the development of a model of analysis that considers the semiotic complexity of songs. A new approach based on themes will allow for a more holistic view of the song and of its content.

A Social Semiotic Approach to the Attitudinal Meanings in Multimodal Texts

As a special type of multimodal text, picture books for children are highly valued in the creation of meaning by the integrative use of verbal and visual semiotic resources. Informed by Painter and Martin’s framework of visual narratives, this paper primarily deals with the interpersonal meanings encoded and expressed by the two semiotics (image and verbiage) within the Chinese picture books. It aims to analyse the visual and verbal choices available for writers to establish engagement between various participants. In the hope of investigating the collaboration and interplay of verbal and visual semiotics to construe interpersonal meanings, it examines the attitudinal meanings inscribed or invoked in picture books, exploring the ways in which visual and verbal resources are co-instantiated to encode attitudinal convergence and also divergence.

Communicative functions and characterization in the Chinese graphic novel adaptations of "Little Women"

This article attempts to use an integrated theoretical framework to examine the three graphic novels published in Taiwan based on Louisa May Alcott’s Little Women. After the concept of intertextuality (Kristeva, 1980), Kress and van Leeuwen’s (1996) visual grammar helps us to explore how the visual semiotic resources in three Taiwanese graphic novel editions of Little Women ascribe meanings, achieve functions and communicate ideologies to enhance the characterization of Jo March. Progressing from the images, image–text relationships are discussed in terms of how these signifiers shape signs and meaning. Finally, by exploring speech or thought representation, the researcher investigates the translators’ awareness of the readers and situates their use of signs socially and culturally. In this study, the integrated framework reveals how these semiotic resources characterize Jo March, and the double meanings (Kristeva, 1980, pp. 65–66) they present, differently. These characterisations may, in turn, may have affected the communicative function of each version. The study concludes with a discussion of the strengths and weaknesses of these applications.

A multimodal discourse analysis of the textual and logical relations in marketing texts written by international undergraduate students

AbstractThematic progression and the composition of information value contribute to the development of well-structured meaningful text. Discourse-based research of the textual features has been confined to language learning and workplace contexts, and in particular advertising and marketing research. However, no published study has explored and analysed the multimodal textual and the logic-semantic features in tertiary marketing texts. This case study employed a Systemic Functional Multimodal Discourse Analysis (SF-MDA) to investigate and compare thematic progression patterns and composition of information value in five business marketing plan reports. It also aimed to investigate the logico–semantic expansions between the visual semiotic resources and the text surrounding them. The SF-MDA of the marketing plan texts is underpinned by Halliday’s (1994) systemic functional linguistics, Kress and van Leeuwen’s (1996) composition of information value and Martinec and Salway’s (2005) system for multimodal discourse analysis of logico–semantic relations. The findings of the SF-MDA revealed the extensive use of Theme reiteration pattern, followed by linear thematic progression pattern. Although multiple-Theme pattern was minimally employed, disciplinary-specific uses of this pattern in marketing discourse emerged. The findings suggest that the students have managed professional workplace practices by showing their understandings of marketing as part of overall business management. A number of other interesting findings related to textual and logico–semantic relations that exist between the tables and the graphs and the accompanying text were revealed. The SF-MDA of informational choices in the marketing texts extends Kress and van Leeuwen’s (1996) functional interpretations of visual artefacts in terms of compositional zones. As the intersemiotic logic-semantic relations between graphs and tables and the orthographic texts surrounding them play a vital role in marketing discourse, tutors can introduce ways of expanding the meaning-making potential in these semiotic resources. Other pedagogical implications for the teaching and learning of writing are presented, particularly in the teaching of English for Business students.

A Multimodal Analysis of Car Billboard Advertisements: Audi/BMW Rivalry

Within the plethora of advertising media, billboard advertising is gaining precedence due to its high visibilityandaccessibility to all viewers. This study explores the multimodal nature of billboard advertising by focusingon the interplay between theverbal and visualmodes employed by BMW and Audi advertisers.It accounts forthemeaning potentials and affordances of the semiotic resources and uncovers the advertisers’persuasive strategies. In doing so, the study draws on Hallidayand Matthiessen’s(2014)systemic functional grammar and Kressand van Leeuwen’s (2006) visual grammar to examine five billboards and a zeppelin. The study examines the headlines in terms of the grammatical structures of Theme/Rheme, Mood, and transitivity. It also explores the advertisers’ choices of visual semiotic resources to orchestrate representational, interactive, and compositional meanings. The study concludes that the two rivals: BMW and Audisucceeded in deploying the visual semiotic resources to communicate signified messages, except for one image.Regarding the verbal mode, Audi successfully manifestsmanipulation of the meaning-making metafunctionsto convey verbal messages;BMW conveys messages via lexical items only.Themultimodal approach to the analysis of billboard advertisements offers an insight into the integration of the verbal and visual modes to create signified meanings

A multimodal framework for tracking Sesshomaru's character development in an anime movie – Inuyasha: Swords of an Honourable Ruler – an appraisal and gestural perspective

In this article, I propose an integrated multimodal framework for tracking a Japanese character's development, that of Sesshomaru, in the anime movie, Inuyasha: Swords of an Honourable Ruler. The language part of the framework focuses specifically on the appraisal analysis of the language spoken by the characters. The visual framework builds on Lim's system of gesture by integrating Martin and White's system nodes for appraisal analysis into Lim's gesture framework for a nuanced analysis of the characters' gestures. The framework is then illustrated through a close comparative analysis of two scenes from the anime movie. I explain how the relationship between the language and visual semiotic resources based on appraisal analysis changes from convergence between the language and visuals to divergence, which could be correlated with character development. I conclude by discussing how the narrative context influences the semiotic representation of the character Sesshomaru.

Social semiotics and literacy: A case study about the social meanings constructed by ads of a children’s magazine

This article aims to contribute to studies on literacy and multimodality from the theoretical perspective of social semiotics and the methodological framework of visual grammar. The objective is to investigate the types of representations, interactions and compositions produced by visual semiotic resources configured by advertising images promoted by Visão Júnior, a Portuguese magazine dedicated to children and youth. The results show: a) the promotion of television channels and news publications owned by the media group of which Visão Júnior is also part which are aimed at adults; b) the encouragement of reading and the consumption of cultural products and events; c) the association between the consumption of food and sport; d) the introduction of new values in the child’s universe, such as the use and consumption of mobile phones, as well as familiarising the child with banks and their possible relationships with money.

Young students’ uses of visual composing resources in literacy classroom contexts: a cross-case analysis

In this study, the author draws upon a social semiotic framework, exploring the ways in which students can use visual semiotic resources while composing texts in literacy classroom contexts. His multi-case study involving three cases of students in primary-level classrooms focuses specifically on how several focal students created, accessed, and assembled sets of visual semiotic resources that they used while engaged in the literacy processes at hand. This study defines and develops the concept of a visual composing resource, and qualifies the range, scope, and type of visual resources that were available to the students while they engaged in literacy processes in various classroom settings. The author also illustrates how the students brought these sets of visual composing resources into a complex interaction with the semiotic assemblages that they were producing in each context, revealing important aspects of early literacy processes that incorporate visuality.

Visual discourses of the role of women in war commemoration

Discourses of war are disseminated and legitimised not only through speeches and written texts, but through visual semiotic resources. One important vehicle for this has been the war monument. Evidence shows that from WW1 in Europe and the US monuments have been used systematically by authorities to recontextualise the realities of war and soldiery, suppressing much of what comprises war, avoiding any critical stance, while fostering celebratory discourses of nation, protection and noble sacrifice. The representation of women on the war monument has been particularly important in this recontexualisation. Using Multimodal Critical Discourse Analysis to examine the discourses realised by British war monuments, this paper shows that while much of the way women participate in and experience war has been suppressed on war monuments. Their representation has been a key part of the legitimation of one particular discourse of war, a representation that has helped to sideline other possible discourse in British society and which is still used in the commemoration of the death of ‘our boys’ — such as the young men killed in Afghanistan.