Semiotic Modes Research Articles

Brand transcreation as multimodal configuration

Abstract This article aims at revisiting some key concepts in translation studies by taking into account a new translation or multimodal practice called brand transcreation. Unlike translation proper or brand design proper, brand transcreation involves interlingual and multimodal production in the target culture. It relies on verbal instructions rather than the source text, and intersemiotically, transforms verbal messages into polished multimodal products. Using the systemic-functional sociosemiotic framework, this study delves into the types and dynamic orchestration of semiotic modes involved in two successful brand transcreations from a producer-researcher perspective. The findings reveal that brand transcreation can be defined as a goal-oriented textual production that configures multiple modes and serves specific communicative purposes. In this regard, the concepts of texts, contexts, meaning, and particularly (the categorization of) translation have evolved and require reconsideration.

Recurrent gestures in organic chemistry in tertiary education: creating emblems through material and embodied actions

ABSTRACT Background Science teaching and learning, as discursive practices, occur through the utilization of various semiotic modes. Gestures play a pivotal role as an embodied mode that acts in conjunction with speech in oral communication settings, such as classrooms. Previous research on gestures in chemistry and science education has illuminated how interactions involving different types of gestures can enhance learning opportunities for students. However, the investigation of recurrent gestures and their potential function as emblems in classroom discourse remains unexplored. Purpose The objective of this article is to illustrate how catchments or recurrent gestures were constructed, used, and, to some extent, evolved into emblems within the interaction between an organic chemistry teacher and her students throughout an entire semester. Sample, design and methods For the duration of an entire semester, we non-intrusively observed and recorded video footage of one class from an introductory organic chemistry course taught by an experienced university teacher. Our analysis to identify recurrent gestures involved a thorough examination of the lesson recordings. In addition, we scrutinized other semiotic modes that coincided with the recurrent gestures and assessed the metafunction of the words spoken during gesturing. Results During our observation, we noted the occurrence of 23 catchments, with four of them recurring more than 100 times throughout the semester. These four recurrent gestures were closely tied to fundamental concepts considered crucial in the realm of organic chemistry. Remarkably, three out of these four catchments were consistently associated with visual representations. Conclusion Recurrent gestures serve as a means to embody and impart materiality to exceedingly abstract content. By employing such gestures, the teacher managed to associate unfamiliar and abstract domains with familiar and concrete domains. This process of materialization and embodiment aids in perceiving these concepts as tangible entities that can be manipulated in accordance with their own rules.

The Denotative and Connotative Meanings on the Advertisement of iPhone 13 Relax, It’s Iphone

A creative and innovative advertisement is required by a company to attract customers. So, advertisers must utilize an attractive technique to convince the viewers to buy their products and they need to deliver the messages to the customers. To deliver the messages, advertisers employ semiotic modes such as visual mode, linguistic mode, and gestural mode. We analyze the denotative and connotative meanings as expressed through the semiotic resources used and the messages delivered in the iPhone 13 Relax, it’s iPhone video advertisement. We use the theory of the process of signification by Chandler (2017) and semiotic mode as supporting theory. The denotative and connotative meanings of the advertisement is that the iPhone 13 has three features namely water-resistance, long battery life, and durability. The messages that these advertisements create is a brand myth, that is, people do not have to worry when they use iPhone 13 in an unfavorable situation.

A Multimodal Analysis on Rick Singer from Operation Varsity Blues: The College Admissions Scandal

Multimodal communication employs multiple semiotic modes to convey meaning. Operation Varsity Blues: The College Admissions Scandal is a documentary about Rick Singer, an education entrepreneur who committed the bribes. This study analyzed his representation through the oral and visual modes and was aimed at showing how multimodality could help reveal particular character representations. This study applies multimodal theory (Kress & Leeuwen, 2006) and semiotic modes (Chia & Chan, 2017). The oral monologues and dialogues, and the visuals of people, objects, and places were scrutinized to observe how Rick Singer was depicted. As a result, he was seen being portrayed as a wealthy and successful entrepreneur with many high-ranking connections, who strategically manipulated his clients and his university partners in a clever scheming maneuver. The interplay of both the oral and visual modes has resulted in a relatively complete picture of the main actor of this documentary.

Rubrics for Girls in Arab Children’s Magazines: A Multimodal Discourse Analysis Approach

The present study aims at examining the semiotic modes used in the rubrics for girls and the topics they reflect in Arab children’s magazines. To this end, eight girls’ sections were extracted from two Arab children’s magazines, namely Majid (active) and Hatem (non-active). We adopted Kress and Van Leeuwen’s (2006) visual grammar in our data analysis. We focused on representational, interactive, and compositional metafunctions. The results show that although both publications used a variety of semiotic modes to express meaning, there are some differences between them. That is, Majid is richer than Hatem in terms of layout, the use of pictures, and vibrant colors. The findings also indicate that the two periodicals discuss similar topics, mainly recipes, fashion, and etiquette, but they vary in the attention given to each topic. While Majid concentrates on fashion and beauty, Hatem places much more focus on recipes and handcrafts. These results suggest that the variations in focus on specific topics could be related to the socio-economic contexts in which the two magazines are published.

Subtitles and Captions: An Instrument for Intentional or Incidental Learning?

Audio-visual materials such as TV series and films offer multiple semiotic modes and authentic language input for language learning. This case study sought to depict English as a Foreign Language (EFL) learners’ habits of utilising subtitles and captions for incidental and intentional learning of grammar and vocabulary through on-screen texts, subtitles and captions. The target group was determined through criterion sampling because they were expected to watch English TV series or films to be eligible to take part in the study. A survey and inter-view were employed to gather data from 113 Turkish EFL learners studying at the tertiary level. The quantitative data were analysed descriptively, while the qualitative data were analysed using a constant comparative method (Creswell & Poth, 2016). The findings revealed that, by watching captioned TV series and films, most of the participants performed intentional learning of vocabulary and grammar, whereas some are engaged in incidental learning. The strategies used by EFL learners for intentional learning of vocabulary and grammar were also provided.

Modes of Social Closure in Morten Tyldum’s Film The Imitation Game

Abstract This article discusses Morten Tyldum’s film The Imitation Game in terms of its media-specific representation of social closure at three spatio-temporal levels. They involve Alan Turing’s boarding school in the late 1920s, Bletchley Park, the home of the cryptanalysts during the Second World War, and a homophobic Britain of the early 1950s. The paper borrows from an expanded neo-Weberian theoretical frame and attempts to address power-based processes of social closure by means of an approach that relates the storyline to selected semiotic modes used by the film language. The analysis foregrounds some scenic juxtapositions and modal ties to illustrate how the two codebreakers, Alan Turing (Benedict Cumberbatch) and Joan Clarke (Keira Knightley), cope with an exclusionary society, its homophobic legislation, sexist attitudes, and, in the case of young Alan, its practice of physical bullying.

Soundscapes of two rural communities in Papua New Guinea

This article presents an analysis of the soundscapes in two rural communities in northwestern Papua New Guinea where the endangered languages Srenge and Walman are spoken. These languages have no written tradition and therefore are not publicly displayed in signage. In the last few years, the discipline of linguistic landscape studies has fostered the study of aspects of the visual semiotics of public signs together with aspects of other semiotic modalities including interactional semiotics and auditory semiotics. By discussing the sounds of instruments, conch shells, place names, recitations, and messenger birds in areas where Srenge, Walman, and other languages are spoken, the aim is twofold. The first goal is to fill a gap in the study of semiotic systems that play a role in linguistic landscapes by focusing on a rural area and on auditory semiotics. The second goal is to document anthropological knowledge in an area that is extremely rich culturally and linguistically but where language attrition and cultural loss have been greatly accelerated in the last few decades.

Theatre in Transition: Lars Elleström's Media Modalities and the Rise of “Video Theatre”

ABSTRACT During the COVID-19 pandemic, pre-recorded and recorded videos were extensively live-streamed or displayed on online platforms, usually as a substitute for the “real thing”. Online theatre became an ambassador for theatre performance, remaining selectively oblivious to theatre’s innovative digital paths of the past. The synergy between theatre and the digital was significantly reconfigured and its intermedial dimension escaped our collective attention, despite a silent process of media translation and appropriation (or even total transformation) unfolding. A comparative approach based on Lars Elleström’s (2021, 2010) media modalities model suggests that a video recording of a theatre performance destined for the web differs from the actual theatre performance in terms of its material, spatiotemporal, sensorial and semiotic modalities to such an extent that it constitutes a different media type. An online video of a live theatre performance emerges as an intermedial phenomenon with a strong transmedial dynamic and is found to belong to the sphere of media transformation. This new media type of “video theatre/drama” may acquire artistic value per se, when it develops self-awareness, addresses aesthetics and establishes its own poetics.

The role of multimodality and intertextuality in accentuating humor in Algerian Hirak’s posters

Abstract This study investigates how the interaction between multimodal modes and intertextual resources accentuates humor functions. Kress and van Leeuwen’s Visual Grammar (2006) was adopted to analyze a set of sixty humorous online posters of Hirak’s movements. The results revealed that humor was generated from the purposeful interplay of various semiotic modes and the reproduction and recontextualization of shared socio-cultural and political resources. The protesters utilized cartoon characters, religious discourse, folk traditions, and cultural mundane to represent the authority’s members humorously as lawbreakers, prisoners, robbers, and gangsters. This humorous exposition has placed the authority members outside the Algerian societal in-group boundaries. In contrast, such a device has enhanced conformity among the protesters and exhibited their superiority over the ruling outer group. We hope such an investigation will pave the grounds for further studies and provide insights into multimodal discourse analysis.

Acceptance and perception of multimodal expression of Israeli-Jewish culture in China: ‘The Realist’ comic strip as a case study

ABSTRACT This article analyzes the acceptance and perception of Israeli-Jewish culture in China through Chinese translation of ‘The Realist’. It points that most topics Chinese readers could interpret are those shared as general common sense or used in common terminology in all semiotic modes. However, the Israeli-Jewish tradition, literacy and specific practices remained unrecognised for the most part. It argues that the translator’s mediation of comic texts is grounded in processes of multimodal analysis that facilitates comprehension of whether and how a message is understood in a certain context. The translator has the task of making up the target readers’ contextual deficiency.

Multimodal rhythm in TikTok videos: Exploring a recontextualization of the Gillard ‘misogyny speech’

This article presents a multimodal rhythmic analysis of a TikTok video, adopting a social semiotic perspective on embodied meaning-making. We highlight the importance of rhythm in coordinating and intertwining semiotic modes to produce meaning. The study develops a method for undertaking an integrated multimodal analysis of rhythm across speech, bodily action, gesture and music, and develops a transcription convention for representing this rhythmic unfolding. The data considered is a TikTok ‘glambot/boss challenge’ video featuring a lip sync to audio sampled from former Australian Prime Minister Julia Gillard’s 2012 culturally iconic ‘Misogyny Speech’ condemning misogynist and sexist men, particularly those in positions of power. This speech achieved viral prominence internationally and continues to be a key feminist text in Australian political history. The paper demonstrates how end-accented rhythmic groups create anticipation and lead to the main event in both the speech and glambot sections of the video. Alongside the rhythmic analysis, the article examines the intertextual meanings established with other TikTok videos that iterate the glambot meme and glambot challenge.

Translingual Englishes, participatory hip‐hop and social media in Nepal

AbstractThis article discusses some key functions and features of English in participatory popular culture and social media in Nepal. Analytical attention is paid to how English use in rap battles is entangled with other local languages and semiotic modes to create translingual practices and how online metapragmatic comments about the rap battles give rise to diverse language ideologies. The study shows that by creating their social life offline and online, Nepali young adults project themselves as individuals who have access to niche Englishes via popular culture. English use by Nepali youth functions not only as an instrument to understand Western popular culture but also as a symbolic resource to index a range of such social identities as ‘educated’, ‘civilized’ and ‘competent’. The article concludes by arguing that Nepali young adults create new subjectivities that are often suggestive of social transformation.

Parent–child shared reading of scratch-and-sniff books: the communicative affordance of olfaction

ABSTRACT This study extends the research on shared book reading (SBR) by specifically examining the role of smell in parent–child SBR sessions observed at home. Drawing on qualitative methods, we analysed the verbal engagement of ten Norwegian families and their three to five-year-olds reading an olfactory book (the scratch-and-sniff book Peter Follows His Nose). We followed socio-semiotic theorisations (Kress and Van Leeuwen 2002), to critically evaluate the role of olfaction as a communicative resource in SBR. We outline the principal ways in which smell fulfils three linguistic metafunctions during adult–child SBR with olfactory storybooks: the interpersonal function of signalling individual interests, the textual function of creating a dialogic space and the ideational function of revealing divergent interpretations. We propose that olfaction can be seen as a semiotic mode in SBR, which similarly to colour, has distinct grammar and systematic communication properties, with unique multimodal qualities.

Exploring the materiality of science learning: analytical frameworks for examining interactions with material objects in science meaning-making

ABSTRACT Background With a growing new materialism paradigm and research on multimodality, there is an increasing attention on the role of material objects in science teaching and learning. However, there is currently no available framework, coding scheme, or method of inquiry to specifically analyze the use of material objects in science meaning-making. Purpose This paper presents and discusses two analytical frameworks developed to examine teachers’ and students’ interaction with material objects during classroom discourse. Design and methods The first analytical framework was theoretically informed by multimodal interaction analysis (MIA), focusing on humans’ situated interaction using embodied and disembodied modes located in the material world. The second framework was informed by social semiotics multimodal analysis (SSMA), focusing on material interaction as a semiotic mode with unique affordances for sign-makers to make meaning alongside other semiotic modes (e.g. speech, gesture). Sample Both frameworks are used to analyze two vignettes of classroom discourse generated from video data to illustrate their applications, coding procedures, and interpretative potential. Results The MIA-informed framework highlighted the dynamic and reciprocal relations between students’ experiences and the transformation of materials, while the SSMA-informed framework highlighted the subtle meanings constructed with the unique affordances of the material mode. Conclusion The affordances and limitations of the two frameworks are discussed, along with their compatibility and complementary relations. This work contributes to the emerging field of materiality in science education research and provides a foundation for future studies on the analysis of material interaction in science meaning-making.

Transmediation, Transgression and Popularization: A Study of the Cantonese Opera Film White Snake

The preservation of declining xiqu heritage through remediation and transmediation has often been ineffective in attracting a wider audience. The Cantonese opera film White Snake (2021), with its unusual utilization of computer-generated technology and transgressive combination of various media, offers a fresh approach to popularizing xiqu among younger audiences. To overcome aesthetic differences between cinema and xiqu, the film transforms medium specificities and transmedial potentials of Song landscape paintings and martial-arts film, adapting them to a new scenic and choreographic setting. By altering the original play’s sensory and semiotic modalities, White Snake’s transmedia aspects blaze a new trail for popularizing xiqu heritage for a new generation.

Drawing and Play as Windows into a Child’s Multimodal Meaning-Making Development during COVID-19

Framed within multimodality and situated in a bounded socio-geographical context (i.e., Vancouver), this ethnographic case study provides an in-depth analysis of a bilingual 8-year-old girl’s literacy practices of meaning-making established across varied semiotic modes (i.e., linguistic, visual, audio, spatial, embodied, kinesthetic) during COVID-19. The study draws upon 13 open-ended informal interviews, three sessions of imaginative play, and 16 participant-generated artifacts. The findings revealed two themes (i.e., drawing as collective meaning-making; play as embodied, anthropomorphic meaning-making) that show how the child’s interactions with humans and nonhumans (e.g., toys, objects) contributed to her multimodal meaning-making during the pandemic, which might be beneficial for children in different contexts.

A Stylistic Study on the Selected Poems of Rupi Kaur’s “Milk and Honey”

The emergence of cyberspace and digital means of communicating has resulted in the flourishing of writers who became famous by publishing their literary works online. One of the successful writers with this humble beginning is Rupi Kaur whose style is subjectively described as bizarre, peculiar, an illusion of profoundness, lacks nuance, and unconventional. To avoid jeopardizing the aesthetics and genuine meaning of her poetry, this research aims to explore the objective analysis and interpretation of her 12 micro poems selected from her book “Milk and Honey”. It specifically seeks to stylistically investigate the presence of foregrounded parts found in her poems and their linguistic forms. The obtained verbal data are analyzed through the lens of Geoffrey Leech’s foregrounding theory which focuses on deviation and parallelism. The study reveals that the foregrounded parts in terms of deviation are enjambment, omission of syllable sounds, hyperbaton, the personification of abstract nouns, and metaphorization which are linguistically categorized as graphological, phonological, grammatical, and semantic. In terms of parallelism, they are the repetition of sounds, lexeme, words, and phrases manifested through anadiplosis, epanalepsis, epistrophe, anaphora, and conduplicatio which are lexical, phonological, and syntactic. The foregrounded parts relay Kaur’s message on the importance of equalness among gender and women's empowerment as she retains her sense of heritage, and culture very evident from the interdependence of various semiotic modes that she used in her poetry. It is recommended that the objectivity of literary analysis and interpretation should be discussed intensively for the awareness of language/literature teachers and learners.

Visual representation of the menopause in Iran

This study investigates the visual representation of menopause in Iranian health promotion resources. The author uses Kress and Van Leeuwen’s approach to visual analysis (Kress G and Van Leeuwen T (2002) Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication 2(3): 343–368 and Kress G and Van Leeuwen T (2006) Reading Images: The Grammar of Visual Design, 2nd edn. London: Routledge.) to study publications about the menopause by the Iranian Government and two major medical universities. Two research questions guide the investigation: How do Iranian health promotion resources visually represent the menopause? How do these representations reflect and reproduce cultural values and religious beliefs? Findings reveal that: the publications address both men and women, but exclude widows, single and divorced women; the publications encourage women to stay healthy and lead a fulfilling post-menopausal life; and that they have a distinctly Iranian approach to the use of visual communication, characterized by abstract modality, visual symbolism and the emotive expressivity of a vivid use of colour. However, the publications have not been updated since 2008 and are currently not distributed so that their generally positive message cannot, at present, challenge some traditional masculine beliefs which restrict women to the roles of wife and mother.

Representations of science content in a primary classroom: Combining long and short timescales for multimodal analysis

AbstractThis study reports on a case study about multimodal work in a primary physics classroom focusing on forces. Previous research reports that students benefit from multimodal work in science classrooms. Yet, few systematic studies have been performed to reveal how students represent their experiences and ideas of science phenomena over time within and across different semiotic modes (e.g., through action, speech, writing, and image, including multimodal ensembles). The design of the lessons was built around a number of experimental activities, starting with a puzzling phenomenon and where students for each experiment predicted, observed, described, and tried to come up with explanations. Based on social‐semiotics, we combined analysis at an overall level (long timescale and large grain size) and a detailed level (short timescale and small grain size) to shed light on how the science content was represented within and between modes over the teaching and learning period, and what content was expressed in these representations. Our findings reveal how students moved from focusing on attributes such as a “heavy” regarding artifacts used in the experiments, towards the central physics processes, such as one object exerting force on another object. Furthermore, we were able to detect that such “signs of learning” were shown in students’ small‐group discussions and multimodal texts following a carefully orchestrated multimodal exposition by the teacher. Hence, such a careful multimodal orchestration appeared to be critical for the students’ meaning‐making about the science content.