Social Semiotic Approach Research Articles

The Impact of Multimodal Discourse Analysis on the Improvement of Iraqi EFL Learners' Reading Comprehension Skill

The present study was conducted to investigate the impact of multimodal discourse on Iraqi EFL learners' reading comprehension skill. To achieve the goal, a group including 40 Iraqi EFL learners participated in the free Reading Comprehension course based on their scores in a Placement Test. Selected participants were randomly assigned to control (N=20) and experimental (N=20) groups. Prior to the start of the course, both groups were given the pretest (reading comprehension test). The two groups received the same language instruction, but the experimental group was exposed to multimodal treatment (multimodal use of discourse, e.g. video and picture based on Halliday’s social semiotics approach) while the control one did not receive any multimodal treatment. Having accomplished the experiment, participants of the two groups completed two reading comprehension tasks as the posttest. Results of t-test indicate significant outperformance of the experimental group over the control one.

A multimodal critical discourse analysis of city as text: investigation of meaning metafunctions of Rasht’s Imam Khomeini Street

Studying the city as text is influenced by a phenomenon called the linguistic turn. This is because the text of the city is composed of different semiotic systems. This study employs multimodal discourse analysis as its theoretical framework. Through a close examination of the organizational, interactional and representational metafunction of the city text, the authors aim to highlight its characteristics as a communicational resource to provide a multimodal reading of the city. They demonstrate how applying a social semiotic approach to the physical dimensions of a street contributes to a particular understanding of how this public space is used or navigated and how the formal management of the flow of cars and people is materialized. Here the city is classified into different levels of resolution based on urban morphology’s description of the urban tissue. The text of the city is constructed through dialogue among meaning metafunctions on one level and among different levels. Examining different aspects of meaning in the text of Imam Khomeini Street and the city of Rasht indicates how, in such a text, the producers and participants communicate through the material aspect of the city. Uses of semiotic materials are ideological and their affordances have evolved in specific socio-political moments. In this street, the city as an institution and as a place of authority having the biggest share in the production of this text persuades other participants to follow and reproduce a certain narration of the reality. Controlling and training the body, on one hand, and using the urban space for advertising the dominant discourse, on the other hand, are the boldest motives of this street’s design, and the formation of the different dimensions of meaning is somehow at the service of the consolidation of a particular kind of everyday life.

Semiotic staging of the ideological point of view in Amiri Baraka’sSlave Ship: A social-semiotic approach

Semiotic staging of the ideological point of view in Amiri Baraka’s<i>Slave Ship</i>: A social-semiotic approach

Michel Pastoureau and the history of visual communication

Despite early and ongoing calls for a systematic engagement with history, social semiotics has largely emphasized research on the synchronic rather than diachronic dimensions of meaning-making. And while the ‘instability’ of semiotic practices (see Kress’s Multimodality: A Social Semiotic Approach to Contemporary Communication, 2010) and the importance of semiotic change (see Van Leeuwen’s Introducing Social Semiotics, 2005) have become key themes in semiotics, there is still a need for a dynamic approach to the study of visual and multimodal communication, focusing not only on describing how meaning-making resources and their uses are changing, but also on why they are changing. In this article, the authors focus on the importance of the work of medieval historian Michel Pastoureau for the social semiotic study of visual communication, highlighting that this work can help us further refine and even rethink key social semiotic concepts such as modes and media, provenance, and context. Pastoureau’s work shows how we can make theoretical statements about instability, change and innovation more concrete and, ultimately, empirically based. His approach can also help us understand semiotic change and its relation to social and cultural (and also economic and technological) change more broadly, often with the aid of the (crucial) normative discourses that shape semiotic practices over time.

‘Smash this sissy boy’s mouth, Cuiguo!’: Framing and humor in gay Taiwanese YouTubers’ self-disclosures about being bullied

Sharing one’s past experience of being bullied is expected to be painful, but gay Taiwanese YouTubers’ self-disclosures about how they became victims of homophobic bullying are frequently accompanied by humor and shared laughter. Their interactions are also presented multimodally with other visual and audible semiotic resources. Adopting a multimodal social semiotic approach and a framing approach, this study analyzes three YouTube videos about homophobic bullying. Findings suggest that the gay Taiwanese YouTubers frequently shifted between the sociable frame and the audience-targeted frame when interacting with their interactants and audience. They also assumed different narrative roles, serving as the story-teller, the commentator, and the fictionist. How these gay influencers articulate their opinions on behalf of the LGBTQ community also helps them to assert themselves as eloquent, knowledgeable, and humorous sexual moderns.

The representation of cultural identity in Melaka Portuguese cookbooks

ABSTRACT The Melaka Portuguese are descendants of unions between the Portuguese, who conquered Melaka in the early 16th Century, and locals. The language and culture of this minority hybrid community have survived to this day in Malaysia. Over the last decade, there has been a push to distinguish their food, and hence, cultural identity, from other communities in Malaysia. However, there has not been much focus on the cultural representation of this “home-grown” group of people in discourses related to food. To begin addressing this gap, this paper addresses the question of cultural identity represented in four Melaka Portuguese cookbooks. The focus of this paper is on the use of linguistic and visual elements related to food heritage in four Melaka Portuguese cookbooks to examine how they are used in the construction of cultural identity among the community. These elements were analyzed using a social semiotic approach. The linguistic elements in the texts were found to be related to the cultural identity of the Melaka Portuguese people through inclusion of their family narratives and stories in the cookbooks. Visual elements were also used to portray the Melaka Portuguese culture through depictions of traditional kitchenware and food preparation techniques.

Covidly humorous memes

The analysis of memes posted on Moroccan Facebook pages during the first wave of Covid-19 pandemic shows that the use of humour by Moroccans is not only motivated by achieving mirth but it also vehicles critical views about issues of common concern debated in the digital public sphere. Some of these memes were used to cope with fear and uncertainty. However, most memes harboured mixed feelings about the situation and were used for social control and the expression of conflict and resistance, addressing issues of behaviour, governance and communication. This article uses a social semiotic approach to analyse the collected memes (460 from personal and communal pages) as a multimodal discourse in terms of context, culture, and media affordances. This article contends that the study of these memes can be a key to understanding how Moroccans used humour to cope with danger and radical uncertainty, build identification and strengthen social cohesion. It also highlights the polyvocality of humour in times of the pandemic and the gradual shift from inclusive, conformist and sympathetic humour to disparaging, exclusive and challenging humour as the pandemic lingered, consensus began to crack, social control was challenged and injunctive norms were replaced by survival values. The results show how these memes are indicative of the way humour changes mechanisms and functions in terms of contingent motivations.

Towards a social-semiotic approach to visual analysis of two-dimensional games

Nowadays, video games are one of the most influential media from an economic and sociocultural point of view. Therefore, it is important to have tools to analyze such texts and the meanings and discourses they potentially convey. Based on the social-semiotic approach, this paper proposes a set of analytical categories to study the visual construction of 2D (two-dimensional) games. Although the 2D paradigm is no longer prevalent in video games — 3D (three-dimensional) games became gradually common as early as 1995 with the introduction of the Sony PlayStation — it is still used in several current titles. Furthermore, social-semiotic studies on video games are still not numerous; therefore, a focused study that systematizes methodological categories for the analysis of video games is a relevant step towards understanding the semiotic constitution of these artifacts. The present study focuses on Marvel Super Heroes in War of the Gems for the Super Nintendo (CAPCOM, 1996). By applying the categories presented here on the Brazilian stage of the game, this article suggests ways to analyze the visual mode of video games, taking into account their multimodal nature and discursive potential.

How to brand a border despite its wall? A social semiotics approach to cross-border place branding

This paper explores the ways in which borders are likely to be used as a semiotic resource in cross-border place branding. We develop a conceptual framework, distinguishing two meaning-making strategies: one that seeks to valorize borders as assets for cross-border place branding and another that relies on their invisibilization in order to promote, implicitly, the legitimacy of a seamless cross-border regional imaginary. For each meaning-making strategy, specific border discourses are identified. Based on a social semiotics approach, we analyze the case of Cali Baja, a place branding initiative that seeks to promote a bi-national mega-region straddling the US-Mexico border between San Diego and Tijuana. Despite the militarization of the border and the highly visible physical wall that divides Cali Baja, a selective branding discourse framed around favorable border images and representations is brought to the fore. It is therefore at the cost of the partial (in)visibilization of the border that the mega-region can be given value and promoted on the international scene, but it is also the discrepancy with the identity of the bi-national region that precipitated the failure of this place branding initiative.

Digital Storytelling: A Didactic Approach to Multimodal Coherence

The aim of this article is to explain multimodal coherence-making as a transcribing practice and how this can be used to teach multimodal, narrative, and media competences in different genres. In multimodal arrangements, language makes images readable in specific ways and images make language understandable in different ways. This results in an abductive understanding process that can be used in teaching and learning contexts. This idea of meaning-making is based on the social semiotic approach of style. According to the understanding of semiotic meta functions, this approach considers style as the practice of selecting, forming, and composing semiotic resources. These stylistic practices realize a subjective appropriation of discursive and habitual patterns, which are carried out within the semiotic and technological dispositions (affordances) of the situationally used media infrastructures. In this sense, digital storytelling is a multimodal style practice with digital tools. Multimodal storytelling in educational contexts means that teachers and learners are prompted to bring the communicative functions of text, image, video, and audio into narrative coherence. Based on a journalistic Instagram story, this article reconstructs the media-practical, multimodal, and narrative skills that are prototypically necessary. Based on this analysis, these competencies are operationalized to make them usable for new teaching/learning arrangements using digital storytelling.

Sculpting the interpersonal: towards a social semiotic framework for analysing interpersonal meaning in statues

The underlying question of this article is ‘how do statues convey interpersonal meaning?’ To answer this question, the authors briefly critically examine the current social semiotic analytical framework for statues and develop a revised framework for analysing interpersonal meaning in which features from Kress and van Leeuwen’s Reading Images: The Grammar of Visual Design (2006) and selected features from O’Toole’s framework from The Language of Displayed Art (2011) for analysing sculpture are integrated. These features are also extended and/or complemented by incorporating features obtained from research into the fields of gesture, body language and facial expression. Further, in keeping with Systemic Functional Linguistic-inspired research, system networks are used to map out the potential material and semantic (interpersonal) features for figurative statues and to present possible configurations among these features. Although the focus of this article is on interpersonal meaning, it is acknowledged that within a social semiotic approach, there is an interdependency among interpersonal, experiential (representational) and textual (compositional) meanings, and that these configure within a specific context of situation. This interdependency is only briefly attended to in the article itself, but the proposed framework provides a starting point for developing an account of the way that interpersonal features and their realizations in statues may configure with representational and compositional features and their realizations. With the current world focus on statues and their sometimes controversial social meanings, this article offers a timely opportunity for a range of users such as social semioticians or art educators and students to consider, through a systematic analytical framework, the way in which statues may relate interpersonally with viewers, and provides a key step towards accounting for the way that configurations of interpersonal, representational and compositional features may construe contextual tensions in relation to the overall message conveyed by a statue.

Developing multimodal communicative competence: adolescent English learners' multimodal composition in an after‐school programme

AbstractThis qualitative study aimed to investigate adolescent English learners' multimodal communicative competence. Grounded upon a social semiotic approach with the concepts of affordances and transmediation, this research was carried out in an after‐school programme at a secondary school in Korea. Three high school students participated in the multimodal composition project and produced narrative and argumentative multimodal texts. Multiple sources of data were gathered from interviews, video recordings with fieldnotes and student artefacts. The data were analysed using an inductive and deductive approach, along three orientations: modes as social action; modes as framing devices; and modes as agency representation. The students' multimodal texts were analysed based on multimodal text analysis. This study demonstrated that the participants personalised, avoided, modified and orchestrated semiotic modes to engage in multimodal composition. The study also revealed that the students' selection and use of semiotic modes were influenced by their audience awareness, genre knowledge and identities, and vice versa, all of which contributed to developing their multimodal communicative competence. The paper concludes with pedagogical implications for multimodal composition practices and suggestions for future multimodal composition studies.

Intersemiotic projection and academic comics: towards a social semiotic framework of multimodal paratactic and hypotactic projection

Abstract Intersemiotic projection is one of the most common configurations in the knowledge construction process of academic comics. Although previous studies address some general features of intersemiotic projection, further research on interdependency relations of intersemiotic projection is needed in order to map out the whole system. This study, based on the social-semiotic approach to multimodal studies, proposes a systemic framework of image-text paratactic and hypotactic projection in academic comics. This framework identifies three sub-categories of paratactic projection and hypotactic projection, respectively: (1) Emergent, (2) Adjoined, (3) Sequential, (4) Compound, (5) Subsumed, and (6) Fused. The use of these projection configurations contributes to the realization of the communicative purposes of academic comics to evidence, exemplify, emphasize, explain, elaborate, and visualize the discussed knowledge. This research attempts not only to contribute to the theoretical development of multimodality from the social-semiotic perspective by refining and enlarging the multimodal hypotactic projection framework but also to provide an effective and applicable tool for comics analysis.

Recovery rhetoric: a critical discourse analysis of substance use recovery

ABSTRACT Dominant discourses of recovery permeate substance use research, policy, and practice recommendations around the world, exercising the power to shape the identities of people with substance use issues. Drawing on Foucauldian and intersectionality theories, this study explores power operations embedded in recovery discourses constructing a certain type of recovery for a certain type of subject. A critical discourse analysis using van Leeuwen’s social semiotic approach examines two Canadian federal recovery documents to consider the discursive representation of people with substance use issues, legitimization of recovery, and interpretation of space. Findings suggest that substance use recovery discourses are entwined in a complex web of neoliberal, biomedical, and moral discourses by which stigmatized identities are (co)constructed at micro, mezzo, and macro levels. This study offers a unique and relevant opportunity to complicate the dominant recovery literature by challenging oppression, deconstructing normative identities, and facilitating the empowerment of marginalized populations.

A Spatial-Semiotic Narrative of the Martyr’s Memorial Site in Amman

Social semiotic approaches to spatial discourse analysis are employed to analyse the predefined arrangements of spaces and semiotic resources in three-dimensional spaces. These spaces and resources are recognized as spatial-texts that construct a discourse which communicates predefined meanings and ideologies. Aiming to advance the growing interest in spatial semiotics in different socio-cultural contexts, this study scrutinizes the spatial semiotic resources in the Martyr’s Memorial and site in Amman, the Capital city of Jordan. The analysis is constructed through an approach of locative and reflective observation of the researcher’s, and the visitors’, experience with the different semiotic resources in the Memorial site. Then, these resources are interpreted on the basis of social semiotic approaches of spatial-discourse analysis. The study demonstrates that the Martyr’s Memorial and its site encompass manifold semiotic resources which make a series of spatial-texts which are organized and arranged along a predefined narrative path. These spatial-texts make a coherent discourse that communicates and unfold well-established meanings and ideologies about the Arabic, and Islamic, conception of martyrdom, the role of the Jordanian Armed Forces, and the legacy of the Hashemite ruling dynasty. Such meanings and ideologies are reproduced on the basis of social, religious, and historical backgrounds.

Multimodal Gestalts and Post-pandemic Utopianism: Drum and Bass on the Bike

Framed within a society insidiously damaged by the novel coronavirus pandemic and associated lockdown restrictions, this article examines the drum and bass on the bike initiative of British DJ Dom Whiting. The initiative comprises of Dom riding a tricycle through various urban landscapes while broadcasting live to social media from on-board mixing decks. Since the first two solo rides in early 2021, thousands of individuals have accompanied him through twelve urban landscapes in addition to millions more participating across social media through views, comments, likes, and shares. Situating YouTube uploads as text, and positioning Dom as a harmonic navigator of change and reformation in the urban space, the article draws from a social semiotic multimodal approach to communication and details the development of three multimodal gestalts indicative of post-pandemic society. Against a soundtrack of thunderous drum and bass music, the three emergent gestalts communicate post-pandemic utopianism through the reformation of the relationship between motor vehicle drivers and cyclists, the reformation of urban road space use, and the reformation of diversity in organized cycling events.

Systemic functional linguistics as a resource for teacher education and writing development

AbstractThis article provides an illustrative and necessarily selective review of scholarship in systemic functional linguistics (SFL) since the 1960s, examining two major strands of application within the institution of education: teacher education and writing development. We explore properties of SFL that have been significant in this institutional setting, asking: “What are the properties of SFL being applied in research in teacher education and writing development?” Five foundational properties emerged from the review (i.e. language as meaning-making resource, the cline of instantiation, ontogenesis or the process of learning how to mean, the hierarchy of stratification and the spectrum of different metafunctional modes of meaning). From this basis, we explicate how SFL-based pedagogies promote language learning and literacy practices in these educational domains. The article ends by suggesting how further explorations can be undertaken to enhance understanding of the potential of this social semiotic approach and its applied contributions to solving real-world problems.

Visual Images and Interpretations: A Semiotic Analysis of Textbook Covers

This article presents a comparative visual interpretation of the cover images of the selected textbooks. A social semiotic approach was used to analyse various compositional and thematic differences among two sets of textbook covers. The compositional interpretation was applied to explore the differences and similarities in an image composition and discourse analysis was used to explore their implied social discourses. The findings show some significant shift as well as continuum in the image composition and underlined social discourses. For example, the newly published textbooks focus on settings, character details and character connection, and use various colours in contrast to the old textbook covers. More activities promoting gender equality were found in the new textbook covers than in the old textbook covers. Overall, the new textbooks propose positive changes in their cover image composition in quality and gender discourses. However, the images only feature-specific religious symbols, regional culture and geography rather than multicultural, multiregional and multi-religious context of Nepali society. It seems that minority religion, ethnicity and regional variations that are found in Nepal are missing in the cover images across the curriculum. This article is expected to address the gap seen in the Nepali school educational research and helpful to provide insights that the cover images used in the textbooks are important and powerful meaning makers.

Languaging dynamics of classroom interactivity: a distributed view of the pedagogic recontextualization in L2 tertiary settings

AbstractThe current study investigates classroom interactivity in L2 tertiary literature classrooms in Hong Kong and Taiwan when ESL/EFL students engage with and interpret literary texts in classroom talk as a pedagogic process of text recontextualization. It proposes a more ecological-based approach to language and languaging dynamics that is complementary to current social semiotic approaches to multimodality. It also aims to open up a more embodied analysis of the meaning-making process in tertiary literature classrooms. The multimodal investigation of real-time classroom interactivity based on a multi-scalar approach showcases an embodied coordination of vocalization and gesticulation as integral aspects of the dynamic whole-body sense-making activity that arises in the pedagogic process of text recontextualization. The dynamics vary from students’ solo speech in individual presentation to teacher–student interactions in group discussion and to student–student interactions in role-play. The distributed language view of first-order languaging dynamics demonstrates the embodied and distributed dimensions of the real-time classroom interactivity that couples pedagogic subjects to the affordances of their pedagogic environment. It also provides insights into the impact of pedagogic activities on the multi-scalar dynamics of the meaning-making process with reference to embodied speech–gesture coordination. The paper demonstrates the value of applying an ecologically embodied perspective to multimodal studies in classroom research and stimulates a re-thinking of some of the important aspects of classroom interactivity that have received little attention thus far.

Social semiotics comparison between old and new Palestinian mathematics curriculum for the ninth grade

This research aimed to analyze Algebra unit in both the old and new Palestinian curriculum for the ninth grade by using deductive content analysis based on the social semiotic approach. The results of this study showed that, in both books, the mental, behavioral, and verbal processes were more frequent than the relational, existential, and material processes. Moreover, the most frequent object used in both curriculum were the initial description, the rule, and the official definition. The most frequent representation of the mathematical object in both curriculum, was the algebraic representation, then the verbal, the numerical and the graphic representations. The research findings also indicate that in both books, the use of the pronoun “I” was the most frequent, then the pronoun “we” and then other pronouns. The imperative was frequent in the Algebra unit in the old curriculum, but less frequent in the new curriculum. The research findings, regarding the textual function in both books, indicated that the text orientation ‘previous knowledge to new knowledge’ was the most frequent in Algebra. This demonstrates that the author in both books cared about applying the new knowledge and assessing it, what means that the text progresses horizontally in both curriculum.