Semiotic Modes Research Articles

Seeing more than reading:The visual mode in utilities' sustainability reports

The analysis of semiotic modes of communication used in corporate reports is very relevant for understanding how meanings are communicated to readers or viewers. But researchers' interest has mainly focused on verbal and numerical modes of communication, and the focus on the visual mode has been very limited. This paper contributes to filling this gap through an explorative analysis of how the visual artefacts are used with the text (verbal and numerical) in utilities' corporate sustainability reports. Results show that visual artefacts are more used for captivating readers' or viewers' attention, spatializing and materializing concepts than infiltrating meanings.

Deployment of gestures in the semiotic construction of scientific knowledge: a systemic functional approach to pedagogic semiosis

Abstract A variety of semiotic resources makes the construction of scientific knowledge possible and meaning-making resources are conveyed by certain semiotic modes. Next, numerous studies have demonstrated the pedagogical importance of gestures in the demonstration of scientific knowledge in the classroom. Drawing on social semiotic systemic functional theory and legitimation code theory, this study explores the types and the role of gestures in the semiotic construction of scientific knowledge in pedagogic semiosis and their pedagogical values for the meaning-making of science content. With this aim, a gesture observation protocol (GOP) is developed to annotate and analyze data. Two science lectures, which are broadcast remotely for middle school students including gifted students, are observed. Systemic Functional Multimodal Discourse Analysis (SF-MDA) method is employed to analyze and evaluate data. Results demonstrated that the deployment of gestures that contributes to the construction of scientific knowledge is low and deployed gestures mostly do not have pedagogical value. It is recommended that teachers should be more conscious and educated about deploying gestures to present scientific knowledge efficiently.

Student teachers’ reflections on semiotics in Grade 3 isiXhosa literacy lessons

Background: Globally, teaching practice has been at the heart of teacher education programmes. For quality teaching and learning, literacy student teachers are expected to develop metacognitive attributes and critical thinking to integrate theory and practice. Because of the dominance of autonomous models in literacy teaching and learning nationally and internationally, literature continues to report poor literacy attainment, especially for indigenous language learners. Contrasting this deficit view, this article employed languaging as a lens to describe student teachers’ reflections on their interactions with Grade 3 learners using multimodal and linguistic repertoires, which they both bring from socio-cultural contexts as well as utilisation of embodied representational modes.Aim: To explore how Foundation Phase (FP) student teachers used languaging and semiotic modes to enhance literacy teaching and learning in Grade 3 classrooms.Setting: An Eastern Cape Institution of Higher Education.Methods: In this qualitative study, four purposely selected FP isiXhosa Home Language student teachers used reflective journals to articulate their individual and peer classroom literacy practices. Data were thematically analysed.Results: Findings revealed the importance of acknowledging authentic and diverse linguistic resources that learners bring from their socio-cultural backgrounds as well as the use of multimodal literacies in the classroom context.Conclusion: This study concludes that languaging allowed learners and student teachers to exploit multimodalities and linguistic repertoires that they bring from their socio-cultural backgrounds.Contribution: This study demonstrates the pedagogical literacy strategies that created live dialogical engagements between student teachers and learners. These can be useful to teacher educators as well as teachers in the Foundation Phase contexts and thus improve literacy teaching and learning, especially in indigenous languages.

Developing Multimodal Literacy in the Business English Reading Class: A Case Study of Students’ Presentation PowerPoint Slides

This study aims to investigate the students’ multimodal literacies by analyzing their presentation PPT slides in Business English reading class in two dimensions of discourse genre and multimodality. It discovers certain preferences and patterns in their multimodal semiotic practice as well as some drawbacks in the meaning-making process. Students tend to adopt visualizing genre over other genres to demonstrate factual events in their PPT presentations. They are inclined to use language and image more frequently than other semiotic modes in their PPT meaning making. Besides, their technology and media literacy needs to be strengthened in searching, criticizing, analyzing online information resources. Eventually it provides teachers with pedagogical scaffolds to teach the viewing and representing of multimodal texts effectively.

Constructing Ethnic and National Belonging: Ingrian Finnishness in a Museum Exhibition

Abstract A temporary exhibition called Ingrians—The Forgotten Finns was on display at the National Museum of Finland in 2020. The exhibition presented the ethnic minority of Ingrian Finns, concentrating on their memories of persecution in the Soviet Union, ethnic mobilization, and migration to Finland. By examining the exhibition's meaning potentials across different semiotic modes, the article explores the ways in which it discursively constructed Ingrian Finnishness. The findings suggest that the exhibition strongly argued for the inclusion of Ingrian Finns in the Finnish nation and highlighted cultural features associated with Finnishness, but also presented them as a distinct group. Remembering the difficult past was viewed as the very essence of the collective identity.

Realization of Cultural Values and Trans-Languaging in The Visual Images of an Indonesian Picture Book

This study investigated the realization of cultural values and trans-languaging in a picture book. The current discussion of trans-languaging has been focused on the verbal texts found in school textbooks, children’s comics or conventional novels. On the other hand, the study offers insights into trans-languaging that is realized in a different semiotic mode, namely visual images. The primary data for the study is a picture book written by an Indonesian writer Clara Ng entitled ‘Kancil Yang Baik’.’. The main methodological tools for the study are Wei’s (2018) concept of translanguaging and Kress and Van Leuween’s (2006) concept of the grammar of visual design, focusing on the compositional function. The data analysis is conducted to elaborate on the features of cultural values and trans-languaging in the visual image format of the picture book. Further, interpretation will be made on the implications of how these features may be used in various reading activities. The results of the study indicate that visual images in a picture book are a rich source of cultural content. Therefore, it is advisable to optimize reading activities that also focus on the visual images of picture books.
 Keywords: translanguaging, picture books, visual images, cultural values

Multimodal analysis of humour as an engagement strategy in YouTube research dissemination videos

Science popularisation has received widespread interest in the last decade. With the rapid evolution from print to digital modes of information, science outreach has been seen to cross educational boundaries and become integrated into wider contexts such as YouTube. One of the main features of the success of research dissemination videos on YouTube is the ability to establish a meaningful connection with the audience. In this regard, humour may be used as a strategy for engagement. Most studies on humour, however, are conducted solely from a purely linguistic perspective, obviating the complex multimodal reality of communication in the digital era. Considering this background, we set out to explore how humour is used from a multimodal point of view as an engagement strategy in YouTube research dissemination. We selected three research dissemination videos from three distinct YouTube channels to fulfil this aim. After an initial viewing, 22 short humoristic fragments that were particularly engaging were selected. These fragments were further explored using Multimodal Analysis - Video (MAV)[1], a multi-layered annotation tool that allows for fine-grained multimodal analysis. Humoristic strategies and contextual features were explored, as well as two main types of modes: embodied and filmic. Results show the presence of 9 linguistic strategies to introduce humour in YouTube science dissemination videos which are always accompanied by heterogeneous combinations of embodied and filmic modes that contribute to fully achieving humoristic purposes. [1] Multi-layer annotation software used to describe the use of semiotic modes in video files. By using this software, researchers may analyse, for instance, how gestures, gaze, proxemics, head movements, facial expression, etc. are employed in a given file.

Multimodal engagement strategies in science dissemination: A case study of TED talks and YouTube science videos

The growing interest on science dissemination offers new opportunities to communicate science openly to various audiences, but also brings on the challenge of adapting to an audience that does not share the same academic background. This adaptation has been referred to as recontextualization. In the case of the formats that concern this study, that is, TEDx Talks and YouTube science dissemination videos, their multimodal nature suggests that recontextualization, and therefore engagement as a crucial aspect of this process, is likely to go way beyond purely linguistic aspects. The aim of this study is to unveil how engagement strategies in two science dissemination formats (a face to face talk and an online video) are realized through complex multimodal ensembles, and to highlight differences across them. In order to fulfill this aim, two talks by the same presenter and dealing with similar content were selected for analysis: a TEDx talk and a YouTube science dissemination video from the channel PBS Space Time. The recordings were annotated using the software Multimodal Video Analysis. The annotation included engagement strategies; embodied modes, that is, modes carried out using the body; and, in the case of the YouTube video, filmic modes, that is, modes triggered by the editing process of the recorded video. Our results show that the role of both embodied and filmic modes is paramount in the realization of engagement strategies. Our findings also bring to the fore significant differences in the ways in which the two distinct audiences are engaged, concerning the frequency and use of both semiotic modes and engagement strategies.

Transcendental and Material Silence: A Multimodal Study on Silence in Team Meetings

Silence in management and organization studies has been predominantly understood as something negative. However, recent examples have highlighted silence as a positive element in learning and organizing. We contribute to prior literature on positive silence and multimodality by arguing silence can operate as a semiotic mode that mobilizes resources for meaning-making. Ten team meetings in a financial organization in Europe were investigated. Visual ethnography was mobilized to gather data through interviews, observations, and photographs. Our analysis identified two types of silence—transcendental and material—that both function through three mechanisms to resemiotize meaning. A framework is presented to situate silence in relation to verbal and visual modes. Three contributions are made to studies on silence and multimodality: extended conceptualizations of silence, silence as a semiotic mode in itself, and methodological pathways for studying silence. In addition, practical implications for team meetings and silence in the workplace are discussed.

Multimodality of Make Over Power Stay Advertisement

This research entitled "Multimodality of Make Over Power Stay Advertisement" is a multimodal research which contains interactions between verbal and visual semiotic modes to find the generic structure and the intent of the advertisement image. The purpose of this study is to identify potential generic structures in the advertisement image of 'Make Over Power Stay' and this thesis is prepared using a qualitative descriptive method to describe hidden meanings that are displayed in verbal and visual elements. In this thesis, the object that the writer analyzes is "Make Over Power Stay" advertisement. The data collection instrument in this thesis is selecting, focusing, simplifying, and abstracting empirical data. This analysis focuses on the Generic Structure Potential concept by (Cheong, 2004)in analyzing verbal and visual texts. The results of the analysis of the verbal text used in the ad advertisement are Announcements and the visual element most commonly used in the advertisement is the Lead.

Bold and impactful: a reappraisal of Gunther Kress’s (social) semiotic legacy in the light of current multimodality research

Abstract This paper is a critical appreciation of some of Gunther Kress’s central (social) semiotic notions: i.e., motivation, materiality, rhetorical aptness and semiotic mode versus medium. These will be discussed in relation to four landmark models of sign-making and semiosis by Saussure, Peirce, Bühler and Jakobson. Based on these comments, the paper identifies the persistent difficulties current multimodality research faces in defining mode and in devising linguistically unbiased grammars of non-verbal modes. Finally, the argument is advanced that multimodal genre and discourse interpretation in particular deserve to be re-developed. The paper critiques Kress’s insistence on motivation as a universal principle of sign use and his overemphasis on materiality to the detriment of grammar, while praising his overall (social) semiotic legacy for multimodality research as far-sighted and lastingly influential.

Exploring physical sciences learners’ graphical interpretation of rate and extent of reaction graphs

Graphs are very important in all areas of science, and they are an essential part of tertiary, high school, and primary school learning, worldwide. A solid understanding of graphical interpretation is essential for understanding today's world and becoming scientifically literate. The aim of this study was to explore grade 12 learners’ graphical interpretation of the rate and extent of reaction topic. A mixed-method methodology was adopted for this study. A purposive sampling technique was used to sample participants from the accessible population in King Cetshwayo District KwaZulu-Natal province in South Africa. One-hundred and forty-six (146) grade 12 Physical Sciences learners formed the sample. A validated two-tier diagnostic questionnaire and semi-structured interviews were used to collect data. Quantitative data were analysed using SPSS version 25 and inductive coding was used to qualitative data. The Johnstone triangle and the Peircean semiotic modes were employed as the theoretical framework. The findings indicated that learners rely on definitions to interpret graphs. Most of the learners failed to interpret the salient features of the graphs. The findings of this study are diagnostic, and they assist module designers and educators in determining challenges learners face when interpreting graphs in chemistry. Implications for instructional approaches particular to the rate of reaction graphs are discussed. Further studies are needed on instructional practices and their effect on students’ ability to interpret rate and extent of reaction graphs. Keywords: Graphical interpretation; rate and extent of reaction; Johnstone triangle

The role of rhythm in science-animated videos: construing entities and bridging across different semiotic modes

Recent decades have witnessed the increasing popularity of animation used for science education. However, the affordances of animation are still yet to be described in a comprehensive manner. Working on a corpus of online science animated videos, this article firstly proposes a visual rhythm system for animation, and then examines the convergence and divergence of the rhythm in animation and language at different levels of analysis. The author found that different visual rhythms in animation function to construe different quantities and properties of entities. In terms of the intermodal relations between animation and language, at shorter wavelengths, animation can be in sync with the rhythm in language, with different ‘beats’ in animation converging with different phonological units in language; at longer wavelengths, the larger rhythmic pattern of animation can function to scaffold the generic structure of language. This article enriches the systematic description of the meaning-making potential of animation, which also informs empirical studies focusing on the effectiveness of animation used for science education.

Мультимодальна когезія в парфумерній рекламі

The article deals with the investigation of perfume advertisement in the aspect of its semiotic mode cooperation and interactions which are reflected as cohesion within one and the same semiotic mode and between different semiotic modes as well. The objects of the research are perfume advertisements and their formal elements which create the semiotic structure. Within the verbal semiotic mode, such leading formal elements as functional blocks, subblock names and their lexical embodiment were distinguished. The most developed functional block contains name, naming complex with A-1 brand (company), B-2 individual name, C-3 genitive (product) name, and D-4 additional information. These names are the key markers of lexical cohesion. Lexical cohesion within the verbal semiotic mode is realised through the mechanism of reiteration, including repetition and synonym usage. Lexical cohesion occurs within one and the same functional block as well as between different functional blocks. Within the visual semiotic mode, there are such formal elements as composition, plan, balance, and depth. The advertisement visual semiotic mode has such a set of parameters as a plan, visual component, variable and constant details of a visual component. The advertisement depth is realised using front, middle, and back plans. The main visual components are a perfume bottle, model and scenery. The model has the most detailed visual representation including the model's image, code of costume, and code of behaviour (grasping body language – pose, imitation of behaviour and facial expression). Colour as a media language is closely connected with the process of visualisation and occurs in all visual components providing visual repetition as a basic colour or its shade. Visual components (objects) are repeated in different plans, providing further cohesion between different plans. There is a definite correlation between verbal components-naming units and their visualisation. It is the case of further cohesion between verbal and visual semiotic modes. The olfactory semiotic mode uses the mechanism of synaesthesia when smell is represented involving verbal description and optical visualisation but this aspect of research needs more detailed investigation and deeper revealing and analysis.

Accumulation of words: lists in the Italian public discourse

In written and spoken language, lists are flexible instruments that take on different functions, such as grouping elements in a stable set or suggesting others to come. Umberto Eco perceives lists as a semiotic modality of vertigo-like accumulation, which offers a way to speak of something whose boundaries one does not know and to place a limit upon the entropy of the mass of things (Belknap 2004; Eco 2009). In public discourse, lists are a helpful device to express opinions and slogans in a constrained space (for example, on Twitter) through a cohesive and coherent collection of elements placed on the same level around a semantic area that the recipients must interpret. This article examines how verbal lists exploit two basic textual strategies in Italian public discourse. The first strategy consists of a collection with precise extensional boundaries and no alternative items. The second is an assortment of items that allude to a set of possibilities with nuanced extensional boundaries. Through these two strategies, the recipient can create two textual semiotic modalities due to the inference procedures: lists with a closed textual modality, closed fabula, and lists with an open textual modality, open fabula (Eco 1979a).

An Intersemiotic Analysis of the Arabic Dubbed Version of Disney’s Frozen Let it Go

: In today’s digital environment, verbal, visual and sound resources constantly collaborate to construct meaning in the audiovisual mediascape. Audiovisual Translation (AVT) studies have thus shifted focus to examine how such multimodal orchestration and meaning shifts are captured and transferred from one culture to another. Songs’ AVT in animated musicals imposes (non)verbal constraints on translators who have to adapt/adjust to transfer not only the verbal and visual codes but also the musical. This article examines the intersemiotic relations among the three different semiotic modes when dubbing Let it Go from English into Arabic. It builds on Reus’ (2020a/b) Triangle of Aspects to fit both academic and practical purposes. The results suggest that dubbing a song mainly relies on a quadrangular of parameters: meaning-making, technical, interpretative, and performative elements of translation. Verbally, the findings indicated that the translator oscillated between full adherence to disregard of the semantic meaning. Nonverbally, the findings revealed mixed results regarding the OS-DS (in)congruencies in which the translator foregrounded and/or backgrounded certain aspects to conform to imposed constraints. Four different intermodal relations were identified: addition, enhancement, modification, and deletion. This study offers a methodological contribution to the AVT scholarship, positing a framework that can be systematically followed in future research.

Візуальний семіотичний модус парфумерної реклами

The article deals with peculiarities of perfume advertisements, in particular, their visual semiotic mode structure and its constituents creating the recognitive features of this mode. Constituents-details are recognized as visual formal elements which bear main and significative visual information. The central objects of this research are perfume advertisements and their visual formal elements which create the whole visual semiotic structure. All formal elements are divided into three functional groups according to their function in the visual space of advertisements. Primary group of visual formal elements performs the delimitative function. This function represents the contouring of formal elements as separate objects (product, model, and background). Secondary group of visual formal elements provides the embodiment. This function represents the outline, when formal elements of the primary group are filled with structural components – formal elements (line, plane, composition, depth (space), shape). Intermediate group of visual formal elements performs the colourative function when formal elements of primary and secondary groups are coloured. This group is formed with variations of colours. Within the visual semiotic mode each group (primary, secondary, and intermediate) includes formal elements which demonstrate complex and integrative nature of visual inner structural organisation. Visual formal element of any group has its own boundary, shape, or outline, border and function. Another feature of formal elements is a set of parameters. They are recognized as qualitative and quantitative characteristics of formal elements. Parameters are bearers of attributes. The main function of parameters is to accompany any formal element making it distinctive, recognizable, well-developed. Division of formal elements into some related and corresponding smaller sub-elements reflects the integrated and complex nature of visual semiotic mode and its formal elements. Formal elements are separated into sub-elements. Simultaneously parameters are dissociated into sub-parameters. Correlation of formal element-parameter reflects the boundaries between these units as clear-cut. Boundaries in the system of sub-units (sub-element – sub-parameter) are diffusive. Integrated nature of visual semiotic mode is connected with the existence of intermediate sub-state which is formed with corresponding sub-units. These sub-units function together in the state of self-equilibrium which gives the visual semiotic mode some flexibility. It happens when boundary between sub-units becomes diffusive and transparent. Sub-elements and sub-parameters percolate mutually through that weakened boundary. That mutual percolating leads to the mutual mixing when recognitive features of sub-elements and sub-parameters become common and shared for sub-units. Flexibility is supported with the dense net of sub-parameters and colouration of all visual formal elements.

A Multimodal Discourse Analysis of BBC Documentary Chinese New Year: The Biggest Celebration on Earth

With the rapid development of multimedia and the internet, traditional discourse analysis, which only studies text, is gradually transferred to multimodal discourse analysis (MDA), studying image, language, color, sound, etc. Informed by Systemic Functional Linguistics, this paper tries making a multimodal discourse analysis of the BBC documentary Chinese New Year: The Biggest Celebration on Earth (CNY). Based on Kress & Van Leeuwen’s Visual Grammar and Zhang Delu’s visual-verbal Inter-semiotic relations, the paper examines how different semiotic modes combine to provide meaning in a documentary. After a synthesis analysis, it is concluded that the employment of various modes in this dynamic discourse contributes to the spreading of Chinese culture and the representation of “Cultural China” image. It is hoped that this paper will provide a new perspective for the semiotic studies of documentaries. The study's importance lies in demonstrating how numerous modes and sub-modes are employed in the source domains as part of a larger persuasion campaign to skew public perceptions of China. Additionally, we look at the metaphorical implications. Keywords: multimodal discourse analysis, systemic functional linguistics, BBC documentary. DOI: https://doi.org/10.55463/hkjss.issn.1021-3619.60.40

Visual Grammar and Meaning: A Multi-Modal Discourse Analysis of Selected Jubilee Life Insurance Company Advertisements

With the development of technology and advancement of media, the advertisements are no more mono-modal discourses. Instead, advertisements utilize multiple modes to catch the eyes of the viewers and to win the hearts of desired customers. Today traditional literacy alone fails to interpret modern day multimodal texts like advertisements. This paper based on Kress and Van Leeuwen’s (1996) theory of Visual Grammar, analyzes two non-product advertisements of Jubilee Life Insurance Company, Pakistan, to find out how visual communication is carried out through a relation between these semiotic modes. The analysis has shown that semiotic modes are used by the ad makers to represent participants and also to maintain a relation between the represented participants and the viewers. The study also highlights how the ideology of the involved participants works to communicate the desired meanings to the target audience through these the interrelated systems of semiotic modes within a business discourse.

Gesture at the crossroads

Abstract Task-based exchanges are known to have multiple co-occurring interactive structures that prompt interlocutors to integrate various semiotic fields (Goodwin 2000). In addition to underlying norms of cooperative interaction, task-based exchanges include joint accomplishment of some activity. In this study, a group of co-workers (two hearing and one deaf) engaged in a team-building exercise where they jointly constructed a container fit for protecting an egg. They communicated, in part, via a sign language interpreter. Clark’s (2016) methods of communication were applied to the interpreter-mediated interaction (IMI) to determine whether certain semiotic modes were more (or less) accessible to the respective linguistic groups. Micro-analysis of three activity phases revealed moments when the participants responded to and integrated semiotic fields displayed in front of them. While not all content was accessible, like the complex depictive structures produced by the deaf person, some instances of mutual understanding without interpretation were achieved via integration of meaningful components of visibly accessible semiotic fields. Results challenge traditional views of IMIs where interpreters are positioned as channelling communicative content exclusively through them.