Semiotic Modes Research Articles

Multimodality of political digital discourse on Twitter social media

As a communication tool, social media has linguistic aspects ranging from letters to discourse, so it is important to be researched from a linguistic scientific point of view to find its meaning. This research aims to identify, elaborate, and analyze digital discourse on Twitter social media related to political elements. This research is based on the multimodal meaning related to communication modes in the form of signs such as writing, images, photos, videos, and other forms that have visual meaning in finding meaning in a communication in twitter posts. The method of this research is qualitative descriptive. The research data is in the form of posts on Twitter accounts that contain political elements. The data analysis technique in this study goes through the reduction and classification stage based on the use of communication modes in digital political discourse. Data collection techniques are carried out by documenting, listening, and taking notes. The results of the research are in the form of identification and explanation of how multimodal has meaning and meaning in digital political discourse on Twitter social media and the contribution of semiotic fashion in the form of writings, images, photos, videos, and other forms that have visual meaning in digital political discourse on Twitter social media as relevance regarding the views of the interests of discourse producers, digital discourse makers with certain semiotic modes to realize discourse, and interests aimed at the creation of political digital discourses such as identity politics, politics of struggle, and power politics.

Inferential and Macrostructural Ability in Multimodal Texts

Objective: To evaluate inferential and macrostructural ability through three types of multimodal artifacts: verbal, graphic-mathematical, and complete text (which integrates all semiotic modes). Theoretical Framework: This section explores the key concepts of multimodality theory, as well as the notions of inferences and multimodal ability, which are fundamental to addressing the research problem. The main theories and concepts supporting the research are presented, providing a solid foundation for understanding the study's context. Method: A mixed methodology was employed, combining quantitative and qualitative approaches. The quantitative phase adopted a correlational ex post facto design, while the qualitative phase was carried out through a focus group. The quantitative study followed a factorial experimental between-subjects design, considering two dimensions: the participants' professional program and the semiotic systems present in the comprehension test. Results and Discussion: It was concluded that the type of multimodal artifact does not significantly affect the efficient identification of inferences or macrostructural reduction. This suggests that students exhibit a low level of multimodal ability. Research Implications: The practical and theoretical implications of the research are analyzed, providing insights into how the results can influence or be applied to practices related to multimodal comprehension. Originality/Value: The research highlights that universities, in general, do not pay sufficient attention to multimodality, which negatively impacts students' multimodal competencies.

Embodied movement as a stratified semiotic mode: how movement, gaze and speech mean together in the classroom

Embodied movement as a stratified semiotic mode: how movement, gaze and speech mean together in the classroom

Relational Processes and Social Projections in Facebook Selfie-Quotation Juxtaposition: An Exploratory Study

Abstract The juxtaposition between selfie and quotation is an emerging user-generated Facebook content. This exploratory study is the first to show how Facebook users self-represent themselves through Relational Processes, based on the intertextuality between these verbal and visual modes. Relational Processes refer to the process of characterizing, identifying or describing a person or entity. In this study, 132 Facebook quotations were categorized into three types of Relational Processes based on the users’ selfies. The clauses were restated into I-expressions to center the users’ multimodal experiential and egocentric positioning. The findings revealed that Attributive Relational Processes, such as “I am happy,” predominated in the self-quotation juxtaposition. These relational processes are then discussed based on the potential social projections within the Philippine cultural circuit, namely: the Filipino culture of “inggit” (envy); crab mentality; being social climbers; and the neoliberal and neocolonial status of the Philippines as channeled through the Western-driven quotations. Overall, we argue that the juxtaposition of these two semiotic modes can lead to the creation of a new persona of the Facebook users, enhancing their digital self-remodeling and self-representation.

Social Semiotic Analysis of a Pet Protection Post on Instagram

This study applies Kress and Van Leeuwen's Visual Grammar framework to analyze an Instagram post advocating pet protection. The research focuses on representational, interactive, and compositional metafunctions to understand how visual and textual elements combine to convey urgency and empathy. The representational analysis highlights the dog's isolated portrayal and the use of the pronoun "me" to foster an emotional connection. The interactive analysis examines indirect gaze, close-up shots, and low angles to engage viewers and emphasize their role in pet protection. The compositional analysis reveals the strategic placement of verbal and visual elements, with bold text and contrasting colors enhancing the message's salience. This multimodal approach demonstrates the synergy between visual and textual elements, creating a cohesive and persuasive narrative. The study's findings align with existing literature on social semiotics and Visual Grammar, emphasizing the importance of integrated visual strategies in social media advocacy. The research offers practical insights for designing effective pet protection campaigns, highlighting the role of Visual Grammar in promoting responsible pet ownership and animal welfare. Future research should explore the long-term impact of these visual strategies on viewer behavior and the effectiveness of different semiotic modes across various social media platforms.

Assessing multiliteracies: a systematic review and a perspective from Singapore

ABSTRACT The literacy demands on our students in the contemporary digital age goes beyond the reading of linguistic texts to include the viewing and representing with multimodal texts that make meanings across semiotic modes. Students need to possess multiliteracies to navigate the complex multimodal communication landscape and schools are responsible to both teach and assess multiliteracies. In this paper, we perform a systematic review of studies from 2013 to 2023 on multiliteracies and assessment in the English language classrooms and identified the themes of the importance of a metalanguage of multiliteracies for assessment, the productivity of multimodal analytical frameworks for assessment of students’ artefacts, and the preoccupation with language and print literacies in multiliteracies assessment. We reflect on these themes in relation to our multiliteracies project in Singapore and discuss how we have engaged with the issues on multiliteracies and assessment. Our findings attest to the need for a commonly shared pedagogic metalanguage to assess students’ learning of multiliteracies and the need to move beyond the focus on language and print literacy in assessment. Our paper contributes towards initiating a conversation on how multiliteracies can be equitably assessed in anticipation of the changes that must come in today’s digital age.

The Moscow–Tartu Semiotic School and Soviet Art History

Although the problems of art history did not form the core of the Moscow–Tartu semiotic school’s interests, its members often turned to the material of visual art within the framework of general and specific studies of sign systems. In turn, Soviet art history in general did not show interest in semiotics. Meanwhile, the selection of problems and the approach to them in art history (mainly of the Moscow school) indicated that the reflections of art historians and philologists starting from a certain time (in the late 1960s and into the 1970s) began to develop in parallel veins. The present article provides an overview of the main problems of visual art in the works of representatives of the Moscow–Tartu school (Lotman, Uspenskij, Ivanov, Toporov and others), as well as of the adepts of semiotics from the side of art history (Paperny, Daniel, Zlydneva). In addition, the article shows how despite not accepting the semiotic mode of thinking, in their texts art historians approached the semiotic problematics of art raised by philologists (in particular, interest in the problem of the border-zone and marginalia, correlation between a word and an image in visual art, and the poetics of the historical avant-garde, etc.). This antinomic (non-)meeting of semiotics and art history in the realm of Soviet humanities in the 1970s can serve as the manifestation of the power of the unified scientific episteme of the era.

Digital dissemination practices: An analysis of explanatory strategies in the process of recontextualising specialised knowledge

Researchers and scientists are increasingly encouraged by their institutions, by external organizations and by societal demands to foster the global dissemination of their knowledge production. Such dissemination is nowadays very frequently carried out online through different digital practices and texts. The current Web 2.0 and Science 2.0 context requires complex discursive practices to recontextualise and communicate specialised knowledge in a way that is accessed, understood and accepted by multiple audiences. The use of explanatory strategies has been highlighted by previous research as playing a key role in the recontextualisation of scientific findings. Such strategies can be realised verbally and non-verbally through diverse semiotic modes and affordances of the digital medium. A taxonomy of verbal explanatory strategies (elaboration, explicitation, exemplification, enumeration and comparison) and non-verbal (visual representations and spatial organisation) is presented stemming from the data-driven analysis of a sub-corpus of web-hosted practices, which is part of the SciDis Database compiled by the InterGEDI research group at Universidad de Zaragoza. In particular, the sub-corpus consists of 30 texts: 10 author-generated digital texts – from the knowledge dissemination community The Conversation –, and writer-mediated digital texts – 10 feature articles and 10 research digests – on circular economy and sustainability. Results show that verbal explanatory strategies are more frequent than non-verbal ones and that within the latter exemplification and explicitation are most common across all three types of digital dissemination practices. Author-generated scientific digital texts present more non-verbal explanatory strategies than writer-mediated ones – feature articles, and especially than research digests. The findings on the strategies resorted to and their realisation can be used to design tools for researchers, scientists and scriptwriters, mediators of disciplinary knowledge, who need to communicate such knowledge through digital platforms to diversified audiences.

Double strategies for reproducing multimodal puns in interlinguistic translation: Experimental research

The research is dedicated to exposing the specifics of translating multimodal puns based on the retrospective experiment. A multimodal pun is defined as a creolized/ polycoded formation substantiated by a specific type of ambiguity consisting of two inhomogeneous semiotic modes. The material for our research was provided by verbal-visual puns functioning as separate texts. Each multimodal pun results from intersemiotic translation when the signs of one semiotic system are transformed into the signs of another. For the vast majority of multimodal puns, verbal signs are interpreted into pictures, but the opposite cannot be excluded either. The role of the visual component is twofold. It can be creative when the picture is part of the ambiguity mechanism, or it can be amplifying when the picture accentuates the verbal wordplay, not participating directly in the creation of ambiguity. Hence, our first hypothesis is that multimodal puns with an amplifying visual component are a lesser challenge for translators than those with a creative one due to the absence of the necessity to coordinate verbal and visual modes in the target text. The research aims to identify the strategies of interlinguistic translation of multimodal puns and the factors that determine them, particularly the impact of the visual mode on the translator’s decision-making. Hence, our second hypothesis is that multimodal puns require double strategies that would allow to correlate the reproduction of the elements of two different semiotic systems. The analysis of the subjects’ translations, as well as their reports received in the course of the delayed retrospective experiment, confirmed both hypotheses. Disclosure Statement The authors reported no potential conflicts of interest. * Corresponding author: Oleksandr Rebrii, 0000-0002-4912-7489o.v.rebrii@karazin.ua

Multimodality in CLIL assessment: implications for teacher assessment literacy

ABSTRACT Content and Language Integrated Learning (CLIL) features the integration of content learning and additional language learning. With the increasing recognition that representation and construction of content knowledge often involve multiple semiotic modes, it follows that the ‘language’ dimension in CLIL should be extended to a multimodal dimension. Researchers have been examining the affordance of multimodal resources in facilitating CLIL instruction, but little attention has been paid to CLIL assessment, where students’ knowledge is assessed in their less proficient language. It is then worth exploring whether and how CLIL assessment allows students to mobilise multimodal resources to demonstrate their actual learning. Through in-depth individual interviews with 10 purposefully sampled secondary school CLIL teachers in Asian contexts and examining their assessment materials, this study investigates teachers’ perceptions and practices of using multimodal resources in CLIL assessment. It is found that although teachers are aware of the affordance of multimodal resources in making assessment questions accessible to students and mediating students’ content knowledge expression, teachers generally need more systematic knowledge and skills to design effective multimodal assessment tasks, develop multimodal assessment criteria, and provide feedback on students’ multimodal production. These findings have important implications for CLIL assessment design and CLIL teacher assessment literacy.

Exploring TPACK Barriers of EFL Junior High School Pre-service Teachers: Narrative Inquiry

Technology, content, and pedagogy must have a meaningful relationship with each other so that Pre-service English Teachers' literacy can be connected to a variety of semiotic modes beyond written language. Interaction with a greater variety of texts in a variety of modes, media, and environments highlights the need for tools to assess English language instructors' TPACK when it comes to educating students in multimodal literacy. Several pre-service English teachers in EFL classes utilize the use of word wall media for learning. This research aims to identify the obstacles faced by pre-service teachers in utilizing TPACK to teach EFL English at a private university in East Java, Indonesia. By adapting the interview framework (Creswell, 2018), which consists of four discussion targets, including pre-service teachers' beliefs regarding the practice of using TPACK, which is applied in learning activities in junior high schools. This narrative qualitative method (Clandinin & Connelly, 2000) uses semi-structured interviews as its instrument. Systematically using Widodo's (2014) thematic analysis to analyze qualitative data. The findings show that there are many obstacles faced by pre-service teachers and students in implementing learning using TPACK in EFL classes, to be effective for their future teachers, pre-service skills.

Teachers’ perceptions of implementing digital multimodal composing in tertiary classrooms: Voices from Chinese EFL teachers

AbstractIn language education, digital multimodal composing (DMC) pertains to a new literacy practice in which students use digital tools to produce texts incorporating multiple semiotic modes. Despite that many scholars advocate for implementing DMC in language classrooms, it remains marginal in mainstream classrooms, with scant attention being paid to teachers’ perceptions concerning the use of DMC. Situated in Chinese universities, this research probes into the perspectives of five English teachers on implementing DMC as a pedagogical innovation. The analysis of data collected from multiple sources reveals that the teachers perceived DMC positively due to its affordances for students, namely, cultivating digital and multimodal literacies, improving speaking and writing abilities, fostering collaboration and communication skills, and enhancing learning interest and motivation. Teachers themselves harvested benefits regarding deeper insights into students’ learning potentials, enhanced digital literacies, a novel avenue for academic research, and an innovative teaching method. Nonetheless, the teachers expressed concerns about time constraints, students’ engagement in group work, potential plagiarism, assessment of students’ videos, and the long‐term effect of DMC on students’ writing performance. Based on the findings, this research provides implications for integrating curriculum innovations like DMC into conventional curricula.

Frazeologia gestyczna – znaczenie gestu a znaczenie idiomu

Frazeologia gestyczna – znaczenie gestu a znaczenie idiomu

Nonverbal Communication in Classroom Interaction and Its Role in Italian Foreign Language Teaching and Learning

The purpose of this study is to present the state of the art of recent research on nonverbal communication in L2 classroom interaction, in particular on teachers’ and students’ gestures, and then focus on a case of gestures in an L2 Italian classroom. A corpus of video-recorded interactions (CLODIS) were analyzed to answer the following research question: How do L2 Italian native teachers behave when addressing international students? Are there differences with what has been observed in other foreign language (L2) teaching contexts? Both previous data-based research on multimodality in L2 classes and the analysis on CLODIS show that teachers select and coordinate multiple semiotic modes as interactional resources to complete various teaching tasks. Furthermore, Italian native teachers use not only the typical pedagogical gestures (both iconic and metaphorical), but also culturally specific emblems that may cause misunderstandings or inappropriate mirroring effects. For these reasons, it is important that L2 teachers develop a good multimodal awareness, especially if they teach their mother tongue to foreign students and if they belong to a “contact culture”, as is the case observed in L2 Italian classes.

Appraisal Theory: Can It Be Applied to Visual Images in Isolation to the Text?

The role of women in Saudi society is evolving; however, the process of reform is subject to multiple interpretations. This makes the role of resemiosis when interpreting visual media particularly important, perhaps more than a direct focus on the semiotic modes adopted. Data was collected from a variety of on-line media covering the recent changes. Appraisal Theory was used to capture the judgement process in terms of emotions experienced by the reader/viewer and how the wider context such as the full news article, or even just the headline, forms a part of this process. This posits that in contested situations, resemiosis is a critical part of the process and sometimes the apparent interpretation of a single image (intersemiosis) can be different to the interpretation of the whole. In such instances, different semiotic modes convey different meanings but a focus on one in isolation can be misleading as to the overall intent.

Erratum: Refining concepts for empirical multimodal research: defining semiotic modes and semiotic resources

Erratum: Refining concepts for empirical multimodal research: defining semiotic modes and semiotic resources

Despair, Hope, And Politics: A Semiotic Analysis Of Online Media Cartoons On The Turkey And Syria Earthquake 2023

The Turkey and Syria earthquake of 2023 caused widespread devastation and loss of life, resulting in widespread despair and hopelessness among the people. In the aftermath of the disaster, social media was flooded with caricatures that sought to represent the political and social realities of the affected regions. This research paper presents a semiotic analysis of online media cartoons using the Barthesian modal to understand the representation of despair, hope, and politics in the context of the Turkey and Syria earthquake of 2023. Through a qualitative analysis of purposively sampled cartoons, this study explores how dominant emotions and political messages are conveyed through visual and textual elements and how they shape public perception of the event. The findings suggest that the cartoons use a range of semiotic modalities, such as images, text, and symbolic representations, to convey political messages about the earthquake and its aftermath. The paper contributes to the growing research on semiotics and visual communication. It highlights the potential of online cartoons as a rich source of social and political insights in the context of the earthquake and how people respond to it.

Refining concepts for empirical multimodal research: defining semiotic modes and semiotic resources

The issue of defining key concepts in multimodal research is at the same time ongoing and of pivotal importance. Building on John Bateman’s categorisation of modes, and paying special attention to the concept of materiality within the discussion, the paper provides a clear differentiation between semiotic modes and semiotic resources and discusses the relationship between the two. These will be defined by also looking at how they differ from another key concept in multimodal research, i.e., media, and examples will be provided to illustrate how the newly defined concepts can guide empirical investigations of multimodal texts and their reception. The paper aims to continue the discussions around these key concepts amongst multimodal scholars, so that agreement in the field can eventually be reached.

Drawing as a Tool in Metaphor-Led Discourse Analysis

ABSTRACT The use of words to label concepts is a weak point in CMT but one which is little discussed. This article considers the relative merits of image and writing as semiotic modes for identifying conceptual metaphor domains and whether image can offer an alternative to target is source formulae. It reports on an experiment in which MA translation students draw composite drawings to represent orientational metaphoric aspects of a UK government press release. Conventional metaphoric language deriving from the orientational metaphors, especially good is up/bad is down, plays a significant role across this text, forming a “metaphor chain” pattern. The drawings and post-task interviews with the participants reveal details of the internal structure of the source domain up and how it is conceptualized. Particularly of note is the strong horizontal forward aspect of orientation which emerges when the written text is represented pictorially. The article demonstrates how drawing can be used as a tool in metaphor-led discourse analysis.

Tattoos as multimodal semiotic assemblages

Abstract Drawing on interviews with, and images from, a selection of female participants, we develop the notion of semiotic assemblages to examine tattoos as mobile semiotic material for embodied meaning-making, with multi-located trajectories linking the tattooed bodies to different spatial networks. In turn, we draw on the social semiotic approach to multimodality by looking at assemblages of tattoos as vibrant material, pregnant with thing-power which, while re-making the bodies of individual tattoo owners, simultaneously shapes and semiotically remediates physical and affective (e.g. emotional and memorial) objects and feelings. We illustrate how corporeal materials and activities, and the incorporeal conceptual, perceptual and affective senses and feelings are remediated across boundaries of language, geography, semiotic modes, media, and chains of mediation to make meaning. We end with a suggestion that multimodal analysis of semiotic assemblages needs to take a multiscalar approach to account for mobilities, movements and flow, and the continuities and disconnections in forms and meanings the dynamics of the connectedness produce across different scales.