Semiotic Resources Research Articles

Multimodal dairy cow–human interaction in an intensive farming context

In our consideration of how to decentre an anthropocentric view in linguistics, we will address the following research question: how do dairy cows and humans imbue their interspecies interaction as a semiotic resource with meaning that makes sense for both species under specific social conditions (Jørgensen, 2008:167). We address the question by using a social-interactional approach informed by conversation analysis (CA) (Goodwin, 2017, Mondada, 2016, 2018; Mondémé, 2021), which enables us to examine what the dairy cow makes relevant in the sequential organisation when interacting with a human.We show that the dairy cows make gaze important in their interaction. Gaze alone is sufficient to mobilize human interlocutor response, and gaze withdrawal by the human should take place for a successful communication (case-study 1 versus study 2). The case-studies of dairy cow–human interactions show that these interactions include much more than (human) sounds and (human) signs only: language is taken as languaging, as a social practice, embedded in a multimodal interactional exchange (Levinson and Holler 2014) that includes nonhuman animals as well. This also implies that linguists should therefore look beyond ‘sound’ and ‘sign’.

“Talk to the hand”: handling peer conflict through gestural socialization in an elementary classroom

Although researchers of language and communication have become increasingly interested in both embodiment and conflict in recent years, little is known about how elementary students use embodied actions modeled by their teachers as they engage in peer conflicts. This paper addresses such questions, focusing on the “quiet coyote” gesture and the “open hand prone” gesture, two emblems commonly used as classroom management strategies in elementary grades. Building on work in language socialization, gesture studies, and other areas of discourse analysis, I propose what I call a gestural socialization perspective for analyzing the nuanced ways the US second-grade children in this study use and socialize one another to use these gestures, as well as other semiotic resources, to handle peer disputes. An ethnographically informed, multimodal discourse analysis centering on a multiracial group of girls reveals how students' gesture practices draw on their teacher's gestural socialization practices while also diverging from them, especially with regard to gestural form, stance object, intended recipient, and accompanying metapragmatic commentary. These aspects of the participants' appropriations of the “open hand prone” and “quiet coyote” emblems, together with their use of gestural innovations, metagestures, and other semiotic resources, allow them to take more oppositional stances than those made relevant by the teacher's practices. Through these multimodal stances, students take a hands-on approach to starting, continuing, and closing peer disputes on their own terms. Overall, the study highlights how participants' handling of disputes often subverted a local emphasis on conflict avoidance, efficiency, and appropriateness and the developmentalist, neoliberal, and standard language ideologies underpinning these norms. The paper closes with a discussion of implications for research and pedagogy, emphasizing the importance of closely attending to the multimodal, interactionally emergent, and culturally situated nature of conflicts among children and people of all ages.

Multimodal discourse analysis of English reading instruction in colleges and universities based on weighted function algorithm

Abstract This paper proposes a kernel clustering method that uses the weighted function method to address the problem of excessive similarity calculation in the kernel function clustering process. Based on function-based clustering with orthogonal basis expansion, function-based principal component analysis is used to reduce the dimensionality of function-based data and extract the first few principal components that can contain the most information of the original data. The factors of each component are assigned with characteristic weights so that the scores of the principal components replace the original functional data for clustering analysis of English reading teaching in colleges and universities. A systematic functional language framework was used to conduct multimodal discourse analysis based on clustering results. The symbolic annotation of pure language reached 14.5, and the symbolic resources of visual imagery reached 85.7 during the CB stage of English reading instruction. The number of semiotic resources was relatively small, so the complexity of the modality was lower than in other stages.

Keep our mob safe

This paper examines how semiotic resources, including text, song, dance and visual imagery, are utilised in a COVID-19 public health outreach campaign to communicate to Aboriginal and Torres Strait Islander youth in the Australian state of New South Wales (NSW). Content analysis and multimodal critical discourse analysis are employed to study a corpus of multimodal videos with a visible, speaking social actor published between July 2020 and June 2022 on the official NSW Ministry of Health Facebook page (n=1010), including ‘Keep Our Mob Safe’ public health outreach campaign videos (n=38). The analysis focuses on the discursive representation and recontextualisation of social actors in order to ascertain who the state considers a ‘linguistic and cultural asset’, a term used in NSW language and health policies. The findings indicate that social actors use creative and transformational material processes and through their actions reveal covert language policy mechanisms.

Multimodal arguments: language, script, and politics

Abstract This study uses discourse analysis to investigate the connectedness between embodiment and the politics of orthography. Drawing on a video-recorded conversation from a large sociolinguistic dataset on Amazigh/Berber, the study examines the role of embodied acts not only as semiotic resources that elaborate talk but also as consequential discursive entities that organize political arguments about script. The participants use multimodal utterances that extend beyond meaning to iconize social orientation as well as power in conversation. Analysis of the video reveals the way participants map their social perceptions of the language and its script onto their talk and embodied actions. Analysis also shows how script arguments mirror larger political debates in Morocco, in which script preferences are based on ideological and political orientations.

“記念錯過的DUET_T”: Negotiating translanguaging practices in the legitimate graffiti of a multilingual schoolscape in Hong Kong

Hong Kong’s higher education institutions, with their unique socio-political context and global reputation, have presented multiple and diverse schoolscapes where multilingual students can collectively construct a shared repertoire to perform their desired identities and create specific meanings. Recognizing the semi-public whiteboards on a Hong Kong University library wall as a genre of schoolscape, this paper aims to explore the intricate ways in which students negotiate their varied linguistic and semiotic resources to create and engage with multilingual graffiti that is deemed legitimate on these whiteboards. Drawing upon the concept of translanguaging (W. Li, 2011, 2018), we employed a sociolinguistic ethnographic design and collected 151 photographs of graffiti. Through a semiotic and ethnography-informed analysis of the linguistic landscape data, we argue that these graffiti signs encourage students to establish the schoolscape as a collaborative translanguaging space by enabling them to collectively participate in translingual and transmodal practices for fun. The graffiti signs also invite students to perform translanguaging practices to substantiate their sense of affiliation with the institution and its people in the translanguaging space where their affective experiences can be constructed and shared. The study concludes by advocating for further ethnographic investigations to enhance our understanding of translingual practices within multilingual schoolscape environments.

A posthumanist orientation and cross-national, online bilingual digital storytelling

This cross-national, online biliteracy project connected 13 Canadian and Chinese bi/multilingual learners aged 11–15 years. The research was informed by posthuman conceptualizations of literacy, biliteracy, modes, and media, and disrupted binary thinking about mind/body, first language (L1) / second language (L2), and language/materiality in literacy education. A diffractive methodology facilitated exploration of how the dynamic relationship-building between humans and nonhuman entities shaped the bi/multilingual learners’ digital storytelling. The project intended to enable a generative assemblage that positioned languages, media, and places as interconnected, agentive repertoires. It supported bi/multilingual learners’ connection-making with various semiotic resources and their situated lifeworlds and opened opportunities for them to create new forms of meaning. Findings also show that the generative assemblage was affectively rewarding and enabled bi/multilingual learners to be fully engaged in their digital story creation and eager to try more. The article proposes a biliteracy orientation that nurtures ethical meaning-making across languages, modes, and media.

How The Danish Girl was adapted and recontextualized through multimedia

The Danish Girl has three different versions – novel, screenplay and film. Through comparison, this study explored how Lucinda Coxon (2015) and Tom Hooper (2015) adapted David Ebershoff’s novel (2000), The Danish Girl, to the screenplay and the film of the same name, and how multimodal semiotic resources contributed to recontextualizing the screenplay and the film. This research selected a scene, which represented the ‘remade’ type, from The Danish Girl to instantiate recontextualization in depth. The study found that the adaptations of The Danish Girl mixed different methods. The study also found that the way of recontextualizing the film was more complex than that of the screenplay.

REPRESENTATION OF POLICE IN KANJURUHAN STAMPEDE

This study examines the National Police's representation in a Twitter video (@DivHumasPolri) after the Kanjuruhan Stadium stampede, where over 130 lives were lost. The police faced criticism for allegedly using tear gas during the incident, in violation of FIFA regulations. The official National Police account responded with a 1:09 video showing police assisting the stampede victims. This research employs Hall's (1997) representation framework to elucidate the intended meaning of the video producers and multimodal analysis to find that the police use semiotic resources to represent themselves in the Kanjuruhan video. Police tried to identify themselves as key helpers for the Kanjuruhan victims, also victims during the incident, and partners of the supporters. National Police, as the video producer, was able to select relevant messages to maintain their public image and simultaneously communicate smoothly to the public regarding their position on the incident.

Fueling divisions: a multimodal analysis of Canadian petro-nationalism in the social media discourse of “oil sands strong”

ABSTRACT This research conducts a multimodal critical discourse analysis of social media posts by Oil Sands Strong (OSS), examining their role in advocating for resource extraction through the use of “petro-nationalism” discourse. The analysis uncovers three key aspects: First, the posts employ multimodal semiotic resources to create dichotomies and blend nationalist and xenophobic discourses with energy discourse. Second, they integrate diverse multimodal symbolic elements from various discursive frameworks to forge a collective identity for the local Canadian community. Lastly, the posts stigmatize foreign oil producers, new energy industries, and environmentalists, employing multimodal symbols to obfuscate the argument's focal point and generate a hybrid discourse. Through visual symbols, assertive text, and argumentation of topoi, the posts present an exaggerated and distorted image of petro-nationalism. The study concludes that these multimodal posts aim to reshape Canadian energy politics, promote biased sentiments, exploit cultural prejudices, and formulate petro-nationalism narratives for resource extraction advocacy.

The spatiotemporal constitution of Dubai as a semiotically assembled touristscape

Mobile technologies mark an increasing construct of heterogeneous semiotic resources which coexist in a networked symmetrical interrelations. This area of research is still understudied, especially in terms of demonstrating how app-mediated touristscapes are co-told, transduced, and augmented by networked assemblage between participants and mobile interfaces. Drawing on a pragma-semiotic approach, the present study aims to investigate the spatiotemporal constitution of Dubai as a mobile-mediated touristscape. We draw on a newly synthesized approach that combines Cooren and Matte’s (2010) model of constitutive pragmatics and Pennycook’s (2008, 2017) notion of “semiotic assemblages.” Such a methodological synergy has been applied to the Dubai Travel mobile app in a way that revealed how the touristscape of Dubai has been pragmatically constituted of the semiotic assemblage of heterogeneous figures in the app’s interface-human interaction. This form of techno-human interaction was demonstrated to be situated in three spacing practices: (i) presentifying or making materially present hybrid interactions of techno-human figures, (ii) ordering or systematizing the scripted trajectories of Dubai touristscape by creating more space and time across framed intervals, and (iii) accounting or linking spatiotemporal augmentation to affective semiotic assemblages. The study found that Dubai touristscape has been constituted via a human-non-human semiotic assemblage with augmented and multilayered spatiotemporal possibilities. The pragma-semiotic approach has thus helped in arguing against what accounts as a touristscape with fixed spatiotemporal properties. The study contributes to understanding the increasing role of networked communication through developing a dialogue with linguistic pragmatics.

Student’s agency in English as Foreign Language learning in a Virtual Environment: An Interaction Analysis

The COVID-19 pandemic brought about new challenges for class interaction and for English as a Foreign Language teaching instruction in general, and opened a whole world of opportunities to learn about virtual-mediated instruction and about learning altogether. As a reaction to the new social conditions, alternative spaces, and ways of mediating instruction through the use of technology and digital resources were created and promoted in the quest for teaching and learning. As such, this multimodal interaction analysis inquired into the qualities of interaction taking place in an online course of advanced English at a teaching credential undergraduate program at a private university in Bogotá, and the affordances and limitations that such interactions presented for agency, engagement, and ultimately, learning as an effect of communication. To such an end, class observations, student focus groups, and interviews on teachers were conducted. The findings offered insights on the multiple ways students and the teacher resorted to their semiotic resources and communicative modes to configure meaning, construct knowledge, and relate to one another in a virtual environment. Mainly, however, it encourages a debate about the role that teachers play in designing and implementing communicative environments to enrich the teaching and learning experience.

Teachers’ orchestration of semiotic resources in EFL classrooms: a multimodal analysis

Teaching is an activity involving semiotic resources beyond verbal language. Therefore, more studies investigating what semiotic resources are employed and how they are utilized within EFL classroom settings are needed. This study sought to explore the types of semiotic resources employed by two EFL teachers in Indonesia and the roles of these semiotic resources in teaching activities. Through a qualitative case study, two EFL teachers in Bandung and Medan, Indonesia, were purposively selected. Data collection was through a semi-structured interview and document analysis. A six-phase of thematic analysis was employed to analyze the interview results, while a three-step of skimming, reading, and interpretation was used to analyze the documents. This study found that spoken language, pictures, videos, proxemics, gestures, head movements, gaze, and music were the semiotic resources used by both teachers in their teaching. The findings also demonstrated that these semiotic resources were used in the opening, delivering, and closing of the class, explaining learning content, performing interactions with students, checking the students' progress, managing classroom order, showing appreciation, and conducting evaluations. This study implies the crucial roles of all stakeholders in education to help EFL teachers in multimodal knowledge and practical strategy to create better teaching outcomes.

Multimodality and the expression of mathematical ideas through videos by Mathematics undergraduates

The research reported in this article sought to understand how Mathematics undergraduates in Distance Education combine semiotic resources when using videos to express mathematical ideas. Adopting the qualitative methodology, the research was developed with students of the Analytical Geometry discipline and the participant observation in forums of the virtual learning environment was used in the monitoring of the video productions by the participants. Functional Systemic — Multimodal Discourse Analysis supported the analysis of the strategies used in the combinations of semiotic resources in the produced videos. In this article, the analysis of one of the videos produced in the research is discussed. As a result, it was found that the multimodal nature of the video encourages the use of contextualization in the construction of digital mathematical discourse. However, the concepts are formalized from a recontextualization referring to the classroom, which shows the influence of the experience in this environment in the construction of the mathematical discourse shared in the online environment in the context of this research.

Seeds that germinate, sprout, grow and develop in their own time: productions of educational actors inserted in different spaces

As part of the “Digital Videos in Mathematics Education” extension course, we aimed to understand how the discourse of a teacher combines semiotic resources in the video production entitled “About the sphere and the cylinder”. The research approach is qualitative, and the data produced in the investigation is based on the video, on interviews, and on discussion forums. We reflect on spontaneity, the classroom, and the use of digital technology, creating a link between the spreading of seeds and the possibility of exploring this technology in public schools to stimulate students to produce mathematical knowledge through videos. In this sense, when we became aware that our reflections touched the sensibility of the teacher, we oriented the investigation to the product of his experience in the light of multimodality. We conclude that this educational actor, guided by “playfulness” and “valid information”, harmonizes semiotic resources of language, mathematical symbolism, and visual demonstration when producing meanings in his discourse.

Mathematical modelling when the problem is proposed with digital video: multi-voicedness, multimodality, domestication

This article describes research that explored the possibilities of digital video in mathematical modelling practices. It was developed in an online course aimed at in-service teacher education. Its scenario was a modelling practice in which a problem is proposed through a digital video (modelling videoproblem) and the participants are challenged to produce a video as an answer (modelling videoresponse). This is qualitative research based on the theoretical perspectives of the humans-with-media construct, Activity Theory, and Social Semiotics. The results point out that the way of combining semiotic resources in the videoproblem scenes influences the production of meanings and the modelling, and that the underutilization of resources of this media can limit educational possibilities. Furthermore, from the combination of different theoretical approaches, new theorizations emerge to be explored in future investigations.

Resources in the Advertisement of Tertiary Institutions Qualifying Exams

The study adopts van Leeuwen’s social semiotics theory. The study reveals that the advertisers deploy semiotic resources which suit the symbolic context of appealing to the dreams and anxiety of the admission seekers.

“Mi conciencia habla inglés, aunque yo no quiera”: Unearthing sociopolitical wisdom through translingual poetry

ABSTRACT In this study, we examine translingual identity poems written by three focal Latinx Teacher Candidates (TCs) in response to assignments in their Teacher Education Program (TEP). To interpret the focal TCs work, we bring together theories of raciolinguicized subjectivities, translingual literacies, and sociopolitical wisdom. Through thematic analysis, we argue that the use of translingual identity poems provided opportunities for TCs to draw on their emotions as semiotic resources and assert the connections of their identities to broader histories of marginalization and resistance. We also argue that when we, as teacher educators, engaged in the work of reflexively reading the poems while considering our own identities, it created a space for mutual learning and vulnerability. This article demonstrates the value of translingual identity work for both TCs and teacher educators, and counters trends towards “identity-neutral” and monolingual approaches in teacher education.

Wordless Picturebooks as Resources for the Construction of the Pedagogy of Multiliteracies. The Case of Migrants, by Issa Watanabe

AbstractThis article presents the study of Migrants (2019), a wordless picturebook by Peruvian author Issa Watanabe, developed to answer the following research question: To what extent can this visual text become a resource for the learning of multiliteracies? A visual discourse analysis of the narrative was performed, focusing on the identification of key meanings and semiotic resources used in their visual construction. Our findings unveil the prominent role of semiotic resources such as anthropomorphism, vectors, colours, position and unboundedness of the page frame, in the visual realization of key meanings such as diversity, journeys, death, and empathy. Upon these findings, we present Migrants as an extraordinary literary text to initiate children in learning the visual mode, one of the multiple modes currently used in the representation of meaning. We discuss our findings with reference to the main question leading the study, the pedagogical conceptualizations endorsed by the multiliteracies theoretical framework and other relevant research. While the study suggests that wordless picturebooks are called to play a prominent role in the construction of the multiliteracies pedagogical agenda, it furthermore points to the potential of the pedagogy of multiliteracies in the conceptualization of literary pedagogy itself.

Multimodal meaning-making of aggression in English song narrative: A cognitive-pragmatic perspective

This article addresses cognitive and pragmatic aspects of the multimodal meaning-making of AGGRESSION in the English song narrative. The cognitive-pragmatic foundations integrate cognitive and pragmatic aspects with multimodal ones, which allows to explain how signs of different modes and semiotic resources instantiate the concept and which cognitive features are decisive for (re)construction of meaning by performers and viewers. Cognitive features have verbal and non-verbal implementation through visual and auditory modes. The verbal semiotic resource, implemented through auditory and visual modes, contains lexical and pragmatic means of direct/indirect realization of aggression. Lexical means include words that name/describe/express aggression, as well as lexemes that acquire the aggressive meaning in a certain context. Pragmatic means are represented by expressive statements realizing threat, order, reproach, and accusation. The non-verbal semiotic resource is also represented through both modes and is characterized by semiotic heterogeneity, including prosodic/mimic/gestural features, various aspects of performers’ appearance, music clip design and lighting. They include hoarse voice/high exalted vocals/screaming; aggressive look/bulging eyes/wide open mouth/tense distorted face; active movements/fight; dark clothes/ dark make-up/tattoos. The musical component is characterized by the lower register, alternating consonance and dissonance, and a fast tempo. The lighting includes the play of light and shadow, and flashes. The construction of aggression appears as a result of the integration of semiotic elements into a multimodal blend, which is an emergent entity. Semantic relations between the blend components are based on the complementarity or contradiction of their elements, which makes it possible to single out convergent and divergent constructive patterns.