Multiple Semiotic Resources Research Articles

Pedagogical translanguaging and trans-semiotizing in a LOTE Arabic classroom: the coordination of semiotic resources in alphabetic teaching and learning

ABSTRACT Purpose Based on the notion of translanguaging/trans-semiotizing, the study aims to explore an effective pedagogy in a foreign language classroom to respond to the multilingual and multimodal context and facilitate language teaching and learning. Specifically speaking, the study attempts to investigate the coordination of multiple semiotic resources during the teaching and learning of Arabic alphabet in a LOTE (Language other than English) Arabic classroom at a Chinese university. Design Drawing from a classroom ethnography for two academic years, the current study collected data from a focal Arabic teacher and learners via classroom recordings, semi-structured interviews, and stimulated recalls; multimodal conversational analysis and qualitative content analysis were used to analyze the data. Findings The findings illustrate how multilingual resources can be activated to organize the classroom, how multimodal resources can be coordinated to contribute to language learning, and how students’ individual resources can be integrated into the entire semiotic resources to enhance their understanding of Arabic orthography. Value The implications for LOTE education and teacher training programs are presented, including the encouragement of a multilingual, multimodal and multisensory teaching approach in the LOTE classroom. LOTE teacher's agentive role is demonstrated; LOTE learners are expected to make positive response to the teacher’s pedagogy, and become active co-constructors of the language classroom and agents of translanguaging/trans-semiotizing practices.

“Zhuge Kongming becomes reborn as a clownish partygoer!”: Linguistic carnivalization, critical metapragmatics of danmu, and mediatized neoliberal (inter)subjectivity

The Chinese video-sharing platform of bilibili experiences a ‘Kongming fever’, wherein danmu commenters collectively parodize the historical figure Zhuge Kongming, a pre-existing indexical of Confucianist value beliefs institutionalized by the official authority. Drawing upon Bakhtinian carnivalesque (‘free and familiar contact’ and ‘parodic profanation’), the present study proposes the analytical framework of ‘linguistic carnivalization’ as a critical metapragmatic approach (i.e., subjectivity-oriented). The multimodal analysis of danmu comments reveals how digital users appropriate multiple semiotic resources to construct carnivalesque, including vulgar linguistic varieties from media culture, netspeak genre, poetic patterns of textual repetition, and inversive sign vehicles and rescripting. As a discursive-affective effect of such carnivalization processes, Kongming’s ‘serious’ Confucianist personae and indexed ideological expectations become playfully mediatized, profaned, and transformed into new images of personhood (‘livestreaming microcelebrity’ and ‘hedonic partygoer’) according to mass popular culture. In so doing, the Chinese netizens metapragmatically negotiate existing sociocultural hierarchies and reposition themselves as neoliberal subjects. This paper further suggests that the ‘inside-out’ and ‘down-to-earth’ power of linguistic carnivalization does not simply reside in creating aesthetic humor within a local cybercommunity, but importantly owns critical implications for illuminating variegated forms of neoliberal discourse and (re)production of neoliberal subjectivity under large-scale political economic conditions.

Multimodal expressions of disciplinary relevant aspects in inquiry-based physics learning

In this study we are interested in how learners of geometrical optics express disciplinary relevant aspects multimodally in an inquiry-based learning with support of a computer simulation. Students’ discussions and work with the simulation were filmed, transcribed, and we analysed some parts of the transcript where the students show signs of disciplinary relevant aspects by a method based in conversation analysis. We found evidence that the students use multiple semiotic resources for expressing disciplinary relevant aspects about the tasks, especially gestures. The letter are by their visual nature well suited for communication in physics, and are used by the students to transduct semiotic information.

“Looking like a boarding school student”: the construction of unequal personhood in language policy in education

Abstract The purpose of this article is to analyze the construction of unequal personhood in the institutional logics for the implementation of English as a medium of instruction (EMI) policy. I build on the theories of figures of personhood and figured worlds to discuss how institutions (public schools) use multiple semiotic resources to characterize students’ diverse personality traits that reproduce neoliberal subjectivities shaping their EMI policies. The data for this article are drawn from ethnographic observations and interviews with the teachers from two Nepali public schools that have recently introduced a segregated EMI policy. The analysis of data shows that EMI schools use ‘śikṣita’, ‘sabhya’ and ‘yogya’ personality traits to justify the relevance of EMI policy to produce the educated person. The construction of such person types is shaped by sociocultural and political-economic ideologies and build unequal personhood, reinforcing neoliberal subjecthood and epistemic injustice. My recommendation is that we need to pay attention to examining how language policies in education construct unequal personhood by assigning, imposing, and imaging discriminatory personality traits which remain as the foundation of social injustice.

Exploring Children's L2 Disciplinary Literacy Through a Multimodal Science Project in a CLIL Context

AbstractThis study explores the ways in which young second language learners in an intact fourth‐grade content and language integrated (CLIL) science class drew on the affordances of multiple semiotic resources including language, images, sound, movement, etc., to construct disciplinary knowledge in the context of a multimodal project on machines. After an instructional period in which attention to literacy was embedded in their science classes, the learners designed and illustrated their own complex machine, filmed a digital video and produced a written explanation. By integrating a systemic functional linguistics‐informed analysis of the linguistic features of the children's written texts with a semiotic analysis of the intermodal relationships in their projects, we offer insights into the interrelation of different modes (linguistic, visual and aural) in facilitating children's meaning‐making in the L2. The results revealed considerable within‐group variability in the learners' written texts and intermodal relations of concurrence and complementarity in their multimodal projects. The role of multimodal tasks in providing all children with effective tools for communicating knowledge is discussed.

The ‘Struggle for the Narrative’: Cooperation and Conflict in Tiger King’s Intermedial Universe

Abstract This article examines the disputed status of narratives in the Netflix documentary Tiger King (2020—2021), and the ways in which the series’ actors use media to bolster their particular version of a narrative. While classic studies of intermediality have productively analyzed the relations between the multiple semiotic resources employed in narrative forms, I offer an approach to intermediality in documentary art that enriches the structuralist paradigm insofar as it calls additional attention to the various human actors that put the worldmaking power of media to use. Assuming that in filmmaking the creation of a storyworld is a fundamentally cooperative, while also potentially conflictive, endeavor, I examine the Netflix hit show as a documentary in which narrative co-construction is particularly significant. The series introduces its audience to the strange world of ‘big cat owners’ in the United States — a world which is populated by dubious storytellers and full of conflicts of interests. Tiger King’s ‘hyperreal’ world is saturated with media and images that are employed by its actors for storytelling purposes on a contested narrative territory. I argue that the actors’ ‘struggle for the narrative’ resonates with the show’s Darwinian themes and its interests in documenting a world in which the true predators are not the tigers but the human ‘storytelling animals.’ By examining how the various actors boost their own narratives while discrediting those of other players, I aim to illuminate the fine line between narrative co-construction and conflict in the show’s intermedial storyworld.

Ting, tang, tong: Emergent bilingual students investigating and constructing evidence‐based explanations about sound production

AbstractThere is a significant amount of research literature on the importance of identifying and building on students' experiences and ideas for making sense of the natural world, especially when engaging in science practices. Simultaneously, approaches to creating justice‐oriented science education promote the need to focus on the diverse sense‐making repertoires that students, especially those from historically marginalized communities, bring to science classrooms. However, when it comes to emergent bi/multilingual students, science education has favored narrow definitions of what ways of communicating are seen as productive for figuring out natural phenomena, privileging English‐based academic vocabulary. In this article, we investigate the myriad conceptual and semiotic resources that third‐grade emergent bilingual students developed and used when explaining sound production. Additionally, we explore how students investigated the sounds produced by a string instrument and unpacked the how and whys that give rise to the pitch of the sounds they heard. Our analyses indicate that: (1) students created mechanistic explanations that identified how changes to the salient physical features of strings affected the pitch of the sounds; (2) students created and laminated multiple semiotic resources when sharing their observations and explanations, particularly sound symbolisms; and (3) students navigated both semiotic convergence and divergence as they worked toward conceptual convergence. Based on our findings, we argue that justice‐oriented science learning environments must become spaces where emergent bilingual students can build on all their conceptual, semiotic, and cultural resources, without being policed, as they engage science practices.

Textart, identity and the creative process: a case study with Arabic heritage language learners

ABSTRACT Informed by a postmodern perspective on language, culture and visual art education this article examines what a creative, visual art focus can bring to the experience of language-and-culture learning for secondary-age students of Arabic as a heritage language (HL). It builds on our previous research focussing on student interactions with works by renowned artist, Ali Omar Ermes, which sets text in the form of Arabic letter shapes and short poetic inscriptions against a painted background. Here we seek to gain a deeper, more nuanced understanding of the process involved through an in-depth study of the thoughts, feelings and emotions experienced by three intermediate-level students as they engage with works by Ermes and then creatively transform them into ‘textart’ pieces of their own. Methodologically we adopt an ethnographic case study design focussing strongly on process but also incorporating principles of arts-based research. Our findings demonstrate how the approach can extend possibilities for meaning-making and affirmation of identity by connecting with personal experience, by leveraging multiple semiotic resources rhizomically and intertextually, and by making space for affective, spiritual, aesthetic and multisensory dimensions. For heritage language learners this brings a deeper engagement with learning and a strong sense of empowerment as multicompetent speakers.

Children's multimodal language development from an interactional, usage-based, and cognitive perspective.

Through daily exposure to the surrounding input structured in conversations, children's language gradually develops into rich linguistic constructions that contain multiple cross-modal elements subtly used together for rich communicative functions. Children demonstrate their skills to resort to multiple semiotic resources in their daily interactions and expertly use them according to their expressive needs and communicative intents. Usage-based (Tomasello, 2003) and cognitive linguistics (Langacker, 1988) as well as construction grammar (Goldberg, 2006) have enriched our comprehension of the processes at work. Those approaches need to be combined to gesture studies (Kendon, 1988; McNeill, 1992) and multimodal approaches (Andren, 2010; Morgenstern, 2014) to fully capture the orchestration of the semiotic resources at play (Cienki, 2012; Müller, 2009). But child language development cannot be understood outside its interactional, dialogic context (Bakhtin, 1981) and without taking into account the role of expert languagers (Vygotsky, 1934) in routines or formats (Bruner, 1975). The first section thus extensively focuses on a productive combination of theoretical approaches and methods, which have been essential to understand child language development, but analyzing child language is also necessary in turn to ground socio-cognitive and interactional approaches to language. The salient features of the variably multimodal child's development are presented in the second section. The third section illustrates longitudinal pathways into multimodal languaging thanks to detailed analyses of adult-child interactive sequences. This article is categorized under: Cognitive Biology > Cognitive Development Computer Science and Robotics > Natural Language Processing Linguistics > Language Acquisition Linguistics > Cognitive Linguistics.

Introducing a multimodal perspective to emotional variables in second language acquisition education: Systemic functional multimodal discourse analysis.

A systemic functional theory applied to multimodal discourse analysis (SF-MDA) is related to the theoretical and practical meaning extraction from the data showing how multiple semiotic resources are used in different modes of discourse ranging from textual, printed, and electronic texts to visual language and those existing alive as embedded in reality. In the present paper, we provide an introduction to the SF-MDA approach and then justify how it can benefit the second language acquisition (SLA) domain and more specifically the exploration of L2 affective variables. We will show how the SF-MDA approach fits in with the complexity and dynamic systems theory (CDST), and is capable of revealing the nuances of the developmental nature of the affective variables involved in language learning. The exemplary works of research in the SLA domain using the SF-MDA approach are reviewed here as well as those exploring L2 affective variables. The further benefits of this approach will be emphasized at the end along with the implications it can have for the CDST-led line of research in applied linguistics.

Emergent multilingual learners’ meaning negotiation and knowledge building through translingual writing practices in South Korea

ABSTRACT This article reports findings from a three-year qualitative study that explored the nature of translingual writing practices of the emergent multilingual youth who are ethnic Korean migrant children from Central Asia and Russia to South Korea. While settling into a new society, these emergent multilingual children tend to navigate and negotiate multiple languages, literacies, and identities. As a researcher and former teacher of Russian-speaking ethnic Korean migrant children in Korea, I became interested in examining how and why they engaged in translingual writing practices beyond school walls (referring to the capabilities and the processes of translingual language learners’ appreciation, negotiation, and evaluation of multiple semiotic resources). Drawing upon ‘translingualism’ , this study gathered and analysed the data from multiple sources (i.e. observations, interviews, fieldnotes, and artefacts). The findings demonstrate the emergent multilingual youth’s agentive engagement in translingual writing practices for differing purposes such as meaningful interaction, language development, and academic success. This study suggests that the dynamic and spontaneous use of multiple semiotic resources is the key to the emerging multilingual children’s everyday communicative and literate lives as well as in the process of multilingual language development.

Unpacking Translanguaging Practices in Multilingual Business Communication in China: A Qualitative Phenomenological Approach

Abstract This qualitative phenomenological study examines a cohort of multilingual employees’ translanguaging practices in business communication at seven multinational corporations in China. It is found that these employees integrate various linguistic features of named languages stored in their linguistic repertoires; they spontaneously prioritize the linguistic features of different named languages in different contexts to achieve effective communication. They value sense-making more than grammatical correctness in communication practices, so they deploy multiple semiotic resources in translanguaging practices. This paper suggests that multilingual employees’ translanguaging literacy should be valued as their linguistic/human capital since it provides them with communication resources in a globalized business context. This paper advances understandings of translanguaging and enriches research practice, broadening the research scope of transnational business communication by expanding translanguaging practices from the bilingual education field to a multilingual, transnational workplace context.

Citizen Linguistic Landscape, bordering practices, and semiotic ideology in the COVID-19 pandemic

Abstract This article examines how Linguistic Landscapes in the Covid-19 pandemic construct the borders of place and people. We build on ‘semiotic ideology’ (Keane, 2018) and ‘semiotopology’ (Peck, Stroud & Williams, 2018) to analyze the bordering practices in citizen Linguistic Landscapes during the pandemic in Nepal. Our analysis shows that citizens combine multiple semiotic resources, both linguistic and non-linguistic, to create physical boundaries to restrict the mobility of people during the pandemic. However, the findings show that such practices are ideological; they promote the othering of the tenants, returnees from abroad/outside the valley, and non-locals. We argue that keeping place and people at the centre of analysis provides a critical framework to widen the scope of Linguistic Landscapes as a broad visual and semiotic space that embodies the bordering practices and categorization of people and their impacts on emotions, identities, and sense of belonging.

Attending to others’ mathematical ideas: a semiotic alternative to logocentrism in bilingual classrooms

Educational research tends to overemphasize spoken language as the main resource when attending to others’ ideas. In this paper, we problematize hegemonic assumptions of such a logocentric approach and its narrow scope. As an alternative, we propose a semiotic ideology, which focuses on the interplay among multiple semiotic resources, both linguistic and nonverbal. We argue that a semiotic ideology broadens what attending to others’ mathematical ideas in whole class settings entails. Recognizing bilingual children’s heightened sensitivity to nonverbal semiotic resources, we center bilingual classrooms as privileged sites to learn about a semiotic ideology. We draw on data from a third-grade Spanish-immersion classroom to illustrate considerations of temporality, materiality, and participant roles. We call for a continued search for intentional and creative ways to attend to the ideas of those with whom we do research.

Engaging the online audience in the digital era: A multimodal analysis of engagement strategies in TED talk videos

Scientific popularization genres have attracted the attention of researchers in esp. Authors of such genres have been found to employ a variety of strategies to engage the readers. However, most studies of engagement in scientific popularization tend to focus on written genres. For this reason, they have mostly examined only the linguistic mode and rarely gone beyond this to take multimodal resources into consideration. the present study investigates how multimodal semiotic resources are combined in digital media to engage the online audience in an influential online scientific popularization genre, that is, TED talks. drawing on a corpus of 28 highly viewed TED talks from the area of biology, this study provides an in-depth analysis of multimodal engagement strategies in one of those talks. the analysis shows that the engagement of online viewers in a digitally mediated scientific popularization genre like TED talks tends to be achieved by the combination of multiple semiotic resources afforded by digital technologies, such as speech, gestures, gaze, visuals, distance of shot, perspective of shot and angle of shot. the study also reveals five strategic multimodal configurations employed to achieve viewer engagement, with the focus variously on engagement through visual aids, a long shot, gaze, questions and reference to personal emotions.

Translanguaging space and classroom climate created by teacher’s emotional scaffolding and students’ emotional curves about EFL learning

ABSTRACT Translanguaging goes between linguistic and/or modal boundaries and beyond them (Li, W. (2011a). Moment analysis and translanguaging space: Discursive construction of identities by multilingual Chinese youth in Britain. Journal of Pragmatics, 43(5), 1222–1235. https://doi.org/10.1016/j.pragma.2010.07.035). The appearance of this new concept marks a significant development of multilingualism studies. This paper is based on an empirical study of creating a translanguaging space and positive ‘classroom climate’ (Gabryś-Barker, D. (2014). Success: From failure to failure with enthusiasm. Studies in Second Language Learning and Teaching, 4(2), 301–325. https://doi.org/10.14746/ssllt.2014.4.2.7) by allowing for students’ emotional expression and supporting their emotions to benefit EFL (English as a Foreign Language) learning in the Chinese context. The visual data are multilingual and multimodal student-made artefacts called emotional curves completed by newly registered university students as descriptive homework of their emotions about English learning in the classroom. These emotional curves create a translanguaging space where students adopt multiple linguistic and semiotic resources to express emotions and identities freely and creatively. The multimodal discourse analysis of emotional curves is combined with (auto)ethnographic observations and interviews, which shed light upon the emotional motivation for translanguaging and manifest effective teacher-student communication owing to the teacher’s ‘emotional scaffolding’ (Park, M. H. (2014). Increasing English language learners’ engagement in instruction through emotional scaffolding. Multicultural Education, 22(1), 20–29.) strategies. Theoretically, this paper highlights the transformative power of translanguaging space and clarifies emotional representations of translanguaging theory. Pedagogically, this study advocates acknowledging and scaffolding students’ emotions to establish better teacher-student relationships and create a flexible translanguaging space and harmonious classroom climate.

Pragmatic reframing from distress to playfulness: !Xun caregiver responses to infant crying

Using the method and theoretical perspective of language socialization, this paper examines the pragmatic reframing of infant behavior from distress to playfulness among the !Xun San of north–central Namibia. Reframing mobilizes a constellation of multiple semiotic resources. For the !Xun, central among these resources is “gymnastic behavior,” namely, holding infants upright or moving them up and down (bouncing). The analysis shows that !Xun caregivers frequently engage infants in gymnastic behavior in response to infant crying and other displays of distress. They thereby collaboratively create rhythms with infants and encourage them to display joyful reactions. Consequently, infants grow accustomed to pragmatic reframing and rekeying long before they can do so using language. Along with that, caregivers often involve other people in the changing participation framework of family interactions. These findings suggest that pragmatic reframing and rekeying assist infants and caregivers in establishing an intimate intercorporeality. Moreover, by reframing infant behavior in a particular manner, the caregivers display their alignment with !Xun knowledge and value system. Involvement of people around infants in caregiving behavior also enhances the (re)socialization of each interactant into the family and other social identities. It is argued that these features forge the development of !Xun distinct intersubjectivity.

Impersonated personae – paralanguage, dialogism and affiliation in stand-up comedy

Abstract This paper argues that paralinguistic resources employed by stand-up comedians to construe textual personae (impersonated characters) make a substantial contribution to the creation of humor by allowing the comedian to distance themselves from particular social values and by referencing shared cultural stereotypes. A stretch of stand-up comedy discourse is analyzed to explore how gesture and voice quality contribute to the construal of projected personae. These are mapped in relation to the interaction between comedian and audience to discern how they evoke specific social values. The results suggest that textual personae are deployed by the comedian to embody stereotypes that connote particular value positions, and that the comedian can construe blended or hybrid personae through the use of multiple semiotic resources. Impersonation thus constitutes a powerful resource for negotiating social values in order to generate tension and create humor.

Early Responses: An Introduction

ABSTRACT This special issue investigates early responses—responsive actions that (start to) unfold while the production of the responded-to turn and action is still under way. Although timing in human conduct has gained intense interest in research, the early production of responsive actions has so far largely remained unexplored. But what makes early responses possible? What do such responses tell us about the complex interplay between syntax, prosody, and embodied conduct? And what sorts of actions do participants accomplish by means of such early responses? By addressing these questions, the special issue seeks to offer new advances in the systematic analysis of temporal organization in interaction, contributing to broader discussions in the language and cognitive sciences as to the social coordination of human conduct. In this introductory article, we discuss the role of temporality and sequentiality in social interaction, specifically focusing on projective and anticipatory mechanisms and the interplay between multiple semiotic resources, which are crucial for making early responses possible.

Multimodal literacy and language testing: Visual and intersemiotic literacy indicators of reading comprehension texts

This paper is concerned with multimodal literacy involving the different kinds of knowledge required to fully access texts with multiple semiotic resources used in reading comprehension test tasks. Such literacy requirements have not drawn researchers’ attention to date, mainly because the foreign language teaching and testing project has primarily focussed on the verbal features of reading comprehension texts. Drawing on data from the Greek National Foreign Language Exams (known with the acronym KPG) – one of the few high-stakes examination systems which use multimodal reading comprehension texts – the paper approaches reading comprehension as a meaning-making process highly dependent on both image and language used in test tasks and the relations between them (i.e. intersemiosis). Working mainly within the framework of Systemic Functional Linguistics (SFL), this study has attempted a systematic description of the visual and intersemiotic literacy indicators of 86 multimodal media texts used in KPG reading comprehension test tasks. For the SFL-oriented multimodal discourse analysis, Tan et al.’s categories of system choices suggested for the analysis of different types of multimodal media texts, and the Multimodal Analysis Image (MMA) interactive software program were used.