Multimodal Meaning-making Research Articles

Metacognition, Metaphor, and Multimodal Meaning-Making: Giving Shape to Creative Agency in Education

ABSTRACT This paper reviews my work in the specific area of teacher creative development and highlights future directions in my own research as well as for possibilities in the field of creativity in education. I emphasize the importance of understanding and promoting creative agency as a multidimensional framework for teacher and student flourishing in school and on the key role of creative metacognition and self-regulation in that developmental process. I consider recent work on creative metaphor generation in multiple modalities, including embodiment, and consider how embodied creative teaching and learning remain understudied areas of huge potential in the field of creativity in education. Last, I present the need for more critical and transformative perspectives and approaches to the study and promotion of creativity in education.

The integration of virtual reality-enhanced multimodal meaning-making improves knowledge acquisition and disciplinary literacy development in science classrooms

The integration of virtual reality-enhanced multimodal meaning-making improves knowledge acquisition and disciplinary literacy development in science classrooms

A multimodal grammar of artificial intelligence: Measuring the gains and losses in generative AI

This paper analyzes the scope of Artificial Intelligence (AI) from the perspective of a multimodal grammar. Its focal point is Generative AI, a technology that puts so-called Large Language Models to work. The first part of the paper analyzes Generative AI, based as it is on the statistical probability of one token (a word or part of a word) following another. If the relation of tokens is meaningful, this is circumstantial and no more, because its mechanisms of statistical analysis eschew any theory of meaning. This is the case not only for the written text that Generative AI leverages, but by extension image and multimodal forms of meaning that it can generate. The AI can only work with non-textual forms of meaning after applying language labels, and to that extent is captive not only to the limits of probabilistic statistics but the limits of written language as well. While acknowledging gains arising from the brute statistical power of Generative AI, in its second part the paper goes on to map what is lost in its statistical and text-bound approaches to multimodal meaning-making. Our measure of these gains and losses is guided by the concept of grammar, defined here as a theory of the elemental patterns of meaning in the world—not just written text and speech, but also image, space, object, body, and sound. Ironically, a good deal of what is lost by Generative AI is computable. The third and final part of the paper briefly discusses educational applications of Generative AI. Given both its power and intrinsic limitations, we have been experimenting with the application of Generative AI in educational settings and the ways it might be put to pedagogical use. How does a grammatical analysis help us to identify the scope of worthwhile application? Finally, if more of human experience is computable than can be captured in text-bound AI, how might it be possible at the level of code to create a synthesis in which grammatical and multimodal approaches complement Generative AI?

Multimodal meaning-making in student presentations: the impact of explicit feedback in a German as a foreign language classroom

Abstract This study examines the impact of teacher feedback on seven students’ presentations on social and cultural topics held in a German-as-a-foreign language classroom in Japan. Following the submission of self-recorded presentation drafts on an online learning platform, students received explicit feedback on how their spoken commentary related to the visible information on slides derived from the theoretical insights of social semiotics and systemic functional linguistics. After allocating one class to revising their work, students presented their finalized presentations, which were video recorded, cut, and prepared for analysis. Subsequently, a total of 14 sections that underwent alterations in both the draft and in-class presentation were scrutinized following a qualitative multimodal discourse analytical framework. By demonstrating the differences between two students’ early drafts and their final in-class presentations, this study shows how adding verbal language that extended a slide’s content strengthened audience engagement through interpersonal functions in language, gaze, body orientation, and gesture. Further, restructuring and inserting additional entities on the PowerPoint slide helped emphasize the talk’s ideational components. The findings of this study have implications for multiliteracies pedagogy in a foreign language context, highlighting the importance of a holistic approach that considers language and other modes as complementary aspects of communication.

Multimodal meaning-making of aggression in English song narrative: A cognitive-pragmatic perspective

This article addresses cognitive and pragmatic aspects of the multimodal meaning-making of AGGRESSION in the English song narrative. The cognitive-pragmatic foundations integrate cognitive and pragmatic aspects with multimodal ones, which allows to explain how signs of different modes and semiotic resources instantiate the concept and which cognitive features are decisive for (re)construction of meaning by performers and viewers. Cognitive features have verbal and non-verbal implementation through visual and auditory modes. The verbal semiotic resource, implemented through auditory and visual modes, contains lexical and pragmatic means of direct/indirect realization of aggression. Lexical means include words that name/describe/express aggression, as well as lexemes that acquire the aggressive meaning in a certain context. Pragmatic means are represented by expressive statements realizing threat, order, reproach, and accusation. The non-verbal semiotic resource is also represented through both modes and is characterized by semiotic heterogeneity, including prosodic/mimic/gestural features, various aspects of performers’ appearance, music clip design and lighting. They include hoarse voice/high exalted vocals/screaming; aggressive look/bulging eyes/wide open mouth/tense distorted face; active movements/fight; dark clothes/ dark make-up/tattoos. The musical component is characterized by the lower register, alternating consonance and dissonance, and a fast tempo. The lighting includes the play of light and shadow, and flashes. The construction of aggression appears as a result of the integration of semiotic elements into a multimodal blend, which is an emergent entity. Semantic relations between the blend components are based on the complementarity or contradiction of their elements, which makes it possible to single out convergent and divergent constructive patterns.

Drawing and Play as Windows into a Child’s Multimodal Meaning-Making Development during COVID-19

Framed within multimodality and situated in a bounded socio-geographical context (i.e., Vancouver), this ethnographic case study provides an in-depth analysis of a bilingual 8-year-old girl’s literacy practices of meaning-making established across varied semiotic modes (i.e., linguistic, visual, audio, spatial, embodied, kinesthetic) during COVID-19. The study draws upon 13 open-ended informal interviews, three sessions of imaginative play, and 16 participant-generated artifacts. The findings revealed two themes (i.e., drawing as collective meaning-making; play as embodied, anthropomorphic meaning-making) that show how the child’s interactions with humans and nonhumans (e.g., toys, objects) contributed to her multimodal meaning-making during the pandemic, which might be beneficial for children in different contexts.

Supporting multilingual learners’ science learning from the multimodal perspective: the case of a VR-Enhanced science unit

Multilingual learners (MLs) often struggle with science conceptual learning partly due to the abstractness of the concepts and the complexity of scientific texts. This study presents a case of a Virtual Reality (VR) enhanced science learning unit to support middle-school students’ science conceptual learning. Using a transformative mixed methods design, this study examines ML’s science learning and participation in VR educational programing from the multimodal perspective. Combining the strengths of quantitative and qualitative data analysis, the results of this study show that engaging in multimodal meaning-making in multiple ways, including the VR game, can enable multilingual learners to gain conceptual understanding of science content equally as well as their English-speaking counterparts. Based on the findings, a framework for multimodal instructional design to support MLs is also discussed. Overall, this study contributes to the growing body of research on the uses of VR-enhanced multimodal learning in K-12 science education, especially for multilingual learners.

Meeting the Challenges of Designing Multimodal Texts to Promote Multiliteracies Pedagogy

This article aims to add to the understanding of the challenges involved in designing digital texts to promote multiliteracies pedagogy. A multiliteracies approach calls for multimodal meaning-making and cultural diversity to be integrated into new school curriculum content, and accordingly, we analyse an interactive children’s story app, named Mobeybou in Brazil. The research question addressed was: what can we learn about the design of multimodal texts aimed at promoting intercultural learning from the design of this story app? The app incorporates tangible and digital storytelling materials to promote intercultural skills among young children, focusing specifically on Brazil. Mobeybou in Brazil was studied to characterize the design of its multimodal representations of meaning, using categories from the grammar of storytelling and multimodal meaning-making, particularly those concerned with representing the experiential diversity and personal positioning of the app users. The findings provide evidence of the complexity involved in designing multimodal texts to meet the challenges of promoting multiliteracies pedagogy, highlighting the urgent need to narrow the interface between research undertaken in education, semiotics, and digital media design. The article concludes by identifying the study’s limitations and some future developments.

Multimodal composing in the English classroom: recontextualising the curriculum to learning

ABSTRACT As literacy curricula around the world expand to include multimodal meaning-making, the challenge that remains is how teachers can design engaging and effective learning experiences in this context and the nature of their guidance to students in developing their multimodal literacy. Our paper focuses on the topic of multimodal composing, where students create artefacts to learn and demonstrate their learning. We seek to understand how teachers can design for students’ learning through multimodal composing with the use of a pedagogic metalanguage. Our data is drawn from a design-based research project on the teaching and learning of multimodal literacy in two secondary schools in Singapore. We discuss the implications of the design and evaluation of students’ learning through multimodal composing and reflect on the nature of the design work by teachers as they negotiate the curriculum requirements and make sense of their professional learning.

Script-switching in Japanese pop culture: a social semiotic multimodal approach

This article explores the nature of multimodal meaning-making in writing by analyzing the script-switching used in Japanese pop culture. Japanese intermixes multiple script systems in one text. Although official and public documents follow standard orthographic conventions, creative works in pop culture texts freely and regularly break these norms by switching script types to create new meanings. However, while many previous works recognize the semiotic effects of script-switching, to this day, there is no systematic account for how and why such meaning-making operates. Notably, script-switching triggers meanings independent of linguistic meaning. Yet, scripts are a visual manifestation of language, so one cannot say they are independent of language. Accordingly, this work attempts to resolve this dilemma by exploring a social semiotic multimodal approach (see Van Leeuwen ‘Typographic meaning’, 2005, and ‘Towards a semiotics of typography’, 2006, and Stöckl ‘Typography: Body and dress of a text’, 2005), which claims the semiotic independence of typography and its interdependency with language. By recognizing the basic unit of the writing system as graphemes, the author argues that the Japanese script systems are categorized into four types of graphemes depending on the linguistic unit they relate to and she demonstrates how the visual unit of the grapheme and the linguistic unit interact in meaning-making. In this analysis, script-switching occurs when grapheme types are intentionally changed to foreground a visual meaning associated with each script system. This article shows that such a meaning reflects the historical development and discursive social practices of writing with multiple script systems throughout the history of the Japanese writing system.

Semiotic timescapes

AbstractIn linguistic/semiotic landscape (LSL) studies, overall, time and temporalities have occupied a curious space between visibility and invisibility, resulting in time and temporalities being treated as an optional, additive, subordinate, or separate dimension in the linguistic and semiotic study of place-making. The term semiotic timescape serves as a heuristic concept which endeavours to centre time and its imbrication with space/place, with an explicit focus on the semiosis of ‘temporalized place-making’. The five articles in this special issue highlight the ‘complex and multiplex’ dimensions of time (Adam 2008) interconnected with spaces/places, as important aspects in understanding social realities and practices, via entanglements with multimodal meaning-making, materialities, embodiment, and politics of social actors and communities. (Semiotic timescapes, interconnectedness of time and space, complexities and multiplexities, multimodality, materialities)*

Multimodal Meaning-making: A Stylistic Analysis of the Use of Semiotic Resources in Magazine Covers

Multimodal Meaning-making: A Stylistic Analysis of the Use of Semiotic Resources in Magazine Covers

Multimodality in webconference-based language tutoring: An ecological approach integrating eye tracking

AbstractDrawing on existing research with a holistic stance toward multimodal meaning-making, this paper takes an analytic approach to integrating eye-tracking data to study the perception and use of multimodality by teachers and learners. To illustrate this approach, we analyse two webconference tutoring sessions from a telecollaborative project involving pre-service teachers and learners of Mandarin Chinese. The tutoring sessions were recorded and transcribed multimodally, and our analysis of two types of conversational side sequences shows that the integration of eye-tracking data into an ecological approach provides richer results. Specifically, our proposed approach provided a window on the participants’ cognitive management of graphic and visual affordances during interaction and uncovered episodes of joint attention.

Teaching About Electricity in Primary School Multimodality and Variation Theory as Analytical Lenses

AbstractPrimary school topics involving abstract concepts are challenging to teach. Electric circuits can be simply constructed but complex to explain. New approaches in teaching students for understanding are needed to advance practice in primary science. This paper combines the strengths of multimodality research perspectives with variation theory to provide insights into a teaching sequence designed for year 6 students. Application of a representation construction approach (RCA) provides opportunities for multimodal meaning-making of electric circuits. The case study adopted a design-based research method to investigate teaching electric circuits. Data collection involved video capture of classroom practice, teacher and student interviews, student journal entries and assessment artefacts, field notes, and pre-test and post-test results. Design principles included identifying key ideas, devising a lesson sequence emphasising energy transfer and transformation, hands-on exploration using multimodal representations in response to learning challenges, and student journaling. Students’ metarepresentational competence was also developed through evaluation, negotiation, and creation of representations and models of electric circuits. Representational challenges followed by strategic teacher-led discussion facilitated students’ developing understanding through focusing attention on critical features. Deep learning was evidenced by journal records, formative and summative assessment artefacts, and post-test responses. RCA principles were instrumental in the successful design of an effective teaching sequence through focus on critical aspects of energy. We advocate a RCA for the design of a multimodal learning sequence. Variation theory was a useful analytical framework to understand the enactment of the design sequence. The study contributes to the challenge of rethinking traditional teaching practices in primary science.

Exploring Educational Designs by Schematic Models: Visualizing Educational Value and Critical Issues of Recognition and Socio-Material Frames

Drawing on experiences from a large-scale Nordic co-design research project, this paper explores the development of cross-border blended education with the aim to strengthen students’ Nordic identity and knowledge of neighboring Nordic languages illuminating the flow of didactical designs students were offered to engage in, i.e. the knowledge content, activities, spaces and resources. The iterative, three-year educational development of designing for learning proved to provide space for shared and collaborative knowledge development involving students in rich, authentic, and goal-oriented inter-Nordic comprehension practices on multiple levels, addressing content-related as well as contextual frames in teaching situations. Schematic models of the didactical designs revealed initial dis-alignment and diverse appropriation of modes and media in parallel cooperative activities combined with collaborative cross-border face-to-face talk promoting students to take perspective, compare, and contrast. Addressing the socio-material frames of teaching gave an opportunity to discover new learning goals and design activities in new adequate arrangements of the physical classroom and the digital space. The analysis demonstrates that students developed linguistic, cultural, critical, and digital competences of varied educational value based on the teams’ divergent approaches to knowledge processes to incorporate digital technologies in teaching practice. The study highlights the critical factors of recognizing digital, multimodal meaning-making and informal skills as means to promote learning and schools’ remaining insufficient provision of digital infrastructure for learning activities in blended learning environments.

Introduction to the Special Issue: Multimodal Meaning-Making in Science

Introduction to the Special Issue: Multimodal Meaning-Making in Science

Contemplating post-digital narrativity: Co-active, multimodal meaning-making on Instagram and its implications on learning

The paper addresses the post-digital, multimodal narratives of a thirteen-year-old teenage girl on Instagram. Post-digitality refers to the ontological assumption that the online is inseparably intertwined with the offline world. On the other hand, narratives are understood as sociocultural, perspectival, and interactional discursive nodes co-produced by the teenage girl and the platform. The fine-grained interpretation of our data draws upon a transdisciplinary framework combining several theoretical and methodological approaches, most prominently social semiotics and semiotic technology, narrative studies, new literacy studies, and critical sociolinguistics. The qualitative analysis of our data follows the logic of nexus analysis, highlighting the design of the post-digital, multimodal teenage narratives on Instagram Stories and the software’s complex role in co-crafting situated storytelling. The main research findings indicate that Instragram’s affordances, i.e., its technological, semiotic, social, and algorithmic features, function as co-active, non-human agents with which the adolescent girl strategically negotiates to produce her multimodal narrative work. Finally, we argue that educational policy should acknowledge the affordances of the teenage-platform multimodal narrative synergy and the need for a post-digital critical literacies pedagogy.

Deleuze’s meta-cinematic framing: Multimodal meaning-making in Installation Art

The article discusses multimodal meaning-making in the context of video and multi-media installation art and related curatorial practices. It draws on the Deleuzian concept of pure duration or ‘time-image,’ understood as a Foucauldian dispositive (Panagia) and, thus, as a broader heuristic device in discussing the viewer’s experience engulfed in installations. It discusses non-discursive aspects of meaning-making while focusing on the viewer/ participant’s subjectifying, multi-sensorial, kinaesthetic, performative, and time-based experience of an exhibition. We discuss installation art as a temporal situation, constructed on the difference between represented or narrated time and subjective or reception time (Petersen), following the phenomenological category of the artwork as a ‘temporal object’ (Ingarden). The paper aims to offer novel interpretive tools for investigating our current shifting sensorium engaged in meaning-making. It also maintains that installation art constitutes not only a compelling cultural strategy for “imagining and imaging the world” (Rodowick) but also the means to understand how modes of perception converge in producing subjectivity.

Infografías científicas en secundaria: complejos de significados multimodales en ensambles compuestos verbales-visuales

The complexity of science discourse has long been recognized as challenging for many students. Systemic functional linguistic accounts of technicality and meaning aggregation, differentiating scientific and everyday discourse, have explicated the linguistic complexity confronting students. The complexity of images and image-language ensembles in science discourse has not been similarly delineated. Two aspects of multimodal meaning-making have not been sufficiently theorized to support pedagogies of visualization interpretation and creation in science: (1) the role of the verbiage within scientific visualizations has been largely ignored; (2) image analysis has emphasized single-structure images, e.g. narrative or classificational or analytical, whereas multiple structures in a single image is a frequent and significant resource in science. This paper outlines a framework describing the co-deployment of image and verbiage to construct multi-structure image-language ensembles in high school science textbooks. Using this framework two investigations are described: (1) variation among textbook infographics in image-language co-articulation representing meaning complexes of phenomena such as mitosis; (2) the relationship between co-articulation of image-language resources and achievement level in infographics constructed by senior high school students. Implications are drawn for extending transdisciplinary research in educational semiotics and science education and for pedagogies of multimodal disciplinary literacy development in high school science.

The multimodal meaning-making of elementary students in social studies

During their participation in a classroom-based research project, 9-10-year-old students had opportunities to develop their visual meaning-making skills and competences, as well as their aesthetic understanding of and critical thinking about multimodal ensembles. The Grade 4 students read, discussed and wrote about picturebooks during Language Arts, Social Studies and Science, and participated in a range of activities that focused on a selection of elements of visual art and design. For the culminating activity of an interdisciplinary Social Studies unit, which was one component of the case study research, the students designed a multimodal poster to communicate their learning about how interactions between Indigenous Peoples and European explorers lead to change. Photographs of four students’ Exploration posters and excerpts from their written poster descriptions reveal how the students purposefully selected and orchestrated specific semiotic resources of image and layout to represent and express meanings that realized their objectives as sign-makers in a particular context. Indeed, the students designed multifaceted symbolic images that communicated their historical understandings as well as their knowledge about the elements of visual art and design under study. Discussion of the students’ Exploration posters is situated in sociocultural and social semiotics theory, multimodality, and visual competences.