Multimodal Artifacts Research Articles

Inferential and Macrostructural Ability in Multimodal Texts

Objective: To evaluate inferential and macrostructural ability through three types of multimodal artifacts: verbal, graphic-mathematical, and complete text (which integrates all semiotic modes). Theoretical Framework: This section explores the key concepts of multimodality theory, as well as the notions of inferences and multimodal ability, which are fundamental to addressing the research problem. The main theories and concepts supporting the research are presented, providing a solid foundation for understanding the study's context. Method: A mixed methodology was employed, combining quantitative and qualitative approaches. The quantitative phase adopted a correlational ex post facto design, while the qualitative phase was carried out through a focus group. The quantitative study followed a factorial experimental between-subjects design, considering two dimensions: the participants' professional program and the semiotic systems present in the comprehension test. Results and Discussion: It was concluded that the type of multimodal artifact does not significantly affect the efficient identification of inferences or macrostructural reduction. This suggests that students exhibit a low level of multimodal ability. Research Implications: The practical and theoretical implications of the research are analyzed, providing insights into how the results can influence or be applied to practices related to multimodal comprehension. Originality/Value: The research highlights that universities, in general, do not pay sufficient attention to multimodality, which negatively impacts students' multimodal competencies.

Engaging culturally and linguistically diverse youth in semiotic analysis for community‐based inquiry

AbstractThis manuscript examines multimodal storytelling as community inquiry for an urban high school class of 30 first‐ and second‐generation bi/multilingual immigrant students, most of whom maintained transnational connections. We share how these students, in an A.P. Research class, engaged in community‐based inquiry and utilized various multimodal artifacts from a university cultural heritage collection, including a field trip to the cultural heritage archives, to analyze signs and symbols. We asked: How did students engage with semiotic resources and approaches to support inquiry about their school and neighborhood communities? Findings showed that students drew upon their linguistic and cultural experiences to analyze signs and symbols, yet in their digital storytelling presentations some student groups chose signs and symbols that reified stereotypical views of their neighborhood despite their insider status in the local community. We offer recommendations for using cultural heritage artifacts to advance multimodal meaning‐making in teaching and research.

Exuberant and fantastical: How flythroughs are changing the way museums construct their identities and build relationships with their publics

Digital ‘flythroughs’ are increasingly being used by museums to promote themselves to their publics and to extend their spatial and experiential boundaries. These complex, multimodal artefacts use 3D animation, AI imagery, drone footage, sound, music and other modes to construct a parallel museum experience. Such an experience is markedly dissimilar from an ‘ordinary’ embodied relationship to architectural space, and instead constructs an experience of elevated velocity, height and glissade. These movement style choices construct a ‘non-naturalistic’ experience of moving through space that differs considerably from the embodied experience constructed by digital walkthroughs, augmented and virtual reality and ordinary corporeal movement through a museum. Instead, they offer a potential imaginary, a supernatural or ’extracorporeal’ embodiment of a kind typically associated with fantasy films, video games, amusement rides and dreams. This paper analyses 10 flythroughs from a variety of museums and locations through the lens of multimodal social semiotics. The systems of movement styles and visual validity are used to explore how meanings are made in the flythroughs and how they may be understood in terms of the ‘fantasy’ coding orientation. The implications for how museums construct their respective identities and relate to their publics are also unpacked.

Audio-Video Analysis Method of Public Speaking Videos to Detect Deepfake Threat

Aim: The purpose of the article is to present the hypothesis that the use of discrepancies in audiovisual materials can significantly increase the effectiveness of detecting various types of deepfake and related threats. In order to verify this hypothesis, the authors proposed a new method that reveals inconsistencies in both multiple modalities simultaneously and within individual modalities separately, enabling them to effectively distinguish between authentic and altered public speaking videos. Project and methods: The proposed approach is to integrate audio and visual signals in a so-called fine-grained manner, and then carry out binary classification processes based on calculated adjustments to the classification results of each modality. The method has been tested using various network architectures, in particular Capsule networks – for deep anomaly detection and Swin Transformer – for image classification. Pre-processing included frame extraction and face detection using the MTCNN algorithm, as well as conversion of audio to mel spectrograms to better reflect human auditory perception. The proposed technique was tested on multimodal deepfake datasets, namely FakeAVCeleb and TMC, along with a custom dataset containing 4,700 recordings. The method has shown high performance in identifying deepfake threats in various test scenarios. Results: The method proposed by the authors achieved better AUC and accuracy compared to other reference methods, confirming its effectiveness in the analysis of multimodal artefacts. The test results confirm that it is effective in detecting modified videos in a variety of test scenarios which can be considered an advance over existing deepfake detection techniques. The results highlight the adaptability of the method in various architectures of feature extraction networks. Conclusions: The presented method of audiovisual deepfake detection uses fine inconsistencies of multimodal features to distinguish whether the material is authentic or synthetic. It is distinguished by its ability to point out inconsistencies in different types of deepfakes and, within each individual modality, can effectively distinguish authentic content from manipulated counterparts. The adaptability has been confirmed by the successful application of the method in various feature extraction network architectures. Moreover, its effectiveness has been proven in rigorous tests on two different audiovisual deepfake datasets. Keywords: analysis of audio-video stream, detection of deepfake threats, analysis of public speeches

Student writing in the engineering curriculum: discursive rhetorical model of the laboratory report genre in Spanish.

The laboratory report is a widely used genre in the academic training process in civil computer engineering. Students produce this genre in the university classroom for diverse academic and professional purposes. Despite its relevance, empirical rhetorical-discursive descriptions of the value of student writing are still scarce in Spanish. Thus, we describe the rhetorical-discursive organization of the laboratory report genre in this subdiscipline. To fulfil this purpose, we followed a methodological design based on Swalesian Genre Analysis and used a corpus of ninety-eight texts. The sample was collected in a self-compiled form through consultation with teachers and students in the university classroom. The application of this method allowed us to determine the macro-moves, moves, and rhetorical steps of the genre, its communicative functions, and textual features. The resulting rhetorical model consisted of four macro-moves, twelve moves and seventeen steps. This model shows the highly dynamic and mesogeneric nature of this genre, the new functions of multimodal artefacts, and the genre's presence across the curriculum. To know about the teachers' and students' views on the process of training professional writers in engineering, the rhetorical model was complemented with an ethnographic phase (in the terms proposed by Swales: corroboration process with a couple of members of the community) before and after collecting the textual corpus. Finally, the implications for genre theory, Spanish language theory and pedagogy of the Spanish language and genre pedagogy are discussed.

Vocabulary in urban contexts using multimodality

This is the presentation of a new pedagogical tool. The tool is an alternative to the usual presentation of vocabulary focusing on some key concepts: multimodality, theory of space, and collaborative learning. Considering that the study of foreign languages has the difficulty of studying cultural elements that are usually not present in the life of the students, this paper will propose an activity to study vocabulary in a contextualized space. The objective is to understand social spaces through discovering lexical items in a natural environment, using the observation and description of urban photographs, which represents a change to the traditional way of presenting vocabulary. Through a collective and technology mediated process, the students create a multimodal artifact that can be shared with other students. The process of learning occurs, first, from the production of the artifact, and second, from the presentation and exchange of the artifacts produced among students.

Mediated focalisation in video explanations: Implications for the communication of architecture and STEM

In 2019–2020, students of two English for Architects and Civil Engineers courses at a German university were tasked with creating a digital, multimodal video composition explaining a technical concept to a lay audience. The resultant multimodal artefacts, however, often did not exhibit typical semiotic patterns associated with explaining or describing in science-related disciplines. In particular, 78% of artefacts featured ‘mediated focalisation’, a framing technique used to align the composer with their audience and more commonly associated with fictional narrative or social media. The paper describes how this framing technique appeared in the artefacts and explores how and to what effect it was used. It will unpack the implications of using this technique for the performance of professionalism and ‘authenticity’ in architecture and STEM communication. A subsystem of mediated focalisation techniques and a new ‘coding orientation’ will be proposed, so that educators may better prepare students for these emergent shifts.

Blurring the boundaries of current and future selves: students’ STEM identity exploration in a multimodal composing learning environment

ABSTRACT As multimodal composition is gradually integrated into STEM learning, research is needed to examine how to fully connect multimodal composing practices and STEM practices to support students’ STEM identity exploration. To fill in this gap, we conducted a design-based research project to investigate how students explored STEM identities in a multimodal composing environment. A total of 42 fifth- to eighth-grade students participated in the project in which they worked in groups to create multimodal science fiction stories. We identified two cross-cutting themes regarding how students presented STEM-related selves in multimodal artifacts: channeling current and imagined future life experiences into multiple characters and experiencing a hypothetical universe through restorying the self. The findings suggest that students presented a mix of current and future selves in multimodal artifacts. This study sheds light on promoting STEM identity by engaging students in presenting selves through modes of choice.

Multimodal media: Framing climate change

The paper uses examples of multimodal discourse to show how frame analysis can account for the choices of form in a range of multimodal (image or image-plus-text) artifacts addressing issues of climate change. Relying on the concepts of frame metonymy and blending, the article discusses a selection of artifacts representing emergent frames such as Save the Planet and Hourglass, which are pervasive in multimodal representations of climate change. The article describes the advantages and disadvantages of common multimodal stylistic choices, while addressing issues of ‘persuasive bleaching/overload’. It also uncovers some frame-related specificities of multimodal discourse.

Sound Anchors

When watching a film, we engage with much more than combinations of moving images. We combine what we see with what we hear, and what we hear often aids in the construction of a story. Although some researchers endorse the ways sound guides viewer expectations, there is still a need to explain the ways images, sounds, and other available cinematic modes interact to construct meaning. This article engages with research on embodiment, cognition, and multimodal artifacts to reveal how sound aids in the construction of film narratives by focusing on examples where sounds take the primary role in constructions of narrative meaning. Additionally, by discussing recent theories on cognition and multimodality, this article shows how sounds can evoke conceptual and narrative information in ways that stabilize our understanding of cinematic representations through the joint contribution of all of the available modes.

Pre-Service Primary Teachers’ Beliefs, Practices, and Needs Regarding the Teaching of a Second Language Through Book Creator

Several studies have investigated the use of Digital Storytelling to foster Second Language Acquisition at primary school level from pre-service teachers’ perspectives. However, research regarding the employ of Book Creator, a digital book-making tool, to enhance students’ language, digital, and intercultural competences according to student teachers’ perceptions is still lacking. This article presents a case study carried out with 35 student teachers who participated in a workshop on teaching Italian as a Second Language through Digital Storytelling during a single cycle degree programme in Primary Teacher Education, at an Italian university. The study aimed at exploring pre-service teachers’ beliefs, practices, and needs regarding the use of Book Creator to train pupils’ language, digital, and intercultural skills within the multilingual and multicultural Italian primary school. Student teachers were asked to experiment Book Creator and to reflect about its didactic potentialities, from both students’ and teachers’ perspectives. Data were collected through a questionnaire, student teachers’ multimodal artifacts and teaching materials. Results show the efficacy of Book Creator as a learning tool aimed at enhancing pupils’ language competences, digital literacies, and intercultural awareness, as well as pre-service teachers’ interest in discovering further digital resources to foster Second Language Acquisition through Digital Storytelling.

Unveiling the rationale of soft hate speech in multimodal artefacts

This article sets out to outline a methodological framework that enables us to unravel the underlying reasoning of covert hatred (i.e. soft hate speech), through an examination of the ways in which this is realised and, moreover, argumentatively justified in multimodal artefacts of the mainstream press. It focuses on a controversial case study – the release of the Italian hostage Silvia Romano – and how it was covered on the front pages of two right-wing Italian newspapers. It draws on the premises of multimodal critical discourse studies (MCDS), proposing a micro-level cross-fertilisation of a social semiotic discourse–analytical perspective for the analysis of multimodal meaning(s), realised on newspaper front pages, with the Argumentum Model of Topics (AMT), which focuses on the analysis of argumentative inferences that stem from multimodal representational meaning(s).

Materials and Places for Learning: Experiences of Doctoral Students in and around University Spaces

People are more likely to thrive when they feel connected, when they feel they belong to a group, to a place, or when they feel part of a community. Places can play a powerful role in shaping one’s attachment to others and to institutions as part of the development of one’s identity. People’s experiences of places are linked to their sensorial impressions of material and digital elements, and to their perceptions of how multiple elements interconnect and impact lived experiences or imagined futures. This research investigates doctoral students’ experiences of places for learning in and around a university in New Zealand. The analysis combines individual interviews and digital multimodal artefacts created by participants who were studying on campus or studying at distance and remotely located. By acknowledging the diversity of university spaces where learning activity may unfold — in classrooms, at libraries, in the canteen, in the corridors, via online learning management systems, social media and messaging, or in the many in-between spaces such as buses, cafes, or working from home — this paper discusses the connections between people, places, material, and digital artefacts, with a focus on the materiality of learning in and around university spaces. Using socio-material conceptual lenses, the article reveals how students navigate the postdigital university through the spaces they inhabit and the places they value, and how their attachment to materials, feelings of inclusion, and learning purpose interconnect to support their (emerging) professional identity.

One Fourth-Grader’s Orchestration of Modes Through Comic Composition

Language-oriented literacy standards offer mostly linguistic accounts of text complexity. In response, the present article demonstrates that multimodal and visual narratives offer additional ways to understand and discuss text complexity. This descriptive analysis of one fourth-grader’s comic provides an account of the multimodal patterns and orchestration noted across the pages of the comic. Data sources included the published comic, as well as a multimodal artifact elicitation interview conducted with Sabrina, a fourth-grade student. The authors show how Sabrina constructed a complex multimodal text by drawing not only from her knowledge of image and written language but also from her experiences with spoken language, touch, facial expressions, and gesture. These findings suggest that it would be beneficial for teachers and researchers to continue to create curricular space for multimodal composing opportunities and that stakeholders in language arts and communication education might deepen collaborations to develop instructional frameworks that support students as they compose using modes beyond language across the grade levels.

Multimodality and English for Special Purposes: Signification and Transduction in Architecture and Civil Engineering Models

The applied disciplines of architecture and civil engineering require students to communicate multimodally, and to manipulate meaning across media and modes, such as image, writing or moving image. In their disciplinary studies for example, students must be able to transform the language of lectures and textbooks into models and diagrams. In their future workplaces, they will commonly be required to transform reports and legal documents into floor plans and digital & physical 3D models. Such multimodal literacy, however, is not typically reflected in their related subject-specific English language courses, especially in Germany, where a text-centric approach is favored. To better reflect the demands placed upon them, students in two courses of English for Architecture and Civil Engineering were tasked with creating digital, multimodal artifacts to explain a concept from either of these fields to a lay audience. The resultant artifacts used a wide variety of semiotic resources to make meaning, including a total of 26 separate architectural and civil engineering models. This is a quantity sufficiently large enough to invite closer examination, and also reflects the important role models play in the fields of architecture and civil engineering, both at university and in the workplace. This paper suggests that models of this kind exist within a system of signs, in which meaning is created in the relationships between the signs. The process of transforming one resource into another also invites the consideration of the artifacts in terms of the notion of “transduction”, to discern how meaning changes between contexts, practices and modes and to contribute to existing literature on multimodal texts in tertiary education, particularly within a language-learning context.

“Staring Down a Charging Bull”: Reconceptualizing Content and Disciplinary Literacy through Transmediation

This paper illustrates the findings from year three of a five-year research project where participants were asked to multimodally re-conceptualize their understandings about content and disciplinary literacy practices from a mandatory Bachelor of Education literacy course. Data collection includes transcribed interviews, professor feedback, in-class conversations with peers, multimodal artefacts, and participant notes taken during a gallery walk. Findings show that life experiences, transmediation processes, peer group sharing, and facility with modes and media contributed to deep understanding about multiliteracies practices, course content, and assessment techniques. Findings reveal that learning opportunities transcend disciplines, space, and time while enriching identity formation.

Teaching Italian as a second language through digital storytelling: Students’ perceptions towards izi.TRAVEL

The use of technology-enhanced language learning, representing an urgent issue due to the Covid-19 pandemic, has also been promoted by many studies in second language acquisition. Nevertheless, research in this field is only partially developed for the teaching of Italian as a second language (L2) within the university context and for investigating students’ perceptions. This article presents an action research project on the use of izi.TRAVEL, a website housing more than 15,000 audio guides for touring various sites in cities around the world. The aim of the study is to contribute to developing didactic practices relative to Italian as an L2 through digital storytelling, in order to raise and foster students’ linguistic and digital skills. Participants were a small group of students studying Italian as an L2 as part of a master’s program at an Italian university. Data were collected through a questionnaire, a focus group, and students’ multimodal artifacts. Results show the positive impact of project participation on students’ attitudes and perceived learning outcomes, as well as improvements in linguistic, cultural, environmental, and digital competences.

Salient visual foci on human faces in viewers’ engagement with advertisements: Eye-tracking evidence and theoretical implications

Multimodal theories informed by Systemic Functional Linguistics involve postulations on users’ interactions with multimodal artifacts which need to be tested with behavioral data. Studies empirically testing the postulations are of great theoretical and methodological significance for multimodal research but have so far remained nascent. To open up new avenues of research in this area, findings from an eye-tracking experiment designed to test social semiotic postulations are reported and discussed. Specifically, the experiment adopted a between-group matched design and involved 44 viewers and 10 trials in each of which a print advertisement was randomly presented to the viewers who were instructed to perform a group-specific ad viewing task. Eye movement metrics obtained from the experiment were analyzed by descriptive statistics based on Areas of Interest and the Scanpath Trending Algorithm to link fixation metrics to semantic components of the ads and to calculate the common viewing path per group and trial, respectively. Data analysis results revealed, among others, a concentration of visual attention on human faces placed at various regions of the ads. The findings are then channeled into discussions on information value theory in the grammar of visual design to explicate its implications for theorizing multimodal communication.

“Never in my life have I heard such a load of absolute nonsense. Wtf.” Political satire on the handling of the COVID-19 crisis

This paper problematises political satire in a time when the COVID-19 virus has provoked numerous deaths worldwide, and had dramatic effects on social behaviour, on a scale unknown in western nations since World War II. Most populations have endured lockdown, periods of enforced domestic imprisonment, which led to images of the empty streets of big cities appearing in media, symbols of the drastic changes that the health emergency was making necessary. Yet, from the outset, comic memes began to circulate across (social) media, while in mainstream print media political satirists continued to lampoon official responses to the ongoing crisis. The paper thus aims to explore the connection of political satire and humour, asking two principle research questions: firstly, how to explain the humorous effects of these multimodal artefacts in such depressing circumstances; secondly, from a pragmatic perspective, to account for their overall socio-political function.The study uses memes taken from various online sources (Facebook, Twitter, Google) during the crisis, analysed according to a mixed approach that blends notions from Humour studies, especially incongruity (Morreall 2016), with insights from linguistic pragmatics (e.g. Kecskes 2014). The findings emphasise the emotional dimension of this form of satire, as the memes work against the backdrop of a range of feelings (anger, bitterness, disappointment, frustration, despair, etc.), many of which have been widely generated by the COVID-19 crisis and political responses to it. In short, to paraphrase Walter Benjamin (2008: 378), man may run out of tears but not of laughter. The findings contribute to our understanding of online satire as an emergent genre, one that uses the affordances of new media to extend the social potentialities of a traditional subversive discourse form.

Translanguaging space and classroom climate created by teacher’s emotional scaffolding and students’ emotional curves about EFL learning

ABSTRACT Translanguaging goes between linguistic and/or modal boundaries and beyond them (Li, W. (2011a). Moment analysis and translanguaging space: Discursive construction of identities by multilingual Chinese youth in Britain. Journal of Pragmatics, 43(5), 1222–1235. https://doi.org/10.1016/j.pragma.2010.07.035). The appearance of this new concept marks a significant development of multilingualism studies. This paper is based on an empirical study of creating a translanguaging space and positive ‘classroom climate’ (Gabryś-Barker, D. (2014). Success: From failure to failure with enthusiasm. Studies in Second Language Learning and Teaching, 4(2), 301–325. https://doi.org/10.14746/ssllt.2014.4.2.7) by allowing for students’ emotional expression and supporting their emotions to benefit EFL (English as a Foreign Language) learning in the Chinese context. The visual data are multilingual and multimodal student-made artefacts called emotional curves completed by newly registered university students as descriptive homework of their emotions about English learning in the classroom. These emotional curves create a translanguaging space where students adopt multiple linguistic and semiotic resources to express emotions and identities freely and creatively. The multimodal discourse analysis of emotional curves is combined with (auto)ethnographic observations and interviews, which shed light upon the emotional motivation for translanguaging and manifest effective teacher-student communication owing to the teacher’s ‘emotional scaffolding’ (Park, M. H. (2014). Increasing English language learners’ engagement in instruction through emotional scaffolding. Multicultural Education, 22(1), 20–29.) strategies. Theoretically, this paper highlights the transformative power of translanguaging space and clarifies emotional representations of translanguaging theory. Pedagogically, this study advocates acknowledging and scaffolding students’ emotions to establish better teacher-student relationships and create a flexible translanguaging space and harmonious classroom climate.