Multimodal Interaction Analysis Research Articles

Disclaiming knowledge to encourage participation in research group meetings

This study investigates instances where experienced researchers make explicit claims of lack of knowledge (e.g., ‘I have to reveal my ignorance completely’) in the context of paper discussions during research group meetings. Drawing on multimodal interaction analysis, the analysis focuses on epistemic disclaimers in their sequential contexts, and the local management of institutional identities and domains of knowledge. The analyzed data draw from video-ethnographic work involving participant observation and video recordings at a research program in Physical Chemistry at a Swedish university. Focusing on epistemic disclaimers occurring in first pair-parts, the analysis explores how senior researchers employ disclaimers to mark the eliciting function of their questions, as they work to encourage participation by opening the floor while positioning co-participants as (more) knowledgeable. The results evidence how the accomplishment of peer collaboration and knowledge distribution in scientific work involves the management of rights and responsibilities with respect to knowledge and scientific expertise.

First-order sense-making in L2 academic discussions: A distributed view of teacher languaging dynamics in embodied and situated learning context

The study explores teacher's situated and embodied interactions with students in second language (L2) academic discussions and how the teacher languaging dynamics facilitate students' learning when they are engaged in an academic discussion outside the classroom. The situated learning enables learning to take place in contexts other than classroom that affords the environment for embodied interaction and legitimizes the learning community with shared communicative purposes. The first-order teacher languaging dynamics are recontextualized in the situated learning contexts that encourage knowledge exchange as social learning engagements to enable students to bodily experience learning and develop the process of knowing. With a distributed view to language, a multi-scalar approach to multimodal interaction analysis is employed to investigate how the teacher dynamically interacts with students through embodied semiosis in the situated learning context and how academic knowledge is bodily enacted by the teacher languaging behaviours in the dynamic embodied environment. The findings show different embodied strategies are deployed by the teacher to bodily ground the academic knowledge in both vocal tract gesturing and hand movements where the meanings are generalized, visualized, objectified, concretized and symbolized in first-order languaging dynamics to facilitate and afford student sense-making and learning in academic discussions.

Multimodal classroom interaction analysis using video-based methods of the pedagogical tactic of (un)grouping

ABSTRACT Grouping of people and/or things in school can involve challenging pedagogical problems and is a recurrent issue in research literature. Grouping of pupils sometimes aids learning, but detailed video-based analysis of how teachers (and pupils) group or ungroup (termed ‘(un)grouping’) in classrooms is rare. This multimodal classroom interaction analysis study builds on previous work by exploring how the Pedagogy Analysis Framework can help untangle complicated classroom interactions involving (un)grouping and identifies sixteen types of (un)grouping. The sample size is one class of thirty pupils (10-year-olds), their class teacher, and teaching assistant. Four research methods were used (lesson video analysis, teacher verbal protocols, pupil group verbal protocols, and individual teacher interviews). Six hours of data were video recorded (managed using NVivo). Data were analysed by two educational researchers, the class teacher, and two groups of pupils (three girls and three boys). The methodology is Straussian Grounded Theory. Data were recorded in 2019. We present how often participants (un)grouped during a lesson. We propose and use a grounded theory for (un)grouping which we call the ‘Exclusion, Segregation, Integration, and Inclusion (ESII) model’. Additionally, we discuss how misinformation and disinformation can complicate analysis of (un)grouping and examine different perspectives on (un)grouping.

Occupying Another’s Digital Space: Privacy of Smartphone Users as a Situated Practice

AbstractA smartphone’s screen is commonly regarded as a private space, and the action of looking at it is usually considered a violation of one’s privacy both by researchers and designers. However, our study demonstrates how participants in the interaction themselves negotiate moment by moment and achieve an understanding of someone’s screen space as public or private. In this paper, we analyze the interactional sequences of uninvited looks at another participant’s phone. Drawing on visual ethnography and ethnomethodologically informed multimodal interaction analysis, we video-recorded and analyzed everyday interactions between friends and acquaintances. Our findings show that looking at someone’s smartphone display is often performed and oriented to as a resource in interaction rather than an invasion of privacy. We therefore characterize the interactional functions of gazes and glances at another’s screen. We also discuss the research and design implications of approaching privacy as a situated practice.

Repetition for real-time coordination of action: Lexical and non-lexical vocalizations in collaborative time management

Repetition has often been argued to be a semiotic device that iconically signifies ‘more content’, such as intensity and plurality. However, through multimodal interaction analysis of materials in English, Estonian, and Swedish, this paper demonstrates how self-repetition is used to coordinate actions across participants and temporally organize the ongoing activity. The data are taken from infant mealtimes, pilates classes, dance training, boardgames, rock climbing, and opera rehearsals. Repetition of both lexical and non-lexical tokens can prolong, postpone, and generally organize segments of action as well as co-create rhythms and moves in a moment-by-moment reflexive relationship with other (non-vocalizing) participants. A crucial feature of repetitions is that they can be flexibly extended to fit the other’s public performance, its launching, continuation, and projectable completion. We argue that the iconicity of repetition emerges through its indexical relationship to other bodies, as a real-time jointly achieved phenomenon.

Transcribing Facial Gestures

Within multimodal interaction analysis, transcripts serve not only as a tool for managing the volatility of interaction and catalyzing analytic procedures, but also as an essential medium for making analytic results intersubjectively available. While there are already well-established conventions for transcribing verbal and vocal interactional resources, researchers still struggle with adequately aligning and recognizably representing visual-bodily resources. This contribution provides a practical solution for multimodal transcription, combining conventions of the Jeffersonian system with the sign inventory of the International SignWriting Alphabet (ISWA). The result is a standardized, data oriented, expandable system that relies on iconic depictions rather than on verbal descriptions of visual conduct.

Swedish-English preschool as a site for the collaborative discovery of bilingual meanings

This paper examines spontaneous conversations about word meaning in a bilingual preschool in Sweden. This qualitative empirical study is grounded in an ethnomethodological theoretical framework and contributes to research on multilingualism by using a sociocultural lens to examine mundane linguistic experiences of very young children who learn to speak in more than one language. The data comprise video-recordings of naturally occurring interactions among teachers and children in a Swedish-English preschool with a one teacher-one language policy. The data were collected during ethnographic fieldwork in an urban area in Sweden. Approached with multimodal interactional analysis, the data draw attention to teachers’ everyday didactics, including their professional strategies for initiating spontaneous vocabulary work and orchestrating multiparty engagement in the collaborative discovery of meaning, and children’s participation. The analysis discusses strategies for providing word definitions and demonstrates mundane institutional contexts outside of the classroom setting where such interactions were possible. Both teachers and children engaged in vocabulary exploration by using words in a situated, locally meaningful way. The study highlights that the teachers followed the preschool’s language policy and embodied monolingual identities, while orienting to children as multilingual learners and supporting their language development.

Sy ihop

This article explores assessment actions – as an aspect of classroom communication – and how they are expressed in sloyd teaching; with a formative aim centering on pupils’ learning and (the teacher’s) own teaching practice, respectively. In this context, the teacher holds a central role, mediating sloyd knowing through (communicative) verbal and physical actions, acting as an authoritative guide for pupils’ learning, by making perceptually tacit knowing discernable thanks to pre-established confidence for the teacher in question. Besides authoritative guiding, another theoretical starting point of the article is sociomateriality; a perspective considering physical objects as resulting from connections and activity, hence influencing on the (more) social sphere of actions and activity. The method used is multimodal interaction analysis of three video-recorded sequences of teacher/pupil interaction in a sloyd-classroom setting. To sum up, the findings of the article display how formative assessment in the sloyd context consists of the teacher adapting questions posed to ascertain the knowing of the pupil, then waiting to see how/what/if the pupil performs, whilst being provided – gentle but ample – authoritative guiding to help in the process. The main contribution of the study is establishing the role of formative assessment within sloyd education. The research project has been conducted within the framework of Stockholm Teaching and Learning Studies (STLS), entailing the active involvement of the teacher in question as co-author of the article.Key words: assessment actions, formative assessment, sloyd teaching, authoritative guiding, sociomateriality, communicative resources

Multimodal repetitions in children’s co-construction of arguments

AbstractThe study examines the interactive functions of gesture repetitions as a specific form of interpersonal synchronization in argumentative decision-making processes of peers (1–6 graders). Based on a collection of 13 instances, gesture repetitions are investigated in the process of collaboratively co-constructing arguments. Drawing on multimodal interaction analysis, a description is provided of how gestures are aligned and laminated with other multimodal resources, such as body posture and gaze, into multimodal repetitions during the co-construction of both arguments and counterarguments. The study illustrates the way in which multimodal repetitions serve to mark coherence between the participants’ turns and contribute to the argument. As the multimodal resources are highly synchronized with each other, multimodal repetitions also make the collaboration of specific participants publicly visible to all interactants, thereby attaching more importance to the sharedness—and persuasiveness—of the constructed argument. The analysis therefore focuses on both depictive and pragmatic gestures that are conceived as parts of multimodal gestalts (Mondada, 2014), and describes different trajectories of co-constructions that entail interactively successful as well as unsuccessful examples.

Engaging with materials and the body: young plurilingual children’s resource-rich interactions in science investigations

ABSTRACT Background Children’s interactions are inherently multimodal. Accordingly, when children engage in science in classroom communities, their interactions are grounded in material and embodied aspects of interactions with teachers and their peers. Purpose This study explores the nature of children’s science interactions in a classroom community where students are working in a language they are also learning through examination of the ways in which open-ended pedagogical approaches mediated embodied and material science participation and sense-making. Study and method We employed multimodal interaction analysis layered with a critical ethnographic perspective to explore the embodied and materially-grounded experiences of a student working in an early-childhood classroom with peers and teachers in science through a language he is also working to learn. Results Multimodal interaction analysis allowed us to build views of how the classroom interactional spaces afforded embodied and material participation and learning in science, and uncovered the multimodal ways in which this mediated his engagement in science and communication of science meanings and wonderings. Conclusions Open-ended pedagogical approaches afforded spaces in which this student was able to engage with phenomena, materials, and embodied interactions, regardless of his verbal participation. This embodied participation and materially grounded interactions entangle and unfold to become learning. Pedagogical implications for the teaching and learning of science with all children, and in particular with plurilingual children, through the use of open-approaches that afford resource-rich embodied engagement and learning are discussed.

Using literary conversations to design stimulating learning environments for all elementary students

The paper presents results of the LemaS-GRiP project. The focus lies on literary conversations in inclusive literature lessons in primary schools. These conversations link individual aesthetic experiences and interests with a jointly responsible development of literary interpretations in exchanges between pupils and teachers. Based on a Literary Classroom Conversation in a fourth grade, we can show in a preliminary analysis how learners enter into the open-ended process of understanding and approach the ambiguity of the text. At the same time, all participants have a growing responsibility towards the text and for each other. In doing so, they are partly dependent on a competent other, who supports them in approaching the ambiguity of the text in the sense of scaffolding. Central instrument was a multimodal interaction analysis of videotaped lessons following the principles of the documentary method.

Eating Prickly Peas: Sharing Play Worlds During Preschool Meals

AbstractPlay has long been understood as an important pedagogical practice, particularly in early childhood education and care settings. Playing with food, however, has typically been overlooked, and very little is known about food play during mealtimes. The apparent dichotomy between rule-following and playfulness at mealtimes has led to a paucity of research on playing while eating. This paper raises the profile of food play and examines instances in which young children initiate pretend play with their food during shared mealtimes. Data are taken from a large corpus of video-recorded lunches in Swedish preschools, and episodes featuring pretend play with food were analysed using multimodal interaction analysis. The results illustrate how play signalling is multimodally achieved, directed first to teachers, often involves other children, and enables the multiactivity of playing and eating. Children invited teachers into their imaginary worlds and teacher’s responses enabled the play narrative to develop and co-exist with the institutional demands of eating lunch together. The paper provides empirical evidence that pretend play with food during meals offers affordances for generating and sharing imaginary worlds with teachers and peers.

Exploring the materiality of science learning: analytical frameworks for examining interactions with material objects in science meaning-making

ABSTRACT Background With a growing new materialism paradigm and research on multimodality, there is an increasing attention on the role of material objects in science teaching and learning. However, there is currently no available framework, coding scheme, or method of inquiry to specifically analyze the use of material objects in science meaning-making. Purpose This paper presents and discusses two analytical frameworks developed to examine teachers’ and students’ interaction with material objects during classroom discourse. Design and methods The first analytical framework was theoretically informed by multimodal interaction analysis (MIA), focusing on humans’ situated interaction using embodied and disembodied modes located in the material world. The second framework was informed by social semiotics multimodal analysis (SSMA), focusing on material interaction as a semiotic mode with unique affordances for sign-makers to make meaning alongside other semiotic modes (e.g. speech, gesture). Sample Both frameworks are used to analyze two vignettes of classroom discourse generated from video data to illustrate their applications, coding procedures, and interpretative potential. Results The MIA-informed framework highlighted the dynamic and reciprocal relations between students’ experiences and the transformation of materials, while the SSMA-informed framework highlighted the subtle meanings constructed with the unique affordances of the material mode. Conclusion The affordances and limitations of the two frameworks are discussed, along with their compatibility and complementary relations. This work contributes to the emerging field of materiality in science education research and provides a foundation for future studies on the analysis of material interaction in science meaning-making.

A case study of young children’s play negotiations in free play

ABSTRACT This study seeks to understand how children express themselves socially and emotionally in play negotiations in early childhood education and care (ECEC) settings. It addresses the following research questions: What strategies do children employ in play negotiations? How do these strategies manifest themselves socially and emotionally? The study was conducted in eight ECEC centers and one preschool located in Swedish-speaking regions of Finland. The target group consisted of children aged 3–6. Data were collected through video observations that captured children’s social interactions and engagement with their peers during free play. A multimodal interaction analysis approach was employed to identify a prominent category termed ”exclusion”, which refers to how children employ strategies (e.g., fantasy and fiction) to limit the participation of other children in play negotiations. The results of this study contribute to our understanding of children’s complex play negotiations and highlight the significance of addressing their social and emotional development in these contexts.

Cries of Pleasure and Pain: Vocalizations Communicating How Touch Feels in Romantic Relationships

ABSTRACT Research on interaction has recently ventured into the domain of sensoriality, hitherto considered inaccessible for video analysis. This article contributes to this emerging field by targeting the interface of touch and vocal sound, dissecting the intersubjective potential of sounding when bodies are intertwined. Studying naturally occurring affective episodes between romantic partners at their homes with the methods of multimodal interaction analysis, we demonstrate how vocalization can express comfort and thus lead to extension of mutual bodily contact, or inform of discomfort, which leads to swift release of the problematic contact. We argue that, by providing immediate and nuanced access to other participants’ ongoing bodily experiences, haptic sensoriality is partially distributed into the auditory domain. The relevance of progressivity in body contact is continuously negotiated and has moral, social, and relational implications, providing grounds for continuous realization of consent. Data are in Finnish.

Student’s agency in English as Foreign Language learning in a Virtual Environment: An Interaction Analysis

The COVID-19 pandemic brought about new challenges for class interaction and for English as a Foreign Language teaching instruction in general, and opened a whole world of opportunities to learn about virtual-mediated instruction and about learning altogether. As a reaction to the new social conditions, alternative spaces, and ways of mediating instruction through the use of technology and digital resources were created and promoted in the quest for teaching and learning. As such, this multimodal interaction analysis inquired into the qualities of interaction taking place in an online course of advanced English at a teaching credential undergraduate program at a private university in Bogotá, and the affordances and limitations that such interactions presented for agency, engagement, and ultimately, learning as an effect of communication. To such an end, class observations, student focus groups, and interviews on teachers were conducted. The findings offered insights on the multiple ways students and the teacher resorted to their semiotic resources and communicative modes to configure meaning, construct knowledge, and relate to one another in a virtual environment. Mainly, however, it encourages a debate about the role that teachers play in designing and implementing communicative environments to enrich the teaching and learning experience.

Research on the Evaluation Mechanisms of the Design Form and Connotation of Digital Games from a Multimodal Interaction AnalyticalPerspective

This research endeavor primarily focuses on computer, or electronic transport games. Electronic games are based on a computer's ability to utilize a proportional calculation pattern that simulates imagined human behavior. While the game industry has grown substantially in recent years, methods for evaluating such games remain underdeveloped. Existing evaluation systems also tend to overlook gender bias, overt violence, and a lack of sexual imagination. To construct a more appropriate evaluation system in terms of both form and function, and operating from the perspective of linguistic multimodal interaction analysis, this study analyses the form design of video games with regard to game action levels and modal configuration, as well as the multiple functional meanings in game design at the ideational, interpersonal, and textual, and expressive levels. In addition, the principles of multimodal discourse analysis are applied to popular games, such as Harry Potter: Magic Awakened and It Takes Two, to enrich discussions of this theoretical framework and provide systematic suggestions for game design's diversified and innovative development. The goal of this article is for scholars in various fields to use the analytical paradigm offered herein to examine multimodal expressions in more evaluation mechanisms of game design in the future.

Using Video and Multimodal Classroom Interaction Analysis to Investigate How Information, Misinformation, and Disinformation Influence Pedagogy

Abstract Misinformation is accidentally wrong and disinformation is deliberately incorrect (i.e., deception). This article uses the Pedagogy Analysis Framework (paf) to investigate how information, misinformation, and disinformation influence classroom pedagogy. 95 people participated (i.e., one lesson with 7-year-olds, another with 10-year-olds, and three with a class of 13-year-olds). The authors used four video-based methods (lesson video analysis, teacher verbal protocols, pupil group verbal protocols, and teacher interviews). 35 hours of video data (recorded 2013–2020) were analysed using Grounded Theory Methods by the researchers, the class teachers, and groups of pupils (three girls and three boys). The methodology was Straussian Grounded Theory. The authors present how often participants used information, misinformation, and disinformation. They illustrate how the paf helps understand and explain information, misinformation, and disinformation in the classroom by analysing video data transcripts. In addition, the authors discuss participant perceptions of the status of information; overlapping information, misinformation, and disinformation; and information communication difficulties.

Creating a web of multimodal resources: Examining meaning-making during a children’s book project in a multilingual community

While many children grow up in linguistically and symbolically diverse communities, it is still rare that they encounter an early educational experience adapted to the complexities of their everyday communicational reality. This paper takes an ecological and multimodal approach to a preschool’s book project in a multilingual community. The study examines the web of resources that emerges from activities, actors and their interrelations during the book project. It is shown how multimodal resources emerge when supported by active pedagogical community engagement, and how resources underpinning early childhood literacy cross linguistic and modal boundaries. The paper uses a multimodal interaction analysis to show how the socioecological resources emerging during the project come together in multimodal interaction. Pedagogical potentials building on multimodal resources involving the wider ecology of actors in linguistically complex settings are discussed.

Multimodal methods for managing deontic rights in interdisciplinary trials

This paper focuses on interdisciplinary trials as a multiparty and multiactivity setting in which different professions are needed for students with mobility disabilities to try out robot-assisted gait training. Through an ethnomethodological and conversation analytic approach to multimodal interaction analysis, we investigate how a research team collaborates with the employees at a sports high school to get students who are wheelchair users to walk. The paper provides an analysis of the professionals’ employment of multimodal resources for managing deontic rights when organizing interdisciplinary trials. It examines how the professionals initiate and negotiate their own and others’ actions necessary for preparing the student for walking. Based on video recordings, the paper focuses on the professionals’ deontic claims concerning co-participants’ actions and analyzes their temporal and spatial contingencies. Building on prior research on deontics, this paper contributes to our understanding of the interactive accomplishment of deontic authority and its sensitivity to contextual features.