Approaches to Multimodal Discourse Analysis

Abstract

Culture and society are produced through interactions between people, objects, and environments. Within these interactions it has become clear that the modes of spoken and written language are only some of a diverse range of modes involved in producing meaning and experience. As topics of study, the modes of spoken and written language have been joined by modes like gesture, gaze, composition, and layout. Multimodal discourse analysis names a range of approaches to studying social interaction and meaning as multimodal, that is, produced with and through multiple modes. However, multimodal discourse analysis is not about identifying and studying modes as isolated but rather about understanding the world as multimodal. This understanding is developed through theoretical and methodological developments. Three major theoretical bases are in use in multimodal discourse analysis: (a) Hallidayan systemic functional linguistics (SFL), underpinning a social semiotic and SFL approach; (b) mediated discourse analysis, underpinning a mediated action based approach; and (c) conversation analysis (CA), underpinning a turn-taking based approach. The simple idea of identifying modes beyond language belies the growing complexity of research in this area. The multimodal literature contains significantly different definitions of mode and different foci. For example, systemic functional approaches began by analyzing how meaning is embedded within images and artifacts, and these studies have been extended at times to include studies of interaction. Mediated discourse analysis focuses primarily on interaction and understands images and artifacts through how social actors interact with them, rather than seeking to decode possible meanings from the perspective of the analyst. Conversation analytic approaches also focus on interactions and only engage with images and artifacts as they appear in interaction. All three theoretical bases develop from studies of language. Researchers in CA still often argue that talk remains the most important mode, while social semiotics and mediated discourse analysis dispute this. Multimodal (inter)action analysis builds upon mediated discourse analysis but is the only approach designed specifically for the study of multimodal interaction and multimodal action. The ongoing development of frameworks within multimodal discourse analysis is indicative of intellectual diversity. For some scholars, this is a source of concern, and they attempt to unify multimodality, while for others it is desirable, especially for a research domain that offers applications to so many areas of social life. This bibliography explores central texts under the umbrella of multimodal discourse analysis in five domains: mediated discourse analysis, multimodal (inter)action analysis, social semiotics, systemic functional multimodal discourse analysis (SF-MDA), and conversation analytic style multimodality. A sixth area focusing on introductory texts begins the bibliography.