Multisensory integration in audiovisual translation

Abstract

Multisensory integration is the process by which information from two or more sensory modalities is perceptually bound to form a coherent and complete internal cognitive representation, thus enabling the perceiver to experience rich multimodal perception in relative real time and to adapt to a changing external environment. Compared to unimodal and bimodal inputs, multimodal information has been consistently shown to result in more efficient processing and improved outcomes in a multitude of language-processing tasks because of the multimodal interaction of multiple sensory inputs, providing information that would otherwise be unavailable in any one modality in isolation. Research on multisensory integration has expanded considerably in recent years to focus on identifying the factors that determine whether sensory information is integrated and the resultant effects on cognition, affect, and behavior. This chapter identifies how this work is directly relevant to audiovisual translation: The examination of the factors that lead to effective and ineffective integration of static and dynamic sensory information and the resultant effects on the diverse viewership of subtitled and captioned media. In doing so, it provides a contemporary critical review of the body of empirical research on multisensory integration in order to explicitly demonstrate how this knowledge and evidence base can further develop cognitive translation studies and lead to improvements in both the process and products of audiovisual translation.