Chapter 1 - Introduction

Abstract

This chapter provides an introduction to the book Multimodal Signal Processing. A multimodal system can be defined as the one that supports communication through different modalities or types of communication channels. In general, multimodal systems are also considered to use concurrent processing and to perform fusion of multiple, possibly asynchronous, input streams. For example, a framework for multimodal human–computer interfaces can be described as using a combination of modes, channels, media, and styles. Although multimodality has been discussed in research and science for several years, the computer science community is however, just beginning to understand how to design well-integrated and robust multimodal systems. The proposed area of study is vast, covering disciplines such as computer science, engineering, linguistics, cognitive sciences, human–computer interfaces, and psychology. This book is thus a survey of the state of the art in a large area of topics, from video, speech, and language processing to multimodal signal processing, human–computer interaction (HCI) and human–human interaction modeling. The applications of signal processing and statistical machine learning techniques to problems arising in these fields are the two major themes of the book.