Foundations of Multimodal Co-learning

Abstract

In the current state of the field of machine learning, often, real-world phenomena are learned through studies of isolated modalities; such as modeling language exclusively from verbal modality, which is a common theme in natural language processing. This is widely adopted since downstream tasksin different disciplines of machine learning are also often similarly isolated and unimodal. In sharp contrast to this, human learning from real-world experiences is rarely unimodal, and often exhibits a multisensory nature, regardless of any assumptions about downstream tasks. The cognitive constructs in human brain are consistently developed through multisensory reinforcement, and the same constructs generalize to unimodal scenarios. The difference between the trend of unimodal learning and human cognitive development raises the following question: “Even if downstream tasks are unimodal during test time, is it better to learn from the isolated modality or from multimodal information?”. In this paper we focus on an in-depth study of this research question. We study the differences between unimodal learning and Multimodal Co-learning (MCl), both from empirical and theoretical standpoints. Through the lens of information entropy and characteristics of deep neural networks, we demonstrate strong theoretical justifications in favor of MCl.