Synthesize Missing Modality Based on Latent Space Model

Abstract

Multi-modal learning is currently a research hotspot in the field of artificial intelligence. Multi-modal learning effectively improves learning performance since it comprehensively utilizes information from multiple modalities. However, in real-world application scenarios, it is often impossible to collect complete multi-modal data, which limits the wide application of multi-modal learning. High-quality missing modality synthesis is still a challenge. In this work, we propose a novel method to synthesize the missing modality. Specifically, we utilize the common latent representation space model to adaptively fuse the consistent and complementary information in existing modalities, and then the synthesis network with 1d-CNN layers and MLP is employed to synthesize the missing modality. In addition, “threshold-loss” is proposed to tackle the over-optimizing phenomenon during the testing stage. Experiments demonstrate the proposed method outperforms other existing methods.