Cross-modal communication technology: A survey

Abstract

In the 5G era and beyond, multi-modal services that integrate audio, visual, and haptic signals are expected to become dominant applications. To support multi-modal services, the concept of cross-modal communications, which involves collaborative audio-visual and haptic interactions, has emerged. Despite significant research about cross-modal communication technology being conducted, a comprehensive literature review on this topic is lacking. To fill this gap, this paper presents a detailed survey on cross-modal communication technology. First, it provides a highly summarized description of representative research attempts in audio-visual and haptic communications, which serve as the foundation for cross-modal communications. Then, it delves into various aspects of cross-modal communications, including architectural, cross-modal coding, cross-modal transmission, cross-modal signal reconstruction, the essence of semantics, and prototype systems. Finally, it discusses conclusions and future research directions. This paper is expected to promote the theoretical research and practical applications of cross-modal communications.