Haptic Signal Reconstruction for Cross-Modal Communications

Abstract

The emerging multi-modal services, characterized as the integration of audio, visual, and haptic signals, will become the killer applications in 5 G and beyond 5 G era. In order to support multi-modal services, cross-modal communications come into being. However, when adopting cross-modal communications to haptic-dominant multi-modal services, there still face several technical challenges. On the one hand, haptic signals are very sensitive to interference and easy to be damaged or even missing during transmission. On the other hand, it needs to generate virtual haptic signals when real touch sensory information is hard to be gathered. To get over the dilemma, this paper proposes a haptic signal reconstruction strategy for cross-modal communications. First, a cloud-edge collaboration-based cross-modal communication architecture is constructed. Then, an audio-visual-aided haptic signal reconstruction (AVHR) approach under this architecture is designed by leveraging the potential correlation among modalities. It can be further divided into three components: feature extraction by cloud-edge transfer, shared semantic learning by multi-modal fusion, and haptic signal generation by semantic constraints. Finally, experiments on a standard audio-visual-haptic dataset and a practical cross-modal communication platform show that the proposed AVHR approach has better reconstruction performance when compared with the competing schemes.