Cross-Modal Transmission Strategy

Abstract

Multi-modal services, typically integrated by visual, audio, and haptic signals, have been considered as promising killer services in 5G and beyond 5G era. However, due to essential difference among these signals, how to guarantee quality of multi-modal services is a significant technical challenge. Existing transmission schemes, which deliver and process each modality signal separately, cannot meet such requirements as low latency, high reliability, and high throughput. To get over the dilemma, this paper proposes a general cross-modal transmission strategy by taking advantage of the potential correlation among modalities, which consists of a delivery mechanism at the sender and a signal restoration procedure at the receiver. On the one hand, a visual-aided haptic content compression method is designed for the delivery mechanism. By utilizing the category correlation among modalities, haptic signals with similar visual content can be effectively compressed, reducing transmission burden. On the other hand, a fine-grained haptic to image synthesis (FHIS) approach is proposed for realizing signal restoration. Through exploring strong matching properties among modalities, the FHIS can restore the impaired, missing, delayed visual images from the received haptic signals. Experiments on a standard visual-haptic database and a practical platform evaluate the performance of the constructed cross-modal transmission strategy.