Saving Bits Using Multi-Sensor Collaboration

Abstract

In this paper, we propose a new video coding method that saves bits using multi-sensor collaboration. Traditional video coding methods have saved bits by removing redundancy in videos. Recently, multiple types of sensors are being deployed to many solutions and multi-sensor data have significant advantages over single sensor data. The proposed method suggests a new way of video compression that saves bits using multi-sensor collaboration. We apply multi-sensor collaboration to the 3D video coding based on color and depth sensors. Based on the correlation between color and depth images, we design two networks CNN-US and CNN-QE in the proposed video coding method to achieve up-sampling and quality enhancement, respectively. The proposed method combines CNN-US and CNN-QE with 3D-HEVC to save bits using multi-sensor collaboration. Compared with 3D-HEVC anchor, the proposed method achieves average 5.9%, 66.8%, and 71.0% BD-rate reductions for sampling factors 1, 2, and 4 on the depth videos of 3D-HEVC test dataset, respectively.