Multiple Description Coding and Recovery of Free Viewpoint Video for Wireless Multi-Path Streaming

Abstract

By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR from the received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our system can outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.