Basics of stereoscopic video, new compression results with MPEG-2 and a proposal for MPEG-4

Abstract

Abstract Many current as well as emerging applications in areas of entertainment, remote operations, manufacturing industry and medicine can benefit from the depth perception offered by stereoscopic video systems which employ two views of a scene imaged under the constraints imposed by the human visual system. Among the many challenges to be overcome for practical realization and widespread use of 3D/stereoscopic systems are good systems for 3D video capture, display and efficient techniques for digital compression of enormous amounts of data while maintaining compatibility with normal video decoding and display systems. After a brief introduction to the basics of 3D/stereo including issues of depth perception, imaging and display, we present a brief overview of portions of the MPEG-2 video standard that are relevant to our discussion on compression of stereoscopic video. Next, we outline the various approaches for compression of stereoscopic video and then focus on compatible stereoscopic video coding using MPEG-2 Temporal scalability concepts. Compatible coding employing two different types of prediction structures become potentially possible, disparity compensated prediction and combined disparity and motion compensated predictions. To further improve coding performance and display quality, gain and offset preprocessing for reducing mismatch between the two views forming stereoscopic video is considered. We then introduce the various considerations in coding of stereoscopic video at lower bit-rates for the ongoing MPEG-4 standard. A method is proposed that builds on the proven framework of MPEG-2 like coding but introduces additional coding flexibilities to achieve reasonable performance at lower bit-rates for MPEG-4. Next, results of experiments are presented for a variety of combinations of MPEG-2 based coding methods for the left and the right views while employing TV resolution video for a number of sequences and for various bit-rates. The combined disparity and motion-compensated prediction is found to offer the best performance among combinations tested. These results indicate that compression of both views of stereoscopic video of normal TV resolution appears feasible with good quality in a total of 6–8 Mbit/s. Further, results are presented at much lower bit-rates based on the coding method proposed for MPEG-4 on two long test sequences. We then discuss multi-viewpoint video applications, the ongoing efforts towards a multi-viewpoint profile in MPEG-2 and expected direction of multi-viewpoint video coding in MPEG-4.