LDV Generation from Multi-View Hybrid Image and Depth Video

Abstract

The technology around 3D-TV is evolving rapidly. There are already different stereo displays available and auto-stereoscopic displays promise 3D without glasses in the near future. All of the commercially available content today is purely image-based. Depth-based content on the other hand provides better flexibility and scalability regarding future 3D-TV requirements and in the long term is considered to be a better alternative for 3D-TV production. However, depth estimation is a difficult process, which threatens to become the main bottleneck in the whole production chain. There are already different sophisticated depth-based formats such as LDV (layered depth video) or MVD (multi-view video plus depth) available, but no reliable production techniques for these formats exist today. Usually camera systems, consisting of multiple color cameras, are used for capturing. These systems however rely on stereo matching for depth estimation, which often fails in presence of repetitive patterns or textureless regions. Newer, hybrid systems offer a better alternative here. Hybrid systems incorporate active sensors in the depth estimation process and allow to overcome difficulties of the standard multi-camera systems. In this chapter a complete production chain for 2-layer LDV format, based on a hybrid camera system of 5 color cameras and 2 time-of-flight cameras, is presented. It includes real-time preview capabilities for quality control during the shooting and post-production algorithms to generate high-quality LDV content consisting of foreground and occlusion layers.