3D Video Applications Research Articles

Latest developments of dynamic holographic three-dimensional display

Holographic three-dimensional (3D) display is a true 3D display technique, which can provide realistic image of a real object or a scene because holography has the ability to reconstruct both the intensity and phase information, i.e., the wave front of the object or scene. Therefore, it could allow the observers to perceive the light as it is scattered by the real object itself without any special eyewear, which is quite different from other 3D display techniques, such as stereoscopic displays and volumetric 3D displays. In this paper, the achievements and developments of the latest new holographic 3D displays are presented. Holographic 3D displays can be divided into static holographic 3D displays and dynamic holographic 3D displays. Here, we briefly introduce the principle of holographic 3D display technique and static holographic 3D displays, and focus on dynamic holographic 3D displays. Large-size, high-resolution and color static holographic 3D displays have already been successfully fabricated and applied in some areas, such as holographic 3D maps and holographic 3D images. However, dynamic holographic 3D displays based on both optical materials and spatial light modulators (SLMs) are still under research, which is a challenge to their applications. Some holographic researchers study the holographic 3D displays based on the SLMs for large-size and large view angle display, but it is difficult to realize them because of limitations of SLMs and there still needs much effort to solve these problems in SLMs. Other holographic researchers work on dynamic holographic materials, such as inorganic crystals, photorefractive polymer, photochromic material etc. The response time and diffraction efficiency are key factors to these materials. Compared with other holographic media, liquid crystals with super-fast response time (about 1 ms) have been reported, which makes it possible to realize video refresh-rate holographic displays. The achievements of dynamic holography, which are helpful for holographic 3D video applications, are presented. Recently, real-time dynamic holographic display has been obtained in super-fast response liquid crystal films, which makes it possible that large-size, high-definition, color holographic 3D video displayers are developed by using these liquid crystal films in the future.

Objective image quality assessment of 3D synthesized views

Depth-Image-Based-Rendering (DIBR) techniques are essential for three-dimensional (3D) video applications such as 3D Television (3DTV) and Free-Viewpoint Video. However, this process is based on 3D warping and can induce serious distortions whose impact on the perceived quality is far different from the one experienced in the 2D imaging processes. Since quality evaluation of DIBR-synthesized views is fundamental for the design of perceptually friendly 3D video systems, an appropriate objective quality metric targeting the assessment of DIBR-synthesized views is momentous. Most of the 2D objective quality metrics fail in assessing the visual quality of DIBR-synthesized views because they have not been conceived for addressing the specificities of DIBR-related distortions. In this paper, a new full-reference objective quality metric, 3DSwIM (3D Synthesized view Image Quality Metric), dedicated to artifacts detection in DIBR-synthesized view-points is presented. The proposed scheme relies on a comparison of statistical features of wavelet subbands of two input images: the original image and the DIBR-based synthesized image. A registration step is included before the comparison step so that best matching blocks are always compared to ensure “shifting-resilience”. In addition, a skin detection step weights the final quality score in order to penalize distorted blocks containing “skin-pixels” based on the assumption that a human observer is most sensitive to impairments affecting human subjects. Experimental tests show that the proposed method outperforms the conventional 2D and DIBR-dedicated quality metrics under test.

A novel depth image enhancement method based on the linear surface model

In three-dimensional (3D) video applications, structured-light RGB-D cameras are commonly used to capture depth images that convey the per-pixel depth information in a scene. However, these cameras often produce regions with missing pixels (MPs). These regions, referred to as holes, will not contain no any depth information for the captured depth image. In this paper, a novel depth image enhancement method that accurately estimates depth values of MPs is presented. In the proposed method, the neighboring region outside the hole is first segmented into superpixels using simple linear iterative clustering. Subsequently, the depth value trend of each superpixel is modeled as a linear surface. Finally, one of the linear surfaces is selected using a proposed metric, to estimate the depth value of a particular MP in the hole. Experimental results demonstrate that the proposed method provides superior performance, especially around the object boundary, compared with other state-of-theart depth image enhancement methods.

Gradient-domain-based enhancement of multi-view depth video

Multi-view depth is an emerging and attractive 3D representation in recent years, which acts a significant role in rendering numerous viewing angles from a small number of given input views. The performance of those depth-based 3D video applications is strongly dependent on the quality of multi-view depth. However, because of the limitations of depth acquisition and estimation, the quality of multi-view depth suffers from artifacts in spatial dimension and inconsistency in both the temporal and inter-view dimensions. In this paper, we propose a gradient-domain based enhancement method for multi-view depth. Being different from the traditional enhancement methods which intended to process one or two above dimensions, our proposal exploits the coherence of both temporal and inter-view dimensions in addition of the spatial one. It is very challenging to obtain the stable characteristics in multi-dimensions. To solve this problem, we propose to investigate the characteristics in the gradient domain rather than the intensity domain. The enhanced multi-view depth is obtained through the minimization of energy function under the constraint of a joint gradient field (JGF), which is estimated from multiple dimensions through motion estimation and geometric mapping. Therefore, the enhanced multi-view depth is a global optimization in multiple dimensions that ensures the consistency in temporal, inter-view and spatial domains. Furthermore, we also propose an enhancement structure to indicate the process order of depth frames. The experimental results suggest that the proposed method can enhance multi-view depth with desired sharpness and consistency.

Post-processing of 3D Video Extension of H.264/AVC for a Quality Enhancement of Synthesized View Sequences

Since July of 2012, the 3D video extension of H.264/AVC has been under development to support the multi-view video plus depth format. In 3D video applications such as multi-view and free-view point applications, synthesized views are generated using coded texture video and coded depth video. Such synthesized views can be distorted by quantization noise and inaccuracy of 3D wrapping positions, thus it is important to improve their quality where possible. To achieve this, the relationship among the depth video, texture video, and synthesized view is investigated herein. Based on this investigation, an edge noise suppression filtering process to preserve the edges of the depth video and a method based on a total variation approach to maximum a posteriori probability estimates for reducing the quantization noise of the coded texture video. The experiment results show that the proposed methods improve the peak signal-to-noise ratio and visual quality of a synthesized view compared to a synthesized view without post processing methods.

A no reference depth perception assessment metric for 3D video

Recent technological breakthroughs in 3-Dimensional (3D) video capture, display, coding, transmission, rendering, etc. have led the advances of 3D multimedia applications into the consumer market. However, the effect of these technologies on 3D video Quality of Experience (QoE) has not been thoroughly investigated to speed up the wide-spread proliferation of the 3D video applications in this market. Quality and depth perception assessment of 3D video from the view of end users reflects the most important aspect of 3D video QoE. Therefore, evaluating quality and depth perception of 3D video should be given the utmost attention. Currently, the depth perception assessment of 3D video can only be achieved using time consuming and rigorous subjective assessments due to the lack of reliable and efficient objective metrics. Assessing the depth perception using Full-Reference (FR)/Reduced Reference (RR) objective metrics is not efficient for on the fly 3D video applications due to the requirement of original video/extracted information at the receiver side. Thus, a No Reference (NR) metric, which does not need any original video related information at the receiver side to predict the depth perception, is proposed in this paper. Three important cues (i.e., binocular parallax, lateral motion, and aerial perspective) for Human Visual System (HVS) to perceive the depth of a 3D video are utilized to develop the NR metric. Experimental results devised using the proposed metric prove the effectiveness of it to predict the depth perception.

Graph-Based Authentication Design for Color-Depth-Based 3D Video Transmission over Wireless Networks

3D video applications such as 3D-TV and 3D games have become more and more popular in recent years. These applications raised significant challenges in the media security, processing and transmissions. Especially, when 3D videos are delivered over wireless networks, the video streaming suffers the potential malicious attacks. One of the most important security challenging issues is how to guarantee the integrity of media content over error-prone wireless networks. To address this challenge, in the paper, we for the first time propose an authentication approach for 3D video transmission over wireless networks, which can improve the reconstructed media quality under error-prone wireless environment with lower authentication overheads and energy consumption. The proposed method is based on color-depth 3D video coding approach, which can save bandwidth, be tolerable to packet losses and thus satisfy the users' Quality of Experience (QoE) requirements. Our major contribution in this paper includes: (1) designing a joint source-channel-authentication coding framework for color-depth-based 3D video transmission; (2) proposing a media quality prediction model for color-depth-based 3D video transmission; (3) developing optimization for graph-based authentication on 3D video transmission to improve reconstructed media quality, reduce authentication overheads and energy consumption. Experimental results demonstrated the effectiveness of our proposed solutions.

Rate-distortion based reference viewpoints selection for multi-view video plus depth coding

For multi-view video plus depth (MVD) coding in 3D video (3DV) applications, texture videos and depth maps of a few reference viewpoints should be compressed and transmitted. The other intermediate virtual viewpoints are synthesized by depth-image-based rendering algorithm. The videos displayed for terminal users include the decoded reference viewpoints and the synthesized virtual viewpoints. Under a given bit rate constraint, how to choose the optimal reference viewpoints setting to obtain the best video quality of all display viewpoints is an important research issue. This paper proposes a rate distortion model based algorithm to obtain the optimal reference viewpoints selection. First, the impacts of the reference viewpoints setting on the entire quality of all display viewpoints is investigated, and then a quadratic distortion model for all display viewpoints is derived. Based on this distortion model, the optimal viewpoints selection problem is solved by computing a few model parameters. Experimental results demonstrate that the proposed distortion model estimates the actual distortion of all display viewpoints with high accuracy, and the proposed reference viewpoints selection method achieves almost the same rate distortion performance as compared to the full search method with a very low computational cost.

Quality of experience for 3D video streaming

New 3D video applications and services are emerging to fulfill increasing user demand. This effort is well supported by the increasing 3D video content including user generated content (e.g., through 3D capture/display enabled mobile phones), technological advancements (e.g., HD 3D video capture and processing methods), affordable 3D displays, and standardization activities. However, not much attention has been given to how these technologies, along the end-to-end chain from content capture to display, affect user perception and whether the overall experience of 3D video users is satisfactory or not. 3D video streaming also introduces artifacts on the reconstructed 3D video at the receiver end, leading to inferior quality and user experience. In this article we present and discuss in detail how artifacts introduced during 3D video streaming affect the end-user perception and how we could use realtime quality evaluation methodologies to overcome these effects. The observations presented can underpin the design of future QoE-aware 3D video streaming systems.

A study of depth/texture bit-rate allocation in multi-view video plus depth compression

Multi-view video plus depth (MVD) data offer a reliable representation of three-dimensional (3D) scenes for 3D video applications. This is a huge amount of data whose compression is an important challenge for researchers at the current time. Consisting of texture and depth video sequences, the question of the relationship between these two types of data regarding bit-rate allocation often raises. This paper questions the required ratio between texture and depth when encoding MVD data. In particular, the paper investigates the elements impacting on the best bit-rate ratio between depth and color: total bit-rate budget, input data features, encoding strategy, and assessed view.

Texture-adaptive hole-filling algorithm in raster-order for three-dimensional video applications

The technique of image rendering plays a vital role in many 3D applications, such as free view-point television, 2D-to-3D video conversion, etc. Depth image based rendering (DIBR) is considered as an effective way for image rendering. One challenge of DIBR is how to deal with the holes appeared in the procedure of rendering. In this paper, we propose a texture-adaptive hole-filling algorithm for post-processing of rendered image on 3D video applications. Especially, the raster-order is fully considered for hardware design and flexible interaction. The algorithm first determines the type of holes, and then fills the missed pixels in raster-order. The filling method depends on the hole types and texture gradient of neighbors with simple data operation, which benefits for fast processing. Experimental results show that the proposed algorithm outperforms other benchmarks on perceptual quality with considerable savings of computational cost.

30.4: Trilateral Filter for Depth Map Interpolation in 3D Video

AbstractDepth maps are used in many 3D video applications. Often, depth map is available at a lower resolution compared to the corresponding color images. To match the resolution, depth maps must be scaled up to the color image resolution. In this paper, we present an interpolation method of depth maps. The proposed idea is based on the observation that the depth map is highly correlated to the color image and the big correlation offers the reliable knowledge. For this purpose, we develop a trilateral filter that interpolates the low resolution depth map by the temporary high resolution depth map and the high resolution color image. Accordingly, evaluation using the MPEG 3D videos shows that significant improvement of depth quality and rendered view quality.

An effective error resilient 3D view synthesis method

View synthesis is a crucial process in current 3D video applications. Currently, the existing view synthesis techniques may introduce visual artifacts such as corona, pinholes and ghosts into pictures, which degrade the visual experience greatly. In this paper, we will introduce an error resilient 3D view synthesis approach, which is able to effectively remove these artifacts. Specifically, we first detect the regions mixed with foreground and background pixels to avoid corona artifacts. Then, we resize images and conduct projection on the resized images to reduce pinhole artifacts. Finally, an improved view blending algorithm is proposed to reduce ghosting artifacts. Simulation results demonstrate that our proposed method outperforms others significantly in removing view artifacts.

Application-adapted mobile 3D video coding and streaming — A survey

3D video technologies have been gradually matured to be moved into mobile platforms. In the mobile environments, the specific characteristics of wireless network and mobile device present great challenges for 3D video coding and streaming. The application-adapted mobile 3D video coding and streaming technologies are urgently needed. Based on the mobile 3D video application framework, this paper reviews the state-of-the-art technologies of mobile 3D video coding and streaming. Specifically, the mobile 3D video formats and the corresponding coding methods are firstly reviewed and then the streaming adaptation technologies including 3D video transcoding, 3D video rate control and cross-layer optimized 3D video streaming are surveyed.

New Depth Coding Techniques With Utilization of Corresponding Video

With the development of 3D display and interactive multimedia systems, new 3D video applications, such as 3DTV and Free Viewpoint Video, are attracting significant interests. In order to enable these new applications, new data formats including captured 2D video sequences and corresponding depth maps have been proposed. Compared to conventional video frames, depth maps have very different characteristics. First, they typically consist of homogeneous areas partitioned by sharp edges representing depth discontinuities, while the sharp discontinuities play very important roles in view rendering. Second, there exists structure similarity between depth map and corresponding video, in which the edges in depth exhibit quite similar behaviors as the edges in the corresponding video. In conventional video coding techniques with transforms followed by quantization, there usually exist large artifacts along sharp edges and it costs significant more bits to represent the edges with higher accuracy. In order to suppress the coding artifacts while preserving edges, and to better compress depth information, we propose in this paper two new depth compression techniques: Trilateral Filter and Sparse Dyadic Mode. Both techniques utilize the structure similarity between depth and corresponding video while focusing on different aspects in depth compression. As a new in-loop filter, Trilateral Filter is designed to filter depth with coding artifacts based on the proximity of pixel positions, the similarity among depth samples, as well as the similarity among the collocated pixels in the video frame. While Sparse Dyadic Mode is used as an intra mode to reconstruct depth map with sparse representations of depth blocks and effective reference of edge information from video frames. With these two new coding tools, we can achieve up to about 1.5 dB gain on rendering quality as compared to depth sequences coded using MVC under the same coding rate.

A Depth Information Based Fast Mode Decision Algorithm for Color Plus Depth-Map 3D Videos

With the advance of 3D display and related technologies, 3D video applications have been booming up recently. The color plus depth-map 3D video gains the attention for its relatively low storage and transmission bandwidth requirements as compared with stereo and multi-view videos. The color plus depth-map 3D video provides the depth information about the recorded content which brings new rooms for video processing and compression. This work reveals the usage of depth information for speeding up video encoding process. A novel fast mode decision algorithm based on depth information is proposed. The algorithm is also applicable to videos in which the associated depth-maps are not the ground-truth. By utilizing the depth information, we can classify scenes according to the associated depth and assign the appropriate modes. We can also deduce the configuration of the given macroblock by the associated depth information and select the suitable mode candidates; therefore, the encoding process will be speeded-up accordingly. Experimental results show that the proposed algorithm is faster than existing mode decision approaches with similar coding efficiency. The entire encoding time is reduced by about 63% for Standard Definition (SD) and High Definition (HD) formats while the Rate-Distortion (RD) performance is almost the same as compared with the exhaustive full mode decision approach.

A joint encoder–decoder error control framework for stereoscopic video coding

Stereoscopic video coding (SSVC) plays an important role in various 3D video applications. In SSVC, robust stereoscopic video transmission over error-prone networks is still a challenge problem to be solved. In this paper, we propose a joint encoder–decoder error control framework for SSVC, where error-resilient source coding, transmission network conditions, and error concealment scheme are jointly considered to achieve better error robustness performance. The proposed joint encoder–decoder error control framework includes two parts: an error concealment algorithm at the decoder side and a rate–distortion optimized error resilience algorithm at the encoder side. For error concealment at the decoder side, an overlapped block motion and disparity compensation based error concealment scheme is proposed to adaptively utilize inter-view correlations and temporal correlations. For error resilience at the encoder side, first, the inter-view refreshment is proposed for SSVC to suppress error propagations. Then, an end-to-end distortion model for SSVC is derived, which jointly considers the transmission network conditions, inter-view refreshment, and error concealment tools at the decoder side. Finally, based on the derived end-to-end distortion model, the rate–distortion optimized error resilience algorithm is presented to adaptively select inter-view, inter- or intra-coding for SSVC. The experimental results show that the proposed joint encoder–decoder error control framework has superior error robustness performance for stereoscopic video transmission over error-prone networks.

A Scalable Multiple Description Scheme for 3D Video Coding Based on the Interlayer Prediction Structure

The most recent literature indicates multiple description coding (MDC) as a promising coding approach to handle the problem of video transmission over unreliable networks with different quality and bandwidth constraints. Furthermore, following recent commercial availability of autostereoscopic 3D displays that allow 3D visual data to be viewed without the use of special headgear or glasses, it is anticipated that the applications of 3D video will increase rapidly in the near future. Moving from the concept of spatial MDC, in this paper we introduce some efficient algorithms to obtain 3D substreams that also exploit some form of scalability. These algorithms are then applied to both coded stereo sequences and to depth image-based rendering (DIBR). In these algorithms, we first generate four 3D subsequences by subsampling, and then two of these subsequences are jointly used to form each of the two descriptions. For each description, one of the original subsequences is predicted from the other one via some scalable algorithms, focusing on the inter layer prediction scheme. The proposed algorithms can be implemented as pre- and postprocessing of the standard H.264/SVC coder that remains fully compatible with any standard coder. The experimental results presented show that these algorithms provide excellent results.

Object based coding of the depth maps for 3D video coding

The emergence of three dimensional (3D) video applications, based on Depth Image Based Rendering (DIBR) has brought up more requirements of bandwidth, due to the need of depth information. This additional bandwidth requirement need to be tackled to enable the widespread of 3D video applications based on DIBR. Exploiting visual correlations between the color image and the depth image, in depth image coding, will reduce the requirement of high bandwidth required to transmit the additional depth information. In this paper, an object based depth image coding technique is presented which is suitable for low bit rate 3D-TV applications that are based on Depth Image Based Rendering. The proposed method achieves at most 50% bit rate reduction at low bit rates.

A Free-Viewpoint Video Renderer

Free-viewpoint video allows interactive control of the camera viewpoint in video playback using a three-dimensional (3D) scene representation. This paper describes the challenges and the state-of-the-art techniques adopted in developing a free-viewpoint renderer for 3D video production of people. The system requires video streams from a set of fixed or dynamic real-world video cameras and synthesizes arbitrary viewpoints on the GPU using a dynamic 3D scene model. We have released the renderer as an open source application for the research community (see the web information at the end of this paper). The renderer provides (i) the facility to simulate camera viewpoints from publicly available multiple-view video datasets and (ii) a baseline technique for free-viewpoint video synthesis in the development of interactive 3D video and 3DTV applications.