Depth Image Based Rendering Research Articles

Combining Local and Global Measures for DIBR-Synthesized Image Quality Evaluation.

Depth-Image-Based-Rendering (DIBR) techniques are significant for three-dimensional (3D) video applications, e.g., 3D television and free viewpoint video (FVV). Unfortunately, the DIBR-synthesized image suffers from various distortions, which induce an annoying viewing experience for the entire FVV. Proposing a quality evaluator for DIBR-synthesized images is fundamental for the design of perceptual friendly FVV systems. Since the associated reference image is usually not accessible, full-reference (FR) methods cannot be directly applied for quality evaluation of the synthesized image. In addition, most traditional no-reference (NR) methods fail to effectively measure the specifically DIBR-related distortions. In this paper, we propose a novel NR quality evaluation method accounting for two categories of DIBR-related distortions, i.e., geometric distortions and sharpness. First, the disoccluded regions, as one of the most obvious geometric distortions, are captured by analyzing local similarity. Then, another typical geometric distortion (i.e., stretching) is detected and measured by calculating the similarity between it and its equal-size adjacent region. Second, considering the property of scale invariance, the global sharpness is measured as the distance between the distorted image and its downsampled version. Finally, the perceptual quality is estimated by linearly pooling the scores of two geometric distortions and sharpness together. Experimental results verify the superiority of the proposed method over the prevailing FR and NR metrics. More specifically, it is superior to all competing methods except APT in terms of effectiveness, but greatly outmatches APT in terms of implementation time.

Hole Filling With Multiple Reference Views in DIBR View Synthesis

Depth-image-based rendering (DIBR) oriented view synthesis has been widely employed in the current depth-based 3D video systems by synthesizing a virtual view from an arbitrary viewpoint. However, holes may appear in the synthesized view due to disocclusion, thus significantly degrading the quality. Consequently, efforts have been made on developing effective and efficient hole filling algorithms. Current hole filling techniques generally extrapolate/interpolate the hole regions with the neighboring information based on an assumption that the texture pattern in the holes is similar to that of the neighboring background information. However, in many scenarios especially of complex texture, the assumption may not hold. In other words, hole filling techniques can only provide an estimation for a hole which may not be good enough or may even be erroneous considering a wide variety of complex scene of images. In this paper, we first examine the view interpolation with multiple reference views, demonstrating that the problem of emerging holes in a target virtual view can be greatly alleviated by making good use of other neighboring complementary views in addition to its two (commonly used) most neighboring primary views. The effects of using multiple views for view extrapolation in reducing holes are also investigated in this paper. In view of the 3D Video and ongoing free-viewpoint TV standardization, we propose a new view synthesis framework which employs multiple views to synthesize output virtual views. Furthermore, a scheme of selective warping of complementary views is developed by efficiently locating a small number of useful pixels in the complementary views for hole reduction, to avoid a full warping of additional complementary views thus lowering greatly the warping complexity.

Robust Template-Based Watermarking for DIBR 3D Images

Several depth image-based rendering (DIBR) watermarking methods have been proposed, but they have various drawbacks, such as non-blindness, low imperceptibility and vulnerability to signal or geometric distortion. This paper proposes a template-based DIBR watermarking method that overcomes the drawbacks of previous methods. The proposed method exploits two properties to resist DIBR attacks: the pixel is only moved horizontally by DIBR, and the smaller block is not distorted by DIBR. The one-dimensional (1D) discrete cosine transform (DCT) and curvelet domains are adopted to utilize these two properties. A template is inserted in the curvelet domain to restore the synchronization error caused by geometric distortion. A watermark is inserted in the 1D DCT domain to insert and detect a message from the DIBR image. Experimental results of the proposed method show high imperceptibility and robustness to various attacks, such as signal and geometric distortions. The proposed method is also robust to DIBR distortion and DIBR configuration adjustment, such as depth image preprocessing and baseline distance adjustment.

No-reference quality assessment of DIBR-synthesized videos by measuring temporal flickering

Depth-image-based-rendering (DIBR) is the most popular view synthesis method. The rendering process in DIBR introduces artifacts to synthesized views. Temporal inconsistency of these artifacts causes flickering, which is one of the dominant distortions in DIBR-synthesized videos. We propose a no-reference quality index for DIBR-synthesized videos. A two-stage flickering region detection method is first proposed by calculating the gradient distance between the matching blocks in adjacent frames followed by a refinement operation. Subsequently, the distortion intensity of flickering regions is measured in singular value decomposition domain. The average flickering intensity is computed as the quality score. Experimental results show that our method outperforms the state-of-the-arts on the IRCCyN/IVC database and ranks the second on the SIAT database, which is only slightly worse than the best-performing one. Furthermore, the presented method is applied to improve the performances of existing metrics and benchmark DIBR algorithms, which achieves very promising results for both tasks.

Elemental image array generation method by using optimized depth image‐based rendering algorithm for integral imaging display

AbstractWe propose an elemental image array (EIA) generation method by using an optimized depth image‐based rendering (DIBR) algorithm. In this method, the EIA is synthesized by the reference and virtual viewpoint elemental images, and the virtual viewpoint elemental images at the given locations are generated by DIBR algorithm. We optimize the existing DIBR algorithm by adaptively repairing the warped depth images in the processing part and extend the generation dimension of the virtual viewpoint elemental images from one dimension to two dimensions. The optimized DIBR algorithm can effectively solve the problem: the low quality of virtual viewpoint elemental images caused by discontinuous depth values and disocclusion regions. We also implement the generations of virtual viewpoint elemental images and EIA in graph processing unit to reduce the time cost. Experimental results show that the proposed method can not only improve the quality of the virtual viewpoint images but also accelerate the generations of the virtual viewpoint elemental images and EIA.

Reliability-Based View Synthesis for Free Viewpoint Video

View synthesis is a crucial technique for free viewpoint video and multi-view video coding because of its capability to render an unlimited number of virtual viewpoints from adjacent captured texture images and corresponding depth maps. The accuracy of depth maps is very important to the rendering quality, since depth image–based rendering (DIBR) is the most widely used technology among synthesis algorithms. There are some issues due to the fact that stereo depth estimation is error-prone. In addition, filling occlusions is another challenge in producing desirable synthesized images. In this paper, we propose a reliability-based view synthesis framework. A depth refinement method is used to check the reliability of depth values and refine some of the unreliable pixels, and an adaptive background modeling algorithm is utilized to construct a background image aiming to fill the remaining empty regions after a proposed weighted blending process. Finally, the proposed approach is implemented and tested on test video sequences, and experimental results indicate objective and subjective improvements compared to previous view synthesis methods.

Real-time ultra-high definition multiview glasses-free 3D display system

The design and implementation of a real-time ultra-high definition (UHD) multiview glasses-free 3D display system always suffer from the high transmission bandwidth, high memory cost, and high computational complexity. This paper presents a glasses-free 3D display system based on depth-image-based rendering (DIBR) technique by solving these problems. It can convert a V $$+$$ D video (RGBD) stream in real time to a multiview representation suitable for a multiview autostereoscopic display. As V $$+$$ D video format is used for the system, the transmission bandwidth is reduced. For the memory cost, we introduce an asymmetric shift-sensor camera setup to avoid external memory usage and reduce the storage requirement of multiviews. For the computational complexity, as our camera setup ensures that all the virtual views can be generated with the same DIBR algorithm, the computational complexity is reduced. In addition, we simplify the view fusion method so as to rearrange the subpixels from multiviews with lower complexity to form a single glasses-free 3D image. Moreover, we propose a hardware architecture for the system and implement it using field programmable gate array. Simulation results show that our system can support the UHD V $$+$$ D videos for an 8-view glasses-free 3D display. Performance evaluation results show that the proposed system can provide reasonably good stereoscopic image quality if appropriate parameters of the system are applied.

Quality Assessment of DIBR-Synthesized Images by Measuring Local Geometric Distortions and Global Sharpness

Depth-image-based rendering (DIBR) is a fundamental technique in free viewpoint video, which is widely adopted to synthesize virtual viewpoints. The warping and rendering operations in DIBR generally introduce geometric distortions and sharpness change. The state-of-the-art quality indices are limited in dealing with such images since they are sensitive to geometric changes. In this paper, a new quality model for DIBR-synthesized view images is presented by measuring LOcal Geometric distortions in disoccluded regions and global Sharpness (LOGS). A disoccluded region detection method is first proposed using SIFT-flow-based warping. Then, the sizes and distortion strength of local disoccluded regions are combined to generate a score. Furthermore, a reblurring-based strategy is proposed to quantify the global sharpness. Finally, the overall quality score is calculated by pooling the scores of local disoccluded regions and global sharpness. Experiments on four public DIBR-synthesized image/video databases show the superiority of the proposed metric over the state-of-the-art quality models. The proposed method is further adopted for boosting the performances of existing quality metrics and benchmarking DIBR algorithms, both achieving very promising results.

View synthesis for FTV systems based on a minimum spatial distance and correspondence field

The main problems with virtual view synthesis based on the common Depth-Image-Based Rendering (DIBR) algorithms are image rectification, depth map and image de-rectification that lead to additional computational load and image distortion. In this paper an efficient and reliable method based on the concept of Correspondence Field and minimum distance among spatial positions of corresponding pixels is proposed to synthesize virtual view images without image rectification, depth map and image de-rectification steps. Simulated multi-view images are used to evaluate the proposed algorithm. By comparison with DIBR algorithms, simulation results show that on average, PSNR is 4.37 dB (14.8%) higher, SSIM is 0.057 (6.2%) more, UNIQUE is 0.13 (20%) more, the running time is 47.34 s (24.5%) less and wrong pixels are 4.35 (38.5%) less.

A novel method for 2D-to-3D video conversion based on boundary information

This paper proposes a novel method for 2D-to-3D video conversion, based on boundary information to automatically generate the depth map. First, we use the Gaussian model to detect foreground objects and then separate the foreground and background. Second, we employ the superpixel algorithm to find the edge information. According to the superpixels, we will assign corresponding hierarchical depth value to initial depth map. From the result of depth value assignment, we detect the edges by Sobel edge detection with two thresholds to strengthen edge information. To identify the boundary pixels, we use a thinning algorithm to modify edge detection. Following these results, we assign the depth value of foreground to refine it. We use four kinds of scanning path for the entire image to create a more accurate depth map. After that, we have the final depth map. Finally, we utilize depth image-based rendering (DIBR) to synthesize left and right view images. After combining the depth map and the original 2D video, a vivid 3D video is produced.

Object Shape Approximation and Contour Adaptive Depth Image Coding for Virtual View Synthesis

A depth image provides partial geometric information of a 3D scene, namely the shapes of physical objects as observed from a particular viewpoint. This information is important when synthesizing images of different virtual camera viewpoints via depth-image-based rendering (DIBR). It has been shown that depth images can be efficiently coded using contour-adaptive codecs that preserve edge sharpness, resulting in visually pleasing DIBR-synthesized images. However, contours are typically losslessly coded as side information, which is expensive if the object shapes are complex. In this paper, we pursue a new paradigm in depth image coding for color-plus-depth representation of a 3D scene: in a pre-processing step, we pro-actively simplify object shapes in a depth and color image pair to reduce depth coding cost, at a penalty of a slight increase in synthesized view distortion. Specifically, we first mathematically derive a distortion upper-bound proxy for 3DSwIM—a quality metric tailored for DIBR-synthesized images. This proxy reduces inter-dependency among pixel rows in a block to ease optimization. We then approximate object contours via a dynamic programming algorithm to optimally tradeoff coding the cost of contours using arithmetic edge coding with our proposed view synthesis distortion proxy. We modify the depth and color images according to the approximated object contours in an inter-view consistent manner. These are then coded, respectively, using a contour-adaptive image codec based on graph Fourier transform for edge preservation and High Efficiency Video Coding (HEVC) intra. Experimental results show that by maintaining sharp but simplified object contours during contour-adaptive coding, for the same visual quality of DIBR-synthesized virtual views, our proposal can reduce depth image coding rate by up to 22% in 3DSwIM and 42% in peak signal-to-noise ratio compared with alternative coding strategies, such as HEVC intra.

Efficient Multi-View 3D Video Multicast with Depth-Image-Based Rendering in LTE-Advanced Networks with Carrier Aggregation

With the recent emergence of naked-eye 3D mobile devices and various 3D-enabled laptops, service providers now afford the opportunity to provide mobile 3D video streaming in LTE-Advanced networks. Differing from traditional single-view 3D videos, multi-view 3D videos allow users to choose preferred view angles and thus are promising for new applications, such as free-viewpoint television (FTV). Nevertheless, enabling multi-view 3D video services may overwhelm the network resource when transmitting all views of every video. Fortunately, Depth-Image-Based Rendering (DIBR) allows each mobile client to synthesize the desired view from a nearby left view and right view, so that not all views of a video are necessarily transmitted. A new challenge with DIBR, however, is to carefully choose the transmitted views to limit the video distortion and minimize the bandwidth consumption. In this paper, therefore, we first formulate a new optimization problem, called View and MCS Selection (VMS) Problem, to minimize the bandwidth consumption for multi-view 3D video multicast in LTE networks. An algorithm, called View and MCS Aggregation (VMAG) is proposed to find the optimal solution to VMS. For Carrier Aggregation (CA) in LTE-Advanced networks, we formulate a new View, MCS and Carrier Selection (VMCS) Problem and prove that the problem is NP-Hard. We first design a dynamic programming algorithm, called the View Assignment with MCS and Carrier (VAMC) algorithm, to find the optimal solution for small instances. We then propose the View and MCS Aggregation with Carrier (VMAGC) algorithm based on VMAG to effectively find the near-optimal solution to VMCS. The simulation results show that bandwidth consumption can be effectively reduced by over 30 percent in VMS and VMCS.

Hole Filling for View Synthesis Using Depth Guided Global Optimization

View synthesis is an effective way to generate multi-view contents from a limited number of views, and can be utilized for 2-D-to-3-D video conversion, multi-view video compression, and virtual reality. In the view synthesis techniques, depth-image-based rendering (DIBR) is an important method to generate virtual view from video-plus-depth sequence. However, some holes might be produced in the DIBR process. Many hole filling methods have been proposed to tackle this issue, but most of them cannot achieve globally coherent or acquire trusted contents. In this paper, a hole filling method with depth-guided global optimization is proposed for view synthesis. The global optimization is achieved by iterating the spatio-temporal approximate nearest neighbor (ANN) search and video reconstruction step. Directly applying global optimization might introduce some foreground artifacts to the synthesized video. To prevent this problem, some strategies have been developed in this paper. The depth information is applied to guide the spatio-temporal ANN searching and the initialization step is specified in the global optimization procedure. Our experimental results have demonstrated that the proposed method has better performance compared with other methods in terms of visual quality, trusted textures, and temporal consistency in the synthesized video.

Reduced-reference quality assessment of DIBR-synthesized images based on multi-scale edge intensity similarity

Depth-image-based-rendering (DIBR) plays an important role in view synthesis for free-viewpoint videos. The warping process in DIBR causes geometric displacement, which distributes intensively around edges, and the subsequent rendering process results in the impairment of edges. Traditional 2D image quality metrics are limited in the quality evaluation of DIBR-synthesized images. In this paper, we present a reduced-reference quality metric for DIBR-synthesized images by only extracting several feature values, namely multi-scale Edge Intensity Similarity (EIS). The original and synthesized images are first downsampled to generate images with different resolutions. Then an edge detection process is conducted on each scale and the edge intensity is calculated. The similarity of the edge intensity between each downsampled original image and the corresponding synthesized image is computed. Finally, the average similarity is calculated as the quality score of the DIBR-synthesized image. Experiments conducted on IRCCyN/IVC DIBR image and video databases demonstrate that the proposed method overall outperforms traditional 2D and existing DIBR-targeted quality metrics.

Evaluating virtual image quality using the side-views information fusion and depth maps

Three Dimensional (3D) image quality assessment is a challenging problem as compared to 2D images due to their different nature of acquisition, representation, coding, and display. The additional dimension of depth in multiview video plus depth (MVD) format is exploited to obtain images at novel intermediate viewpoints using depth image based rendering (DIBR) techniques, enabling 3D television and free-viewpoint television (FTV) applications. Depth maps introduce various quality artifacts in the DIBR-synthesized (virtual) images. In this paper, we propose a novel methodology to evaluate the quality of synthesized views in absence of the corresponding original reference views. It computes the statistical characteristics of the side views from whom the virtual view is generated, and fuses this information to estimate the statistical characteristics of the cyclopean image which are compared to those of the synthesized image to evaluate its quality. In addition to texture images, the proposed algorithm also considers the depth maps in evaluating the quality of the synthesized images. The algorithm blends two quality metrics, one estimating the texture distortion in the synthesized texture image induced by compression, transmission, 3D warping, or other causes and the second one determining the distortion of the depth maps. The two metrics are combined to obtain an overall quality assessment of the synthesized image. The proposed Synthesized Image Quality Metric (SIQM) is tested on the challenging MCL-3D and SIAT-3D datasets. The evaluation results show that the proposed metric significantly improves over state-of-the-art 3D image quality assessment algorithms.

A Novel DIBR 3D Image Watermarking Algorithm Resist to Geometrical Attacks

A histogram shape based method is utilized to design watermarking algorithm to protect the Depthimage-based rendering (DIBR) 3D images. To make the watermarking method robust to common attacks, much more suitable pixel groups are selected for watermark embedding which are insensitive to geometric attacks. Taking into account the adjusting of baseline distance in DIBR process might affect the watermarking extraction, a postprocessing is proposed after DIBR process to make the proposed watermarking method much more robust to DIBR process. As the experimental results shown, the proposed method is much more robust to the geometric attacks and combined attacks compared with existing methods. In addition, the proposed watermarking method also has good robustness to the adjusting of baseline distance and depth image blurred.

A Novel Hybrid Kinect-Variety-Based High-Quality Multiview Rendering Scheme for Glass-Free 3D Displays

This paper presents a new hybrid Kinect-variety-based synthesis scheme that renders artifact-free multiple views for autostereoscopic/automultiscopic displays. The proposed approach does not explicitly require dense scene depth information for synthesizing novel views from arbitrary viewpoints. Instead, the integrated framework first constructs a consistent minimal image–space parameterization of the underlying 3D scene. The compact representation of scene structure is formed using only implicit sparse depth information of a few reference scene points extracted from raw RGB depth data. The views from arbitrary positions can be inferred by moving the novel camera in parameterized space by enforcing Euclidean constraints on reference scene images under a full-perspective projection model. Unlike the state-of-the-art depth image-based rendering (DIBR) methods, in which input depth map accuracy is crucial for high-quality output, our proposed algorithm does not depend on precise per-pixel geometry information. Therefore, it simply sidesteps to recover and refine the incomplete or noisy depth estimates with advanced filling or upscaling techniques. Our approach performs fairly well in unconstrained indoor/outdoor environments, where the performance of range sensors or dense depth-based algorithms could be seriously affected due to scene complex geometric conditions. We demonstrate that the proposed hybrid scheme provides guarantees on the completeness, optimality with respect to the inter-view consistency of the algorithm. In the experimental validation, we performed a quantitative evaluation as well as subjective assessment of the scene with complex geometric or surface properties. A comparison with the latest representative DIBR methods is additionally performed to demonstrate the superior performance of the proposed scheme.

Model-Based Referenceless Quality Metric of 3D Synthesized Images Using Local Image Description.

New challenges have been brought out along with the emerging of 3D-related technologies, such as virtual reality, augmented reality (AR), and mixed reality. Free viewpoint video (FVV), due to its applications in remote surveillance, remote education, and so on, based on the flexible selection of direction and viewpoint, has been perceived as the development direction of next-generation video technologies and has drawn a wide range of researchers' attention. Since FVV images are synthesized via a depth image-based rendering (DIBR) procedure in the "blind" environment (without reference images), a reliable real-time blind quality evaluation and monitoring system is urgently required. But existing assessment metrics do not render human judgments faithfully mainly because geometric distortions are generated by DIBR. To this end, this paper proposes a novel referenceless quality metric of DIBR-synthesized images using the autoregression (AR)-based local image description. It was found that, after the AR prediction, the reconstructed error between a DIBR-synthesized image and its AR-predicted image can accurately capture the geometry distortion. The visual saliency is then leveraged to modify the proposed blind quality metric to a sizable margin. Experiments validate the superiority of our no-reference quality method as compared with prevailing full-, reduced-, and no-reference models.

A SIFT features based blind watermarking for DIBR 3D images

Depth image based rendering (DIBR) is a promising technique for extending viewpoints with a monoscopic center image and its associated per-pixel depth map. With its numerous advantages including low-cost bandwidth, 2D-to-3D compatibility and adjustment of depth condition, DIBR has received much attention in the 3D research community. In the case of a DIBR-based broadcasting system, a malicious adversary can illegally distribute both a center view and synthesized virtual views as 2D and 3D content, respectively. To deal with the issue of copyright protection for DIBR 3D Images, we propose a scale invariant feature transform (SIFT) features based blind watermarking algorithm. To design the proposed method robust against synchronization attacks from DIBR operation, we exploited the parameters of the SIFT features: the location, scale and orientation. Because the DIBR operation is a type of translation transform, the proposed method uses high similarity between the SIFT parameters extracted from a synthesized virtual view and center view images. To enhance the capacity and security, we propose an orientation of keypoints based watermark pattern selection method. In addition, we use the spread spectrum technique for watermark embedding and perceptual masking taking into consideration the imperceptibility. Finally, the effectiveness of the presented method was experimentally verified by comparing with other previous schemes. The experimental results show that the proposed method is robust against synchronization attacks from DIBR operation. Furthermore, the proposed method is robust against signal distortions and typical attacks from geometric distortions such as translation and cropping.

Analysis of maximum tolerant depth distortion in view synthesis

In view synthesis, pixels in an original view are warped into a virtual view with depth-image-based rendering (DIBR). During the procedure of DIBR, distortions in depth map may lead to geometric errors in the synthesized view which will induce quality degradation of synthesized view. Therefore, how to efficiently preserve the fidelity of depth information is extremely important. In this paper, we explore and develop a maximum tolerable depth distortion (MTDD) model to examine the allowable depth distortion which will not introduce any texture distortion for a rendered virtual view and accordingly develop. Experimental results show that a virtual view can be synthesized without introducing any geometric changes if depth distortions follow the MTDD specified thresholds.