Depth-image-based Rendering Process Research Articles

Quality Prediction of View Synthesis Based on Curriculum-Style Structure Generation

Existing quality metrics of view synthesis usually perform on synthesized images, which are produced based on a computationally expensive depth-image-based rendering (DIBR) process. Moreover, current metrics quantify quality by extracting hand-crafted features, which may fail to fully capture the complex distortion characteristics. With the success of deep learning on numerous computer vision tasks, it has become possible to utilize convolutional neural networks to predict the quality of DIBR-synthesized images. In this letter, we propose a deep model to predict the quality of view synthesis based on Curriculum-style Structure Generation without conducting the DIBR process. Specifically, considering that the distortion of view synthesis is mainly manifested in the destruction of image structure, a structure generation network is first built to learn the structure of the new view from the original one by curriculum-style training. Then, we transfer the prior knowledge learned from the last phase into the quality prediction network for measuring the structure distortion, based on which a regressor is introduced to produce the quality score. Experimental results prove the advantages of the proposed model.

Blind Quality Prediction for View Synthesis Based on Heterogeneous Distortion Perception.

The quality of synthesized images directly affects the practical application of virtual view synthesis technology, which typically uses a depth-image-based rendering (DIBR) algorithm to generate a new viewpoint based on texture and depth images. Current view synthesis quality metrics commonly evaluate the quality of DIBR-synthesized images, where the DIBR process is computationally expensive and time-consuming. In addition, the existing view synthesis quality metrics cannot achieve robustness due to the shallow hand-crafted features. To avoid the complicated DIBR process and learn more efficient features, this paper presents a blind quality prediction model for view synthesis based on HEterogeneous DIstortion Perception, dubbed HEDIP, which predicts the image quality of view synthesis from texture and depth images. Specifically, the texture and depth images are first fused based on discrete cosine transform to simulate the distortion of view synthesis images, and then the spatial and gradient domain features are extracted in a Two-Channel Convolutional Neural Network (TCCNN). Finally, a fully connected layer maps the extracted features to a quality score. Notably, the ground-truth score of the source image cannot effectively represent the labels of each image patch during training due to the presence of local distortions in view synthesis image. So, we design a Heterogeneous Distortion Perception (HDP) module to provide effective training labels for each image patch. Experiments show that with the help of the HDP module, the proposed model can effectively predict the quality of view synthesis. Experimental results demonstrate the effectiveness of the proposed model.

Predicting the Quality of View Synthesis With Color-Depth Image Fusion

With the increasing prevalence of free-viewpoint video applications, virtual view synthesis has attracted extensive attention. In view synthesis, a new viewpoint is generated from the input color and depth images with a depth-image-based rendering (DIBR) algorithm. Current quality evaluation models for view synthesis typically operate on the synthesized images, i.e. after the DIBR process, which is computationally expensive. So a natural question is that can we infer the quality of DIBR-based synthesized images using the input color and depth images directly without performing the intricate DIBR operation. With this motivation, this paper presents a no-reference image quality prediction model for view synthesis via COlor-Depth Image Fusion, dubbed CODIF, where the actual DIBR is not needed. First, object boundary regions are detected from the color image, and a Wavelet-based image fusion method is proposed to imitate the interaction between color and depth images during the DIBR process. Then statistical features of the interactional regions and natural regions are extracted from the fused color-depth image to portray the influences of distortions in color/depth images on the quality of synthesized views. Finally, all statistical features are utilized to learn the quality prediction model for view synthesis. Extensive experiments on public view synthesis databases demonstrate the advantages of the proposed metric in predicting the quality of view synthesis, and it even suppresses the state-of-the-art post-DIBR view synthesis quality metrics.

NSCT-Based Robust and Perceptual Watermarking for DIBR 3D Images

Depth-image-based rendering (DIBR), where arbitrary views are synthesized from a center image and depth image, has received much attention in the three-dimensional (3D) research field. With advances in depth-acquisition techniques and the proliferation of 3D glasses and 3D display devices, there is a growing demand for schemes to protect the copyrights of DIBR 3D images. Digital watermarking is a typical protection technology and designing a watermarking method for DIBR 3D images is a challenging task because the synchronization of watermarks can easily be broken in the process of generating synthetic images. To address this issue, we propose a non-subsampled contourlet transform (NSCT)-based blind watermarking for DIBR 3D images. To ensure the proposed method has properties of robustness against the DIBR process, we conduct an analysis of robustness for NSCT subbands against DIBR attacks. Based on the analysis results, we select subbands that are robust against DIBR attacks and embed watermark in the low coefficients of NSCT subbands using quantization-based embedding. While ensuring robustness, the proposed method also improves the imperceptibility of watermarks by adjusting their embedding strength according to computed perceptual masking values. Through experiments, we show that the proposed method is robust against both desynchronization attacks of the DIBR process and common attacks including signal processing operations and geometric distortions. The high imperceptibility of our method is also verified by several evaluation metrics in a subjective and objective manner.

Local Foreground Removal Disocclusion Filling Method for View Synthesis

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.32-2.43dB, 0.0036-0.0155, 0.0041-0.0198 and 0.0012-0.0057 respectively compared with competitive methods.

Artifact Handling Based on Depth Image for View Synthesis

The depth image based rendering (DIBR) is a popular technology for 3D video and free viewpoint video (FVV) synthesis, by which numerous virtual views can be generated from a single reference view and its depth image. However, some artifacts are produced in the DIBR process and reduce the visual quality of virtual view. Due to the diversity of artifacts, effectively handling them becomes a challenging task. In this paper, an artifact handling method based on depth image is proposed. The reference image and its depth image are extended to fill the holes that belong to the out-of-field regions. A depth image preprocessing method is applied to project the ghosts to their correct place. The 3D warping process is optimized by an adaptive one-to-four method to deal with the cracks and pixel overlapping. For disocclusions, we calculate depth and background terms of the filling priority based on depth information. The search for the best matching patch is performed simultaneously in the reference image and the virtual image. Moreover, adaptive patch size is used in all hole-filling processes. Experimental results demonstrate the effectiveness of the proposed method, which has better performance compared with previous methods in subjective and objective evaluation.

Hole Filling for View Synthesis Using Depth Guided Global Optimization

View synthesis is an effective way to generate multi-view contents from a limited number of views, and can be utilized for 2-D-to-3-D video conversion, multi-view video compression, and virtual reality. In the view synthesis techniques, depth-image-based rendering (DIBR) is an important method to generate virtual view from video-plus-depth sequence. However, some holes might be produced in the DIBR process. Many hole filling methods have been proposed to tackle this issue, but most of them cannot achieve globally coherent or acquire trusted contents. In this paper, a hole filling method with depth-guided global optimization is proposed for view synthesis. The global optimization is achieved by iterating the spatio-temporal approximate nearest neighbor (ANN) search and video reconstruction step. Directly applying global optimization might introduce some foreground artifacts to the synthesized video. To prevent this problem, some strategies have been developed in this paper. The depth information is applied to guide the spatio-temporal ANN searching and the initialization step is specified in the global optimization procedure. Our experimental results have demonstrated that the proposed method has better performance compared with other methods in terms of visual quality, trusted textures, and temporal consistency in the synthesized video.

A Novel DIBR 3D Image Watermarking Algorithm Resist to Geometrical Attacks

A histogram shape based method is utilized to design watermarking algorithm to protect the Depthimage-based rendering (DIBR) 3D images. To make the watermarking method robust to common attacks, much more suitable pixel groups are selected for watermark embedding which are insensitive to geometric attacks. Taking into account the adjusting of baseline distance in DIBR process might affect the watermarking extraction, a postprocessing is proposed after DIBR process to make the proposed watermarking method much more robust to DIBR process. As the experimental results shown, the proposed method is much more robust to the geometric attacks and combined attacks compared with existing methods. In addition, the proposed watermarking method also has good robustness to the adjusting of baseline distance and depth image blurred.

A novel hashing scheme for Depth-image-based-rendering 3D images

Image hashing as an advanced approach has been widely applied in many areas such as image authentication, and content-based identification and retrieval. Being different from the traditional 2D image hashing scheme, virtual images which are generated in the DIBR process should be considered with the same content as the center image in Depth-image-based rendering (DIBR) 3D image hashing scheme. In this paper, we propose a hashing method for DIBR 3D images by applying three-level Dual-tree complex wavelet transform (DT-CWT) to images and selecting suitable DT-CWT coefficients to form the final hash sequence. Experimental results indicate that the proposed method is robust to a wide range of distortions and attacks. When compared with the current state-of-the-art methods, the proposed method can achieve better identification performances in terms of geometric attacks such as rotation attacks, whilst providing comparable performances under classical distortions such as additive noise, blurring, and compression. Furthermore, the proposed method is also able to ensure that the generated virtual images could be identified with the same content as the corresponding center image due to the characteristic approximate shift invariance of DT-CWT.

Robust DT-CWT Watermarking for DIBR 3D Images

The popularity of 3D content is on the rise since it provides an immersive experience to viewers. In this situation, depth-image-based rendering (DIBR) has taken on an important role in 3D technology due to its low bandwidth cost and ease of depth configuration. Noting that the viewer could record provided center view or synthesized views for illegal distribution, it is clear that copyright protection must be taken into account for the DIBR 3D content, including the possibility that one single view could be illegally distributed as 2D content. In this paper, we propose a robust watermarking scheme for DIBR 3D images by quantization on dual-tree complex wavelet transform (DT-CWT) coefficients with consideration of imperceptibility. To make the proposed scheme robust to DIBR process, two characteristics of DT-CWT are employed: approximate shift invariance and directional selectivity. We select certain coefficient sub-blocks and group the coefficient rows based on the properties of DIBR. On the extraction side, the threshold is carefully chosen with a low false positive rate. The simulation results show that the embedded watermark is stably extracted from the center view and the synthesized left and right views. In addition, even if the synthesized left and right views are distorted by general attacks, the watermark is successfully extracted. Furthermore, the proposed scheme is robust to pre-processing of the depth image and baseline adjusting, which are common processing on the DIBR system for better quality of 3D views.