Depth Image-based Rendering Method Research Articles

Coarse- and Fine-Grained Fusion Hierarchical Network for Hole Filling in View Synthesis.

Depth image-based rendering (DIBR) techniques play an essential role in free-viewpoint videos (FVVs), which generate the virtual views from a reference 2D texture video and its associated depth information. However, the background regions occluded by the foreground in the reference view will be exposed in the synthesized view, resulting in obvious irregular holes in the synthesized view. To this end, this paper proposes a novel coarse and fine-grained fusion hierarchical network (CFFHNet) for hole filling, which fills the irregular holes produced by view synthesis using the spatial contextual correlations between the visible and hole regions. CFFHNet adopts recurrent calculation to learn the spatial contextual correlation, while the hierarchical structure and attention mechanism are introduced to guide the fine-grained fusion of cross-scale contextual features. To promote texture generation while maintaining fidelity, we equip CFFHNet with a two-stage framework involving an inference sub-network to generate the coarse synthetic result and a refinement sub-network for refinement. Meanwhile, to make the learned hole-filling model better adaptable and robust to the "foreground penetration" distortion, we trained CFFHNet by generating a batch of training samples by adding irregular holes to the foreground and background connection regions of high-quality images. Extensive experiments show the superiority of our CFFHNet over the current state-of-the-art DIBR methods. The source code will be available at https://github.com/wgc-vsfm/view-synthesis-CFFHNet.

Virtual View Synthesis Based on Asymmetric Bidirectional DIBR for 3D Video and Free Viewpoint Video

Depth image-based rendering (DIBR) plays an important role in 3D video and free viewpoint video synthesis. However, artifacts might occur in the synthesized view due to viewpoint changes and stereo depth estimation errors. Holes are usually out-of-field regions and disocclusions, and filling them appropriately becomes a challenge. In this paper, a virtual view synthesis approach based on asymmetric bidirectional DIBR is proposed. A depth image preprocessing method is applied to detect and correct unreliable depth values around the foreground edges. For the primary view, all pixels are warped to the virtual view by the modified DIBR method. For the auxiliary view, only the selected regions are warped, which contain the contents that are not visible in the primary view. This approach reduces the computational cost and prevents irrelevant foreground pixels from being warped to the holes. During the merging process, a color correction approach is introduced to make the result appear more natural. In addition, a depth-guided inpainting method is proposed to handle the remaining holes in the merged image. Experimental results show that, compared with bidirectional DIBR, the proposed rendering method can reduce about 37% rendering time and achieve 97% hole reduction. In terms of visual quality and objective evaluation, our approach performs better than the previous methods.

Multiscale Natural Scene Statistical Analysis for No-Reference Quality Evaluation of DIBR-Synthesized Views

This paper proposes to blindly evaluate the quality of images synthesized via a depth image-based rendering (DIBR) procedure. As a significant branch of virtual reality (VR), superior DIBR techniques provide free viewpoints in many real applications, including remote surveillance and education; however, limited efforts have been made to measure the performance of DIBR techniques, or equivalently the quality of DIBR-synthesized views, especially in the condition when references are unavailable. To achieve this aim, we develop a novel blind image quality assessment (IQA) method via multiscale natural scene statistical analysis (MNSS). The design principle of our proposed MNSS metric is based on two new natural scene statistics (NSS) models specific to the DBIR-synthesized IQA. First, the DIBR-introduced geometric distortions damage the local self-similarity characteristic of natural images, and the damage degrees of self-similarity present particular variations at different scales. Systematically combining the measurements of the variations mentioned above can gauge the naturalness of the input image and thus indirectly reflect the quality changes of images generated using different DIBR methods. Second, it was found that the degradations in main structures of natural images at different scales remain almost the same, whereas the statistical regularity is destroyed in the DIBR-synthesized views. Estimating the deviation of degradations in main structures at different scales between one DIBR-synthesized image and the statistical model, which is constructed based on a large number of natural images, can quantify how a DIBR method damages the main structures and thus infer the image quality. Via trials, the two NSS-based features extracted above can well predict the quality of DIBR-synthesized images. Further, the two features come from distinct points of view, and we hence integrate them via a straightforward multiplication to derive the proposed blind MNSS metric, which achieves better performance than each component and state-of-the-art quality methods.

View synthesis based on spatio-temporal continuity

Free viewpoint video is generated on the basis of a video plus depth (V+D) virtual point rendering framework. Because of the limited bandwidth of video transmission, depth-image-based rendering (DIBR) has become a common method. Most DIBR methods not only have holes and ghost artifacts but also have problems with time continuity, leading to frequent flickers. In this paper, we make full use of time domain information in video sequences and adjacent frames to extract the static background image of the whole scene. Furthermore, we propose a weighted-fusion hole-filling method based on static background to fill holes and maintain time continuity. Experimental results show that the proposed method can improve the quality of virtual view images and strengthen spatio-temporal continuity.

A Novel Hybrid Kinect-Variety-Based High-Quality Multiview Rendering Scheme for Glass-Free 3D Displays

This paper presents a new hybrid Kinect-variety-based synthesis scheme that renders artifact-free multiple views for autostereoscopic/automultiscopic displays. The proposed approach does not explicitly require dense scene depth information for synthesizing novel views from arbitrary viewpoints. Instead, the integrated framework first constructs a consistent minimal image–space parameterization of the underlying 3D scene. The compact representation of scene structure is formed using only implicit sparse depth information of a few reference scene points extracted from raw RGB depth data. The views from arbitrary positions can be inferred by moving the novel camera in parameterized space by enforcing Euclidean constraints on reference scene images under a full-perspective projection model. Unlike the state-of-the-art depth image-based rendering (DIBR) methods, in which input depth map accuracy is crucial for high-quality output, our proposed algorithm does not depend on precise per-pixel geometry information. Therefore, it simply sidesteps to recover and refine the incomplete or noisy depth estimates with advanced filling or upscaling techniques. Our approach performs fairly well in unconstrained indoor/outdoor environments, where the performance of range sensors or dense depth-based algorithms could be seriously affected due to scene complex geometric conditions. We demonstrate that the proposed hybrid scheme provides guarantees on the completeness, optimality with respect to the inter-view consistency of the algorithm. In the experimental validation, we performed a quantitative evaluation as well as subjective assessment of the scene with complex geometric or surface properties. A comparison with the latest representative DIBR methods is additionally performed to demonstrate the superior performance of the proposed scheme.

Improved Depth Image-Based Rendering Using an Adaptive Compensation Method on an Autostereoscopic 3-D Display for a Kinect Sensor

Depth image-based rendering (DIBR) is a 2-D-3-D view conversion technology in a space domain that corresponds to depth map. DIBR system requires only a single color image and an associated depth map to generate synthesized autostereoscopic 3-D views, but the quality of the result is depreciated due to the disoccluded areas in the depth map. Because of this problem, an improved DIBR method that uses a depth map adaptive compensation method is proposed. Using a fast parallax lookup table, the position of holes is allocated and depth map is split into blocks for statistical analysis and classification in order to estimate the quantity of holes. The result of the classification is used for adaptive multimode filtering to optimize the depth map. The experimental results show that the proposed method not only enhances the quality of depth map by compensating for the bigger and smaller areas of holes referring to analysis results, and by determining the filling method using the type of holes, but also reduces the number of holes that retain better depth map and reduces computation time.

깊이 정보 확장과 메쉬 구성을 이용한 DIBR 기반 다시점 중간 영상 화질 향상 방법에 관한 연구

ABSTRACT In this research, we conducted an experiment on evaluating the extending depth information method and surface reconstruction method and the interaction of these two methods in order to enhance the final intermediate view images, which are acquired using DIBR (Depth-Image-Based Rendering) method. We evaluated the experimental control groups using the Microsoft’s “Ballet” and “Break Dancer” data sets with three different hole-filling algorithms. The result revealed that the quality was improved the most by applying both extending depth information and surface reconstruction method as compared to the previous point clouds only. In addition, it found that the quality of the intermediate images was improved vastly by only applying extending depth information when using no hole-filling algorithm. 키워드 : 깊이 영상 기반 렌더링, 메쉬 구성, 깊이 정보 확장, 다시점 디스플레이 Key word : Depth-Image-Based Rendering, Mesh Construction, Extending depth information, Multiview DisplayJournal of the Korea Institute of Information andCommunication Engineering

輝度と視差に基づく画像の領域分割によるDepth-Image-Based-Rendering

Depth-image-based rendering (DIBR) is a technique for synthesizing novel views of a scene from an image and its depth map. Many DIBR methods in the literature blurred the depth map before 3D warping to make the holes become smaller. However, the PSNR values when using methods with depth map blurring are lower than those when using methods without blurring. On the other hand, conventional methods without blurring failed when the scene was a complex 3D structure. In this paper, a DIBR method without depth map blurring is proposed for synthesizing novel views by 3D warping followed by depth map and hole interpolation. To interpolate holes after 3D warping naturally, it is important to recognize the objects surrounding holes especially when the scene structure is complex. To recognize objects in the image, the input image is segmented into small regions based on color and depth. Each hole is extrapolated by extending the surrounding small regions of the hole while considering the visibility and continuity. The experimental results demonstrate the efficiency of our method.

Depth-image-based rendering with spatial and temporal texture synthesis for 3DTV

AbstractA depth-image-based rendering (DIBR) method with spatial and temporal texture synthesis is presented in this article. Theoretically, the DIBR algorithm can be used to generate arbitrary virtual views of the same scene in a three-dimensional television system. But the disoccluded area, which is occluded in the original views and becomes visible in the virtual views, makes it very difficult to obtain high image quality in the extrapolated views. The proposed view synthesis method combines the temporally stationary scene information extracted from the input video and spatial texture in the current frame to fill the disoccluded areas in the virtual views. Firstly, the current texture image and a stationary scene image, which is extracted from the input video, are warped to the same virtual perspective position by the DIBR method. Then, the two virtual images are merged together to reduce the hole regions and maintain the temporal consistency of these areas. Finally, an oriented exemplar-based inpainting method is utilized to eliminate the remaining holes. Experimental results are shown to demonstrate the performance and advantage of the proposed method compared with other view synthesis methods.

High performance computational integral imaging system using multi-view video plus depth representation

Integral imaging is an attractive auto-stereoscopic three-dimensional (3D) technology for next-generation 3DTV. But its application is obstructed by poor image quality, huge data volume and high processing complexity. In this paper, a new computational integral imaging (CII) system using multi-view video plus depth (MVD) representation is proposed to solve these problems. The originality of this system lies in three aspects. Firstly, a particular depth-image-based rendering (DIBR) technique is used in encoding process to exploit the inter-view correlation between different sub-images (SIs). Thereafter, the same DIBR method is applied in the display side to interpolate virtual SIs and improve the reconstructed 3D image quality. Finally, a novel parallel group projection (PGP) technique is proposed to simplify the reconstruction process. According to experimental results, the proposed CII system improves compression efficiency and displayed image quality, while reducing calculation complexity.

Improving Graph Cuts algorithm to transform sequence of stereo image to depth map

Recently, 3D display systems are getting considerable attentions not only from theater but also from home. 3D multimedia content development plays very important role in helping to setup a visual reality entertainment program. Lenticular Autostereoscopic display is one of the 3D-TV having the following advantages such as improving 3D viewing experience, supporting wider viewing angles for multiple viewers, and no requiring any special glasses. However, most of the current 3D movie and camera do not support the Autostereoscopic function. Therefore, we proposed a system that can transform the current 3D stereoscopic image sequence to the depth map sequence. These sequences can be warped into the multiplexed image by DIBR (Depth Image Based Rendering), and show with Autostereoscopic.Some recent techniques that transform the stereoscopic correspondence problem are based on Graph Cuts. They transform the matching problem to a minimization of a global energy function. However, it has been difficult to include high level information in the formulation of the Graph Cut. In this paper, we describe a new technique for generating depth map sequence from stereoscopic image sequence. We improve the Graph Cuts from pixel-based matching to region-based by using the Mean Shift 3D regions clustering to link the features of images before segmentation. And we also use the result of 3D regions clustering to assign depth values to time domain. After the sequence of depth map has been obtained, the DIBR method was used in transformation process. The experimental result shows that our system not only establishes a mechanism of depth transformation but also improves the accuracy and effectiveness on traditional Graph Cuts.

Depth Image-Based Rendering With Advanced Texture Synthesis for 3-D Video

A depth image-based rendering (DIBR) approach with advanced inpainting methods is presented. The DIBR algorithm can be used in 3-D video applications to synthesize a number of different perspectives of the same scene, e.g., from a multiview-video-plus-depth (MVD) representation. This MVD format consists of video and depth sequences for a limited number of original camera views of the same natural scene. Here, DIBR methods allow the computation of additional new views. An inherent problem of the view synthesis concept is the fact that image information which is occluded in the original views may become visible, especially in extrapolated views beyond the viewing range of the original cameras. The presented algorithm synthesizes these occluded textures. The synthesizer achieves visually satisfying results by taking spatial and temporal consistency measures into account. Detailed experiments show significant objective and subjective gains of the proposed method in comparison to the state-of-the-art methods.

Object-based depth image–based rendering for a three-dimensional video system by color-correction optimization

Three-dimensional (3-D) video technologies are becoming increasingly popular because they can provide high quality and immersive experience to end users. Depth image-based rendering (DIBR) is a key technology in 3-D video systems due to its low bandwidth cost as well as the arbitrary rendering viewpoint. We propose an object-based DIBR method by color-correction optimization. The proposed method first performs temporal consistent rendering to reduce the rendering complexity. Then, by segmenting the depth map into foreground and background, the object-based scalable rendering is performed to improve the rendering quality and reduce the rendering complexity. Finally, the rendered virtual view is further optimized by color-correction operation. Experimental results show that, compared to the results without the above optimization operations, the proposed method can reduce >40% computational complexity while maintaining high rendering quality.

Preserving depth resolution of synthesized images using parallax-map-based dibr for 3D-TV

Due to the flexibility and efficiency of video-plus-depth 3D video representation format, Depth-Image-Based Rendering (DIBR) has been proposed to create content for 3D-TV. Most previous researches have focused on eliminating areas of disocclusion in synthesized images by smoothing depth-images, which is time-consuming and usually leads to the attenuation of depth-resolution and some artifacts. In this paper, we aim to investigate factors degrading visual quality and attenuating depth-resolution. Based on the investigation, a low-cost and depth-resolution-preserved Parallax-Map-Based DIBR method is then presented. Our method efficiently prevents attenuation of depth resolution and geometric distortions in synthesized images with lower computation resources compared to previous studies, as shown in the experimental results.

깊이맵의 전처리와 깊이 정보의 기하학적 분석을 통한 최적의 스테레오스코픽 영상 자동 생성 기법

DIBR(depth image-based rendering) 기법은 한 장의 칼라 영상과 그에 대응하는 깊이맵을 이용하여 수신측에서 가상의 좌우 영상을 생성하여 시청자에게 깊이감을 제공해주는 방법이다. 이 때 생성되는 좌우 영상의 화질과 깊이감은 좌우 영상의 시점에 해당하는 가상 카메라의 간격에 영향을 받게 된다. 본 논문에서는 가상 카메라의 간격 조절을 통해 양질의 깊이감을 제공하는 기법을 제안한다. 영상의 기하 분석에 따르면 스테레오스코픽 영상의 변이가 증가할수록 영상의 왜곡이 줄어들어 깊이감이 좋아지게 되나, 이 경우에는 그에 따른 영상의 열화가 고려되지 않았다. 실험결과에 따르면 영상열화나 시역형성의 제한 등으로 인해 변이의 증가폭에 제한이 있음을 알 수 있다. 변이 증가폭을 늘리기 위해서는 영상의 열화 현상을 줄여야 하므로 깊이맵의 전처리 과정을 추가하였다. 제안하는 방법은 시청자의 요구에 따라 카메라의 위치와 시점을 추가적으로 조절할 수 있게 하는 양방향 서비스를 추가로 제공하므로 3차원 TV 뿐만 아니라, DMB와 같은 이동형 방송 시스템에도 적용이 가능하다. The DIBR(depth image-based rendering) method gives the sense of depth to viewers by using one color image and corresponding depth image. At this time, the qualities of the generated left- and right-image depend on the baseline distance of the virtual cameras corresponding to the view of the generated left- and right-image. In this paper, we present a novel method for enhancing the sense of depth by adjusting baseline distance of virtual cameras. Geometric analysis shows that the sense of depth is better in accordance with the increasing disparity due to the reduction of the image distortion. However, the entailed image degradation is not considered. Experimental results show that there is maximum bound in the disparity increasement due to image degradation and the visual field. Since the image degradation is reduced for increasing that bound, we add a depth map preprocessing. Since the interactive service where the disparity and view position are controlled by viewers can also be provided, the proposed method can be applied to the mobile broadcasting system such as DMB as well as 3DTV system.