Local Foreground Removal Disocclusion Filling Method for View Synthesis

Haitao Liang,Siyu Ren,Yi Wang,Xiaodong Chen,Huaiyuan Xu,Huaiyu Cai

doi:10.1109/access.2020.3036053

Haitao Liang, Siyu Ren + Show 4 more

Open Access

https://doi.org/10.1109/access.2020.3036053

Copy DOI

Abstract

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.32-2.43dB, 0.0036-0.0155, 0.0041-0.0198 and 0.0012-0.0057 respectively compared with competitive methods.

Highlights

With the development of computer science and multimedia technology, 3D video and free viewpoint video (FVV) have drawn more attention
All pixels in the reference image are projected to the world coordinate based on the depth information, and the resulting points are reprojected onto the imaging plane of the target view
Depth value reflects the distance between the object and the camera, which plays an important role in 3D warping

Summary

Introduction

With the development of computer science and multimedia technology, 3D video and free viewpoint video (FVV) have drawn more attention. As the ultimate of 3D video, FVV allows the viewer to freely choose the viewpoint within a certain range [3] It requires the data of multiple viewpoints. For 2D to 3D conversion, only a single viewpoint can be obtained [5] In this case, a practical way is to use a depth-image-based rendering (DIBR) method to generate multiple virtual views, which only requires a single reference view and its associated depth image [6]. The core technology of DIBR is called 3D warping [7] In this process, all pixels in the reference image are projected to the world coordinate based on the depth information, and the resulting points are reprojected onto the imaging plane of the target view

Methods

Results

Conclusion