Depth Image Based View Synthesis: New Insights and Perspectives on Hole Generation and Filling

Abstract

View synthesis with depth-image-based rendering (DIBR) has attracted great interest in that it can provide a virtual image at any arbitrary viewpoint in 3-D video and free-viewpoint TV. An inherent problem in the DIBR view synthesis is occurrence of holes in a synthesized image, which is also known as disocclusion problem. The disoccluded regions need to be handled properly in order to generate a synthesized view of good quality. This paper provides a fundamental examination of hole generation mechanism in the DIBR oriented view synthesis process. A necessary and sufficient condition of hole generation is first shown, and the corresponding hole location and length is obtained analytically. Furthermore, in view that the conventional hole filling algorithms may fail to fill up a hole correctly when lacking (adequate) visible background information, we propose utilizing the occluded (invisible) information to identify and locate the relevant background pixels around a hole. We then make use of the visible and invisible background information together to perform hole filling. Experimental results validate our hole generation model demonstrating agreement to our analytical results, while our proposed hole filling approach shows superior performance in terms of visual quality of synthesized views.