Inverse tensor transfer with applications to novel view synthesis and multi-baseline stereo

Abstract

A new tensor transfer-based novel view synthesis (NVS) method is proposed in this paper. As opposed to conventional tensor transfer methods which transfer the pixel from the real input views to the virtual novel view, our method operates inversely in the sense that it transfers a pixel from the novel view image back to the real images. This inverse tensor-transfer approach offers a simple mechanism for associating corresponding image points across multiple views, resulting in geometrically consistent pixel chains across the input images. A colour consistency metric is used to choose the most likely colour for a pixel in the novel image by analysing the spread of colours in each of the possible pixel chains. By emphasizing colour consistency, rather than depth, our method avoids the need to precompute a dense depth map (which is essential for most conventional transfer methods), therefore alleviating many common problems with conventional methods. Experiments involving NVS on real images give promising results. The synthesized novel view image is not only photo-realistic but also has the right geometric relationship with respect to the other views. Since this method avoids explicit depth map computation, we further investigate its applicability to the multi-baseline stereo matching problem (MBS). By using this inverse transfer idea, we are able to handle non-ideally configured MBS in a natural and efficient way. The new MBS algorithm has been used for stereo vision navigation.