Real-time stereo-based view synthesis algorithms: A unified framework and evaluation on commodity GPUs

Abstract

Novel view synthesis based on dense stereo correspondence is an active research problem. Despite that many algorithms have been proposed recently, this flourishing, cross-area research field still remains relatively less structured than its front-end constituent part, stereo correspondence. Moreover, so far little work has been done to assess different stereo-based view synthesis algorithms, particularly when real-time execution is enforced as a hard application constraint. In this paper, we first propose a unified framework that seamlessly connects stereo correspondence and view synthesis. The proposed framework dissects the typical algorithms into a common set of individual functional modules, allowing the comparison of various design decisions. Aligned with this algorithmic framework, we have developed a flexible GPU-accelerated software model, which contains optimized implementations of several recent real-time algorithms, specifically focusing on local cost aggregation and image warping modules. Based on this common software model running on graphics hardware, we evaluate the relative performance of various design combinations in terms of both view synthesis quality and real-time processing speed. This comparative evaluation leads to a number of observations, and hence offers useful guides to the future design of real-time stereo-based view synthesis algorithms.