Demo: Real-time depth extraction and viewpoint interpolation on FPGA

Abstract

In this demo, we demonstrate a real-time viewpoint interpolation application on FPGA. Viewpoint interpolation is the process of synthesizing plausible in-between views — so-called virtual camera views — from a couple of surrounding fixed camera views. Stereo matching is used to extract depth information, by computing a disparity map from a pair of input images. With the depth information, virtual views at any points between the two cameras are computed through view interpolation. To make viewpoint interpolation possible for low/moderate-power consumer applications, a further quality/complexity tradeoff study is required to conciliate algorithmic quality to architectural performance. In essence, the inter-dependencies between the different algorithmic steps in the processing chain are thoroughly analyzed, aiming at an overall quality-performance model that pinpoints which algorithmic functionalities can be simplified with minor (preferably no) global input-output quality degradation, while maximally reducing their implementation complexity w.r.t. arithmetic and line buffer requirements. Compared to state-of-the-art CPU and GPU platforms running at several GHz clock speed, our low-power 65 MHz FPGA implementation achieves speedups with one order of magnitude over state-of-the-art, without impeding on the visual quality, reaching over 60 frames per second high-definition (1024×768) high-quality, 64-disparity search range stereo matching and enabling viewpoint interpolation in low-power, embedded applications.