Real-time representation and rendering of high-resolution 3D light field based on texture-enhanced optical flow prediction

Abstract

Three-dimensional (3D) light field displays can provide an immersive visual perception and have attracted widespread attention, especially in 3D light field communications, where 3D light field displays can provide face-to-face communication experiences. However, due to limitations in 3D reconstruction and dense views rendering efficiency, generating high-quality 3D light field content in real-time remains a challenge. Traditional 3D light field capturing and reconstruction methods suffer from high reconstruction complexity and low rendering efficiency. Here, a Real-time optical flow representation for the high-resolution light field is proposed. Based on the principle of 3D light field display, we use optical flow to ray trace and multiplex sparse view pixels. We simultaneously synthesize 3D light field images during the real-time interpolation process of views. In addition, we built a complete capturing-display system to verify the effectiveness of our method. The experiments' results show that the proposed method can synthesize 8 K 3D light field videos containing 100 views in real-time. The PSNR of the virtual views is around 32 dB and SSIM is over 0.99, and the rendered frame rate is 32 fps. Qualitative experimental results show that this method can be used for high-resolution 3D light field communication.