Lenslet Image Coding With SAIs Synthesis via 3D CNNs-Based Reinforcement Learning With a Rate Reward

Abstract

The deep learning-based coding schemes for lenslet images combine coding standards and view synthesis through Deep Learning (DL) models, where the compression efficiency is heavily influenced by the coding structure and quality of synthesized views. To exploit the inter-view redundancy among Sub-Aperture Images (SAIs), this paper proposes a hybrid closed-loop coding system that uses a novel coding structure based on checkerboard interleaving at a frame level. The frame-wise checkerboard interleaving method partitions an Original SAIs’ Set (OSS) of images into two mutually exclusive subsets, each consisting of alternating rows and columns of SAIs. We utilize the video coding standard Versatile Video Coding (VVC) to encode one subset while proposing a novel rate constraint-reinforced 3D Convolutional Neural Networks (CNNs) to predict the other subset, referred to as the complement subset. The rate constraint-reinforced 3D CNNs is newly designed with a gradient loss and reinforced rate cost to improve synthesized SAIs’ image quality and bit cost saving simultaneously. Experimental results on the light field image dataset demonstrate that the proposed hybrid coding system outperforms both HEVC_LDP and the previous state-of-the-art (SOTA), achieving an average BD-Bitrate savings of 41.58% and 23.31%, respectively.