Deep Intra Prediction by Jointly Exploiting Local and Non-Local Similarities

Abstract

Intra prediction, which aims to remove the redundancies within a frame, has shown promising performance by simply projecting and interpolating samples along multiple angular directions. Recently, with numerous approaches devoted to learning nonlinear predictors with deep neural networks (DNN) based on local correlations, much less work has been dedicated to exploring non-local self-similarities in intra prediction. In this paper, we propose a unified prediction model that exploits both local and non-local correlations for intra prediction. The proposed model not only supports the nonlinear prediction using local reference samples as input, but also aggregates useful non-local information from a large reconstructed region with a Patch-level Non-local Attention Network (PNA-Net). More specifically, PNA-Net incorporates template matching with attention mechanism in feature domain to obtain the responses of all non-local features to the content to be predicted, leading to the prediction produced with weighted non-local patches. Finally, the predictions in the local and non-local manners are blended adaptively with a trainable network, ensuring the capability to handle a variety of contents. Experimental results on Versatile Video Coding (VVC) software VTM-11.0 show that the proposed model achieves on average 4.69% bit rate savings for natural scene sequences, and 4.24% bit rate savings for screen content sequences under the all intra configuration.