A Deep Learning Approach in Scalable High Efficiency Video Coding for Fast Coding Unit Size Decision

Abstract

The High Efficiency Video Coding (HEVC) with scalable extension known as SHVC allows encoding the same video with different resolutions in a single bitstream. However, this process increases the encoding complexity. The complexity is increased mainly due to the motion estimation process and the Rate-Distortion Optimization (RDO) search process. The Test Zonal (TZ) algorithm with fast search patterns helps to accelerate the motion estimation process. However, the search patterns may get trapped to local minima, leading to inaccurate motion vectors. Moreover, the RDO search process used to determine the Coding Unit (CU) size increases the complexity. We proposed the Horizontal Subsampling Motion Estimation (HSME) method to find the accurate motion vectors with reduced complexity. The experimental results prove that the HSME method saves the encoding time by 53.03% with a 6.45% increase in Bjøntegaard delta bit rate and 0.28 dB loss in Bjøntegaard Delta Peak Signal-to-Noise Ratio (BD-PSNR) compared to standard SHM-12.1. In addition, we designed the Early Terminated Long- and Short-Term Memory (ET-LSTM) network that predicts the CU partition by taking the output features of the Early Terminated Convolutional Neural Network (ET-CNN). The ET-CNN learns the CU partitions from the residual Coding Tree Unit (CTU) using the deep learning approach. Our proposed method (HSME + ET-CNN + ET-LSTM) achieves 53% savings in encoding time, which is significantly higher than state-of-the-art methods.