SHVC CU Processing Aided by a Feedforward Neural Network

Abstract

The development of multimedia and hardware technologies has led to a great number of industrial video applications, such as virtual reality, high-definition video surveillance, and remote monitoring. As complex communication environments and heterogeneous networks are common in industrial applications, industrial videos are required to support a diverse range of display resolutions and transmission channel capacities. Scalable high-efficiency video coding (SHVC) standards provide the tools to meet this requirement. However, it is highly computationally expensive. Coding complexity has a great impact on SHVC performance in industrial applications. Many of these applications are sensitive to time delay and have limited power. Thus, improvements are required to ensure the practical usability of SHVC encoders. In SHVC encoders, intra/interprediction of variable coding unit (CU) sizes is independently performed for the base and enhancement layers (ELs). There are many interlayer similarities that can be exploited to speed up the procedure for EL coding. In this paper, we propose a feedforward neural network aided model for CU size and mode decisions for SHVC, which utilizes base layer coding information and the coding data of spatiotemporal neighboring CUs to decide which CU sizes or prediction modes can be bypassed for certain EL CUs. Two feedforward neural network based learning models are built for CU classification, which are introduced in the procedures for CU size and mode decisions, respectively. According to the analysis from a large number of video sequences, the representative features are directly extracted from the coding information of previously coded neighboring CUs to avoid computational overheads. After the training is finished, these two models are designed and integrated to build classifiers. Then, two online classification approaches are designed for the CU size and mode decision procedures to classify each CU's type. Finally, different candidate CU sizes and prediction modes are adaptively assigned for each type of CU. This approach outperforms the state-of-the-art fast SHVC/high-efficiency video coding (HEVC) algorithms with approximately 19–42% coding time savings or better compression efficiency, which will be beneficial for the realization of real-time scalable video coding.