An efficient inter prediction mode selection scheme for advanced video coding based on motion homogeneity and residual complexity

Abstract

Modern video coding standards such as High Efficiency Video Coding (HEVC) and H.264/MPEG‐4 advanced video coding (AVC) supersede the previous coding standards because of their improved coding efficiency. These standards adopt variable block sizes in frame coding ranging from 4 × 4 to 16 × 16 and 4 × 4 to 64 × 64 for H.264/AVC and HEVC, respectively. The use of variable block sizes for inter prediction provides a significant coding gain compared to coding a macroblock (MB) using regular block size. However, this new feature greatly increases the computational complexity of the encoder when brute‐force rate distortion optimization (RDO) algorithm is used for coding parameter selection. This paper proposes an efficient inter prediction mode selection scheme based on motion homogeneity and residual complexity measures of an MB to speed up the encoding process. The motion homogeneity is assessed through the normalized motion vector (MV) field, and residual complexity is evaluated by the sum of absolute difference (SAD). To acquire the MVs and SADs, motion estimation at 8 × 8 block size is performed using a lightweight recursive motion estimator in which the vector field tends toward true object motion. Based on motion homogeneity and residual complexity of an MB, only a small number of inter prediction modes are selected for the RDO process. The experimental results for H.264/AVC show that the proposed scheme reduces the encoding time by 64% on average without any significant degradation of coding efficiency. © 2016 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.