Multiple Reference Frame Motion Estimation Research Articles

Temporal correlation based general search strategy for multiple reference frame motion estimation

Multiple reference frame motion estimation (MRFME) is one of the main features of recent video coding standards such as H.264/AVC and high efficiency video coding. This paper presents a fast search strategy for avoiding redundant computations during MRFME. The proposed method reduces the number of pixels to be examined using the precalculated rate-distortion costs of previously searched frames. Experimental results show that the proposed method reduces the complexity of the motion estimation (ME) process by up to 81.43 %.

The Research Based on H.264 Embedded Video Systems of Optimized Compression Algorithm

Analyzes the basic principles of H.264 video compression algorithm, such as integer DCT, multiple reference frame motion estimation, intra prediction, inter prediction; describes the characteristics of embedded systems, propose a feasible method of video signal optimization, by comparing with the experimental results, the influence of the H.264 coding bit rate bottleneck optimization function obtains a good result. These methods and ideas have practical significance, and it provides a strong reference for other development of video systems.

Level-set-based motion estimation algorithm for multiple reference frame motion estimation

Motion estimation (ME) has a variety of applications in image processing, pattern recognition, target tracking, and video compression. In modern video compression standards such as H.264/AVC and HEVC, multiple reference frame ME (MRFME) is adopted to reduce the temporal redundancy between successive frames in a video sequence. In MRFME, the motion search process is conducted using additional reference frames, thereby obtaining better prediction signal as compared to single reference frame ME (SRFME). However, its high computational complexity makes it difficult to be utilized in real-world applications. In order to reduce the computational complexity of MRFME, this paper proposes a level-set-based ME algorithm (LSME) without any penalty in the rate-distortion (RD) performance. First, the proposed algorithm partitions the motion search space into multiple level sets based on a rate constraint. The proposed algorithm then controls the ME process on the basis of the predetermined level sets. Experimental results show that the proposed algorithm reduces the ME time by up to 83.46% as compared to the conventional full search (FS) algorithm.

Multiple weighted prediction models for video coding with brightness variations

Weighted prediction (WP) is a tool introduced in H.264 to improve motion-compensation performance for video sequences with brightness variations. Various WP models to estimate the parameter set have been discussed in the literature. However, no single WP model works well for all types of brightness variations. A single reference frame multiple WP models (SRefMWP) scheme is proposed to facilitate the use of multiple WP models in a single reference frame. The proposed scheme makes a new arrangement of the multiple frame buffers in multiple reference frame motion estimation. It enables different macroblocks in the same frame to use different WP models even when they are predicted from the same reference frame. Furthermore, a new re-ordering mechanism for SRefMWP is also proposed to guarantee that the list of the reference picture is in the best order for further decreasing the bit rate. To reduce the implementation cost, a reduction in the memory requirement is achieved via look-up tables (LUTs). Experimental results show that SRefMWP can achieve significant coding gain in scenes with different types of brightness variations. Furthermore, SRefMWP with LUTs can reduce the memory requirement by about 80% while keeping the same coding efficiency as SRefMWP.

A low energy adaptive hardware for H.264 multiple reference frame motion estimation

Multiple reference frame motion estimation (MRF ME) increases the video coding efficiency at the expense of increased computational complexity and energy consumption. Therefore, in this paper, a low complexity H.264 MRF ME algorithm and a low energy adaptive hardware for its real-time implementation are proposed. The proposed MRF ME algorithm reduces the computational complexity of MRF ME by using a dynamically determined number of reference frames for each Macroblock (MB) and early termination. The proposed H.264 MRF ME hardware is implemented in Verilog HDL. The proposed H.264 MRF ME hardware has 29-72% less energy consumption than an H.264 MRF ME hardware using 5 reference frames for all MBs with a negligible PSNR loss. Therefore, it can be used in consumer electronics products that require real-time video processing or compression with low power consumption.

Reliable tracking algorithm for multiple reference frame motion estimation

Multiple reference frame motion estimation (MRF-ME) is one of the most crucial tools in H.264/AVC to improve coding efficiency. However, it disciplines an encoder by giving extra computational complexity. The required computation proportionally expands when the number of reference frames used for motion estimation increases. Aiming to reduce the computational complexity of the encoder, various motion vector (MV) composition algorithms for MRF-ME have been proposed. However, these algorithms only perform well in a limited range of reference frames. The performance deteriorates when motion vector composition is processed from the current frame to a distant reference frame. In this paper, a reliable tracking mechanism for MV composition is proposed by utilizing only the relevant areas in the target macroblock and taking different paths through a novel selection process from a set of candidate motion vectors. The proposed algorithm is especially suited for temporally remote reference frames in MRF-ME. Experimental results show that compared with the existing MV composition algorithms, the proposed one can deliver a remarkable improvement on the rate-distortion performance with similar computational complexity.

An Efficient Priority-Based Reference Frame Selection Method for Fast Motion Estimation in H.264/AVC

The H.264/AVC achieves higher coding performance than the previous video coding standards, such as MPEG-2 and MPEG-4. In order to achieve this improved coding performance, H.264 adopts various advanced techniques including multiple reference frame motion estimation (MRF-ME), which requires very heavy encoding complexity in proportion to the number of reference frames. In this letter, we propose an efficient priority-based reference frame selection (PRFS) method to reduce the computational complexity. The PRFS method computes the priorities of all reference frames using spatial and temporal correlation of the reference frame index and motion vectors, and selects candidate reference frames to reduce the number of reference frames for each ME block. Experimental results show that the proposed method reduces the computational complexity of the MRF-ME and achieves similar coding performance to that of previous methods.

Fast multi-frame motion estimation algorithm for H.264

The multiple reference frame motion compensation was introduced into H.264/AVC to improve video coding performance.However,the resulted Multiple Reference Frame Motion Estimation (MRF-ME) leads to much more computational cost.To improve the encoding speed and reduce the computational complexity,a new algorithm called DDS based on spatial correlation was proposed.First,the algorithm updated predictive motion vector constantly using Forward Dominant Vector Selection (FDVS),then searched different reference frame by different template according to the statistical characteristic of the location of optimal reference frame.The experimental results show that the algorithm can decrease the search points by 80% and reduce the complexity of the encoder considerably while maintaining nearly unchanged Peak Signal-to-Noise Ratio (PSNR) of pictures and the bit rate compared with the H.264 reference software JM10.2.

High performance hardware architectures for one bit transform based single and multiple reference frame motion estimation

Motion Estimation (ME) is the most computationally intensive part of video compression and video enhancement systems. One bit transform (1BT) based ME algorithms have low computational complexity. Therefore, in this paper, we propose high performance systolic hardware architectures for 1BT based fixed block size (FBS) single reference frame (SRF) ME, variable block size (VBS) SRF ME, and multiple reference frame (MRF) ME. The proposed FBS-SRF ME hardware performs full search ME for 4 Macroblocks in parallel and it is faster than the 1BT based ME hardware reported in the literature. In addition, it uses less on-chip memory than the previous 1BT based ME hardware by using a novel data reuse scheme and memory organization. The proposed VBS-SRF ME hardware is also faster and uses less on-chip memory than previous 1BT based VBS-SRF ME hardware. The proposed MRF ME hardware is the first 1BT based MRF ME hardware in the literature. In order to trade-off ME performance and computational complexity, the proposed MRF ME hardware is designed as reconfigurable in order to statically configure the number and selection of reference frames based on the application requirements. The proposed hardware architectures are implemented in Verilog HDL. They are capable of processing 83 1920×1080 full High Definition frames per second. Therefore, they can be used in consumer electronics products that require real-time video processing or compression.

Selective Search Area Reuse Algorithm for Low External Memory Access Motion Estimation

In motion estimation for video codec, reducing the amount of external memory access is critical to reduce power consumption and to minimize performance degradation. Previous search area reuse algorithms to reduce the memory access still suffer from coding efficiency degradation in fast motion video. Previously, we proposed a selective search area reuse (SSAR) algorithm to reduce the amount of external memory access with minimal coding efficiency degradation. In this letter, we extend SSAR algorithm to multiple reference frame motion estimation with a method to utilize multiple on-chip memories. Then, we propose a frame-level dynamic search range algorithm based on the SSAR algorithm. Finally, we propose a memory usage switching method to increase the utilization of the limited-size on-chip memory. Experimental results show that the proposed algorithm with a search range of 16 achieves 28.64-56.24% reduction according to the number of on-chip memories in multiple reference frames. In the results of the Foreman video sequence, our algorithm operating with a fixed-size on-chip memory compensated for quality degradation by up to 2.7 dB in the frames of fast camera motion, and reduced the amount of memory access by 22.6% with a peak signal-to-noise ratio gain of 1 dB in the frames of camera shaking.

Fast reference frame selection algorithm for H.264/AVC

The H.264/AVC video coding standard newly adopted multiple reference frame motion estimation to improve video coding performance. However, these features result in a considerable increase in encoder complexity. Therefore, we propose a fast reference frame selection algorithm based on information from the reference region for a motion estimation process. Experimental results show that the proposed algorithm effectively reduces the complexity of multiple reference frame motion estimation. The reduction in the coding gain is negligible.

Selective VS-MRF-ME and intra coding in H.264 based on spatiotemporal continuity of motion field

The new video coding standard, H.264 uses variable size motion estimation (VS-ME), multiple reference frame motion estimation (MRF-ME) and spatial-based intra prediction with selectable block size in inter frame coding. These tools have achieved significant coding efficiency compared to coding a macroblock (MB) only based on motion-compensation in regular size with single reference frame. However, these new features also give rise to an exhaustive computation in the coding procedure since there are so many combinations of coding modes and reference frames to be tried. In this paper, a fast motion estimation algorithm based on the selective VS-MRF-ME and intra prediction is proposed to reduce H.264 coding computational complexity. The basic idea of the method is to utilize the spatiotemporal property of motion field in predicting where VS-MRF-ME and intra prediction are needed, and only in these regions VS-MRF-ME and intra coding are enabled. The motion field is generated by motion vectors from 16×16 motion estimation on the nearest reference frame. Simulation results show that the proposed algorithm can save 50% computational complexity on average, with negligible loss of coding efficiency.

Motion Feature and Hadamard Coefficient-Based Fast Multiple Reference Frame Motion Estimation for H.264

In the state-of-the-art video coding standard, H.264/AVC, the encoder is allowed to search for its prediction signals among a large number of reference pictures that have been decoded and stored in the decoder to enhance its coding efficiency. Therefore, the computation complexity of the motion estimation (ME) increases linearly with the number of reference picture. Many fast multiple reference frame ME algorithms have been proposed, whose performance, however, will be considerably degraded in the hardwired encoder design due to the macroblock (MB) pipelining architecture. Considering the limitations of the traditional four-stage MB pipelining architecture, two fast multiple reference frame ME algorithms are proposed here. First, on the basis of mathematical analysis, which reveals that the efficiency of multiple reference frames will be degraded by the relative motion between the camera and the objects, for the slow-moving MB, the authors adopt the multiple reference frames but reduce their search range. On the other hand, for the fast-moving MB, the first previous reference frame is used with the full search range during the ME processing. The mutually exclusive feature between the large search range and the multiple reference frames makes the computation saving performance of the proposed algorithm insensitive to the nature of video sequence. Second, following the Hadamard transform coefficient-based all_zeros block early detection algorithm, two early termination criteria are proposed. These methods ensure the pronounced computation saving efficiency when the encoded video has strong spatial homogeneity or temporal stationarity. Experimental results show that 72.7%-93.7% computation can be saved by the proposed fast algorithms with an average of 0.0899 dB coding quality degradation. Moreover, these fast algorithms can be combined with fast block matching algorithms to further improve their speedup performance.

A fast feature-assisted adaptive early termination approach for multiple reference frames motion estimation in H.264

The multiple reference frames motion estimation approach used in H.264 is computationally intensive. This paper presents a fast or computationally efficient feature-assisted adaptive early termination approach in order to reduce the computational complexity while maintaining more or less the same video quality. The introduced feature-assisted approach consists of three parts: (1) reduction of the number of available reference frames using predicted motion activity, extracted texture information, and skip mode from neighboring macroblocks, (2) the most probable reference frame prediction based on neighboring macroblocks, and (3) an adaptive early termination threshold derived from a theoretical analysis of all zero block detection. Extensive experimental results are performed to demonstrate the computational gain of the introduced approach over the standard approach for the multiple reference frames motion estimation.

Single Reference Frame Multiple Current Macroblocks Scheme for Multiple Reference Frame Motion Estimation in H.264/AVC

Due to the multiple reference frame motion estimation (MRF-ME), an H.264/AVC encoder requires ultrahigh memory bandwidth. Conventional multiple reference frames single current macroblock (MRSC) scheme only considers the data reuse within one frame, and requires on-chip memory size and off-chip memory bandwidth in proportional to the reference frame number. In this paper, a single reference frame multiple current macroblocks (SRMC) scheme is presented to further exploit the data reuse at frame level. With frame-level rescheduling of the motion estimation ME procedures in different reference frames, one loaded search window can be utilized by multiple current MBs in different original frames. The demanded on-chip memory size and off-chip memory bandwidth for MRF-ME can thus be reduced to those supporting only one reference frame. Moreover, based on SRMC scheme, an architecture prototype with two-stage mode decision flow is proposed. For HDTV specifications, 62.21 KB (74.8%) of SRAM and 364.3 MB/s (62.6%) of system bandwidth are saved in comparison with the MRSC scheme

Data reuse algorithm for multiple reference frame motion estimation

A data reuse algorithm for multiple reference frame motion estimation is described. The proposed algorithm reduces memory access by modifying the reference frame search order and search centre such that the likelihood of data reuse is increased. Experimental results show that the algorithm reduces memory access by 15–30% compared to the conventional fast reference frame selection algorithm, while maintaining similar bit rate and PSNR.

Analysis and complexity reduction of multiple reference frames motion estimation in H.264/AVC

In the new video coding standard H.264/AVC, motion estimation (ME) is allowed to search multiple reference frames. Therefore, the required computation is highly increased, and it is in proportion to the number of searched reference frames. However, the reduction in prediction residues is mostly dependent on the nature of sequences, not on the number of searched frames. Sometimes the prediction residues can be greatly reduced, but frequently a lot of computation is wasted without achieving any better coding performance. In this paper, we propose a context-based adaptive method to speed up the multiple reference frames ME. Statistical analysis is first applied to the available information for each macroblock (MB) after intra-prediction and inter-prediction from the previous frame. Context-based adaptive criteria are then derived to determine whether it is necessary to search more reference frames. The reference frame selection criteria are related to selected MB modes, inter-prediction residues, intra-prediction residues, motion vectors of subpartitioned blocks, and quantization parameters. Many available standard video sequences are tested as examples. The simulation results show that the proposed algorithm can maintain competitively the same video quality as exhaustive search of multiple reference frames. Meanwhile, 76 %-96 % of computation for searching unnecessary reference frames can be avoided. Moreover, our fast reference frame selection is orthogonal to conventional fast block matching algorithms, and they can be easily combined to achieve further efficient implementations.

Fast Multiple Reference Frame Selection Method Using Correlation of Sequence in JVT/H.264

H.264 video coding standard has a significant performance better than the other standards are the adoption of variable block sizes, multiple reference frames, and the consideration of rate distortion optimization within the codec. However, these features incur a considerable complexity in the encoder for motion estimation. As for the multiple reference frames motion estimation, the increased computation is in proportion to the number of searched reference frames. In this paper, a fast multiple frame reference frames selection method is proposed for H.264 video coding. The proposed algorithm can efficiently determine the best reference frame from the allowed five reference frames. As determine the number of reference frames to search the motion using the correlation of the different block between the block of current frame and that of previous frame, this scheme can efficiently reduce the computational cost while keeping the similar quality and bit-rate. Simulation results show that the speed of the proposed method is faster than that of the original scheme adapted in JVT reference software JM95 while keeping the similar video quality and bit-rate.

Fast multiple reference frame motion estimation for H.264/AVC

Multiple reference frame motion compensation is a new feature introduced in H.264/MPEG-4 AVC to improve video coding performance. However, the computational cost of multiple reference frame motion estimation (MRF-ME) is very high. In this paper, we propose an algorithm that takes into account the correlation/continuity of motion vectors among different reference frames. We show that the algorithm effectively reduces the computations of MRF-ME, and achieves similar coding gain compared to the motion search approaches in the reference software.

Adaptive Search Range Decision and Early Termination for Multiple Reference Frame Motion Estimation for H.264

The newest video coding standard called H.264 provides considerable performance improvement over a wide range of bit rates and video resolutions compared to previous standards. However, these features result in an extraordinary increase in encoder complexity, mainly regarding to mode decision and multiple reference frame motion estimation (ME). This letter presents two algorithms to reduce the computational complexity caused by motion estimation. The adaptive search range decision method determines the search range size according to the motion vector predictor dynamically and the early termination scheme defines a criterion to early terminate the search processing for multiple reference frames. Experimental results show that the proposed algorithms provide significant improvement of coding speed with negligible objective quality degradation compared to the fast motion estimation algorithms adopted by reference software.