Multiresolution Motion Estimation Research Articles

Three-level pipelined multi-resolution integer motion estimation engine with optimized reference data sharing search for AVS

Integer motion estimation (IME), which acts as a key component in video encoder, is to remove temporal redundancies by searching the best integer motion vectors for dynamic partition blocks in a macro-block (MB). Huge memory bandwidth requirements and unbearable computational resource demanding are two key bottlenecks in IME engine design, especially for large search window (SW) cases. In this paper, a three-level pipelined VLSI architecture design is proposed, where efficiently integrates the reference data sharing search (RDSS) into multi-resolution motion estimation algorithm (MMEA). First, a hardware-friendly MMEA algorithm is mapped into three-level pipelined architecture with neglected coding quality loss. Second, sub-sampled RDSS coupled with Level C + are adopted to reduce on-chip memory and bandwidth at the coarsest and middle level. Data sharing between IME and fractional motion estimation (FME) is achieved by loading only a local predictive SW at the finest level. Finally, the three levels are parallelized and pipelined to guarantee the gradual refinement of MMEA and the hardware utilization. Experimental results show that the proposed architecture can reach a good balance among complexity, on-chip memory, bandwidth, and the data flow regularity. Only 320 processing elements (PE) within 550 cycles are required for IME search, where the SW is set to 256 × 256. Our architecture can achieve 1080P@30 fps real-time processing at the working frequency of 134.6 MHz, with 135 K gates and 8.93 KB on-chip memory.

Nonlinear Transform for Robust Dense Block-Based Motion Estimation

We present a noniterative multiresolution motion estimation strategy, involving block-based comparisons in each detail band of a Laplacian pyramid. A novel matching score is developed and analyzed. The proposed matching score is based on a class of nonlinear transformations of Laplacian detail bands, yielding 1-bit or 2-bit representations. The matching score is evaluated in a dense full-search motion estimation setting, with synthetic video frames and an optical flow data set. Together with a strategy for combining the matching scores across resolutions, the proposed method is shown to produce smoother and more robust estimates than mean square error (MSE) in each detail band and combined. It tolerates more of nontranslational motion, such as rotation, validating the analysis, while providing much better localization of the motion discontinuities. We also provide an efficient implementation of the motion estimation strategy and show that the computational complexity of the approach is closely related to the traditional MSE block-based full-search motion estimation procedure.

Hierarchical Coding Structure for Video Coding and its Applicability in Scalable Video Coding

With the advancement in the video compression and internet technology, the application of video streaming has increased. Because of different types of devices used for accessing the video application and the heterogeneous nature of the network the scalable video coding has become important in order to fulfill the needs of the user. Discrete wavelet transform (DWT) is best tool for scalable video coding because of its property of multiresolution. The coding and display order of pictures in H.264/MPEG4-AVC is completely decoupled and any picture can be marked as reference picture and is used for prediction of following picture independent of corresponding slice type. A wavelet based hierarchical coding structure has been proposed which selects best matched reference frame for the current frame to be encoded based on the mean square error (MSE) in the group of picture(GOP). The scalable encoding of hierarchical prediction has also been exploited. This algorithm has been compared with the performance of traditional multiresolution motion estimation technique based on peak signal to noise ratio. The proposed algorithm show an improvement of 13% in PSNR value over the traditional MRME technique.

Performance of Wavelet-based Multiresolution Motion Estimation for Inbetweening in Old Animated Films

This paper investigates the performance of wavelet-based multiresolution motion estimation (MRME) for inbetweening in old animated films using three different MRME schemes. The three schemes are: coarse-to fine with a wavelet-based MRME, one of Zhang’s MRMEs, and an MRME in the spatial domain. In order to make a performance comparison of these MRME schemes, two video sequences were used for a simulation. The experimental results show that the coarse-to-fine method performed better than Zhang’s MRME and the MRME in the spatial domain. The evaluation results on block size 9x9 indicate that the coarse-to-fine method had an average peak signal-to-noise ratio (PSNR) of 23.48 dB for the first sequence and 29.84 for the second sequence.

A Hardware-Efficient Multi-Resolution Block Matching Algorithm and its VLSI Architecture for High Definition MPEG-Like Video Encoders

High throughput, heavy bandwidth requirement, huge on-chip memory consumption, and complex data flow control are major challenges in high definition integer motion estimation hardware implementation. This paper proposes an efficient very large scale integration architecture for integer multi-resolution motion estimation based on optimized algorithm. There are three major contributions in this paper. First, this paper proposes a hardware friendly multi-resolution motion estimation algorithm well-suited for high definition video encoder. Second, parallel processing element (PE) array structure is proposed to implement three-level hierarchical motion estimation, only 256PEs are enough for one reference frame real-time high definition motion estimation by efficient PE reuse. Third, efficient on-chip reference pixel buffer sharing mechanism between integer and fractional motion estimation is proposed with almost 50% SRAM saving and memory bandwidth reduction. The proposed multi-resolution motion estimation algorithm reached a good balance between complexity and performance with rate distortion optimized variable block size motion estimation support. Also, we have achieved moderate logic circuit and on-chip SRAM consumption. The proposed architecture is well-suited for all MPEG-like video coding standards such as H.264, audio video coding standard, and VC-1.

Multi-resolution motion estimation and compensation based on adjacent prediction of frame difference in wavelet domain

Aiming at the higher bit-rate occupation of motion vector encoding and more time load of full-searching strategies, a multi-resolution motion estimation and compensation algorithm based on adjacent prediction of frame difference was proposed. Differential motion detection was employed to image sequences and proper threshold was adopted to identify the connected region. Then the motion region was extracted to carry out motion estimation and motion compensation on it. The experiment results show that the encoding efficiency of motion vector is promoted, the complexity of motion estimation is reduced and the quality of the reconstruction image at the same bit-rate as Multi-Resolution Motion Estimation (MRME) is improved.

Improved algorithm on fast multi-resolution motion estimation

In this paper,an improved algorithm on fast multi-resolution motion estimation was proposed,which made use of the multi-resolution property and wavelet matching error characteristic.Based on Partial Distortion Elimination(PDE) algorithm,we improved the speed of motion estimation by improving searching order,matching order and comparison interval.Experimental results show that the proposed algorithm can achieve the same estimate accuracy as Full Search Algorithm(FSA),while the computation complexity is reduced extremely.

Fast multiresolution motion estimation algorithms for wavelet-based scalable video coding

Motion estimation and compensation in wavelet domain have received much attention recently. To overcome the inefficiency of motion estimation in critically sampled wavelet domain, the low-band-shift (LBS) method and the complete-to-overcomplete discrete wavelet transform (CODWT) method are proposed for motion estimation in shift-invariant wavelet domain. However, a major disadvantage of these methods is the computational complexity. Although the CODWT method has reduced the computational complexity by skipping the inverse wavelet transform and making the direct link between the critically sampled subbands and the shift-invariant subbands, the full search algorithm (FSA) increases it. In this paper, we proposed two fast multiresolution motion estimation algorithms in shift-invariant wavelet domain: one is the wavelet matching error characteristic based partial distortion search (WMEC-PDS) algorithm, which improves computational efficiency of conventional partial distortion search algorithms while keeping the same estimate accuracy as the FSA; another is the anisotropic double cross search (ADCS) algorithm using multiresolution-spatio-temporal context, which provides a significantly computational load reduction while only introducing negligible distortion compared with the FSA. Due to the multiresolution nature, both the proposed approaches can be applied to wavelet-based scalable video coding. Experimental results show the superiority of the proposed fast motion estimation algorithms against other fast algorithms in terms of speed-up and quality.

A Nonrigid Registration of MR Breast Images Using Complex-valued Wavelet Transform

In this paper, a fast, slice-by-slice, nonrigid registration algorithm of dynamic magnetic resonance breast images is presented. The method is based on a multiresolution motion estimation of the breast using complex discrete wavelet transform (CDWT): the pyramid of oriented complex subimages is used to implement a hierarchical phase-matching-based motion estimation algorithm. The resulting motion estimate is nonrigid and pixel-independent. To assess the method performance, we computed the correlation coefficient and the normalized mutual information between pre- and postcontrast images with and without realignment. The indices increased after using our approach and the improvement was superior to rigid or affine registration. A set of clinical scores was also evaluated. The clinical validation demonstrated an increased readability in the subtraction images. In particular, CDWT registration allowed a best definition of breast and lesion borders and greater detail detectability.

DESIGN AND IMPLEMENTATION OF WAVELET-DOMAIN VIDEO COMPRESSION USING MULTIRESOLUTION MOTION ESTIMATION AND COMPENSATION

Video compression techniques have been applied routinely in order to conserve storage space and minimize bandwidth utilization in various video applications. To reduce inter-pixel redundancies inside and between video frames, video codecs (coder/decoder or compression/decompression) are mainly characterized by applying motion estimation and motion compensation (MEMC) in combination with discrete cosine transform (DCT). However, blocking artifacts are obvious from the block-based motion estimation and transformation, especially at low bit rates. Due to the intrinsic advantages of multiresolution and scalability of discrete wavelet transform (DWT), video compression techniques have been infused with exciting prospects by the beneficial integration of motion estimation and wavelet transformation. The contributions of this paper are in (1) proposing a design architecture of the wavelet-domain encoder and decoder; (2) implementing the multiresolution motion estimation and compensation method; and in (3) realizing the dynamic entropy encoding and decoding, so as to achieve more efficient video compression than the conventional spatial domain methods for low bit rate applications without the annoying blocking artifacts.

Comparison of wavelets for multiresolution motion estimation

The performance of various wavelets, including those known to be well suited for the coding of still images, has been evaluated for the multiresolution motion estimation of video sequences. The multiresolution motion estimation scheme proposed by Zhang and Zafar, which has been widely cited in the literature, is used as the simulation scheme in this study. In our study, the prediction mean square error in the wavelet transform coefficient domain is used as the measure for prediction performance. In order to show the overall rate distortion performance, the number of bits needed to encode the motion vectors is also calculated. Simulation results show that the 7/9 biorthogonal wavelet, one of the best wavelets for the coding of still images, is the best wavelet for the task of multiresolution motion estimation among the wavelets evaluated in this study.

Wavelet Video Denoising with Regularized Multiresolution Motion Estimation

This paper develops a new approach to video denoising, in which motion estimation/compensation, temporal filtering, and spatial smoothing are all undertaken in the wavelet domain. The key to making this possible is the use of a shift-invariant, overcomplete wavelet transform, which allows motion between image frames to be manifested as an equivalent motion of coefficients in the wavelet domain. Our focus is on minimizing spatial blurring, restricting to temporal filtering when motion estimates are reliable, and spatially shrinking only insignificant coefficients when the motion is unreliable. Tests on standard video sequences show that our results yield comparable PSNR to the state of the art in the literature, but with considerably improved preservation of fine spatial details.

A multiresolution motion estimation technique with indexing

In the multiresolution motion estimation (MRME) techniques originally proposed by Zhang and Zafar, four MRME algorithms have been proposed. In one of these algorithms, the motion vectors in the low-pass subband are properly scaled and used as the final motion vectors for all the other subbands, and, in another, the properly scaled motion vectors in the first algorithm are used as predictions and further refined. The former algorithm requires a much lighter computational load and fewer coding bits for the motion vectors than the latter; on the other hand, the latter is able to provide a better MRME performance than the former. In this paper, we propose a new MRME technique that takes advantage of both of the above algorithms. In the proposed algorithm, the sum of absolute difference associated with each of the scaled motion vectors as in the first algorithm is calculated, and the result compared with the sum of the absolute values of the amplitudes of the wavelet coefficients within the motion block to be compensated. The outcome of the comparison decides if these scaled motion vectors are accepted as the final ones. For the coding of motion information, the motion vectors used for the prediction and their patterns of applicability to higher resolution levels, called the indices, are coded.

Fast motion search with efficient inter-prediction mode decision for H.264

A fast inter-prediction mode decision and motion search algorithm is proposed for the H.264 video coding standard. The multi-resolution motion estimation scheme and an adaptive rate-distortion model are employed with early termination rules to accelerate the search process. With the new algorithm, the amount of computation involved in the motion search can be substantially reduced. Experimental results show that the proposed algorithm can achieve a speed-up factor ranging from 60 to 150 times as compared to the full-search algorithm with little quality degradation.

Multiresolution Motion Estimation for Low-Rate Video Frame Interpolation

Interpolation of video frames with the purpose of increasing the frame rate requires the estimation of motion in the image so as to interpolate pixels along the path of the objects. In this paper, the specific challenges of low-rate video frame interpolation are illustrated by choosing one well-performing algorithm for high-frame-rate interpolation (Castango 1996) and applying it to low frame rates. The degradation of performance is illustrated by comparing the original algorithm, the algorithm adapted to low frame rate, and simple averaging. To overcome the particular challenges of low-frame-rate interpolation, two algorithms based on multiresolution motion estimation are developed and compared on objective and subjective basis and shown to provide an elegant solution to the specific challenges of low-frame-rate video interpolation.

Multi-Resolution Block Matching Algorithm and Its VLSI Architecture for Fast Motion Estimation in an MPEG-2 Video Encoder

This paper proposes a high-performance multi-resolution motion estimation algorithm (HMRME) for MPEG-2 video encoding, which satisfies high estimation performance and efficient very large scale integration (VLSI) implementation. HMRME is based on a characteristic that field motion vectors (MVs) are very similar to their corresponding frame MV. Firstly, HMRME performs frame-based motion estimation (ME) as follows: at the coarsest level, two MV candidates are found on the basis of minimum matching error. The two MV candidates from the coarsest level search and the other one based on spatial MV correlation are used as center points for three local searches at the middle level. At the finest level, a frame MV is obtained from a local search around a single candidate from the middle level search. Field MVs are estimated with the single MV candidate from the middle level search of frame ME as initial estimates at the finest level, without any coarser level searches. This paper also describes a VLSI architecture based on HMRME. This architecture is designed to provide a good tradeoff between on-chip memory size and I/O bandwidth with high throughput. We implemented this architecture with about 140 K gates and 2.5 K bytes static random access memory for a large search range of [-192.0, +191.5] by using a synthesizable Verilog HDL.

신뢰성있는 웨이블릿 비디오 전송을 위한 패킷화 기법

Since Wavelet Transform decomposes a video frame into subbands with various frequencies and resolutions, the reconstructed video qualify at a receiver fluctuates according to the location of transmission errors within frames. This deteriorates the whole visual duality of the video. Specifically, for a wavelet based video which exploits the motion estimation prediction scheme, the transmission errors of a subband not only have a bad effect on other subbands within a same frame but also propagates to the subsequent frames. In this paper, we propose BDP(Block Based Dispersive Packetization) scheme, for a wavelet based video stream, which maintains constant video quality despite packet location that a transmission error occurs. To evaluate the performance of the proposed scheme, we use MRME(Multi-Resolution Motion Estimation) scheme to compress a video in Inter coding mode and Gilbert´s error model to generate the error patterns in wireless network environment. The simulation results show that BDP is more efficient than BP (Block based Packetization) or DP (Dispersive Packetization) in both PSNR and visual quality.

ACTIVE CONTOUR MODEL USING WAVELET MODULUS FOR OBJECT SEGMENTATION AND TRACKING IN VIDEO SEQUENCES

We propose an integrated wavelet-based framework of the active contour model (snake) for segmentation and motion tracking of deformable objects in video sequences. The input image frame is represented by means of wavelet transform. First, the moduli of wavelet transform coefficients are used in a multi-resolution motion estimation process to find the initial snake contour in the current frame. The presented multi-resolution motion estimation methods allow a larger movement of the tracked object than does traditional image-based motion estimation. Secondly, the wavelet transform modulus at each scale is considered in the energy function of the snake model. The snake computation is based on a coarse-to-fine scale continuation method. Application of the proposed methods to biological cell tracking is demonstrated in experiments.

New techniques for multi-resolution motion estimation

We investigate three new methods to predict motion vectors (MVs) across subbands for multi-resolution motion estimation (MRME): linear prediction, median filtering (MF), and multi-candidate techniques. Compared to the conventional MRME techniques, the proposed linear-prediction-based and the MF-based techniques effectively overcome the problem of propagation of false MVs. A significant feature of these two techniques is that they impose no demand for additional bandwidth, and simulation studies show that they not only improve prediction performance, but also reduce the number of bits needed to encode the motion information. It is further shown that the improvement in the performance thus achieved involves little increase in computational complexity.

Methods cooperation for multiresolution motion estimation

For a medical application, we are interested in an estimation of optical flow on a patient's face, particularly around the eyes. Among the methods of optical flow estimation, gradient estimation and block matching are the main methods. However, the gradient-based approach can only be applied for small displacements (one or two pixels). Gener- ally, the process of block matching leads to good results only if the searching strategy is judiciously selected. Our approach is based on a Markov random field model, combined with an algorithm of block match- ing in a multiresolution scheme. The multiresolution approach allows de- tection of a large range of speeds. The large displacements are detected on coarse scales and small displacements are detected successively on finer scales in a coarse to fine strategy. The Markov random fields allow the initialization and control of motion estimation across all scales. The tracking of motion is achieved by a block matching algorithm. This method gives the optical flow, whatever the amplitude of motion is, if pertaining to the range defined by the multiresolution approach. The re- sults clearly show the complement of Markov random field estimation and block matching across the scales. © 2002 Society of Photo-Optical Instru-