Motion Vector Coding Research Articles

Motion Vector Coding and Block Merging in the Versatile Video Coding Standard

This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.

A video steganalysis method based on coding cost variation

Aiming at the problems existing in existing steganalysis algorithms, this article proposes Motion Vector Coding Cost Change video steganalysis features based on Improved Motion Vector Reversion-Based features and Subtractive Probability of Coding Cost Optimal Matching features based on Subtractive Probability of Optimal Matching features from the perspective of the change of coding cost. Motion Vector Coding Cost Change features can be well consistent with the coding cost before recoding by analyzing the sub-pixel coding cost of recoding. By counting the sub-pixel coding costs of motion vectors before and after video recoding, the Sum of Absolute Difference values of motion vectors instead of predicted residuals are applied to steganalysis and detection, and the steganographic algorithm based on motion vectors is effectively detected. Experiments show that Motion Vector Coding Cost Change features have higher detection accuracy than Add-or-Subtract-One, Improved Motion Vector Reversion-Based, and other typical features in various steganography methods, and Subtractive Probability of Coding Cost Optimal Matching features have higher detection effect and better robustness than Subtractive Probability of Optimal Matching features.

Tools for Video Coding Beyond HEVC: Flexible Partitioning, Motion Vector Coding, Luma Adaptive Quantization, and Improved Deblocking

This paper provides a description and analysis of technology included in two contributions to the Call for Proposals for Video Compression with Capability beyond HEVC. This Call for Proposals was issued jointly by the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG). The contribution emphasized a flexible and rectangular partitioning structure, which was combined with both new and existing coding tools. New coding tools included methods for improved motion vector coding, quantization signaling, and deblocking; while existing tools were largely methods studied in the Joint Exploration Model software. Results show the efficacy of the approach. Using the sequences and test conditions defined in the Call for Proposals, the described approach provided an average bitrate reduction, relative to an HEVC anchor, of 41.2% and 35.7% for 4K and HD test sequences, respectively. Moreover, the method achieved a compression performance of 34.3% and 32.2% for high dynamic range content using the perceptual quantizer (PQ) and Hybrid-Log Gamma transfer functions, respectively.

A Consideration of A High-Speed Video Identification Method Using Histogram Information of Coded Motion Vectors

A Consideration of A High-Speed Video Identification Method Using Histogram Information of Coded Motion Vectors

Inter-frame Prediction with Fast Weighted Low-rank Matrix Approximation

Abstract In the field of video coding, inter-frame prediction plays an important role in improving compression efficiency. The improved efficiency is achieved by finding predictors for video blocks such that the residual data can be close to zero as much as possible. For recent video coding standards, motion vectors are required for a decoder to locate the predictors during video reconstruction. Block matching algorithms are usually utilized in the stage of motion estimation to find such motion vectors. For decoder-side motion derivation, proper templates are defined and template matching algorithms are used to produce a predictor for each block such that the overhead of embedding coded motion vectors in bit-stream can be avoided. However, the conventional criteria of either block matching or template matching algorithms may lead to the generation of worse predictors. To enhance coding efficiency, a fast weighted low-rank matrix approximation approach to deriving decoder-side motion vectors for inter frame video coding is proposed in this paper. The proposed method first finds the dominating block candidates and their corresponding importance factors. Then, finding a predictor for each block is treated as a weighted low-rank matrix approximation problem, which is solved by the proposed column-repetition approach. Together with mode decision, the coder can switch to a better mode between the motion compensation by using either block matching or the proposed template matching scheme.

Advanced motion vector coding framework for multiview video sequences

The inconsistency between the temporal motion vector and the disparity vector fields causes a drastic loss in coding performance during multiview video coding. In order to resolve this inconsistency problem, we investigate a new approach for motion vector coding. In this paper, degradation in accuracy using a conventional method is experimentally analyzed using mathematical tools. Then, we introduce a new framework for motion vector coding, including modified Inter-mode and Adaptive Direct-mode, in which an advanced motion vector prediction method is employed. The main idea of the proposed motion vector prediction method is to separate each motion vector field from the others. For this purpose, we introduce three alternative motion vectors; a virtual disparity vector, a virtual temporal motion vector and a scaled motion vector. The experimental results show that the proposed method can resolve the inconsistency problem and consequently improve the overall rate-distortion performance. The average bit-rate savings is about 7.3% compared to JMVC 6.0. At maximum, the gain obtained is about 20%, corresponding to a PSNR of 1.42 dB.

Hybrid motion estimation scheme for secondary SP-frame coding using inter-frame correlation and FMO

To cope with considerable size of secondary SP-frames, quantized-transform domain motion estimation has recently been proved to be appropriate for the coding of secondary SP-frames in H.264/AVC. Nevertheless, its computational complexity is tremendous and there are still some situations that pixel-domain motion estimation can perform better. Both techniques are therefore not implemented solely in secondary SP-frame coding. In this paper, a hybrid scheme is proposed to effectively combine two existing motion estimation techniques. The combination is based on a new measurement of inter-frame correlation using the bit-counts of the macroblocks in SP-frames, so that the hybrid scheme is dominated by employing quantized-transform domain motion estimation in the macroblocks with weaker inter-frame correlation; otherwise, it approaches to pixel-domain motion estimation. With the further help of the explicit mode in Flexible Macroblock Ordering (FMO), the proposed hybrid scheme classifies MBs into two slice groups by examining the domain used in motion estimation prior to coding motion vectors in a secondary SP-frame. The slice structure of a secondary SP-frame using the explicit FMO mode is flexible and can be changed during the encoding of each new frame. Simulation results show that our proposed scheme overwhelmingly outperforms the quantized-transform domain motion estimation scheme. As a consequence, the size of secondary SP-frames can be reduced remarkably with significant computational reduction.

Efficient multiview depth video coding using depth synthesis prediction

The view synthesis prediction (VSP) method utilizes interview correlations between views by generating an additional reference frame in the multiview video coding. This paper describes a multiview depth video coding scheme that incorporates depth view synthesis and additional prediction modes. In the proposed scheme, we exploit the reconstructed neighboring depth frame to generate an additional reference depth image for the current viewpoint to be coded using the depth image-based-rendering technique. In order to generate high-quality reference depth images, we used pre-processing on depth, depth image warping, and two types of hole filling methods depending on the number of available reference views. After synthesizing the additional depth image, we encode the depth video using the proposed additional prediction modes named VSP modes; those additional modes refer to the synthesized depth image. In particular, the VSP_SKIP mode refers to the co-located block of the synthesized frame without the coding motion vectors and residual data, which gives most of the coding gains. Experimental results demonstrate that the proposed depth view synthesis method provides high-quality depth images for the current view and the proposed VSP modes provide high coding gains, especially on the anchor frames.

A New Coding Mode for Hybrid Video Coders Based on Quantized Motion Vectors

The rate allocation tradeoff between motion vectors and transform coefficients has a major importance when it comes to efficient video compression. This paper introduces a new coding mode for an H.264/AVC-like video coder, which improves the management of this resource allocation. The proposed technique can be used within any hybrid video encoder allowing a different coding mode for any macroblock. The key tool of the new mode is the lossy coding of motion vectors, obtained via quantization: while the transformed motion-compensated residual is computed with a high-precision motion vector, the motion vector itself is quantized before being sent to the decoder, in a rate/distortion optimized way. Several problems have to be faced with in order to get an efficient implementation of the coding mode, especially the coding and prediction of the quantized motion vectors, and the selection and encoding of the quantization steps. This new coding mode improves the performance of the hybrid video encoder over several sequences at different resolutions.

Visually Lossless Accuracy of Motion Vector in Overcomplete Wavelet-based Scalable Video Coding

In this paper, we propose a visually lossless accuracy model for scalable coding of motion vectors in overcomplete wavelet-based scalable video codec. By exploiting theory of stationary random process, we first estimate motion compensation errors in spatial domain due to inaccurate motion vectors. We then extend the results to overcomplete wavelet domain, and further derive the errors caused by fraction-pixel motion vectors. Finally, combining with a visibility model of wavelet coefficient errors, we propose a novel algorithm to estimate the accuracy threshold of motion vectors with which motion compensation errors will be invisible. Experimental results show that our algorithm is effective in estimating visually lossless accuracy threshold of motion vectors. The proposed algorithm can also be used in scalable motion estimation in wavelet domain. It can accelerate motion estimation speed by stopping halfway at the accuracy that will not cause any visible errors.

Motion vector coding with selection of an optimal predictive motion vector

A new motion vector coding method with optimal predictive motion vector selection is proposed. To improve compression performance, the proposed encoder selects an optimal predictive motion vector that produces minimum bits for motion vector coding. The proposed decoder estimates the optimal predictive motion vector without additional information for indicating which predictor is to be used at the encoder side. Experimental results show that compared to the H.264/AVC standard, the proposed scheme improves coding efficiency for various video sequences.

An Efficient Motion Vector Coding Scheme Based on Prioritized Reference Decision

In the latest video coding frameworks, efficiency of motion vector (MV) coding is becoming increasingly important because of the growing bit rate portion of motion information. However, neither the conventional median predictor, nor the newer schemes such as the minimum bit rate prediction scheme and the hybrid scheme, can effectively eliminate the local redundancy of motion vectors. In this paper, we present the prioritized reference decision scheme for efficient motion vector coding, based on the H.264/AVC framework. This scheme makes use of a boolean indicator to specify whether the median predictor is to be used for the current MV or not. If not, the median prediction is considered not suitable for the current MV, and this information is used for refining the possible space of a group of reference MVs including 4 neighboring MVs and the zero MV. This group of MVs is organized to be a prioritized list so that the reference MV with highest priority is to be selected as the prediction value. Furthermore, the boolean indicators are coded into the modified code words of mb_type and sub_mb_type, so as to reduce the overhead. By applying the proposed scheme, the structure and the applicability problems with the state-of-the-art MBP scheme have been overcome. Experimental result shows that the proposed scheme achieves a considerable reduction of bits for MVDs, compared with the conventional median prediction algorithm. It also achieves a better and much stabler performance than MBP-based MV coding.

Simultaneous MAP-Based Video Denoising and Rate-Distortion Optimized Video Encoding

In this paper, a simultaneous MAP-based video denoising and rate-distortion optimized video encoding algorithm is proposed. We begin with formulating the denoising problem as a maximum a posteriori (MAP) estimate problem. Then, according to the Bayes rule, we show that the MAP estimate is determined by two terms: noise conditional density model and priori conditional density model. Based on the assumptions that the noise satisfies Gaussian distribution and the priori model is measured by the bit-rate, the MAP estimate can be expressed as a rate distortion optimization problem. With this, we are able to simultaneously perform MAP-based video denoising and rate-distortion optimized video encoding under some assumptions. Moreover, we describe in details how to select suitable coding parameters, i.e., quantization parameter, mode, motion vector, reference index, and regularization parameter. Finally, we conduct several experiments to verify our proposed algorithm.

Inter-View Direct Mode for Multiview Video Coding

Global disparity between views is usually caused by the displacement between cameras, which can be accurately represented by a global geometric transformation. In this paper, we first propose an inter-view motion model in terms of the global geometric transformation to represent the motion correlation between two adjacent views. Specifically, the motion vector of a pixel in one view may be directly derived from that in another view according to the inter-view motion model. Further, we propose an inter-view direct mode to signal the decoder that the motion of a macroblock (MB) can be achieved from the coded view without any coding bits. The proposed inter-view direct mode is further incorporated in the existing multiview video coding (MVC) schemes (i.e., AVC-based MVC and 4-D wavelet-based MVC), working together with the other classical coding modes. The mode selection at each MB is accomplished with the rate-distortion optimization technique. The proposed inter-view direct mode can significantly reduce bits to code motion vectors especially at low bit rates, thus improving the coding efficiency

Adaptive irregular pattern search with matching prejudgment for fast block-matching motion estimation

In this paper, a simple and effective fast block matching algorithm (BMA) is proposed, called adaptive irregular pattern search with matching prejudgment (AIPS-MP). In the AIPS-MP, a dynamic search pattern, adaptive irregular pattern (AIP), is constructed for each block based on its spatial and temporal neighboring motion vectors (MVs). The AIP is used to quickly identify the best center point for the follow-up refined search to effectively reduce unnecessary intermediate searches and avoid the search trapping at local-minimum matching error position. The construction of the AIP jointly takes the tradeoff of rate and distortion into consideration and favors the differential coding of MVs. The matching prejudgment (MP) is imbedded in the AIPS and exploits an adaptive threshold determined for each block based on the prediction error of the temporal neighboring block. The simulation results show that the proposed AIPS-MP consistently achieves very close, and sometimes even higher, peak signal-to-noise ratio (PSNR) than the full search, while achieving 167.6-871.5 times of speed-up or computational gain, measured in terms of the average number of search points per MV generation. It also greatly outperforms two well-referenced fast BMAs, the diamond search and the motion vector field adaptive search technique (MVFAST), adopted in the MPEG-4 Verification Model and the Optimization Model, respectively, on the average PSNR and the search speed.

Motion and texture rate-allocation for prediction-based scalable motion-vector coding

Modern video coding applications require data transmission over variable-bandwidth wired and wireless network channels to a variety of terminals, possibly having different screen resolutions and available computing power. Scalable video coding technology is needed to optimally support these applications. Recently proposed wavelet-based video codecs employing spatial-domain motion-compensated temporal filtering (SDMCTF) provide quality, resolution and frame-rate scalability while delivering compression performance comparable to that of H.264, the state-of-the-art in single-layer video coding. These codecs require quality-scalable coding of the motion vectors to support a large range of bit-rates with optimal compression efficiency. In this paper, the practical use of prediction-based scalable motion-vector coding in the context of scalable SDMCTF-based video coding is investigated. Extensive experimental results demonstrate that, irrespective of the employed motion model, our prediction-based scalable motion-vector codec (MVC) systematically outperforms state-of-the-art wavelet-based solutions for both lossy and lossless compression. A new rate-distortion optimized rate-allocation strategy is proposed, capable of optimally distributing the available bit-budget between the different frames and between the texture and motion information, making the integration of the scalable MVC into a scalable video codec possible. This rate-allocation scheme systematically outperforms heuristic approaches previously employed in the literature. Experiments confirm that by using a scalable MVC, lower bit-rates can be attained without sacrificing motion-estimation efficiency and that the overall coding performance at low rates is significantly improved by a better distribution of the available rate between texture and motion information. The only downside of scalable motion-vector coding is a slight performance loss incurred at high bit-rates.

Advanced motion threading for 3D wavelet video coding

This paper presents an advanced motion threading technique for improved performance in 3D wavelet coding. First, we extend an original motion threading idea of ours to a lifting- based implementation. Methods for enabling fractional-pixel alignment in motion threading and for processing many-to-one pixel mapping and non-referred pixels are proposed to reduce the wavelet boundary effects. Second, we devise an advanced motion threading technique, in which one set of motion vectors is generated for each temporal layer of wavelet coefficients for temporal scalability. In order to reduce the motion overhead information, especially at low bit rates, several correlated motion prediction modes at the macroblock level are defined to exploit the intra/inter layer correlation in motion vector coding. Finally, rate-distortion optimization is utilized in motion estimation to select the best motion prediction mode for each macroblock. With the new motion threading technique, we are able to achieve 1.5–6.0 dB gain in average PSNR in 3D wavelet coding over our previous implementation of motion threading.

Hierarchical 2-D mesh representation, tracking, and compression for object-based video

This paper proposes methods for designing, tracking and coding hierarchical two-dimensional (2-D) content-based mesh representations. The design procedure consists of constructing a fine-to-coarse hierarchy of Delaunay meshes, using image- and shape-based criteria for mesh geometry simplification. Hierarchical tracking employs a coarse-to-fine strategy with mesh-based motion vector optimization. We introduce new techniques to maintain the initial mesh hierarchy and topology during tracking by imposing certain constraints at each stage of the procedure. The hierarchical compression technique is based on a nearest neighbor ordering of mesh node points. This ordering serves to identify the mesh boundary nodes as well as establish spatial predictors for differential coding of node coordinates and motion vectors. The proposed hierarchical mesh representation, which has applications in object-based video manipulation, investing, and compression, provides improved tracking performance (compared to a nonhierarchical representation) and allows progressive (scalable) transmission of the object geometry (including shape) and motion information, as well as variable level-of-detail rendering. Experimental results are presented to compare the tracking and compression performance of hierarchical versus nonhierarchical mesh representations and to demonstrate the tradeoff between image quality and mesh bit rate for 2-D mesh-based video object rendering.

Motion-compensated transform coding of video using adaptive displacement fields

To find good trade-offs between the rate-distortion efficiency of motion-compensated, discrete-cosine-transform video coding and its computational complexity, we explore a scheme which uses an adaptive displacement vector field for motion compensation. The sampling density of the field is nonuniform and is adapted to the image scene using a Lagrangian optimization algorithm. The algorithm is formulated to minimize an overall distortion for a given bit rate constraint on the video frame. The algorithm takes into account the contributions to the overall bit rate and distortion due to motion vector and prediction residual coding. Our simulation results demonstrate that, for various common video sequences, the proposed scheme outperforms a popular H.263 test model by about 0.5 dB of peak signal-to-noise ratio. Significant reduction in coding artifacts is observed, and visual quality improvement is highly palpable. The work demonstrates that a suitably crafted rate-distortion optimization scheme can improve performance without exacerbating complexity.

Video object motion representation using run-lengthcodes

A method of runlength coding motion vector data for object based video coding is proposed. It is shown that significant improvements in coding efficiency can be gained for certain classes of source material.