Block-based Motion Compensation Research Articles

Deep Affine Motion Compensation Network for Inter Prediction in VVC

In video coding, it is a challenge to deal with scenes with complex motions, such as rotation and zooming. Although affine motion compensation (AMC) is employed in Versatile Video Coding (VVC), it is still difficult to handle non-translational motions due to the adopted hand-craft block-based motion compensation. In this paper, we propose a deep affine motion compensation network (DAMC-Net) for inter prediction in video coding to effectively improve the prediction accuracy. To the best of our knowledge, our work is the first attempt to deal with the deformable motion compensation based on CNN in VVC. Specifically, a deformable motion-compensated prediction (DMCP) module is proposed to compensate the current encoding block through a learnable way to estimate accurate motion fields. Meanwhile, the spatial neighboring information and the temporal reference block as well as the initial motion field are fully exploited. By effectively fusing the multi-channel feature maps from DMCP, an attention-based fusion and reconstruction (AFR) module is designed to reconstruct the output block. The proposed DAMC-Net is integrated into VVC and the experimental results demonstrate that the proposed method considerably enhances the coding performance.

Coarse-to-fine SVD-GAN based framework for enhanced frame synthesis

Frame interpolation and synthesis are growing topics in the field of computer vision. Hence, these topics gained more attention recently where several deep-learning architectures were proposed to enhance the quality of the synthesized frames. In this paper, an efficient handcrafted deep approach is proposed for better frame synthesis. The proposed approach takes advantage of singular value decomposition (SVD) framework and Generative Adversarial Networks (GAN). The proposed approach does not require any computationally expensive feature extraction steps such optical flow techniques and block-based motion compensation techniques. Nonetheless, the SVD components still carry the relevant motion information needed to deal with the challenges such as large motion and occlusion. Thus, the frames are temporally upscaled via SVD based construction procedure where new middle frames are interpolated for further enhancement using a GAN based approach that eliminates most of the visual artifacts. The proposed frame synthesis approach is comprehensively evaluated in different scenarios where its performance is assessed and compared with the state-of-the-art. Our framework outperforms the majority of the deep learning approaches in terms quantitative results in addition to qualitative results where our framework can generate smoother frames with less visual artifacts.

RETRACTED: Optimization of Motion Estimation Algorithm Based on FPGA Hardware System and Video Tracking

Motion Estimation (ME) is a computationally expensive step in video tracking. A thorough search technique produces the best execution time and the best accuracy. Computational burden, many fast search algorithms limit and report the number of places to search. Motion estimation, analysis of image sequences, computer vision, tracking a target, and is a critical process in a wide range of fields and applications such as video coding. Precision motion estimation lowers significantly bit rate, but can require a complexity of higher computational. This is different reference frames, sub-pixel estimation, and video compression standards, including techniques such as variable block size, MPEG, especially true for a new generation. Only the motion estimation in H.261 is limited to an integer pixel accuracy, rather than the pixel grid, not in the object is often located, move to move between pixels. The system proposed in the motion vector optimization motion estimation (OME) pixel accuracy. By estimating the amount of displacement at a finer resolution, improve prediction, it can expect better performance than motion estimation using an integer pixel precision. Optimization motion estimation (OME) and the compensation method is very similar to that employed in other standards. The main is that block-based motion estimation and compensation is suitable for optimization structures in MPEG-4. Motion estimation (ME) is one of the most computationally-intensive companies in the video compression technology. Video compression algorithm, MPEG1, MPEG4, such as H.261 and H.264, a lot of the standard used. Improved compression performance. Energy effective motion estimation technique is reduced to the rest of the frame involved in motion compensation.

Motion segmentation-based surveillance video compression using adaptive particle swarm optimization

Video surveillance is one of the widely used and most active research applications of computer vision. Although lots of works have been done in the area of smart surveillance, but still there is a need of effective compression technique for compact archival and efficient transmission of vast amount of surveillance video data. In this work, we propose a hybrid video compression approach with the help of foreground motion compensation for the above application. This method works effectively by including the advantages of both block-based and object-based coding techniques as well as reducing the drawbacks of both. The proposed method first segments the foreground moving objects from the background with the help of adaptive thresholding-based optical flow techniques. Next, it determines the contour of the segmented foreground regions with the help of Freeman chain code. Subsequently, block-based motion estimation and compensation using variants of particle swarm optimization are computed. After that, motion failure areas are detected using change detection method, and finally, DCT and Huffman coding-based entropy encoding are done to compactly represent the data. Experimental results and analyses on different surveillance video sequences using Wilcoxon’s rank-sum test, PSNR and SSID show that our method outperforms other recent and relevant existing techniques.

Design of Efficient Perspective Affine Motion Estimation/Compensation for Versatile Video Coding (VVC) Standard

The fundamental motion model of the conventional block-based motion compensation in High Efficiency Video Coding (HEVC) is a translational motion model. However, in the real world, the motion of an object exists in the form of combining many kinds of motions. In Versatile Video Coding (VVC), a block-based 4-parameter and 6-parameter affine motion compensation (AMC) is being applied. In natural videos, in the majority of cases, a rigid object moves without any regularity rather than maintains the shape or transform with a certain rate. For this reason, the AMC still has a limit to compute complex motions. Therefore, more flexible motion model is desired for new video coding tool. In this paper, we design a perspective affine motion compensation (PAMC) method which can cope with more complex motions such as shear and shape distortion. The proposed PAMC utilizes perspective and affine motion model. The perspective motion model-based method uses four control point motion vectors (CPMVs) to give degree of freedom to all four corner vertices. Besides, the proposed algorithm is integrated into the AMC structure so that the existing affine mode and the proposed perspective mode can be executed adaptively. Because the block with the perspective motion model is a rectangle without specific feature, the proposed PAMC shows effective encoding performance for the test sequence containing irregular object distortions or dynamic rapid motions in particular. Our proposed algorithm is implemented on VTM 2.0. The experimental results show that the BD-rate reduction of the proposed technique can be achieved up to 0.45% and 0.30% on Y component for random access (RA) and low delay P (LDP) configurations, respectively.

Distributed video coding with adaptive two-step side information generation for smart and interactive media

Conventional video coding standards, such as MPEGx and H.26x, use a hybrid architecture of block-based motion compensation and discrete cosine transform (DCT) within the structure of a complex encoder and a simple decoder. Contrary to conventional video codecs, Wyner-Ziv (WZ) video coding, which is a practical application of distributed video coding (DVC) based on the Slepian-Wolf and WZ theorems, exploits the source correlation at the decoder, thereby allowing for the development of simpler encoders. However, the current WZ video coding algorithms cannot match the coding performance of conventional video coding. In order to improve the coding performance of transform-domain WZ video coding, an adaptive two-step side-information generation method is thus proposed for evaluation and analysis in this study. The proposed method uses decoded WZ frames in the down-sampled spatial resolution along with already decoded coefficients to successively improve rate-distortion (RD) performance as the decoding progresses. The experimental results show that the proposed method, compared to previous WZ video coding method, shows improved coding performance, particularly under critical conditions, such as cases with high motion content.

Three-Zone Segmentation Based Motion Compensation for Video Compression.

Motion compensation has been widely employed for removing temporal redundancies in typical hybrid video coding framework. The popular video compression standards, such as H.264/AVC and HEVC, adopt the block based partitioning model to describe the motion filed due to its high compression efficiency and relatively low computational complexity. However, block based motion compensation may not align with the actual object motion boundaries, potentially limiting the compression efficiency. In view of this, we propose a three-zone segmentation based motion compensation scheme to improve the description accuracy of motion field as well as the coding efficiency. In particular, the segmentation information is implied in the reference frame instead of being explicitly signalled. Based on the segmentation information, three motion compensation zones can be identified, including one edge, one foreground, and one background zone. The foreground zone is motion compensated by the signalled motion vector of the block, and the background zone is motion compensated by the motion information implicitly derived from the local motion field. Regarding the edge zone, it is viewed as an overlapped area and the weighted compensation strategy is adopted. The proposed algorithm is implemented into the reference software VTM-1.0 of Versatile Video Coding (VVC), and simulation results show that the algorithm can achieve 1.14% and 1.06% bitrate savings for random access and lowdelay configurations, respectively.

Novel Image Set Compression Algorithm Using Rate-Distortion Optimized Multiple Reference Image Selection

Image set compression has recently become an active research topic due to the explosion of digital photographs. In order to efficiently compress the image sets of similar images with moving objects, in this paper, we propose a novel algorithm for image set compression using multiple reference images. First, for an image set, its depth-constrained minimum arborescence is generated. We then present a reference image candidate determination method to build the reference image candidates for the images of the set. Furthermore, we propose a rate-distortion optimized multiple reference image selection method. This method compares the correlation between every image and each of its reference image candidates to produce its multiple reference images. Finally, compressed image data are achieved by employing block-based motion compensation and residue coding. In addition, we also give a new way of access to images to keep the same access delay with single reference image-based schemes. Compared with the state-of-the-art image compression algorithms, experimental results show that our proposed algorithm can significantly improve the image compression performance.

Algorithm and Architecture Design of Multirate Frame Rate Up-conversion for Ultra-HD LCD Systems

In current liquid crystal display (LCD) systems, the frame size becomes larger than the ultra-HD ( $3840\times 2160$ ) resolution, and the refresh rate becomes higher than 120 Hz or more. However, the available video frame rates are usually at 24, 30, or 60 frames/s only, which are lower than the refresh rate of LCDs. To fill the gap between the video and LCD systems, frame interpolation techniques are usually adopted. Although frame rate up-conversion (FRUC) is regarded as the most efficient method, many design challenges are encountered in the current high-resolution and high-frame-rate LCD systems. In this paper, we developed a hardware-efficient multirate FRUC, which is capable of increasing the video frame rate from 24 or 60 to 120 frames/s. We improved the accuracy of motion vectors (MVs) between the video frames using predictive square search motion estimation (ME), followed by Markov random field (MRF) correction. Subsequently, we applied the block-based forward motion compensation (MC) to interpolate the intermediate frames. Thereafter, we performed subblock refinement to enhance the visual quality. The experiments show that the proposed algorithm performed well in both subjective and objective evaluations. We also designed hardware architecture for our multirate FRUC to support the current ultra-HD LCD systems. We proposed ping-pong two-way scheduling to eliminate the dependence among blocks. We realized 54%, 70%, and 35% cycle reduction and 62%, 82%, and 35% bandwidth reduction in the ME, MRF MV correction, and MC, respectively. The SRAM was shared by all modules, and its total size was reduced by 88%. We also implemented our design to a chip using the TSMC 90-nm cell library.

Motion-compensated coding and frame rate up-conversion: models and analysis.

Block-based motion estimation (ME) and motion compensation (MC) techniques are widely used in modern video processing algorithms and compression systems. The great variety of video applications and devices results in diverse compression specifications, such as frame rates and bit rates. In this paper, we study the effect of frame rate and compression bit rate on block-based ME and MC as commonly utilized in inter-frame coding and frame rate up-conversion (FRUC). This joint examination yields a theoretical foundation for comparing MC procedures in coding and FRUC. First, the video signal is locally modeled as a noisy translational motion of an image. Then, we theoretically model the motion-compensated prediction of available and absent frames as in coding and FRUC applications, respectively. The theoretic MC-prediction error is studied further and its autocorrelation function is calculated, yielding useful separable-simplifications for the coding application. We argue that a linear relation exists between the variance of the MC-prediction error and temporal distance. While the relevant distance in MC coding is between the predicted and reference frames, MC-FRUC is affected by the distance between the frames available for interpolation. We compare our estimates with experimental results and show that the theory explains qualitatively the empirical behavior. Then, we use the models proposed to analyze a system for improving of video coding at low bit rates, using a spatio-temporal scaling. Although this concept is practically employed in various forms, so far it lacked a theoretical justification. We here harness the proposed MC models and present a comprehensive analysis of the system, to qualitatively predict the experimental results.

Photo Album Compression for Cloud Storage Using Local Features

This paper proposes a novel feature-based photo album compression scheme for cloud storage. The main contribution of this paper is the use of local features rather than pixel values for analyzing and exploring the correlation between images. Unlike previously established image set compression schemes, we adopt content-based feature matching, which is invariant to scale, rotation, and robust to illumination changes, in both correlation estimation and redundancy reduction. Based on feature-based correlation analysis, we organize correlated images as a pseudo sequence by minimizing the predictive cost. Since pseudo sequences have much more complicated correlations than natural videos, we further propose a three-step prediction to reduce inter-image redundancy based on local features. We first propose a feature-based multi-model approach to reduce local geometric deformations between images, resulting in multiple deformed predictions. We then decrease the illumination difference between each deformed prediction and the target image with a photometric transformation. Finally, we reduce redundancy between images using a block-based motion compensation, similar to video compression. Experimental results show that our proposed feature-based coding scheme can be 10 times more efficient than individual JPEG compression.

Exploiting Long-Term Redundancies in Reconstructed Video

In this paper we propose an enhancement to HEVC that allows the usage of homographies for motion compensation between temporally far distant frames. First, we show the benefit of using homographies compared to block-based motion compensation, particularly for long-term motion compensation in the presence of zoom. Then we demonstrate that beyond the case where content is temporally repeated, long-term global motion compensation leads to a better compression ratio in the presence of zooming out. We show that this is due to the change in scale on one hand, and the effects of compression on the reference pictures on the other. Using a model for the distortion of images caused by compression, it will be shown that a finer scale reconstructed image that is warped into the perspective of the image to be encoded can constitute a better predictor than an immediately neighboring picture. We compared the HM software with an enhanced system using long-term global motion compensation with homographies and found that the enhanced system achieves Bjontegaard Delta Rate savings of up to 38%. With the mechanism behind the gains unveiled, we finally try to find an optimal scale ratio between the frame to be encoded and the reference frame.

Joint Overlapped Block Motion Compensation Using Eight-Neighbor Block Motion Vectors for Frame Rate Up-Conversion

The traditional block-based motion compensation methods in frame rate up-conversion (FRUC) only use a single uniquely motion vector field. However, there will always be some mistakes in the motion vector field whether the advanced motion estimation (ME) and motion vector analysis (MA) algorithms are performed or not. Once the motion vector field has many mistakes, the quality of the interpolated frame is severely affected. In order to solve the problem, this paper proposes a novel joint overlapped block motion compensation method (8J-OBMC) which adopts motion vectors of the interpolated block and its 8-neighbor blocks to jointly interpolate the target block. Since the smoothness of motion filed makes the motion vectors of 8-neighbor blocks around the interpolated block quite close to the true motion vector of the interpolated block, the proposed compensation algorithm has the better fault-tolerant capability than traditional ones. Besides, the annoying blocking artifacts can also be effectively suppressed by using overlapped blocks. Experimental results show that the proposed method is not only robust to motion vectors estimated wrongly, but also can to reduce blocking artifacts in comparison with existing popular compensation methods.

An Interframe Prediction Technique Combining Template Matching Prediction and Block-Motion Compensation for High-Efficiency Video Coding

This paper introduces an interframe prediction technique that combines two motion vectors (MVs) derived respectively from template and block matching for overlapped block motion compensation (OBMC). It has a salient feature of not having to signal the template MV, while achieving a prediction performance close to that of bi-prediction. We begin by studying template matching prediction (TMP) from a theoretical perspective. Based on two signal models, the template MV is shown to approximate the pixel true motion around the template centroid, through which we explain why TMP generally outperforms SKIP prediction but is inferior to block-based motion compensation in terms of prediction performance. We then approach the problem of finding another MV to best complement the template MV from both deterministic and statistical viewpoints, the latter leading to the search of its optimal sampling location in the motion field. The result is a search criterion with OBMC window functions forming a geometry-like motion partitioning when the template area is straddled on the top and to the left of a target block. Generalizations to adaptive template design, multihypothesis prediction and motion merging are made to explore the complexity and performance trade-offs. Extensive experiments based on the HM-6.0 software show that the best of them, in terms of compression performance, achieves 1.7-2.0% BD-rate reductions at a cost of 26% and 39% increases in encoding and decoding times, respectively.

이진화기반 영역분할을 이용한 3D입체영상의 밝기보정

본 논문에서는 이진화기반 영역분할을 이용한 3D입체영상의 밝기 보정방법을 제안한다. 제안된 방법은 입력된 좌우 3D입체영상 중 우 영상을 이진화를 통한 영역분할을 하고 크기가 작은 영역들은 제거한다. 다음 우 영상의 각 영역들에 대해 대응되는 좌 영상내의 영역을 상관계수(correlation coefficient)를 이용한 정합을 통해 추출한다. 영역단위의 매칭을 할 때 영역들 간의 겹침을 방지하기 위하여 모폴로지 필터로 영역경계에 인접한 일정부분을 제거한다. 그리고 좌우 영상의 영역 간 히스토그램 명세화를 수행함으로써 우 영상의 밝기 보정을 한다. 실험에서 좌 영상으로부터 블록단위 움직임보상으로 우 영상을 생성했을 때 제안한 방법이 블록평균 정합오차가 가장 작은 것을 확인 할 수 있었다. In this paper, we propose a method for intensity correction using binarization-based region segmentation in 3D stereoscopic images. In the proposed method, 3D stereoscopic right image is segmented using binarizarion. Small regions in the segmented image are eliminated. For each region in right image, a corresponding region in left image is decided through region matching using correlation coefficient. When region-based matching, in order to prevent overlap between regions, we remove a portion of the area closed to the region boundary using morphological filter. The intensity correction in left and right image can be performed through histogram specification between the corresponding regions. Simulation results show the proposed method has the smallest matching error than the conventional method when we generate the right image from the left image using block based motion compensation.

Layered Motion Compensation for Moving Image Compression

Block-based motion compensation began with MPEG-1 and H.261 and has continued into MPEG-2, MPEG-4_part 2, MPEG-4_AVC_part 10, H.264, and SMPTE VC1. The general principle used by these conventional motion compensation systems is that there is one motion vector for a square or rectangular group of pixels. The group of pixels associated with the motion vector is generally referred as a “macroblock.” It is common in MPEG and other compression systems to use reduced resolution for Red-Y (U) and Blue-Y (V) compared to Y (luminance), where Y usually equals approximately .59Green + .29Red + .12Blue. Intra-coded frames and the motion-compensated-difference frames use a common quantized discrete cosine transform (DCT). The macroblock structure provides a common boundary for the Y, U, and V DCT-coded regions. The DCT size is usually 8 × 8 pixels, but may be anywhere from 4 × 4 up to 16 × 16 (e.g., in MPEG-4 AVC_part 10 and H.264). The nature of the DCT is that it is a self-contained regional transform that does not extend to any pixel outside its block (usually 8 × 8). In a sense, quantization errors in the DCT wrap from one edge to the other (i.e., left to right, and top to bottom) of the DCT block. Differing quantization errors in adjacent DCT blocks yield block edge discontinuities. Thus, because macroblock boundaries coincide with DCT block boundaries, they share a common edge. When a motion vector differs between adjacent macroblocks, the inherent edge discontinuity does not appear within the DCT block, but rather at its edge. The DCT block edge “wraps around” to see its opposite edges, so the DCT is quite tolerant of the edges inherent in macroblock-based motion compensation.

Block Wiener-based image registration for moving target indication

This paper develops a block Wiener-based image registration method to align consecutive frames in a video sequence. The main purpose of this registration technique is to serve as a processing step in a moving target indication (MTI) algorithm, where subsequent steps include frame-to-frame differencing and thresholding. Experiments show that the block Wiener-based approach is computationally efficient and effective in terms of facilitating a MTI. These experiments compare the block Wiener-based against traditional global parametric registration, pixel-based Wiener registration, block-based motion compensation, and state-of-the-art phase-based registration.

Extended Analysis of Motion-Compensated Frame Difference for Block-Based Motion Prediction Error

In the past, most design and optimization work on hybrid video codecs relied mainly on experimental evidence. A proper theoretical model is always desirable, since this allows us to explain the phenomena of existing codecs and to design better ones. In this paper, we make use of the first-order Markov model to derive an approximated separable autocorrelation model for the block-based motion compensation frame difference (MCFD) signal. A major assumption of our derivation is that the net deformation of pixels is directional, in general, rather than a uniform error distribution in a block. We have also shown that the imperfect block-based motion compensation is significant to the theoretical study and the behavior of motion-compensated codecs. Results of our experimental work show that the derived model can describe the statistical characteristics of the MCFD signals accurately. The model also shows that the imperfectly formulated block-based motion compensation can result in an incorrect MCFD autocorrelation function while, conversely, it can form a better block-based motion compensation scheme.

An efficient low bit-rate video-coding algorithm focusing on moving regions

Block-based motion estimation and compensation are the most popular techniques for video coding. However, as the shape and the structure of an object in a picture are arbitrary, the performance of such conventional block-based methods may not be satisfactory. In this paper, a very low bit-rate video coding algorithm that focuses on moving regions is proposed. The aim is to improve the coding performance, which gives better subjective and objective quality than that of the conventional coding methods at the same bit rate. Eight patterns are pre-defined to approximate the moving regions in a macroblock. The patterns are then used for motion estimation and compensation to reduce the prediction errors. Furthermore, in order to increase the compression performance, the residual errors of a macroblock are rearranged into a block with no significant increase of high-order DCT coefficients. As a result, both the prediction efficiency and the compression efficiency are improved.

Low bit-rate video coding with implicit multiscale segmentation

Discusses a multiscale segmentation based video compression algorithm aimed at very low bit-rate applications such as video teleconferencing and video phones. We introduce novel techniques for multiscale segmentation based motion compensation and residual coding. Our region based forward motion compensation strategy (in terms of direction of motion vector, which is from the previous frame to the current frame) regulates the size and number of regions used, by pruning a multiscale segmentation of video frames. Since regions used for motion compensation are obtained by segmenting the previously decoded frame, the shape of the regions need not be transmitted to the decoder. Furthermore, our hierarchical motion compensation strategy refines an initial region level, coarse motion field to obtain a dense motion field which provides pixel level motion vectors. The refinement procedure does not require any additional information to be transmitted. This motion compensation technique effectively addresses the problem of dealing with "holes" and "overlapping regions" which are inherent to forward motion compensation. Residual coding is performed using a novel method which exploits the fact that the energy of the residual resulting from motion compensation is concentrated in a priori predictable positions. We show that this residual coding technique can also be extrapolated to improve the performance of coders using a block based motion compensation strategy. A fusion of these concepts leads to a gain of 2-3 dB in peak signal-to-noise ratio, apart from significant perceptual improvement, over a generic video coding algorithm using a block based motion compensation strategy (such as H.261 or H.263) for a variety of test sequences.