Overlapped Block Motion Compensation Research Articles

Paper] Memory Bandwidth Constrained Overlapped Block Motion Compensation for Video Coding

Overlapped block motion compensation (OBMC) is an inter prediction tool that improves coding performance by blending reference samples of the current and neighboring block across the block boundaries. For this mechanism, OBMC increases the reference samples fetched from the external memory on hardware decoders, i.e., memory bandwidth. This is a disadvantage when introducing OBMC to video coding standards such as VVC, especially for mobile devices with limited batteries, because the extended memory bandwidth increases decoders' power consumption. In this paper, we propose a memory bandwidth constrained OBMC method with adaptive number of motion vectors and interpolation filter taps of the neighboring blocks depending on the current block sizes. Simulation results show that the proposed method achieves -0.22 % performance improvements over VVC reference software without exceeding the maximum memory bandwidth of VVC, which is comparable to the full performance of OBMC (-0.33 %), requiring 3.8 times its memory bandwidth.

Algorithm and Architecture Design of a Hardware-Efficient Frame Rate Upconversion Engine

Owing to the limited response time of liquid crystal displays (LCDs), the motion-blurring effect is a critical problem to overcome for LCD applications. Among the available motion-blurring reduction methods, frame-rate upconversion (FRUC) presents an outstanding performance; however, it also yields a high computational load, which causes a design challenge for a hardware-efficient FRUC engine running advanced specifications. In this paper, a hardware-efficient FRUC algorithm and architecture were proposed, comprising three core techniques: bidirectional 3-D recursive search (BD-3DRS), spatial and temporal motion vector refinement (ST-MVR), and spatial and temporal overlapped-block motion compensation (ST-OBMC). BD-3DRS provides an initial MV with minor computational overhead. Subsequently, ST-MVR refines the initial MV to generate a more accurate final MV. With this final MV, ST-OBMC has finely processed the interpolated block, especially for the textural smoothness across block boundaries. The experiment results indicated that compared with other sophisticated FRUC methods, computation was reduced by 80% with a minor degradation in a peak signal-to-noise ratio of less than 2%. The hardware architecture was realized using the CMOS technology, and the chip size was $1.20 \times 1.20$ mm2. The working frequency was 270 MHz with a power consumption of 128.08 mW. The maximum throughput was as high as 508 Mpixels/s, which can support a resolution of $3840 \times 2160$ from 60 to 120 Hz. Compared with other FRUC architectures, this paper has demonstrated superior hardware efficiencies in terms of area usage and energy consumption.

Hierarchical Motion-Compensated Frame Rate Up-Conversion Using Gaussian Pyramid and Guidance Motion Vector

This paper proposes fast and accurate frame rate up-conversion algorithm for both software and hardware. The proposed algorithm first construct Gaussian pyramid structure using 3×3 Gaussian mask. After constructing Gaussian pyramid, forward and backward unilateral motion estimation is implemented at the top level of the pyramid. Then, motion vector field that is obtained at the top level is projected onto the middle level and the bottom level of the pyramid with unilateral motion vector refinement. Next, on the bottom level, we implement motion vector smoothing and proposed motion vector shifting with bilateral motion vector refinement. By shifting unilateral motion vector to bilateral motion vector, we could avoid generating holes and overlapped regions. Finally, we select final MV between forward and backward MVs according to their reliability. After all the processes, we interpolate new frames by using overlapped block motion compensation. Experimental results show that proposed algorithm is up to 80 times faster than conventional algorithm with 0.1dB peak signal-to-noise gain in average.

A Novel Frame Rate Up-Conversion Algorithm Based on Soft Threshold Bandelet Transform

The most important part of frame rate up-conversion (FRUC) is block matching. The geometric properties of the image were not taken into consideration in traditional block matching algorithm, so the matching result of motion estimation cannot reach the optimal. A novel FRUC algorithm based on Bandelet was proposed in this paper. The algorithm includes: Firstly, a soft threshold Bandelet transform of matching block was performed. The optimal matching block was determined through detection of direction similarity and Bandelet coefficient similarity; secondly, vector median filtering (VMF) and overlapped block motion compensation (OBMC) were carried out by adopting motion vector to realize interpolated frame algorithm. Experimental results show that the FRUC algorithm based on Bandelet can further promote the quality of FRUC.

An improved 3D wavelet-based scalable video coding codec for MC-EZBC

With the rapid growth of modern multimedia applications, 3D wavelet-based scalable video coding (SVC) codec has received considerable attention lately because of its high coding performance and flexibility in bitstream scalability. It combines the motion-compensated temporal filtering (MCTF) together with the spatial decomposition to produce an embedded bitstream offering various levels of video quality over the heterogeneous networks. However, in the existing 3D wavelet-based SVC schemes, where the block types for block matching algorithms are limited, weighting matrices for block-wise motion compensation are fixed, and variations in activities of temporal subbands are not considered in the selection of the Lagrange multiplier for mode decision. In this paper, our major contribution is to provide some recent extensions to the well-known scalable subband/wavelet video codec Motion-Compensated Embedded Zero Block Coding (MC-EZBC) using three novel and content adaptive algorithms. Firstly, the enhanced hierarchical variable size block matching (Enhanced HVSBM) algorithm is proposed for the variable block size motion estimation. Then, the rate-distortion optimization (RDO) based adaptive Lagrange multiplier selection model for mode decision is presented. Finally, we introduce the adaptive weighting matrices design for overlapped block motion compensation (OBMC). Experimental results show that all the three proposed algorithms significantly improve the overall coding performance of MC-EZBC. Comparisons with other popular wavelet-based SVC codecs demonstrate the effectiveness of our improved codec in terms of both video quality assessment and computational complexity.

Motion-Compensated Frame Interpolation With Multiframe-Based Occlusion Handling

In this paper, an occlusion handling algorithm is presented for motion-compensated frame interpolation (MCFI). The proposed algorithm first estimates the unidirectional motion vector fields (MVFs) between two reference frames in both forward and backward directions. Then, a multiframe-based occlusion detection method is presented to identify the uncovered, covered, and motion ambiguity regions within the intermediate frame plane. Based on the detection results, bidirectional motion vectors of the intermediate frame are assigned accurately from unidirectional MVFs. For bidirectional motion compensation, the conventional overlapped block motion compensation (OMBC) method is improved by adjusting the support weight of each pixel within the overlapped window to suppress the blurring and ghost artifacts. Experimental results show that the proposed method not only provides interpolation results with better subjective visual quality, but can also improve the average peak signal-to-noise ratio of the interpolated frames by up to 1.788 dB, when compared to benchmark algorithms.

A spatiotemporal super-resolution algorithm for a hybrid stereo video system

This paper presents a spatiotemporal super-resolution method to enhance both the spatial resolution and the frame rate in a hybrid stereo video system. In this system, a scene is captured by two cameras to form two videos, including a low spatial resolution with high-frame-rate video and a high spatial resolution with low-frame-rate video. For the low-spatial-resolution video, the low-resolution frames are spatially super-resolved by the high-resolution video via the stereo matching, the bilateral overlapped block motion estimation, and the adaptive overlapped block motion compensation algorithms, while for the low-frame-rate video, those missed frames are interpolated using the high-resolution frames obtained by fusing the disparity compensation and the motion compensation frame rate up-conversion. Experimental results demonstrate that the proposed mixed spatiotemporal super-resolution method has a more significant contribution to both the subjective and objective qualities than the pure spatial super-resolution or the frame rate up-conversion.

Temporally coherent superresolution of textured video via dynamic texture synthesis.

This paper addresses the problem of hallucinating the missing high-resolution (HR) details of a low-resolution (LR) video while maintaining the temporal coherence of the reconstructed HR details using dynamic texture synthesis (DTS). Most existing multiframe-based video superresolution (SR) methods suffer from the problem of limited reconstructed visual quality due to inaccurate subpixel motion estimation between frames in an LR video. To achieve high-quality reconstruction of HR details for an LR video, we propose a texture-synthesis (TS)-based video SR method, in which a novel DTS scheme is proposed to render the reconstructed HR details in a temporally coherent way, which effectively addresses the temporal incoherence problem caused by traditional TS-based image SR methods. To further reduce the complexity of the proposed method, our method only performs the TS-based SR on a set of key frames, while the HR details of the remaining nonkey frames are simply predicted using the bidirectional overlapped block motion compensation. After all frames are upscaled, the proposed DTS-SR is applied to maintain the temporal coherence in the HR video. Experimental results demonstrate that the proposed method achieves significant subjective and objective visual quality improvement over state-of-the-art video SR methods.

Multi-Channel Mixed-Pattern Based Frame Rate Up-Conversion Using Spatio-Temporal Motion Vector Refinement and Dual-Weighted Overlapped Block Motion Compensation

In this paper, a novel motion compensated frame rate up-conversion (MC-FRUC) algorithm is proposed to enhance the visual quality of video sequences. First of all, the multi-channel mixed pattern (MCMP) is proposed to design a block matching criterion which uses a few computations to reveal the variances of one luminance channel and two chrominance channels in a video frame. Second, in basis of the forward and backward initial motion vector fields (MVFs) estimated by a variant of 3DRS algorithm, the spatio-temporal motion vector refinement (ST-MVR) algorithm is proposed to obtain the more smoother MVF by implicitly adding the spatio-temporal smooth constraint into the process of motion vector refinement, and then the highly fault-tolerant motion vector smoothing (HFT-MVS) algorithm is proposed to prevent the emergence of outliers in MVF. Finally, in order to reduce the edge blurring and occlusions, the proposed dual-weighted overlapped block motion compensation (DW-OBMC) algorithm uses the forward and backward MVFs to jointly produce the interpolated frame. Experimental results show that the proposed algorithm can significantly improve both objective and subjective quality of the interpolated frame with a low computational complexity, and provide the better performance than the existing algorithms.

Frame rate up-conversion based on dual criterion motion estimation and motion vector adjustment

This paper presents a novel motion-compensated frame rate up-conversion algorithm that uses dual criterion motion estimation to obtain accurate motion vectors in the circumstance of scene luminance changing. In the proposed algorithm, we also present a selective vector median filtering method and a motion vector refinement method to address block mismatching problem. Finally, a motion-compensated interpolation algorithm based on adaptive overlapped block motion compensation is proposed to reduce block artifact. Experimental results demonstrate that the proposed algorithm improves visual qualities compared with the existing algorithms.

Joint Overlapped Block Motion Compensation Using Eight-Neighbor Block Motion Vectors for Frame Rate Up-Conversion

The traditional block-based motion compensation methods in frame rate up-conversion (FRUC) only use a single uniquely motion vector field. However, there will always be some mistakes in the motion vector field whether the advanced motion estimation (ME) and motion vector analysis (MA) algorithms are performed or not. Once the motion vector field has many mistakes, the quality of the interpolated frame is severely affected. In order to solve the problem, this paper proposes a novel joint overlapped block motion compensation method (8J-OBMC) which adopts motion vectors of the interpolated block and its 8-neighbor blocks to jointly interpolate the target block. Since the smoothness of motion filed makes the motion vectors of 8-neighbor blocks around the interpolated block quite close to the true motion vector of the interpolated block, the proposed compensation algorithm has the better fault-tolerant capability than traditional ones. Besides, the annoying blocking artifacts can also be effectively suppressed by using overlapped blocks. Experimental results show that the proposed method is not only robust to motion vectors estimated wrongly, but also can to reduce blocking artifacts in comparison with existing popular compensation methods.

An Interframe Prediction Technique Combining Template Matching Prediction and Block-Motion Compensation for High-Efficiency Video Coding

This paper introduces an interframe prediction technique that combines two motion vectors (MVs) derived respectively from template and block matching for overlapped block motion compensation (OBMC). It has a salient feature of not having to signal the template MV, while achieving a prediction performance close to that of bi-prediction. We begin by studying template matching prediction (TMP) from a theoretical perspective. Based on two signal models, the template MV is shown to approximate the pixel true motion around the template centroid, through which we explain why TMP generally outperforms SKIP prediction but is inferior to block-based motion compensation in terms of prediction performance. We then approach the problem of finding another MV to best complement the template MV from both deterministic and statistical viewpoints, the latter leading to the search of its optimal sampling location in the motion field. The result is a search criterion with OBMC window functions forming a geometry-like motion partitioning when the template area is straddled on the top and to the left of a target block. Generalizations to adaptive template design, multihypothesis prediction and motion merging are made to explore the complexity and performance trade-offs. Extensive experiments based on the HM-6.0 software show that the best of them, in terms of compression performance, achieves 1.7-2.0% BD-rate reductions at a cost of 26% and 39% increases in encoding and decoding times, respectively.

블록의 성질과 프레임 움직임을 고려한 적응적 확장 블록을 사용하는 프레임율 증강 기법

본 논문에서는 적응적인 확장 블록을 사용하는 프레임율 증강 기법인 AEBME (Adaptive Extended Bilateral Motion Estimation)을 제안하고자 한다. 기존의 EBME (Extended Bilateral Motion Estimation) 알고리듬은 동일한 구역에 두 번의 움직임 예측을 수행함으로 인해 높은 계산량이 요구되었다. 본 논문에서는 영상의 edge 정보를 활용한 블록 유형의 일치 유무를 고려하여 EBME 수행여부를 결정함으로써 움직임 예측 과정을 보다 빠르게 수행하도록 하였다. 움직임 벡터 평활화 과정이 적용되어 움직임 벡터 필드 내의 이상 벡터를 찾아 수정한다. 최종적으로 OBMC (Overlapped Block Motion Compensation)와 MCFI (Motion Compensated Frame Interpolation)이 프레임 움직임의 성질에 따라 적용되어 중간 프레임을 보간하게 된다. 실험 결과를 통해 제안하는 알고리듬이 기존의 알고리듬인 EBME에 비해 향상된 성능과 빠른 속도를 보임을 알 수 있다. In this paper, a novel frame rate up conversion (FRUC) algorithm using adaptive extended bilateral motion estimation (AEBME) is proposed. Conventionally, extended bilateral motion estimation (EBME) conducts dual motion estimation (ME) processes on the same region, therefore involves high complexity. However, in this proposed scheme, a novel block type matching procedure is suggested to accelerate the ME procedure. We calculate the edge information using sobel mask, and the calculated edge information is used in block type matching procedure. Based on the block type matching, decision will be made whether to use EBME. Motion vector smoothing (MVS) is adopted to detect outliers and correct outliers in the motion vector field. Finally, overlapped block motion compensation (OBMC) and motion compensated frame interpolation (MCFI) are adopted to interpolate the intermediate frame in which OBMC is employed adaptively based on frame motion activity. Experimental results show that this proposed algorithm has outstanding performance and fast computation comparing with EBME.

Sparse Spatio-Temporal Representation With Adaptive Regularized Dictionary Learning for Low Bit-Rate Video Coding

For promising vision-based video coding on low-quality data, this paper proposes a sparse spatio-temporal representation with adaptive regularized dictionary learning and develops a low bit-rate video coding scheme. In a reversed-complexity Wyner–Ziv coding manner, it selects a subset of key frames to code at original resolution, while the rest are down sampled and reconstructed by a sparse spatio-temporal approximation using key frames as a training dataset. Since primitive patches (geometry) are of low dimensionality and can be well learned from the primitive patches across frames in a scale space, a video frame is divided into three layers: a primitive layer, a nonprimitive coarse layer, and a nonprimitive smooth layer. The multiscale differential feature representations are invertible to facilitate reconstruction with dictionary learning, and the target is formulated as an optimization problem by constructing a sparse representation of 2-D patches and 3-D volumes over adaptive regularized dictionaries, a set of 2-D subdictionary pairs trained from primitive patches, and a 3-D dictionary trained from nonprimitive volumes. Specifically, the nonprimitive layer is constructed as volumes in to order keep it consistent along the motion trajectory, which enables sparse representations over a learned 3-D spatio-temporal dictionary. Through hierarchical bidirectional motion estimation and adaptive overlapped block motion compensation, the 3-D low-frequency and high-frequency dictionary pair is designed by the K-SVD algorithm to update the atoms for optimal sparse representation and convergence. In reconstruction, the lost high-frequency information of the down-sampled frames can be synthesized from the sparse spatio-temporal representation over the adaptive regularized dictionaries. Extensive experiments validate the compression efficiency of the proposed scheme versus H.264/AVC in terms of both objective and subjective comparisons.

Parametric OBMC for Pixel-Adaptive Temporal Prediction on Irregular Motion Sampling Grids

This paper adapts overlapped block motion compensation (OBMC) to suit variable block-size motion partitioning. The motion vectors (MVs) for various partitions are formalized as motion samples taken on an irregular grid. From this viewpoint, determining OBMC weights to associate with these samples becomes an under-determined problem since a distinct solution has to be sought for each prediction pixel. In this paper, we tackle this problem by expressing the optimal weights in closed form based on parametric signal assumptions. In particular, the computation of this solution requires only the geometric relations between the prediction pixel and its nearby block centers, leading to a generic framework capable of reconstructing temporal predictors from any irregularly sampled MVs. A modified implementation is also proposed to address the MV location uncertainty and to reduce computational complexity. Experimental results demonstrate that our scheme performs better than similar previous works, and when compared to the recently proposed Quadtree-based adaptive loop filter and enhanced adaptive interpolation filter, show a comparable gain. Furthermore, the combination of it with either of them gives a combined effect that is almost the sum of their separate improvements.

Video Super-Resolution Algorithm Using Bi-Directional Overlapped Block Motion Compensation and On-the-Fly Dictionary Training

This paper presents a video super-resolution algorithm to interpolate an arbitrary frame in a low resolution video sequence from sparsely existing high resolution key-frames. First, a hierarchical block-based motion estimation is performed between an input and low resolution key-frames. If the motion-compensated error is small, then an input low resolution patch is temporally super-resolved via bi-directional overlapped block motion compensation. Otherwise, the input patch is spatially super-resolved using the dictionary that has been already learned from the low resolution and its corresponding high resolution key-frame pair. Finally, possible blocking artifacts between temporally super-resolved patches and spatially super-resolved patches are concealed using a specific de-blocking filter. The experimental results show that the proposed algorithm provides significantly better subjective visual quality as well as higher peak-to-peak signal-to-noise ratio than those by previous interpolation algorithms.

High-resolution image scaler using hierarchical motion estimation and overlapped block motion compensation

This paper presents a motion-compensated scaler to interpolate an arbitrary frame in a low resolution video sequence by using sparsely existing high resolution key frames. In order to minimize visually annoying blocking artifact caused by block motion compensation, hierarchical motion estimation is bi-directionally performed for forward and backward key frames. Also, bi-directional overlapped block motion compensation based on the predicted motion vectors is applied to the high resolution key frames. Simulation results show that the proposed algorithm provides significantly better subjective visual quality as well as higher PSNR than previous interpolation algorithms. The proposed algorithm can provide a differentiated functionality to cutting-edge digital cameras, camcorders, etc.

Whole Frame Error Concealment with Refined Motion Extrapolation and Prescription at Encoder for H.264/AVC

Packet losses usually result in frame losses in low bit rate video streaming over error prone networks. The whole frame error concealment (EC) algorithm is then necessary for the decoder to improve the video quality. A refined bidirectional motion vector extrapolation was developed to improve the motion vector estimation precision by alleviating the effect of overlapped and hole regions with an adaptive overlapped block motion compensation to reduce block artifacts. A prescription-based framework was then developed to improve the error concealment at the encoder side. Simulations show that the EC algorithm at the decoder side outperforms the existing methods by 2–8 dB. Moreover, the prescription-based framework at the encoder side provides further peak signal to noise ratio improvement.

Irregular-Grid-Overlapped Block Motion Compensation and its Practical Application

In this letter, we present a hybrid motion compensation scheme integrating overlapped block motion compensation (OBMC) and control grid interpolation (CGI), called irregular grid-based OBMC (IG-OBMC). The objective of this letter is to develop both subjectively and objectively improved motion compensation scheme by combining distinct advantages of OBMC and CGI without additional motion vectors or a complicated motion estimation processing. To evaluate the performance, IG-OBMC is applied to error concealment. The simulation results demonstrate that the proposed hybrid motion compensation method achieves a better performance than the conventional motion compensation methods.

Enhanced MC-EZBC Scalable Video Coder

In this paper, we provide some recent extension to the scalable subband/wavelet video coder MC-EZBC. The enhanced MC-EZBC employs an adaptive motion-compensated temporal filter (MCTF) framework. Directional I-BLOCKs and overlapped block motion compensation (OBMC) further improve MCTF efficiency. A scalable motion vector coder based on CABAC is shown to improve the overall performance at low bitrates/resolution. Frequency rolloff is incorporated to reduce spatial aliasing at low resolution without PSNR loss at full resolution. Experimental results show that the new features significantly improve the performance of MC-EZBC. We provide several comparisons to other recent coders.