Bidirectional Motion Estimation Research Articles

IBVC: Interpolation-driven B-frame video compression

Learned B-frame video compression aims to adopt bi-directional motion estimation and motion compensation (MEMC) coding for middle frame reconstruction. However, previous learned approaches often directly extend neural P-frame codecs to B-frame relying on bi-directional optical-flow estimation or video frame interpolation. They suffer from inaccurate quantized motions and inefficient motion compensation. To address these issues, we propose a simple yet effective structure called Interpolation-driven B-frame Video Compression (IBVC). Our approach only involves two major operations: video frame interpolation and artifact reduction compression. IBVC introduces a bit-rate free MEMC based on interpolation, which avoids optical-flow quantization and additional compression distortions. Later, to reduce duplicate bit-rate consumption and focus on unaligned artifacts, a residual guided masking encoder is deployed to adaptively select the meaningful contexts with interpolated multi-scale dependencies. In addition, a conditional spatio-temporal decoder is proposed to eliminate location errors and artifacts instead of using MEMC coding in other methods. The experimental results on B-frame coding demonstrate that IBVC has significant improvements compared to the relevant state-of-the-art methods. Meanwhile, our approach can save bit rates compared with the random access (RA) configuration of H.266 (VTM). The code will be available at https://github.com/ruhig6/IBVC.

Two-dimensional structural motion estimation via analytical signal enhancement of phase-based video processing

This paper concerns the use of phase information from image sequences for the extraction of structural two-directional motions. The phase-based motion estimation (PME) estimates local phase information via Gabor wavelet transform, and can only estimate the motion in single direction at a time, which will reduce the applicability and efficiency for many structures subject to more sophisticated bidirectional motions. This paper introduces a two-dimensional partial analytical signal to estimate the instantaneous phase and theoretically derives the relationship between the phase variations and the structure motion to achieve bidirectional motion estimations simultaneously. In addition to the analytical signal phase-based motion estimation (APME), this paper designs an ideal narrow-band filter (INF) addressing the limitation of the narrow frequency band in the analytical signals. In the proposed APME method, the acquisition of instantaneous phase improves the accuracy of structural bidirectional motion estimation. Moreover, the realization of the partial analytical signal is based on the fast Fourier transform (FFT) of the image and the dot product in the frequency domain, which bypasses the traditional PME convolution operation in the spatial domain. Therefore, the calculation efficiency of structural motion estimation is greatly improved. The proposed method is validated using a numerical simulation of a moving Gaussian surface with random motions. Results prove a good accuracy of the proposed APME method, and outperforming the traditional PME, the APME demonstrates an outstanding consistence to the true values (correlation coefficient 99.99%) four times better calculation efficiency. Further comparison with the experimental results of a cantilever beam also shows that the proposed method can better identify the structure bidirectional motion estimation.

Motion‐Compensated Frame Interpolation Using Cellular Automata‐Based Motion Vector Smoothing

Motion‐Compensated Frame Interpolation (MCFI) is one of the common temporal‐domain tamper operations, and it is used to produce faked video frames for improving the visual qualities of video sequences. The instability of temporal symmetry results in many incorrect Motion Vectors (MVs) for Bidirectional Motion Estimation (BME) in MCFI. The existing Motion Vector Smoothing (MVS) works often oversmooth or revise correct MVs as wrong ones. To overcome this problem, we propose a Cellular Automata‐based MVS (CA‐MVS) algorithm to smooth the Motion Vector Field (MVF) output by BME. In our work, a cellular automaton is constructed to deduce MV outliers according to a defined local evolution rule. By performing CA‐based evolution in a loop iteration, we gradually expose MV outliers and reduce incorrect MVs resulting from oversmoothing as many as possible. Experimental results show the proposed algorithm can improve the accuracy of BME and provide better objective and subjective interpolation qualities when compared with the traditional MVS algorithms.

Frame rate up-conversion in cardiac ultrasound

The temporal resolution of echocardiographic sequences affects the interpretation of cardiac motion as well as the viewing experience. Temporal resolution, however, is limited by the propagation velocity of sound in tissue. Altering the acquisition system can improve temporal resolution but normally reduces image quality. Alternatively, motion compensated frame interpolation enhances perceived temporal resolution without reducing spatial resolution. There are however significant challenges with these techniques due to the simultaneous presence of large and small motions of the cardiac structures. In this paper, we propose a new motion compensated frame interpolation algorithm based on a variational bidirectional motion estimation model. We derive the Euler-Lagrange equations characterizing the optimal dense bidirectional motion field and develop a multiscale iterative procedure to obtain the motion field and the missing intermediate frames. The performance of the proposed algorithm was evaluated on real cardiac ultrasound recordings, using both subjective and objective evaluation criteria. Experimental results show that the interpolated frames generated by the proposed algorithm do not have the ghosting or blurring artifacts that are commonly present in the existing motion compensated frame interpolation algorithms, especially near fast moving structures like the cardiac valves. Quantitatively, the proposed algorithm achieves an improvement in the average peak signal-to-noise ratio of 1.16 dB and in the structural similarity index of 3.0%.

A Low-Complex Frame Rate Up-Conversion with Edge-Preserved Filtering

The improvement of resolution of digital video requires a continuous increase of computation invested into Frame Rate Up-Conversion (FRUC). In this paper, we combine the advantages of Edge-Preserved Filtering (EPF) and Bidirectional Motion Estimation (BME) in an attempt to reduce the computational complexity. The inaccuracy of BME results from the existing similar structures in the texture regions, which can be avoided by using EPF to remove the texture details of video frames. EPF filters out by the high-frequency components, so each video frame can be subsampled before BME, at the same time, with the least accuracy degradation. EPF also preserves the edges, which prevents the deformation of object in the process of subsampling. Besides, we use predictive search to reduce the redundant search points according to the local smoothness of Motion Vector Field (MVF) to speed up BME. The experimental results show that the proposed FRUC algorithm brings good objective and subjective qualities of the interpolated frames with a low computational complexity.

Triple-Frame-Based Bi-Directional Motion Estimation for Motion-Compensated Frame Interpolation

In this paper, new motion estimation (ME) and motion vector refinement (MVR) methods for motion-compensated frame interpolation (MCFI) are proposed. To estimate motion vectors (MVs) with great reliability, triple-frame-based bi-directional ME (TFBME), which employs three sequential frames, is developed. In TFBME, the middle frame works as a reference to estimate MVs between two end frames, which leads to performance improvement especially when objects move linearly through three sequential frames. In addition, a modified bi-directional ME method, which employs only two successive frames, is combined with TFBME to handle cases in which objects move non-linearly through three sequential frames. In our proposed MVR, artifacts on interpolated frames are first detected by a convolutional neural network and then corrected by weighted vector median filtering. Experimental results show that the proposed MCFI method outperforms conventional methods in both subjective and objective evaluations.

Residual Reconstruction Algorithm Based on Half-Pixel Multi-Hypothesis Prediction for Distributed Compressive Video Sensing

Compressed sensing (CS) provides a method to sample and reconstruct sparse signals far below the Nyquist sampling rate, which has great potential in image/video acquisition and processing. In order to fully exploit the spatial and temporal characteristics of video frame and the coherence between successive frames, we propose a half-pixel interpolation based residual reconstruction method for distributed compressive video sensing (DCVS). At the decoding end, half-pixel interpolation and bi-directional motion estimation helps refine the side information for joint decoding of the non-key-frames. We apply a multi-hypothesis based on residual reconstruction algorithms to reconstruct the non-key-frames. Performance analysis and simulation experiments show that the quality of side information generated by the proposed algorithm is increased by about 1.5dB, with video reconstruction quality increased 0.3~2dB in PSNR, when compared with prior works on DCVS.

Multi-hypothesis compressed video sensing by two-step iterative thresholding

Traditional distributed video coding schemes achieved compression by using a single reference frame for side information, which plays a key-role in determining the performance of the codec. To refine side information, we propose a novel side information generation algorithm by using two neighbouring key frames as reference. The proposed algorithm uses multi-hypothesis predictions to fully utilise the intra and inter frame correlations. Three side information candidates are obtained by applying multi-hypothesis predictions and bi-directional motion estimation on the non-key frame and its bi-directional reference frames. The most similar candidate is chosen by calculating the correlation coefficients between the non-key frame and the three candidates. With the refined side information, the video frames are reconstructed by blocked compressed sensing with a smoothed project-landweber (BCS-SPL) algorithm. Experimental results demonstrate that the proposed algorithm outperforms the original schemes in refining the side information as well as improving the reconstruction performance.

Multi-hypothesis compressed video sensing by two-step iterative thresholding

Traditional distributed video coding schemes achieved compression by using a single reference frame for side information, which plays a key-role in determining the performance of the codec. To refine side information, we propose a novel side information generation algorithm by using two neighbouring key frames as reference. The proposed algorithm uses multi-hypothesis predictions to fully utilise the intra and inter frame correlations. Three side information candidates are obtained by applying multi-hypothesis predictions and bi-directional motion estimation on the non-key frame and its bi-directional reference frames. The most similar candidate is chosen by calculating the correlation coefficients between the non-key frame and the three candidates. With the refined side information, the video frames are reconstructed by blocked compressed sensing with a smoothed project-landweber (BCS-SPL) algorithm. Experimental results demonstrate that the proposed algorithm outperforms the original schemes in refining the side information as well as improving the reconstruction performance.

Motion-compensated frame interpolation using patch-based sparseland model

Using patch-based sparseland model, we design a novel Motion-Compensated Frame Interpolation (MCFI) method. Optical-flow estimation is first adopted to generate a reliable Motion Vector Field (MVF) from the previous frame to the following frame. Then we use patch-based bidirectional motion estimation to generate a smooth Motion Vector Felid (MVF). In the last step, we extract patches from reference frames along the motion trajectories, and perform Principle Component Analysis (PCA) to generate dictionaries that code the patches with various local structures. The sparseland model becomes the prior knowledge of the intermediate frame to fuse into the Motion Compensated Interpolation (MCI) by maximum a posteriori (MAP) criterion. By iterative numerical computing, we solve this sparseland-prior reconstruction model, and acquire a sparsity-preferred interpolated frame. Experimental results show that our method outperforms the existing algorithms in both objective and subjective picture qualities, but it also introduces a high computational complexity in the meantime.

Motion-Compensated Temporal Frame Interpolation Algorithm Based on Global Entirety Unidirectional Motion Estimation and Local Fast Bidirectional Motion Estimation

To solve the uncertainty of motion for objects displayed on liquid crystal display (LCD), the motion estimation (ME) algorithm on the basis of global entirety unidirectional motion and local fast bidirectional motion (GEU-LFB) is proposed for motion-compensated temporal frame interpolation (MCTFI). Firstly, by observing the moving objects in the scene, we will obtain a set of global motion vectors (MVs). Secondly, the fast local search based on the bidirectional ME is executed. Additionally, to make up for the defect of the bidirectional ME, a method to generate the exposure mask and the occlusion mask is proposed by cleverly using the matching criterion of absolute difference and the set of global MVs. Next, the optimal MV field is smoothed by the vector median filter. Finally, the temporal interpolation frame obtained through the weighted filtering compensation method is further improved by using masks. Overall, the experimental results show that the proposed algorithm has better performance than the existing methods in terms of objective and subjective criteria. Moreover, it can solve problems of exposure and occlusion, and get better frame interpolation for the video sequence with fast moving targets.

시공간 적응적인 예측에 기초한 다시점 위너-지브 비디오 부호화 기법

본 논문에서는 시공간 적응적인 예측에 기초한 다시점 위너-지브 비디오 부호화 기법을 제안한다. 제안하는 알고리즘은 기존 움직임 추정 방법을 보완하여 가중치를 부여한 결합 양방향 움직임 추정을 수행하고, 각 시점 영상의 에지 검출 및 합성을 통해 관심영역을 효과적으로 분류하여 움직임 벡터 분석을 통해 최종 참조 프레임을 선택하여 보간 한다. 제안하는 알고리즘은 단일 시점 내의 움직임 정보와 인접 카메라 프레임의 정보를 적응적으로 이용함으로써 영상 내 다양한 폐색, 반사 영역에 대해 효율적으로 처리하고 더 나은 성능을 갖는다. 다양한 다시점 영상 시퀀스에 대한 실험을 통하여, 제안하는 알고리즘은 보조정보 생성하는 기존 알고리즘에 비해 평균 비트율 감소와 더불어 우수한 객관적 화질 향상을 얻었다. This paper proposes a multi-view Wyner-Ziv Video coding scheme based on spatio-temporal adaptive estimation. The proposed algorithm is designed to search for a better estimated block with joint bi-directional motion estimation by introducing weights between temporal and spatial directions, and by classifying effectively the region of interest blocks, which is based on the edge detection and the synthesis, and by selecting the reference estimation block from the effective motion vector analysis. The proposed algorithm exploits the information of a single frame viewpoint and adjacent frame viewpoints, simultaneously and then generates adaptively side information in a variety of closure, and reflection regions to have a better performance. Through several simulations with multi-view video sequences, it is shown that the proposed algorithm performs visual quality improvement as well as bit-rate reduction, compared to the conventional methods.

Bilateral frame rate up‐conversion algorithm based on the comparison of texture complexity

A new bilateral frame rate up-conversion (FRUC) algorithm based on the bi-directional comparison of the texture complexity in the reference frame is proposed. In the proposed algorithm, the accuracy of the motion estimation (ME) process is determined by the comparison of the prediction size in the reference region of the bi-directional ME process. Base on this comparison, the adaptive refinement process is adjusted. In the experimental results, the proposed algorithm outperforms the conventional bilateral FRUC algorithm by 0.27 dB.

Robust Motion Compensated Frame Interpolation Using Weight-Overlapped Block Motion Compensation with Variable Block Sizes to Reduce LCD Motion Blurs

Liquid crystal displays (LCDs) have slow responses, so motion blurs are often perceived in fast moving scenes. To reduce this motion blur, we propose a novel method of robust motion compensated frame interpolation (MCFI) based on bidirectional motion estimation (BME) and weight-overlapped block motion compensation (WOBMC) with variable block sizes. In most MCFI methods, a static block size is used, so some block artefacts and motion blurs are observed. However, the proposed method adjusts motion block sizes and search ranges by comparing matching scores, so the precise motion vectors can be estimated in accordance with motions. In the MCFI, overlapping ranges for WOBMC are also determined by adjusted block sizes, so the accurate MCFI can be performed. In the experimental results, the proposed method strongly reduced motion blurs arisen from large motions, and yielded interpolated images with high visual performance and peak signal-to-noise ratio (PSNR).

An Advanced Parallel FPGA Architecture for Bi-Directional Motion Estimation

Motion estimation (ME) and motion compensation (MC) are the key elements for frame rate up-conversion (FRUC) system. Fast and accurate motion estimation is the premise of high quality motion compensation. Unlike conventional unidirectional motion estimation which brings holes, overlaps and blocking artifacts, the bi-directional motion estimation does not produce any overlapped pixel or hole in the interpolated frames. As a result, the bi-directional motion estimation has better performance than conventional unidirectional motion estimation. This paper presents an efficient FPGA architecture targeting bi-directional motion estimation hardware implementation. This proposed architecture can achieve real-time processing for 1280x720@60Hz under 200MHz operating frequency. The design is described in Verilog HDL, verified in Virtex5 FPGA platform. Experimental results show that the proposed architecture has high performance and low cost for bi-directional motion estimation algorithm. This architecture can be used for video post-processing system.

Bidirectional Mesh-Based Frame Rate Up-Conversion

The authors propose a new frame rate up-conversion (FRUC) method for temporal video quality enhancement. To cope with translation as well as scale and rotation changes, the algorithm generates an interpolated frame between two given frames on the basis of bidirectional mesh interpolation (BMI). BMI performance is highly influenced by the accuracy of the correspondences between the control points in the two frames. To achieve an accurate, dense motion vector map (MVM) through bidirectional and unidirectional motion estimation, the algorithm forms an initial MVM using motion vectors transmitted from the coded bitstream. Then, the algorithm generates the interpolated frame using frame-based BMI with the dense MVM. In experiments, the authors found that the proposed algorithm is about two decibels better than several conventional FRUC methods. Furthermore, the proposed algorithm significantly diminishes block artifacts and blur artifacts.

Alternating asymmetric search range assignment for bidirectional motion estimation in H.265/HEVC and H.264/AVC

Bidirectional motion estimation (ME) significantly enhances video coding efficiency, whereas its huge complexity is also a critical problem for implementation. This paper presents alternating asymmetric search range assignment (AASRA) schemes to reduce the complexity by switching the use of a large and a small search ranges (SR.L and SR.S). A temporal AASRA (T-AASRA) scheme performs search range switching between past and future reference directions. A temporal-spatial AASRA (TS-AASRA) scheme performs more aggressive switching on the two dimensions of reference direction and MB/CTB index. T-AASRA and TS-AASRA achieve 43.5% and 65.2% complexity reduction, respectively, with small coding efficiency drop. Even after removing the factor of coding efficiency drop, the two schemes still show 38.4% and 54.7% equivalent complexity reduction. AASRA can also be combined with existing fast algorithms for further complexity saving, which has been demonstrated on hierarchical ME and dynamic search range selection algorithms.

Temporal Frame Interpolation Based on Multiframe Feature Trajectory

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and to preserve the spatial correlation of MVs. Furthermore, the proposed method can detect occluded regions, where a continuously tracked feature does not have a corresponding point in the current frame. If an occluded region is detected, the intermediate frame is generated from either the previous frame or the current frame. In the non-occluded region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results showed that the proposed temporal frame interpolation (TFI) method can improve the visual quality compared with conventional TFI methods, both objectively and subjectively.

Side Information Refinement for Transform domain Distributed Video Coding

A new side information refinement method for transform domain distributed video coding (DVC) is proposed. In DVC, side information is the frame constructed from key frames which are intra coded. It is utilized as the initial estimate of source symbols of Wyner-Ziv coded frame which is not transmitted to the decoder. Errors in the estimated frames are to be corrected by the decoder using the parity bits which are the only information received from the encoder. Quality of decoder generated side information is one of the most important factors deciding the overall performance of the decoder. A new refinement method of improving the quality of side information of the DVC decoder is proposed. The side information frame is refined at the end of each transform band decoding stage followed by a conventional bi-directional motion estimation and compensation technique. Proposed side information refinement technique consequently improves the quality of final reconstructed pictures.

Sparse Spatio-Temporal Representation With Adaptive Regularized Dictionary Learning for Low Bit-Rate Video Coding

For promising vision-based video coding on low-quality data, this paper proposes a sparse spatio-temporal representation with adaptive regularized dictionary learning and develops a low bit-rate video coding scheme. In a reversed-complexity Wyner–Ziv coding manner, it selects a subset of key frames to code at original resolution, while the rest are down sampled and reconstructed by a sparse spatio-temporal approximation using key frames as a training dataset. Since primitive patches (geometry) are of low dimensionality and can be well learned from the primitive patches across frames in a scale space, a video frame is divided into three layers: a primitive layer, a nonprimitive coarse layer, and a nonprimitive smooth layer. The multiscale differential feature representations are invertible to facilitate reconstruction with dictionary learning, and the target is formulated as an optimization problem by constructing a sparse representation of 2-D patches and 3-D volumes over adaptive regularized dictionaries, a set of 2-D subdictionary pairs trained from primitive patches, and a 3-D dictionary trained from nonprimitive volumes. Specifically, the nonprimitive layer is constructed as volumes in to order keep it consistent along the motion trajectory, which enables sparse representations over a learned 3-D spatio-temporal dictionary. Through hierarchical bidirectional motion estimation and adaptive overlapped block motion compensation, the 3-D low-frequency and high-frequency dictionary pair is designed by the K-SVD algorithm to update the atoms for optimal sparse representation and convergence. In reconstruction, the lost high-frequency information of the down-sampled frames can be synthesized from the sparse spatio-temporal representation over the adaptive regularized dictionaries. Extensive experiments validate the compression efficiency of the proposed scheme versus H.264/AVC in terms of both objective and subjective comparisons.