Motion-compensated Temporal Interpolation Research Articles

Motion-Compensated Temporal Frame Interpolation Algorithm Based on Global Entirety Unidirectional Motion Estimation and Local Fast Bidirectional Motion Estimation

To solve the uncertainty of motion for objects displayed on liquid crystal display (LCD), the motion estimation (ME) algorithm on the basis of global entirety unidirectional motion and local fast bidirectional motion (GEU-LFB) is proposed for motion-compensated temporal frame interpolation (MCTFI). Firstly, by observing the moving objects in the scene, we will obtain a set of global motion vectors (MVs). Secondly, the fast local search based on the bidirectional ME is executed. Additionally, to make up for the defect of the bidirectional ME, a method to generate the exposure mask and the occlusion mask is proposed by cleverly using the matching criterion of absolute difference and the set of global MVs. Next, the optimal MV field is smoothed by the vector median filter. Finally, the temporal interpolation frame obtained through the weighted filtering compensation method is further improved by using masks. Overall, the experimental results show that the proposed algorithm has better performance than the existing methods in terms of objective and subjective criteria. Moreover, it can solve problems of exposure and occlusion, and get better frame interpolation for the video sequence with fast moving targets.

Distributed Coding Scheme for Multi-view Video through Efficient Side Information Generation

In this paper, a distributed image coding scheme for multi-view video through an efficient generation of side information is proposed. A distributed video coding technique corrects the errors in the side information, which is generated with the original image, by using the channel coding technique at the decoder. Therefore, the more correct the generated side information is, the better the performance of distributed video coding. The proposed technique is to apply the distributed video coding schemes to the image coding for multi-view video. It generates side information by selectively and efficiently using both 3-dimensional warping based on the depth map with spatially adjacent frames and motion-compensated temporal interpolation with temporally adjacent frames. In this scheme the difference between the adjacent frames, the sizes of the motion vectors for the adjacent blocks, and the edge information are used as the selection criteria. From the experiments, it was observed that the quality of the side information generated by the proposed technique was improved by the average peak signal-to-noise ratio of 0.97dB than the one by motion-compensated temporal interpolation or 3-dimensional warping. The result from analyzing the rate-distortion curves revealed that the proposed scheme could reduce the bit-rate by 8.01% on average at the same peak signal-tonoise ratio value, compared to previous work.

Spatial correlation-based side information refinement for distributed video coding

Distributed video coding (DVC) architecture designs, based on distributed source coding principles, have benefitted from significant progresses lately, notably in terms of achievable rate-distortion performances. However, a significant performance gap still remains when compared to prediction-based video coding schemes such as H.264/AVC. This is mainly due to the non-ideal exploitation of the video sequence temporal correlation properties during the generation of side information (SI). In fact, the decoder side motion estimation provides only an approximation of the true motion. In this paper, a progressive DVC architecture is proposed, which exploits the spatial correlation of the video frames to improve the motion-compensated temporal interpolation (MCTI). Specifically, Wyner-Ziv (WZ) frames are divided into several spatially correlated groups that are then sent progressively to the receiver. SI refinement (SIR) is performed as long as these groups are being decoded, thus providing more accurate SI for the next groups. It is shown that the proposed progressive SIR method leads to significant improvements over the Discover DVC codec as well as other SIR schemes recently introduced in the literature.

Fusion of Global and Local Motion Estimation for Distributed Video Coding

The quality of side information plays a key role in distributed video coding. In this paper, we propose a new approach that consists of combining global and local motion compensation at the decoder side. The parameters of the global motion are estimated at the encoder using scale invariant feature transform features. Those estimated parameters are sent to the decoder in order to generate a globally motion compensated side information. Conversely, a locally motion compensated side information is generated at the decoder based on motion-compensated temporal interpolation of neighboring reference frames. Moreover, an improved fusion of global and local side information during the decoding process is achieved using the partially decoded Wyner-Ziv frame and decoded reference frames. The proposed technique improves significantly the quality of the side information, especially for sequences containing high global motion. Experimental results show that, as far as the rate-distortion performance is concerned, the proposed approach can achieve a PSNR improvement of up to 1.9 dB for a Group of Pictures (GOP) size of 2, and up to 4.65 dB for larger GOP sizes, with respect to the reference DISCOVER codec.

Novel True-Motion Estimation Algorithm and Its Application to Motion-Compensated Temporal Frame Interpolation

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

A Content-Adaptive Side Information Generation Method for Distributed Video Coding

In this paper, a content-adaptive method to generate side information at the block level is presented. First, motion compensated temporal interpolation (MCTI) algorithm is used between the reconstructed key frames at the decoder to acquire initial motion vectors. Second, the image is segmented and the edge of moving region is detected by obtained the residual frame between two consecutive key frames. Furthermore, hierarchical motion estimation (HME) and motion vector filter (MVF) are adopted for edge region and an adaptive motion vector filter (AMVF) is introduced in non-edge region to correct the false estimated motion vectors. The proposal is tested and compared with the results of the state-of-the-art DISCOVER codec and RD improvements on the set of test sequences are observed.

Wyner–Ziv coding of video with unsupervised motion vector learning

Distributed source coding theory has long promised a new method of encoding video that is much lower in complexity than conventional methods. In the distributed framework, the decoder is tasked with exploiting the redundancy of the video signal. Among the difficulties in realizing a practical codec has been the problem of motion estimation at the decoder. In this paper, we propose a technique for unsupervised learning of forward motion vectors during the decoding of a frame with reference to its previous reconstructed frame. The technique, described for both pixel-domain and transform-domain coding, is an instance of the expectation maximization algorithm. The performance of our transform-domain motion learning video codec improves as GOP size grows. It is better than using motion-compensated temporal interpolation by 0.5 dB when GOP size is 2, and by even more when GOP size is larger. It performs within about 0.25 dB of a codec that knows the motion vectors through an oracle, but is hundreds of orders of magnitude less complex than a corresponding brute-force decoder motion search approach would be.

Estimation and compensation of accelerated motion for temporal sequence interpolation

Abstract This paper makes two contributions to the area of motion-compensated processing of image sequences. First contribution is the development of a framework for the modeling and estimation of dense 2-D motion trajectories with acceleration. Therefore, Gibbs-Markov models are proposed and linked together by the maximum a posteriori probability (MAP) criterion, and the resulting objective function is minimized using multiresolution deterministic relaxation. Accuracy of the method is demonstrated by measuring the mean-squared error of estimated motion parameters for images with synthetic motion. Second contribution is the demonstration of a significant gain resulting from the use of trajectories with acceleration in motion-compensated temporal interpolation of videoconferencing/videophone images. An even higher gain is demonstrated when the accelerated motion trajectory model is augmented with occlusion and motion discontinuity models. The very good performance of the method suggests a potential application of the proposed framework in the next generation of video coding algorithms.

Gradient-based algorithms for block-oriented MAP estimation of motion and application to motion-compensated temporal interpolation

A study of block-oriented motion estimation algorithms is presented, and their application to motion-compensated temporal interpolation is described. In the proposed approach, the motion field within each block is described by a function of a few parameters that can represent typical local motion vector fields.. A probabilistic formulation is then used to develop maximum-likelihood (ML) and maximum a posteriori probability (MAP) estimation criteria. The MAP criterion takes into account the dependence of the motion fields in adjacent blocks. A procedure for minimizing the resulting objective function based on the Gauss-Newton algorithm is presented. The use of multiresolution techniques, essential for satisfactory performance, is discussed. Experimental results evaluating the algorithms for the task of motion-compensated temporal interpolation are presented. The relative complexity of the algorithms is also discussed. >