Motion-compensated Interpolation Research Articles

Adaptive motion-compensated interpolation for frame rate up-conversion

We propose a new frame rate up-conversion (FRC) algorithm using the adaptive motion compensation (MC) that reduces the blocking artifacts due to block-based processing. In the proposed scheme, after conventional motion estimation (ME) between two adjacent frames is performed to construct the motion vectors for the frame to be interpolated, the motion analysis is used to determine the type of motion and the motion-compensated interpolation (MCI) is applied adaptively. Unlike conventional MCI algorithms, the proposed technique utilizes similar neighboring motion vectors to produce multiple motion trajectories. When the proposed MCI is applied, multiple motion trajectories are considered in order to increase the accuracy of the MCI. The proposed method provides high quality format conversion with significantly reduced blocking artifacts.

Adaptive motion-compensated frame rate up-conversion

A frame rate up-conversion algorithm using the adaptive motion compensation that reduces the blocking artifacts due to block-based processing is proposed. The proposed method uses the motion analysis to determine the type of motion and applies the motion-compensated interpolation (MCI) adaptively. When the MCI is applied, multiple motion trajectories are considered in order to increase the accuracy of the MCI. Experiment results indicate good performance of the proposed MCI scheme with significantly reduced blocking artifacts.

Wavelet-based video coder via bit allocation

We present a video coder based on a bit allocation (BA) scheme for a coding approach where a wavelet-based video compression scheme is used to encode a group of frames (GOF). The proposed new BA method utilizes a Lagrangian multiplier technique to determine the optimal bit rate for each frame of a given GOF in a rate versus weighted distortion sense. The proposed baseline coder sequentially employs the application of the wavelet transform on each frame in the group. In the wavelet domain, an optimal-vector BA (rate-allocation strategy) is being used to distribute the bit budget among the frames. The manipulated frames in each group are quantized and coded using techniques from still-image compression. We describe and demonstrate the performance of the proposed algorithm in wavelet-based coding approaches. The first one, a combination of wavelet still compression and optimal BA (WCBA), is based on frame differences and exploits well the wavelet compression characteristics which keeps, for a given bit budget, the most important features. It does not utilize motion-compensation interpolation or a 3-D wavelet decomposition. In cases where heavy motion is detected, we extend WCBA into a combination of wavelet still compression, optimal BA, motion compensation with block matching, and error correction (WCBAM), which uses block matching for motion estimation, while the residual error of the compression is reduced by applying error correction in the wavelet domain to reduce temporal redundancies.

New frame rate up-conversion using bi-directional motion estimation

We propose a new frame rate up-conversion algorithm for high quality video. In the proposed scheme, bi-directional motion estimation (ME) is performed to construct the motion vector (MV) field for the frame to be interpolated. Unlike conventional motion-compensated interpolation (MCI) algorithms, the proposed technique does not produce any overlapped pixel and hole region in the interpolated frame, and thus can utilize the overlapped block motion compensation technique to reduce the blocking artifacts. The proposed algorithm is very simple to implement on consumer products when compared to conventional MCI methods. Computer simulation shows a high visual performance of the proposed frame rate up-conversion algorithm.

A comparison of motion-compensated interlace-to-progressive conversion methods

The conversion of interlaced video to progressive format may be carried out in a number of ways, including vertical interpolation, fixed or motion-compensated vertical-temporal interpolation, and approaches based on motion compensation using non-uniform sampling theory. Motion-compensated methods potentially offer the best performance, although the accuracy and reliability of the vectors has a significant impact on their performance. For applications such as display conversion using vectors recovered from an MPEG-2 bitstream, the vectors available may not be of the highest quality. This paper considers several motion-compensated interlace-to-progressive conversion methods, and assesses their performance with both accurate and inaccurate vectors using objective and subjective techniques. The method found to work best uses a motion-compensated three-field vertical-temporal filter, in which the vertical component of the vector is rounded to the nearest even number of picture lines per field period. This method reduced the RMS error of converted picture sequences by almost 20% compared to simple vertical filtering when using vectors from an MPEG-2 bitstream. Conversely, some other motion-compensated methods were found to increase the error by a similar amount.

Adaptive motion-compensated interpolation based on spatiotemporal segmentation

This paper reports our study of a novel adaptive motion-compensated interpolation technique based on spatiotemporal segmentation. The number of reference frames and the intervals between them are adjusted to improve the temporal compensability of the input video in a GOP (group of pictures). The bit-rate allocation takes the advantage of temporal masking of human visual system. Experimental results show that this proposed adaptive coding scheme is more efficient than normal fixed GOP structure coding algorithm and may be an efficient development of the prior algorithm suggested by Lee and Dickinison (1994).

An optimal quadtree-based motion estimation and motion-compensated interpolation scheme for video compression

We propose an optimal quadtree (QT)-based motion estimator for video compression. It is optimal in the sense that for a given bit budget for encoding the displacement vector field (DVF) and the QT segmentation, the scheme finds a DVF and a QT segmentation which minimizes the energy of the resulting displaced frame difference (DFD). We find the optimal QT decomposition and the optimal DVF jointly using the Lagrangian multiplier method and a multilevel dynamic program. We introduce a new, very fast convex search for the optimal Lagrangian multiplier lambda(*), which results in a very fast convergence of the Lagrangian multiplier method. The resulting DVF is spatially inhomogeneous, since large blocks are used in areas with simple motion and small blocks in areas with complex motion. We also propose a novel motion-compensated interpolation scheme which uses the same mathematical tools developed for the QT-based motion estimator. One of the advantages of this scheme is the globally optimal control of the tradeoff between the interpolation error energy and the DVF smoothness. Another advantage is that no interpolation of the DVF is required since we directly estimate the DVF and the QT-segmentation for the frame which needs to be interpolated. We present results with the proposed QT-based motion estimator which show that for the same DFD energy the proposed estimator uses about 25% fewer bits than the commonly used block matching algorithm. We also experimentally compare the interpolated frames using the proposed motion compensated interpolation scheme with the reconstructed original frames.

Space-time interpolation of oceanic fronts

Oceanic temperature fronts observed through composite infrared images from the AVHRR satellite data are fragmented due mostly to cloud occlusion. The sampling frequency of such frontal position observations tends to be insufficiently high to resolve dynamics of the meandering features associated with the frontal contour, so that contour reconstruction using a standard space-time smoothing often leads to introduction of spurious features. Augmenting space-time smoothing with a simple point-feature detection/matching scheme, however, can dramatically improve the reconstruction product. This paper presents such a motion-compensated interpolation algorithm, for reconstruction of open contours evolving in time given fragmented position data. The reconstruction task is formulated as an optimization problem, and a time-sequential solution which adaptively estimates feature motion is provided. The resulting algorithm reliably interpolates position measurements of the surface temperature fronts associated with the highly convoluted portions of strong ocean currents such as the Gulf Stream and Kuroshio.

Motion‐compensated interpolation for scan rate up‐conversion

There are two kinds of scan rate up-conversion for a cinematic sequence: frame-based motion-compensated interpolation and fieldbased scan rate up-conversion. The latter deinterlaces every field in an interlaced sequence which can be displayed using a progressive format or interlaced format to double the field rate. This paper proposes a new frame-based interpolation method that consists of segmentation and interpolation modules. The first module divides the difference between two successive frames into changed and unchanged regions. Each changed region is further differentiated into moving objects, covered, and uncovered regions. The second module generates the intermediate frames by using the direct motion-based interpolation method and indirect motionbased interpolation method to fill the changed regions, and the forward/ backward insertion method to fill the covered and uncovered regions. The proposed interpolation method can deal with the interframe zooming effect as well as the object-background ambiguity problem. Finally, it is shown that this new method is computationally simpler than the other frame interpolation methods and the interpolated frames are qualitatively and quantitatively better than results obtained with the other methods.

Estimation and compensation of accelerated motion for temporal sequence interpolation

Abstract This paper makes two contributions to the area of motion-compensated processing of image sequences. First contribution is the development of a framework for the modeling and estimation of dense 2-D motion trajectories with acceleration. Therefore, Gibbs-Markov models are proposed and linked together by the maximum a posteriori probability (MAP) criterion, and the resulting objective function is minimized using multiresolution deterministic relaxation. Accuracy of the method is demonstrated by measuring the mean-squared error of estimated motion parameters for images with synthetic motion. Second contribution is the demonstration of a significant gain resulting from the use of trajectories with acceleration in motion-compensated temporal interpolation of videoconferencing/videophone images. An even higher gain is demonstrated when the accelerated motion trajectory model is augmented with occlusion and motion discontinuity models. The very good performance of the method suggests a potential application of the proposed framework in the next generation of video coding algorithms.

Determining accurate and reliable motion fields for motion-compensated interpolation

How to obtain accurate and reliable motion fields is a key problem for motion-compensated (MC) interpolation in video coding. An improved pel-based motion estimation algorithm with smoothness constraints is presented. In order to cope with various moving objects and their complex motion, a hierarchical motion estimation algorithm is discussed. The experimental results show that the motion fields obtained are quite smooth and approximate to real motion.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Motion-compensated conversion from interlaced to progressive formats

This paper presents a new system for the conversion of interlaced formats to progressive ones. Like other proposals, it is motion compensation-based, but a substantial 2-fold improvement is added. First, assuming translational motions, the problem in the vertical direction is studied as a generalized interpolation problem. As a result, we derive two sets of linear filters which take into account the aliasing existing inside the fields. The first set of filters allows one to efficiently perform the interpolation which is required for subpel motion estimation. The second set of filters improves the estimation of the lines needed to obtain the progressive format, namely the deinterlacing process itself. Second, an improved block matching algorithm based on a split and merge procedure is used to estimate the motion and correctly propagate the motion vectors from blocks with reliable motion to blocks with uncertain motion. In order to tackle the problem of covered/uncovered objects, the whole process of estimation and motion-compensated interpolation is applied forward and backward. Simulation results and objective measurements are provided for artificially moving interlaced sequences obtained from a fixed picture and for a progressive sequence first converted to interlaced. The global system has also been tested on other sequences and visually assessed.

Motion-compensated conversion from interlaced to progressive formats

This paper presents a new system for the conversion of interlaced formats to progressive ones. Like other proposals, it is motion compensation-based, but a substantial 2-fold improvement is added. First, assuming translational motions, the problem in the vertical direction is studied as a generalized interpolation problem. As a result, we derive two sets of linear filters which take into account the aliasing existing inside the fields. The first set of filters allows one to efficiently perform the interpolation which is required for subpel motion estimation. The second set of filters improves the estimation of the lines needed to obtain the progressive format, namely the deinterlacing process itself. Second, an improved block matching algorithm based on a split and merge procedure is used to estimate the motion and correctly propagate the motion vectors from blocks with reliable motion to blocks with uncertain motion. In order to tackle the problem of covered/uncovered objects, the whole process of estimation and motion-compensated interpolation is applied forward and backward. Simulation results and objective measurements are provided for artificially moving interlaced sequences obtained from a fixed picture and for a progressive sequence first converted to interlaced. The global system has also been tested on other sequences and visually assessed.

An error-weighted regularization algorithm for image motion-field estimation

Local motion measurement errors are used to guide the global smoothing process in order to preserve motion-field discontinuities. A field-smoothing algorithm based on matching-error weighting is proposed. The added computation is minimal, since it uses byproducts of the local measurement process. The error-weighting functional provides significantly improved motion field estimates, as measured by motion-compensated interpolation performance. However, the mean-square reconstruction error is somewhat higher than that obtained by performing the much more computationally expensive stochastic optimization.

Modified a posteriori mode decision for HD-MAC system

A mode decision method for the motion-compensated three-branch HD-MAC system is described. The conventional mode decision method consists of a priori and a posteriori decisions. The correctness of the mode decision has a considerable influence on the picture quality at HD-MAC receivers. The conventional scheme, however, does not provide a good decision result. Thus, a new method by which the modes of blocks ca be determined more correctly is proposed. This method is based on frame-exchange and motion-compensated interpolation. The performance of this method is shown to be better than that of the conventional ones. >

Gradient-based algorithms for block-oriented MAP estimation of motion and application to motion-compensated temporal interpolation

A study of block-oriented motion estimation algorithms is presented, and their application to motion-compensated temporal interpolation is described. In the proposed approach, the motion field within each block is described by a function of a few parameters that can represent typical local motion vector fields.. A probabilistic formulation is then used to develop maximum-likelihood (ML) and maximum a posteriori probability (MAP) estimation criteria. The MAP criterion takes into account the dependence of the motion fields in adjacent blocks. A procedure for minimizing the resulting objective function based on the Gauss-Newton algorithm is presented. The use of multiresolution techniques, essential for satisfactory performance, is discussed. Experimental results evaluating the algorithms for the task of motion-compensated temporal interpolation are presented. The relative complexity of the algorithms is also discussed. >

Video coding with motion-compensated interpolation for CD-ROM applications

This paper discusses the compression of digital video signals at bit-rates around 1 Mbps for interactive playback applications. The coding algorithm is required not only to provide good-quality reconstruction of complex material but also to facilitate interactivity with the bit-stream at the decoder. The algorithm proposed in the paper is based on the well-known technique of motion-compensated prediction and DCT-coding of the prediction error. This basic approach is considerably enhanced with motion-compensated interpolation of skipped video frames and selective coding of the interpolation errors. The interactivity requirements are met by partitioning the video sequence into segments each comprised of a small number of frames. Different ways of encoding a segment are examined. An arrangement is selected that has one intra-coded frame in the center of the segment and a symmetrical pattern of predicted and interpolated frames on the two sides of the intra-coded frame. Segments of length 9 and 15 frames are evaluated. While the shorter segment leads to faster interactivity and simpler decoder implementation, the associated picture quality is much inferior to that obtained with the longer segment. Finally, a rough design of the decoder, suitable for VLSI implementation, is outlined.

Considerations on the possibility to exchange temporal against spatial resolution in image coding

In order to transmit high definition television signals economically some kind of data compression has to be utilized. We investigate the question how spatio-temporal properties of human vision can be exploited for data reduction. By means of pyramid coding we implement coding schemes which transmit spatial detail information with reduced temporal resolution. In moving areas with correlated motion we introduce motion-compensated interpolation. In the case of uncorrelated motion, where motion estimation often fails and the eye cannot follow the movement, high spatial frequencies are suppressed. The different systems are rated by means of subjective tests. Furthermore motion-compensated interpolation of the highpass signals of a subband codec using quadrature mirror filters will be analysed.