Frame Rate Up-conversion Research Articles

Direction-Select Motion Estimation for Motion-Compensated Frame Rate Up-Conversion

In this paper, we propose a new motion estimation algorithm to be used for motion-compensated frame rate up-conversion. The proposed algorithm independently carries out motion estimations in both forward and backward directions, and selects a more reliable one between forward and backward motion vectors by evaluating the motion vector reliability from the viewpoint of the interpolated frame. The proposed algorithm smooths and refines both the forward and backward motion vectors before selecting the reliable one. This procedure helps to select the reasonable motion estimation direction. In identifying the motion vector outliers, the proposed algorithm uses a circular range of which center is located at the mean of the eight neighboring motion vectors of the motion vector being processed. Experimental results using 1720 test images show that the proposed motion estimation algorithm improves the average peak signal-to-noise ratio and the average structural similarity of the interpolated frames by up to 5.31 dB and 0.053, respectively, compared to conventional motion estimation algorithms.

Feature-based motion compensated interpolation for frame rate up-conversion

In this paper, we propose to use feature point matching and mesh-based motion estimation for frame rate up-conversion. Matched local feature points between successive frames in a video sequence are employed to constitute a discrete and sparse motion field. Afterwards, a proximity-constraint marking process regulates the raw correspondence set to a content/motion adaptive nodal set. These correspondences are organized by a Delaunay triangular mesh so as to generate a dense motion field through the affine transformation. The experimental results demonstrate that the proposed scheme outperforms three other popular methods in terms of both subjective and objective evaluations. They also suggest that the proposed method is more suitable for videos containing a complex motion field.

Video Watermarking With Empirical PCA-Based Decoding

A new method for video watermarking is presented in this paper. In the proposed method, data are embedded in the LL subband of wavelet coefficients, and decoding is performed based on the comparison among the elements of the first principal component resulting from empirical principal component analysis (PCA). The locations for data embedding are selected such that they offer the most robust PCA-based decoding. Data are inserted in the LL subband in an adaptive manner based on the energy of high frequency subbands and visual saliency. Extensive testing was performed under various types of attacks, such as spatial attacks (uniform and Gaussian noise and median filtering), compression attacks (MPEG-2, H. 263, and H. 264), and temporal attacks (frame repetition, frame averaging, frame swapping, and frame rate conversion). The results show that the proposed method offers improved performance compared with several methods from the literature, especially under additive noise and compression attacks.

Optimal Analysis Window Size Investigation in Accord with Number of Edge Direction

The display monitors have been developed rapidly. The traditional broadcasting systems employed interlaced scanning format to hide motion artifacts. The procedure of converting interlaced video into a progressive formatted video called frame rate conversion. However, due to the variety of progressive scanning formats, all interlaced scanned contents must be transformed progressive format. In this paper, we study optimal number of window size for frame format conversion method. The objective and subjective performance are shown in performance analysis section.

Joint trilateral motion vector filter for bidirectional motion compensation

Bidirectional motion compensation is an efficient method for frame rate upconversion application. To enhance bidirectional motion vector field (MVF) accuracy, most recent approaches derive it from unidirectional motion vectors (MVs), which may introduce blocking artifacts around object boundaries due to the block-wise motion assignment. Presented is a joint trilateral motion vector filter to improve the MV accuracy of boundary pixels by increasing the resolution of the block-wise unidirectional MVF to a pixel basis. The MV of each pixel is computed as a weighted average of its neighbouring block MVs. Experimental results show that the proposed method can improve the MV accuracy of boundary pixels effectively. Frames interpolated by the proposed method achieve better performance compared with conventional methods.

블록의 성질과 프레임 움직임을 고려한 적응적 확장 블록을 사용하는 프레임율 증강 기법

본 논문에서는 적응적인 확장 블록을 사용하는 프레임율 증강 기법인 AEBME (Adaptive Extended Bilateral Motion Estimation)을 제안하고자 한다. 기존의 EBME (Extended Bilateral Motion Estimation) 알고리듬은 동일한 구역에 두 번의 움직임 예측을 수행함으로 인해 높은 계산량이 요구되었다. 본 논문에서는 영상의 edge 정보를 활용한 블록 유형의 일치 유무를 고려하여 EBME 수행여부를 결정함으로써 움직임 예측 과정을 보다 빠르게 수행하도록 하였다. 움직임 벡터 평활화 과정이 적용되어 움직임 벡터 필드 내의 이상 벡터를 찾아 수정한다. 최종적으로 OBMC (Overlapped Block Motion Compensation)와 MCFI (Motion Compensated Frame Interpolation)이 프레임 움직임의 성질에 따라 적용되어 중간 프레임을 보간하게 된다. 실험 결과를 통해 제안하는 알고리듬이 기존의 알고리듬인 EBME에 비해 향상된 성능과 빠른 속도를 보임을 알 수 있다. In this paper, a novel frame rate up conversion (FRUC) algorithm using adaptive extended bilateral motion estimation (AEBME) is proposed. Conventionally, extended bilateral motion estimation (EBME) conducts dual motion estimation (ME) processes on the same region, therefore involves high complexity. However, in this proposed scheme, a novel block type matching procedure is suggested to accelerate the ME procedure. We calculate the edge information using sobel mask, and the calculated edge information is used in block type matching procedure. Based on the block type matching, decision will be made whether to use EBME. Motion vector smoothing (MVS) is adopted to detect outliers and correct outliers in the motion vector field. Finally, overlapped block motion compensation (OBMC) and motion compensated frame interpolation (MCFI) are adopted to interpolate the intermediate frame in which OBMC is employed adaptively based on frame motion activity. Experimental results show that this proposed algorithm has outstanding performance and fast computation comparing with EBME.

움직임 보상 보간 프레임에 대한 시공간적 통계특성에 기초한 블록기반의 신뢰도 평가 방법

Motion-compensated frame interpolation (MCFI) techniques in video signal processing have many application areas. Frame rate up-conversion (FRUC) or distributed video coding (DVC) technique needs an effective MCFI algorithm. For these applications, it is necessary to develop an effective post-processing technique to improve visual qualities or to reduce virtual channel noises, resulting in the reduced channel bit rate. This paper proposes a reliability evaluation method based on spatio-temporal characteristics for motion-compensated interpolated blocks. The proposed algorithm investigates the temporal matching characteristics for current frame and then is designed in such a way that it can measure temporal characteristics as well as the spatial ones. Through computer simulations, it is shown that the proposed method outperforms the conventional temporal matching method.

Adaptive luminance coding-based scene-change detection for frame rate up-conversion

This paper presents a new scene-change detection method that uses adaptive luminance coding for frame rate up-conversion. The proposed scene-change detection method splits a frame into several blocks and converts the gray levels of pixels in each block to bit codes. Then, it computes the difference between the bit codes in previous and current blocks. In addition, directional distribution analysis is applied to correct the areas falsely detected as a scene change. The experimental results show that the average F1 score of the proposed method was up to 0.479 higher than those of the benchmark methods (a 108.53% improvement). The proposed method also reduced the average computation time per pixel by up to 5.572 μs compared to the benchmark methods (a 73.14% reduction).

Detecting video frame-rate up-conversion based on periodic properties of inter-frame similarity

Video frame-rate up-conversion is one of the common operations for tampering digital videos in the temporal domain, such as creating fake high-quality videos and splicing two video clips with different frame rates. However, few existing works have been proposed for detecting this form of tampering operation. Based on the analysis of extensive experiments, we found that frame-rate up-conversion algorithms employed in most current video editing softwares will inevitably introduce some periodic artifacts into inter-frame similarity in the resulting video frame sequence. By analyzing such artifacts, we propose a simple yet very effective method to expose video after frame-rate up-conversion, and further estimate its original frame rate. The experimental results evaluated on 100 original videos at different frame rates have shown the effectiveness of the proposed method. The average detection accuracy can achieve as high as 99 % on noise-free videos in uncompressed and H.264/AVC formats. Besides, the proposed method is robust to noise as the detection accuracy could reach over 85 % and 95 % on noised videos with Gaussian white noise when SNR is 33 db and 36 db respectively.

Confidence-based adaptive frame rate up-conversion

In this article, we propose a new frame rate up-conversion (FRU) method for temporal quality enhancement. The proposed FRU algorithm employs gradual and adaptive motion estimation based on confidence priority for selecting more accurate motion vectors (MVs). In order to estimate accurate MVs, we adaptively alternate a search range and the order of blocks to be searched depending on the confidence level in hierarchical motion estimation. The precedence of the proposed algorithm is conducted based on the confidence level that is decided by the complexity of pixel values in a block. In addition, we perform bi-directional motion compensation and spatial linear interpolation to fill occlusion regions. In our experiments, we found that the proposed algorithm is about 2 dB better than several conventional methods. Furthermore, block artifacts and blur artifacts are significantly diminished by the proposed algorithm.

Frame rate up-conversion for high-definition video applications

In this paper, a novel frame rate up-conversion (FRUC) scheme to achieve better visual quality for high-definition videos is proposed. The proposed algorithm has three main advantages over conventional FRUC algorithms. First, to increase the accuracy of motion estimation, this paper considers the diversity of motions, the retiming motion vectors of the previous motion field, and an adaptive update strategy. Second, the motion vector that is selected by using the sum of the absolute differences criterion is outputted after a geometric-based motion vector restriction, thereby improving the consistency of the motion vectors. Third, post-processing is performed on the estimated motion vectors based on localized global motions which are calculated by using motion histograms. Experimental results show that the performance of the proposed algorithm significantly outperforms that of traditional methods1.

Improved matching criterion for frame rate upconversion with trilateral filtering

Frame rate upconversion (FRUC) is an important post-processing technique to enhance the visual quality of low frame rate video. A major, recent advance in this area is FRUC based on trilateral filtering which novelty mainly derives from the combination of an edge-based motion estimation block matching criterion with the trilateral filter. However, there is still room for improvement, notably towards reducing the size of the uncovered regions in the initial estimated frame, this means the estimated frame before trilateral filtering. In this context, proposed is an improved motion estimation block matching criterion where a combined luminance and edge error metric is weighted according to the motion vector components, notably to regularise the motion field. Experimental results confirm that significant improvements are achieved for the final interpolated frames, reaching PSNR gains up to 2.73 dB, on average, regarding recent alternative solutions, for video content with varied motion characteristics.

Joint Motion-Compensated Interpolation Using Eight-Neighbor Block Motion Vectors

Novel joint motion-compensated interpolation using eight-neighbor block motion vectors (8J-MCI) is presented. The proposed method uses bi-directional motion estimation (BME) to obtain the motion vector field of the interpolated frame and adopts motion vectors of the interpolated block and its 8-neighbor blocks to jointly predict the target block. Since the smoothness of the motion vector filed makes the motion vectors of 8-neighbor blocks quite close to the true motion vector of the interpolated block, the proposed algorithm has the better fault-tolerancy than traditional ones. Experiments show that the proposed algorithm outperforms the motion-aligned auto-regressive algorithm (MAAR, one of the state-of-the-art frame rate up-conversion (FRUC) schemes) in terms of the average PSNR for the test image sequence and offers better subjective visual quality.

Design and Implementation of PAL to LVDS Video Adapter Based on FPGA

In this paper, based on the FPGA and with a PAL decoder chip, LVDS coding chip and large capacity SRAM, the design and implementation of a PAL system for analog signals to the LVDS conversion of the video signal interface board. First of all, convert the analog signal of PAL to RGB565 digital video signal, and transform the interlaced scan into progressive scan. Then, through the frame rate conversion, resolution expansion etc. algorithm method to handle. Finally, to achieve the interlaced scanning, a resolution of 720×576, 25Hz PAL analog video signal, is converted to a progressive scan, a resolution of 1024×768, 60Hz LVDS video signal transmission. After the measurement, the resolution and frame rate of the video signal conversion interface board are all meet the design requirements. It has been verified the effectiveness of scheme.

Novel True-Motion Estimation Algorithm and Its Application to Motion-Compensated Temporal Frame Interpolation

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

Multiple Hypotheses Bayesian Frame Rate Up-Conversion by Adaptive Fusion of Motion-Compensated Interpolations

Frame rate up-conversion (FRUC) improves the viewing experience of a video because the motion in a FRUC-constructed high frame-rate video looks more smooth and continuous. This paper proposes a multiple hypotheses Bayesian FRUC scheme for estimating the intermediate frame with maximum a posteriori probability, in which both temporal motion model and spatial image model are incorporated into the optimization criterion. The image model describes the spatial structure of neighboring pixels while the motion model describes the temporal correlation of pixels along motion trajectories. Instead of employing a single uniquely optimal motion, multiple “optimal” motion trajectories are utilized to form a group of motion hypotheses. To obtain accurate estimation for the pixels in missing intermediate frames, the motion-compensated interpolations generated by all these motion hypotheses are adaptively fused according to the reliability of each hypothesis. We revealed by numerical analysis that this reliability (i.e., the variance of interpolation errors along the hypothesized motion trajectory) can be measured by the variation of reference pixels along the motion trajectory. To obtain the multiple motion fields, a set of block-matching sizes is used and the motion fields are estimated by progressively reducing the size of matching block. Experimental results show that the proposed method can significantly improve both the objective and the subjective quality of the constructed high frame rate video.

A continuous, editable representation for deforming mesh sequences with separate signals for time, pose and shape

AbstractIt is increasingly popular to represent non‐rigid motion using a deforming mesh sequence: a discrete sequence of frames, each of which is given as a mesh with a common graph structure. Such sequences have the flexibility to represent a wide range of mesh deformations used in practice, but they are also highly redundant, expensive to store, and difficult to edit in a time‐coherent manner. We address these limitations with a continuous representation that extracts redundancy in three separate phases, leading to separate editable signals in time, pose and shape. The representation can be applied to any deforming mesh sequence, in contrast to previous domain‐specific approaches. By modifying the three signal components, we demonstrate time‐coherent editing operations such as local repetition of part of a sequence, frame rate conversion and deformation transfer. We also show that our representation makes it possible to design new deforming sequences simply by sketching a curve in a 2D pose space.

An adaptive bilateral motion estimation algorithm and its hardware architecture

In this paper, an adaptive bilateral motion estimation (ABIME) algorithm is proposed for frame rate up-conversion of high definition (HD) video. The proposed ABIME algorithm can be used as a refinement step after a true motion estimation algorithm. It refines the motion vector field between successive frames by employing a spiral search pattern and by adaptively assigning weights to candidate search locations. An adaptive early termination technique is also proposed for reducing the computational complexity of the ABIME algorithm. In addition, a high performance hardware architecture is proposed for implementing the ABIME algorithm. The proposed hardware uses an efficient memory organization and a novel data reuse scheme in order to reduce the memory bandwidth and control overhead. The proposed hardware is capable of processing 278 1920x1080 full HD frames per second (fps), therefore doubling the frame rate to 556 fps. The proposed early termination technique reduces the energy consumption of the proposed hardware by 29%1.

움직임 보상 보간 프레임에 대한 프레임 적응적 왜곡 예측 기법

비디오 프레임 율 증가 변환은 가전 분야에서 매우 다양한 응용으로 인해 매우 많은 관심을 받아 오고 있다. 대 부분의 진보된 FRUC 알고리즘은 보간된 프레임들의 움직임 벡터장을 결정하는 움직임 보간 기술을 사용하고 있다. 그러나 몇 개의 응용 분야에서는 움직임 보상 보간 프레임이 얼마나 잘 복원되었는지에 대한 정보를 필요로 한다. 이와 같은 목적을 위해 본 논문에서는 프레임 기반의 적응적 예측에 기초한 움직임 보상 보간 프레임의 왜곡 예측 기법을 제안한다. 제안된 기법은 대칭형 움직임 탐색 및 보상 보간 기법에 적용되며, 세 가지 다른 예측 기법 즉, 순방향, 역방향 그리고 적응적 양방향 예측 기법으로 분석된다. 모의 실험을 통하여 제안된 적응적 양방향 왜곡 예측 방식이 다른 두 방식에 비해 성능이 우수함을 보인다. Video FRUC (Frame Rate Up Conversion) has been a technique of great interest due to its diversified applications in consumer electronics. Most advanced FRUC algorithms adopt a motion interpolation technique to determine the motion vector field of interpolated frames. But, in some applications, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame is reconstructed. For this aim, this paper proposes a distortion estimation for motion compensated interpolation frame using frame-adaptive distortion estimation. The proposed method is applied for the symmetric motion estimation and compensated scheme and then analyzed by three different approaches, that is, forward estimation, backward estimation and adaptive bi-directional estimation schemes. Through computer simulations, it is shown that the proposed bi-directional estimation method outperforms others and can be effectively applied for FRUC.

A Performance Comparison of Block-Based Matching Cost Evaluation Models for FRUC Techniques

DVC (Distributed Video Coding) and FRUC (Frame Rate Up Conversion) techniques need to have an efficient motion compensated frame interpolation algorithms. Conventional works of these applications have mainly focused on the performance improvement of overall system. But, in some applications, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame matches the original frame. For this aim, this paper deals with the modeling methods for evaluating the block-based matching cost. First, several matching criteria, which have already been dealt with the motion compensated frame interpolation, are introduced and then combined to make estimate models for the size of MSE (Mean Square Error) noise of the MCI frame to original one. Through computer simulations, it is shown that the block-based matching criteria are evaluated and the proposed model can be effectively used for estimating the MSE noise.