Video Frame Rate Up-conversion Research Articles

Frame rate up-conversion algorithm based on adaptive-agent motion compensation combined with semantic feature analysis

In this paper, a novel video frame rate up-conversion (FRUC) technique based on adaptive motion compensation is presented by combining the semantic feature analysis for image sequences. Firstly, the image is divided into the sub-patch with the same size and their feature areextracted. And, semantic feature analysis is adopted to generate a more accurate motion vector field from the previous frame to the following frame, and then we select these suitable classification algorithms for patial-temporal patches to divide the frame into moving areas, static areas and fault areas. By using the semantic-based adaptive interpolation, we developed a hierarchical refinement strategy to adaptively correct these motion vector so as to get the interpolation frame. Experimental results show that the performance of our method is better than those of the popular I-FRUC, B-FRUC, F-FRUC and A-FRUC methods in both objective and subjective quality, which has relative advantage for engineering applications.

Detecting video frame rate up-conversion based on frame-level analysis of average texture variation

Frame rate up-conversion (FRUC) refers to frame interpolation between adjacent video frames to increase the motion continuity of low frame rate video, which can improve the visual quality on hand-held displays. However, FRUC can also be used for video forgery purposes such as splicing two videos with different frame-rates. We found that most FRUC approaches introduce visual artifacts into texture regions of interpolated frames. Based on this observation, a two-stage blind detection approach is proposed for video FRUC based on the frame-level analysis of average texture variation (ATV). First, the ATV value is computed for each frame to obtain an ATV curve of candidate video. Second, the ATV curve is further processed to highlight its periodic property, which indicates the existence of FRUC operation and further estimates the original frame rate. Thus, the positions of interpolated frames can be inferred as well. Extensive experimental results show that the proposed forensics approach is efficient and effective for the detection of existing typical FRUC approaches such as linear frame averaging and motion-compensated interpolation (MCI). The detection performance is superior to the existing approaches in terms of time efficiency and detection accuracy.

Detecting video frame-rate up-conversion based on periodic properties of edge-intensity

Video frame-rate up-conversion (FRUC) is one of the common temporal-domain operations. From the earlier frame repetition and linear interpolation, FRUC has been developed to motion compensated frame interpolation (MCFI), which effectively overcomes the temporal jerkiness and ghosting shadows. In a broad sense, FRUC can be regarded as a video forgery operation. By experiments, it is observed that FRUC still leads to edge dis-continuity or over-smoothing artifacts around object boundaries. In this paper, an edge-intensity based passive forensics approach is proposed to detect the possible FRUC operation in candidate video. After computing the edge intensities of every frame, Kaufman adaptive moving average (KAMA) is exploited to define an adaptive threshold to distinguish the interpolating frames by FRUC from the original frames. Moreover, the original frame-rate of up-converted video can be inferred. Experimental results show that the proposed approach is not only effective for simple frame repetition and linear interpolation, but also valid for advanced FRUC techniques such as MCFI. The detection accuracy is up to 94.5% on average. Its computation is simple as well.

폐색영역탐지 기능을 갖는 프레임율 변환

A new technology on video frame rate up-conversion (FRUC) is presented by combining the median filter and motion estimation (ME) with an occlusion detection (OD) method. First, ME is performed to have a motion vector. Then, the OD method is used to refine motion vector in the occlusion region. Since the wrong motion vector can be obtained with high possibility in the occluded area, a median filtering that less depends on the motion vector is applied to that area, and since the motion vector is continuous and robust in the non-occluded area, BDMC(Bi-Directional Motion Compensated interpolation) is applied to obtain interpolated image in that area. BDMC using the bi-directional motion vectors achieves good results when continuity and robustness of the motion vector is higher. Experimental results show that the proposed algorithm provides better performance than the conventional approach. The average gain of PSNR (Peak Signal to Noise Ratio) is approximately 0.16 dB in the test sequences compared with BDMC.

Frame Rate Up-Conversion Using Median Filter and Motion Estimation with Occlusion Detection

A new technology for video frame rate up-conversion (FRUC) is presented by combining a median filter and motion estimation (ME) with an occlusion detection (OD) method. First, ME is performed to obtain a motion vector. Then, the OD method is used to refine the MV in the occlusion region. When occlusion occurs, median filtering is applied. Otherwise, bidirectional motion compensated interpolation (BDMC) is applied to create the interpolated frames. The experimental results show that the proposed algorithm provides better performance than the conventional approach. The average gain in the PSNR (Peak Signal to Noise Ratio) is always better than the other methods in the Full HD test sequences.

Detecting video frame-rate up-conversion based on periodic properties of inter-frame similarity

Video frame-rate up-conversion is one of the common operations for tampering digital videos in the temporal domain, such as creating fake high-quality videos and splicing two video clips with different frame rates. However, few existing works have been proposed for detecting this form of tampering operation. Based on the analysis of extensive experiments, we found that frame-rate up-conversion algorithms employed in most current video editing softwares will inevitably introduce some periodic artifacts into inter-frame similarity in the resulting video frame sequence. By analyzing such artifacts, we propose a simple yet very effective method to expose video after frame-rate up-conversion, and further estimate its original frame rate. The experimental results evaluated on 100 original videos at different frame rates have shown the effectiveness of the proposed method. The average detection accuracy can achieve as high as 99 % on noise-free videos in uncompressed and H.264/AVC formats. Besides, the proposed method is robust to noise as the detection accuracy could reach over 85 % and 95 % on noised videos with Gaussian white noise when SNR is 33 db and 36 db respectively.

움직임 보상 보간 프레임에 대한 프레임 적응적 왜곡 예측 기법

비디오 프레임 율 증가 변환은 가전 분야에서 매우 다양한 응용으로 인해 매우 많은 관심을 받아 오고 있다. 대 부분의 진보된 FRUC 알고리즘은 보간된 프레임들의 움직임 벡터장을 결정하는 움직임 보간 기술을 사용하고 있다. 그러나 몇 개의 응용 분야에서는 움직임 보상 보간 프레임이 얼마나 잘 복원되었는지에 대한 정보를 필요로 한다. 이와 같은 목적을 위해 본 논문에서는 프레임 기반의 적응적 예측에 기초한 움직임 보상 보간 프레임의 왜곡 예측 기법을 제안한다. 제안된 기법은 대칭형 움직임 탐색 및 보상 보간 기법에 적용되며, 세 가지 다른 예측 기법 즉, 순방향, 역방향 그리고 적응적 양방향 예측 기법으로 분석된다. 모의 실험을 통하여 제안된 적응적 양방향 왜곡 예측 방식이 다른 두 방식에 비해 성능이 우수함을 보인다. Video FRUC (Frame Rate Up Conversion) has been a technique of great interest due to its diversified applications in consumer electronics. Most advanced FRUC algorithms adopt a motion interpolation technique to determine the motion vector field of interpolated frames. But, in some applications, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame is reconstructed. For this aim, this paper proposes a distortion estimation for motion compensated interpolation frame using frame-adaptive distortion estimation. The proposed method is applied for the symmetric motion estimation and compensated scheme and then analyzed by three different approaches, that is, forward estimation, backward estimation and adaptive bi-directional estimation schemes. Through computer simulations, it is shown that the proposed bi-directional estimation method outperforms others and can be effectively applied for FRUC.

움직임 보상 보간 프레임에 대한 블록 적응적 정합 특성을 이용한 왜곡 예측 기법

비디오 프레임 율 증가 변환은 최근에 가전 분야에서 비디오 정보원과 더불어 디스플레이 포맷의 폭발적인 증가에 힘입어 매우 많은 관심의 대상이 되었다. 대부분의 진보된 FRUC 알고리즘은 보간된 프레임들의 움직임 벡터장을 결정하는 효과적인 움직임 보간 기술을 사용하고 있다. 그러나 수신기의 후처리 분야와 같은 응용 분야에서는 움직임 보상 보간 프레임이 얼마나 잘 복원되었는지에 대한 정보를 필요로 한다. 이와 같은 목적을 달성하기 위해, 먼저, 본 논문에서는 움직임 보상 보간 프레임의 블록 단위로 신뢰도를 측정할 수 있는 비용 함수들을 도입한다. 그런 후에 이러한 함수들을 사용하여 움직임 보상 보간 프레임에서 얼마나 많은 잡음이 포함되어 있는지를 평가할 수 있는 두 가지의 왜곡 예측 모델을 제안한다. 모의실험을 통하여 제안된 왜곡 예측 방식은 움직임 보상 보간 프레임의 잡음을 효과적으로 예측할 수 있음을 보인다. Video FRUC (Frame Rate Up Conversion) is one of the main issues that have arisen in recent years with the explosive growth of video sources and display formats in consumer electronics. Most advanced FRUC algorithms adopt an efficient motion interpolation technique to determine the motion vector field of interpolated frames. But, in some application areas such as post processing in receiver side, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame was reconstructed. In order to achieve this aim, first, this paper introduces some cost functions to estimate the reliability of a block in the MCI frame. Then, by using these functions, this paper proposes two distortion estimation models for evaluating how much noise was produced in the MCI frame. Through computer simulations, it is shown that the proposed estimation methods perform effectively in estimating the noises of the MCI frame.

A high performance hardware architecture for multi-frame hierarchical motion estimation

This paper presents the architecture design and FPGA implementation of a multi-frame hierarchical motion estimation (MFHME) circuit. The target application of the circuit is high quality motion-compensated video frame rate up-conversion that requires dense motion fields (MF) and accurate motion trajectories. To obtain accurate motion trajectories, the circuit uses two frames as references and calculates the block matching errors for both the luminance and chrominance components of the images. In addition, the sum of squared pixel differences, instead of the sum of the absolute pixel differences, is used as the metric of the block matching errors in order to further improve the accuracy of the estimated motion trajectories. To achieve low computation complexity, the circuit has been designed based on a hierarchical structure and a pre-computed lookup table is used to provide the squared pixel differences. The implementation result shows that the circuit is able to support the frame rate up-conversion of high definition video (1080P format) from 30 to 60 frames per second at a clock frequency of 55 MHz.

Low complexity algorithm for robust video frame rate up-conversion (FRUC) technique

Two challenging situations for video frame rate up-conversion (FRUC) are first identified and analyzed; namely, when the input video has abrupt illumination change and/or a low frame rate. Then, a low-complexity processing technique and robust FRUC algorithm are proposed to address these two issues. The proposed algorithm utilizes a translational motion vector model of the first- and the second-order and detects the continuity of these motion vectors. Additionally, in order to improve perceptual quality of interpolated frame, spatial smoothness criterion is employed. The superior performance of the proposed algorithm has been tested extensively and representative examples are given in this work.

Adaptive frame rate up-conversion based on motion classification

In this paper, a new technique on video frame rate up-conversion (FRUC) is presented by combining the adaptive motion classification (AMC) for image sequences and the mixed motion estimation (ME). In the proposed FRUC scheme, the AMC classifies sequences into global and local motion sequences (G/LMS) whose advantages are fully utilized. The mixture of bidirectional ME (BDME) and unidirectional ME (UDME) is proposed to avoid hole and “background replace object” phenomenon in interpolated frames. Moreover, the motion vectors (MVs) refinement technique is employed to smooth MVs field. For LMS, the motion detection is used to deal with static background and occlusions are located by using forward prediction of motion regions. Correctional sum of absolute difference (CSAD) which gives more precise physical MVs field than conventional SAD is applied during all of the ME process. Simulation results demonstrate the effectiveness of the proposed FRUC algorithm.

Video frame rate up conversion under inconsistent camera motion

In this paper, we address the problem of video frame rate up-conversion (FRC) in the compressed domain. FRC is often recognized as video temporal interpolation. This problem is very challenging when targeted for video sequences with inconsistent camera and object motion, such as sports videos. A novel compressed domain motion compensation scheme is presented and applied in this paper, aiming at up-sampling frame rates in sports videos. MPEG-2 encoded motion vectors (MVs) are utilized as inputs in the proposed algorithm. The decoded MVs undergo a cumulative spatiotemporal interpolation. An iterative rejection scheme based on the dense motion vector field (MVF) and the generalized affine motion model is exploited to detect global camera motion. Subsequently, the foreground object separation is performed by additionally examining the temporal consistency of the output of iterative rejections. This consistency check process helps coalesce the resulting foreground blocks and weed out the unqualified blocks. Finally, different compensation strategies for the camera and object motions are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block and non-block based frame interpolation approaches.