Motion Compensated Frame Rate Up-conversion Research Articles

A fast quaternion wavelet-based motion compensated frame rate up-conversion with fuzzy smoothing: application to echocardiography temporal enhancement

In this paper, we propose a fast Frame Rate Up-Conversion (FRUC) method based on Quaternion Wavelet Transform (QWT) motion estimation to improve the motion estimation accuracy and reduce the computational complexity. The proposed method contains three key elements: motion estimation, motion post-processing, and motion compensated frame interpolation. This paper considers the QWT motion estimation using the phase of image sequences, which leads to more accurate motion estimation, less post-processing procedures, and low complexity as well. Moreover, we proposed a modified fuzzy vector mean filtering to cope with motion outliers and made use of bilateral motion vector modification in the post-processing part. We have assessed our proposed method performance using six widely available benchmark test sequences and three echocardiography image sequences. The evaluations confirm that while our proposed method keeps the PSNR and SSIM performance suitable, it is at least twice faster in comparison with reference methods on the benchmark dataset. The echocardiography evaluations reveal our proposed method superiority in terms of PSNR, SSIM, and computation time, which make the proposed method suitable as an echocardiographic-specific method.

Detection of motion-compensated frame-rate up-conversion via optical flow-based prediction residue

To increase the temporal continuity of low frame-rate videos, Motion-Compensated Frame-Rate Up-Conversion (MC-FRUC), which is a special frame based video editing manipulation, can be employed to synthesize new intermediate frame between two successive frames. However, MC-FRUC can also be utilized to fake high frame-rate videos causing the wastes of storage space or users misleading. Although existing MC-FRUC detectors can obtain desirable performance for uncompressed videos or compressed videos with high perceptual quality, theirs performance degrades when test videos are encoded with higher quantization parameters. To deal with this problem, this paper presents a new optical flow-based detection method. Our method is divided into four steps. First, the forward or backward optical flow map is estimated by optical flow algorithm between the candidate frame, and its previous frame or following frame, respectively. Second, the prediction residue is calculated on the basis of these estimated optical flow maps to eliminate the side effect of texture information in the non-motion regions and the motion regions. Third, Markov statistic features are extracted. Finally, these features are fed into the ensemble classifier for identification of interpolated frames. We test different types of MC-FRUC methods under higher quantization parameters, and the results have shown the effectiveness of the proposed method compared with the existing relative ones.

A 4K$\times$ 2K@60fps Multifunctional Video Display Processor for High Perceptual Image Quality

Video display processing is an indispensable module in a variety of multimedia systems. With the increase of video format to 4K $\times$ 2K@60fps, the design of efficient display processing circuit faces severe challenges. This paper presents a video display processor which supports a variety of video formats and integrates multiple advanced algorithms for perpetual image quality improvement. By leveraging algorithm and architecture co-design, this paper efficiently implements these computational and memory intensive display processing algorithms in a single chip. In particular, we describe three representative design modules in detail, including motion compensated frame rate up-conversion, content-adaptive contrast enhancement, and backlight local dimming. Extensive experiments demonstrate that the proposed design modules outperform the previous works in terms of either image quality or hardware efficiency. The overall chip is fabricated in GF 55nm CMOS technology, and can work at the maximum operating frequency of 594 MHz. The maximum input and output video formats reach up to 4K $\times$ 2K@60fps.

A Spatial Prediction-Based Motion-Compensated Frame Rate Up-Conversion

In Multimedia Internet of Things (IoT), in order to reduce the bandwidth consumption of wireless channels, Motion-Compensated Frame Rate Up-Conversion (MC-FRUC) is often used to support the low-bitrate video communication. In this paper, we propose a spatial predictive algorithm which is used to improve the performance of MC-FRUC. The core of the proposed algorithm is a predictive model to split a frame into two kinds of blocks: basic blocks and absent blocks. Then an improved bilateral motion estimation is proposed to compute the Motion Vectors (MVs) of basic blocks. Finally, with the spatial correlation of Motion Vector Field (MVF), the MV of an absent block is predicted based on the MVs of its neighboring basic blocks. Experimental results show that the proposed spatial prediction algorithm can improve both the objective and the subjective quality of the interpolated frame, with a low computational complexity.

Identification of Motion-Compensated Frame Rate Up-Conversion Based on Residual Signals

Motion-compensated frame rate up-conversion (MC-FRUC) is originally presented to increase the motion continuity of low frame rate videos by periodically inserting new frames, which improves the viewing experience. However, MC-FRUC can also be exploited to fake high frame rate videos or splice two videos with different frame rates for malicious purposes. A blind forensics approach is proposed for the identification of various MC-FRUC techniques. A theoretical model is first built for residual signal, which is exploited as tampering trace for blind forensics. The identification of various MC-FRUC techniques is then converted into a problem of discriminating the differences of residual signals among them. A pre-classifier is designed to suppress the side effects of original frames and static interpolated frames in candidate videos. Then, spatial and temporal Markov statistics features are extracted from the residual signals inside the interpolated frames for MC-FRUC identification. Five open MC-FRUC softwares and six representative MC-FRUC techniques have been tested, and experimental results show that the proposed approach can effectively locate interpolated frames and further identify the adopted MC-FRUC technique for both uncompressed videos and compressed videos with high perceptual qualities.

Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion

Motion-compensated frame-rate up-conversion (MC-FRUC) often exploits either bilateral motion estimation (ME) or unidirectional ME with a fixed block size, which constrains the perceptual quality of up-converted video. In this paper, an advanced MC-FRUC approach is proposed by exploiting hierarchical prediction-based motion vector refinement. To reduce block mismatching in texture regions and color areas, an adaptive multi-layered block matching criterion is designed to extract color and edge information, which is integrated with motion information as constraint term. A hierarchical prediction-based motion vector refinement approach is proposed to obtain more accurate and dense motion vector fields (MVFs). To eliminate the outliers of MVFs, a robust dual-weighted motion vector smoothing scheme is adopted by using both spatial correlation and reliability of neighboring blocks. Experimental results show that the proposed approach has low computational complexity and outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames.

Bi‐directional deformable block‐based motion estimation for frame rate‐up conversion of 360‐degree videos

A bi-directional deformable block-based motion-estimation algorithm to improve the accuracy of a motion prediction in a 360-degree video, used for motion compensated frame-rate up conversion (MC-FRUC) is proposed. The block shapes are adaptively deformed to predict a motion based on the coordinate of the current block, considering the geometric property of the equi-rectangular projection of 360-degree videos. Experimental results demonstrate that the proposed technique can significantly improve the visual quality of the videos as compared with the conventional MC-FRUC.

Blocking-artifact detection in frequency domain for frame-rate up-conversion

In this paper, a blocking-artifact detection method is proposed for motion-compensated frame-rate up-conversion (MC-FRUC) algorithms. In the MC-FRUC process, most of the notable blocking artifacts occur in unreliable motion-compensated regions. To find the unreliable motion-compensated regions, an occlusion detection (OD) method for which the neighboring motion vectors are used is applied. If an occlusion occurs, then the use of median filtering reduces any inappropriate motion-compensated errors; otherwise, the normal motion compensated interpolation (MCI) is applied. In addition, to reduce the occurrence of the blocking artifacts in the MC-FRUC algorithms, a blocking-artifact-reduction method for which the blocking-artifact strength in the frequency domain is applied to the occluded regions is utilized. The experiment results show that the proposed method shows more-favorable peak signal-to-noise ratio (PSNR) values compared with those of the conventional FRUC algorithms regarding full-HD (1920 × 1080) and WQXGA (2560 × 1600) sequences.

Multidirectional motion estimation algorithm for frame rate up-conversion

In this paper, we propose a new motion estimation algorithm for motion-compensated frame rate up-conversion. To reduce computational complexity, it constructs a hierarchical structure using Gaussian image pyramid and unidirectionally estimates motion vectors (MVs) between two original frames at the top level. Since the accuracy of the motion vector may deteriorate at low resolution, each block has five candidates for motion vector so that disadvantages of hierarchical motion estimation are reduced. After motion estimation (ME), MVs of bottom level are corrected based on the sum of absolute difference (SAD) values of the blocks. All unidirectional MVs are assigned to bidirectional MVs, and new frames are interpolated using these MVs. As a result, proposed algorithm shows up to 1.3 dB increase in PSNR compared with conventional algorithm with low computational complexity, and it shows clearer images than conventional algorithms.

프레임율 변환을 위한 주파수 영역에서의 블로킹 현상 검출

In this paper, a blocking-artifact detection method is proposed for motion-compensated frame-rate up-conversion (MC-FRUC) algorithms. In the MC-FRUC process, most of the notable blocking artifacts occur in unreliable motion-compensated regions. To find the unreliable motion-compensated regions, an occlusion detection (OD) method for which the neighboring motion vectors are used is applied. If an occlusion occurs, then the use of median filtering reduces any inappropriate motion-compensated errors; otherwise, the normal motion compensated interpolation (MCI) is applied. In addition, to reduce the occurrence of the blocking artifacts in the MC-FRUC algorithms, a blocking-artifact-reduction method for which the blocking-artifact strength in the frequency domain is applied to the occluded regions is utilized. The experiment results show that the proposed method shows more-favorable peak signal-to-noise ratio (PSNR) values compared with those of the conventional FRUC algorithms regarding full-HD (1920 × 1080) and WQXGA (2560 × 1600) sequences.

Con-Patch: When a Patch Meets Its Context

Measuring the similarity between the patches in images is a fundamental building block in various tasks. Naturally, the patch size has a major impact on the matching quality and on the consequent application performance. Under the assumption that our patch database is sufficiently sampled, using large patches (e.g., 21 × 21 ) should be preferred over small ones (e.g., 7 × 7 ). However, this dense-sampling assumption is rarely true; in most cases, large patches cannot find relevant nearby examples. This phenomenon is a consequence of the curse of dimensionality, stating that the database size should grow exponentially with the patch size to ensure proper matches. This explains the favored choice of small patch size in most applications. Is there a way to keep the simplicity and work with small patches while getting some of the benefits that large patches provide? In this paper, we offer such an approach. We propose to concatenate the regular content of a conventional (small) patch with a compact representation of its (large) surroundings-its context. Therefore, with a minor increase of the dimensions (e.g., with additional ten values to the patch representation), we implicitly/softly describe the information of a large patch. The additional descriptors are computed based on a self-similarity behavior of the patch surrounding. We show that this approach achieves better matches, compared with the use of conventional-size patches, without the need to increase the database-size. Also, the effectiveness of the proposed method is tested on three distinct problems: 1) external natural image denoising; 2) depth image super-resolution; and 3) motion-compensated frame-rate up conversion.

Accurate Frame Rate Up-Conversion for Advanced Visual Quality

Motion-compensated frame rate up-conversion (FRUC) is a method for providing better video image quality than non-motion-based methods. However, incorrect motion vector (MV) prediction can lead to continued induction of debris and artifacts. In this paper, a motion compensated FRUC algorithm is proposed, which is able to acquire more accurate MVs by referencing neighboring frames with hierarchical motion filed construction block size and an MV mapping stage. The block size of hierarchical MV filed construction has been enlarged in order to acquire more motion information, and the size of the MV mapping and image interpolation stage has been reduced to obtain a more accurate interpolated result. In order to smooth the MV, a median filter is considered with sum of absolute difference value and constructed with an adaptive MV smoothing method. For the purpose of suppressing the debris, a noise removal filter is applied after the interpolation operation. According to the experimental result, the proposed approach is able to improve visual quality performance by 0.82 dB on average. Subjective analysis shows that annoying artifacts are significantly reduced.

Wavelet Pyramid Based Multi-Resolution Bilateral Motion Estimation for Frame Rate Up-Conversion

The conventional bilateral motion estimation (BME) for motion-compensated frame rate up-conversion (MC-FRUC) can avoid the problem of overlapped areas and holes but usually results in lots of inaccurate motion vectors (MVs) since 1) the MV of an object between the previous and following frames is more likely to have no temporal symmetry with respect to the target block of the interpolated frame and 2) the repetitive patterns existing in video frame lead to the problem of mismatch due to the lack of the interpolated block. In this paper, a new BME algorithm with a low computational complexity is proposed to resolve the above problems. The proposed algorithm incorporates multi-resolution search into BME, since it can easily utilize the MV consistency between two adjacent pyramid levels and spatial neighboring MVs to correct the inaccurate MVs resulting from no temporal symmetry while guaranteeing low computational cost. Besides, the multi-resolution search uses the fast wavelet transform to construct the wavelet pyramid, which not only can guarantee low computational complexity but also can reserve the high-frequency components of image at each level while sub-sampling. The high-frequency components are used to regularize the traditional block matching criterion for reducing the probability of mismatch in BME. Experiments show that the proposed algorithm can significantly improve both the objective and subjective quality of the interpolated frame with low computational complexity, and provide the better performance than the existing BME algorithms.

Multi-Channel Mixed-Pattern Based Frame Rate Up-Conversion Using Spatio-Temporal Motion Vector Refinement and Dual-Weighted Overlapped Block Motion Compensation

In this paper, a novel motion compensated frame rate up-conversion (MC-FRUC) algorithm is proposed to enhance the visual quality of video sequences. First of all, the multi-channel mixed pattern (MCMP) is proposed to design a block matching criterion which uses a few computations to reveal the variances of one luminance channel and two chrominance channels in a video frame. Second, in basis of the forward and backward initial motion vector fields (MVFs) estimated by a variant of 3DRS algorithm, the spatio-temporal motion vector refinement (ST-MVR) algorithm is proposed to obtain the more smoother MVF by implicitly adding the spatio-temporal smooth constraint into the process of motion vector refinement, and then the highly fault-tolerant motion vector smoothing (HFT-MVS) algorithm is proposed to prevent the emergence of outliers in MVF. Finally, in order to reduce the edge blurring and occlusions, the proposed dual-weighted overlapped block motion compensation (DW-OBMC) algorithm uses the forward and backward MVFs to jointly produce the interpolated frame. Experimental results show that the proposed algorithm can significantly improve both objective and subjective quality of the interpolated frame with a low computational complexity, and provide the better performance than the existing algorithms.

Frame rate up-conversion based on dual criterion motion estimation and motion vector adjustment

This paper presents a novel motion-compensated frame rate up-conversion algorithm that uses dual criterion motion estimation to obtain accurate motion vectors in the circumstance of scene luminance changing. In the proposed algorithm, we also present a selective vector median filtering method and a motion vector refinement method to address block mismatching problem. Finally, a motion-compensated interpolation algorithm based on adaptive overlapped block motion compensation is proposed to reduce block artifact. Experimental results demonstrate that the proposed algorithm improves visual qualities compared with the existing algorithms.

Hybrid Frame Rate Upconversion Method Based on Motion Vector Mapping

Liquid crystal displays (LCDs), which serve as receivers of high visual quality video, suffer from motion blur issues. One of the methods in terminating motion blur is motion compensated frame rate upconversion, which is widely adopted in LCDs. In the previous work, i.e., particle-based frame rate upconversion, the computation complexity is high while repeated operations and some improper cost evaluation setup are observed. Therefore, in this paper, hybrid frame rate upconversion is proposed with two features. First, the cost evaluation for particle-based motion trajectory calibration is modified based on the possible noise sources and video resolution variations. Second, repeated operations in particle-based motion trajectory calibration are observed. Therefore, original particle-based motion trajectory calibration is replaced by initial motion vector assignment and subsequent motion vector mapping to achieve computation complexity reduction, while the effective search range is relatively expanded. According to the experiment results, the visual quality is enhanced by 1.87 dB on average, compared with state-of-the-art unidirectional-based frame rate upconversion approaches. On the other hand, the computation complexity of the proposed design is reduced by 25-90% based on target video resolution, concluding a high visual quality and low computation complexity frame rate upconversion design.

Direction-Select Motion Estimation for Motion-Compensated Frame Rate Up-Conversion

In this paper, we propose a new motion estimation algorithm to be used for motion-compensated frame rate up-conversion. The proposed algorithm independently carries out motion estimations in both forward and backward directions, and selects a more reliable one between forward and backward motion vectors by evaluating the motion vector reliability from the viewpoint of the interpolated frame. The proposed algorithm smooths and refines both the forward and backward motion vectors before selecting the reliable one. This procedure helps to select the reasonable motion estimation direction. In identifying the motion vector outliers, the proposed algorithm uses a circular range of which center is located at the mean of the eight neighboring motion vectors of the motion vector being processed. Experimental results using 1720 test images show that the proposed motion estimation algorithm improves the average peak signal-to-noise ratio and the average structural similarity of the interpolated frames by up to 5.31 dB and 0.053, respectively, compared to conventional motion estimation algorithms.

Novel True-Motion Estimation Algorithm and Its Application to Motion-Compensated Temporal Frame Interpolation

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

Motion-Compensated Frame Rate Up-Conversion—Part II: New Algorithms for Frame Interpolation

Motion-compensated frame rate up-conversion (MC-FRUC) consists of two key elements: motion estimation and motion-compensated frame interpolation. The motion estimation algorithm presented in , which is used in the MC-FRUC method proposed in this paper, provides unidirectional motion trajectories. The advantage of this motion estimation algorithm, besides its accuracy, is that it provides information on occlusions. However, motion compensation along unidirectional motion trajectories yields overlaps, holes, and blocking artifacts. To solve these problems, this paper presents two new algorithms for unidirectional motion-compensated frame interpolation: irregular-grid expanded-block weighted motion compensation (IEWMC) and block-wise directional hole interpolation (BDHI). The IEWMC is used to reduce the blocking artifacts and solve the problem of overlapping blocks. The BDHI preserves local texture and edges while filling holes. Experimental results show that the IEWMC outperforms conventional motion compensation, and the BDHI is better than the repeated median filter that is often used to fill holes. The performance of the proposed MC-FRUC, that uses the two new algorithms and the unidirectional motion estimation algorithm, is evaluated against three existing MC-FRUC techniques: a typical bi-directional algorithm, an object-based algorithm, and a commercial plug-in product. Experimental results show that the quality of the pictures interpolated using the proposed MC-FRUC method is much higher than those interpolated using the three existing MC-FRUC techniques.

Motion-Compensated Frame Rate Up-Conversion—Part I: Fast Multi-Frame Motion Estimation

Motion-compensated frame rate up-conversion is used to convert video/film materials of low frame rates to a higher frame rate so that the materials can be displayed with smooth motion and high-perceived quality. It consists of two key elements: motion estimation and motion-compensated frame interpolation. It requires accurate motion trajectories to ensure quality results and low computational cost to ensure practical applications. This paper presents a novel motion estimation algorithm that combines the accuracy of maximum a posteriori probability (MAP) estimation with the speed of hierarchical block-matching algorithm (BMA). This MAP estimation uses three consecutive pictures, instead of the conventional two, and one previously estimated motion field to exploit the temporal correlation between motion fields and to determine motion in occluded areas. The optimization of the MAP estimation is performed using full-search and implemented by means of look-up tables. The full search ensures that the optimization converges to the global minimum, while the look-up tables dramatically reduce the computational cost. Experimental results show that the proposed algorithm provides motion trajectories that are much more accurate than those obtained using either the full-search BMA or hierarchical BMA alone. Also, it is much faster than the full-search BMA.