Motion Compensated Frame Interpolation Research Articles

SIGNAL TRANSMISSION OPTIMIZATION MOVING-VECTOR CLASSIFICATION MPEG STANDARD VIDEO CODING

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and also Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase-based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

Signal transmission optimization moving-vector classification Mpeg standard video coding

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

Signal transmission optimization moving-vector classification mpeg standard video coding

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

Quality improvement of motion-compensated frame interpolation by self-similarity based context feature

Quality improvement of motion-compensated frame interpolation by self-similarity based context feature

Identification of Interpolated Frames by Motion-Compensated Frame-Interpolation via Measuring Irregularity of Optical Flow

Motion-compensated frame-interpolation (MCFI), synthesize intermediate frames between input frames guided by estimated motion, can be employed to falsify high bit-rate videos or high frame-rate videos with different frame-rates. Although existing MCFI identification methods have obtained satisfactory results, they are seriously degraded by stronger compression. Therefore, to conquer this issue, a blind forensics method is proposed to identify the adopted MCFI methods by considering the irregularities of optical flow produced by various MCFIs. In this paper, a set of compact features are constructed from the motion-aligned frame difference-weighted histogram of local binary pattern on the basis of optical flow (MAFD-WHLBP). Experimental results show that the proposed approach outperforms existing MCFI detectors under stronger compression.

Low Complex Block Level Correlation and Registration for Video Frame Interpolation

In this paper, we propose a low complex block level correlation and registration for Video Frame Interpolation (VFI) algorithm. The proposed method finds the correlation between the corresponding non-overlapping blocks. The correlation coefficients indicate the amount of similarity between the blocks. Using the median of correlation coefficients as threshold, the blocks are blended using image registration process. Compared to existing block based Motion Compensated Frame Interpolation (MCFI) methods, proposed algorithm is a low complex algorithm. In the proposed, there is no need of Motion Estimation (ME), smoothing of unreliable Motion Vectors (MVs) and Motion Prediction. Extensive experiments are conducted on various benchmark video sequences. Subjective assessment shows that the artefacts like blocky artefacts, holes, blurry artefacts, and ghost effects in MCFI are overcome by this algorithm, as there is no ME, preserving the true motion of the object to a greater extent is possible. The metrics PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Metric) provide the objective assessment.

Motion‐Compensated Frame Interpolation Using Cellular Automata‐Based Motion Vector Smoothing

Motion‐Compensated Frame Interpolation (MCFI) is one of the common temporal‐domain tamper operations, and it is used to produce faked video frames for improving the visual qualities of video sequences. The instability of temporal symmetry results in many incorrect Motion Vectors (MVs) for Bidirectional Motion Estimation (BME) in MCFI. The existing Motion Vector Smoothing (MVS) works often oversmooth or revise correct MVs as wrong ones. To overcome this problem, we propose a Cellular Automata‐based MVS (CA‐MVS) algorithm to smooth the Motion Vector Field (MVF) output by BME. In our work, a cellular automaton is constructed to deduce MV outliers according to a defined local evolution rule. By performing CA‐based evolution in a loop iteration, we gradually expose MV outliers and reduce incorrect MVs resulting from oversmoothing as many as possible. Experimental results show the proposed algorithm can improve the accuracy of BME and provide better objective and subjective interpolation qualities when compared with the traditional MVS algorithms.

A robust bidirectional motion-compensated interpolation algorithm to enhance temporal resolution of 3D echocardiography

High frame rate three-dimensional echocardiography (3DE) improves the assessment of the cardiac dynamics by providing a realistic view of the cardiac structures. However, since the field of view in 3D is large and the speed of sound in tissue is limited, one must sacrifice some spatial resolution to achieve a temporally smoothed sequence of 3DE data. Motion-compensated frame interpolation provides a means to enhance perceived temporal resolution without degrading the spatial resolution. Among the existing interpolation techniques, motion-compensated interpolation based on 3D speckle tracking (3DST) is very simple to implement and can be applied in real time since it is well suited for parallel processing. However, the performance of this approach deteriorates in low frame rate 3DE due to large inter-frame displacements. In this paper, we propose a new 3DST-based motion-compensated interpolation algorithm based on a multiscale bidirectional motion estimator. The proposed estimator improves the reliability of the motion estimate at the coarsest scale by utilizing a multidirectional speckle tracker. Moreover, the proposed 3DST algorithm requires smaller search regions for an equivalent search effect compared to the existing algorithms; therefore, it reduces the computation time of 3DST. To reduce the interpolation artifacts caused by decorrelation of the speckle pattern and inaccurate motion estimates, we develop an adaptive interpolation scheme based on robust statistics and the structural similarity index. Experimental results demonstrate that the proposed algorithm, compared to an existing 3DST-based algorithm, greatly reduces the interpolation artifacts such as ghosting and blurring, especially near fast-moving cardiac structures.

A fast quaternion wavelet-based motion compensated frame rate up-conversion with fuzzy smoothing: application to echocardiography temporal enhancement

In this paper, we propose a fast Frame Rate Up-Conversion (FRUC) method based on Quaternion Wavelet Transform (QWT) motion estimation to improve the motion estimation accuracy and reduce the computational complexity. The proposed method contains three key elements: motion estimation, motion post-processing, and motion compensated frame interpolation. This paper considers the QWT motion estimation using the phase of image sequences, which leads to more accurate motion estimation, less post-processing procedures, and low complexity as well. Moreover, we proposed a modified fuzzy vector mean filtering to cope with motion outliers and made use of bilateral motion vector modification in the post-processing part. We have assessed our proposed method performance using six widely available benchmark test sequences and three echocardiography image sequences. The evaluations confirm that while our proposed method keeps the PSNR and SSIM performance suitable, it is at least twice faster in comparison with reference methods on the benchmark dataset. The echocardiography evaluations reveal our proposed method superiority in terms of PSNR, SSIM, and computation time, which make the proposed method suitable as an echocardiographic-specific method.

Forgery detection of motion compensation interpolated frames based on discontinuity of optical flow

Motion Compensated Frame Interpolation (MCFI), a frame-based video operation to increase the motion continuity of low frame rate video, can be adopted by falsifiers for forging high bitrate video or splicing videos with different frame-rates. For existing MCFI detectors, their performance are degraded by stronger video compression, and noise. To deal with this problem, we propose a blind forensics method to detect the adopted MCFI operation. After investigating the synthetic process of interpolated frames, we discover that motion regions of interpolated frames exist some local slight artifacts, causing the optical flow based inter-frame discontinuity. To capture this irregularities introduced by various MCFI techniques, compact features are designed, which are calculated as Temporal Frame Difference-weighted histogram of Local Binary Pattern computed on Optical Flow field (TFD-OFLBP). Meanwhile, Local Inter-block and Edge-block difference Features (LIEF) are further proposed to detect interpolation frames with stable content. Besides, a set of forensics tools are adopted to eliminate the side effects of possible interferences of the scenes change, sudden lighting change, focus vibration, and some original frames with inherent local artifacts. Experimental results on four representative MCFI software and techniques show that the proposed approach outperforms existing MCFI detectors and also with robustness to compression, and noise.

Volumetric interpolation of tomographic sequences for accurate 3D reconstruction of anatomical parts

Background and objective:Tomographic sequences of biomedical images are commonly used to achieve a three-dimensional visualization of the human anatomy. In some cases, the number of images contained in the sequence is limited, e.g., in low-dose computed tomography acquired on neonatal patients, resulting in a coarse and inaccurate 3D reconstruction.Methods:In this paper, volumetric image interpolation methods, devised to increase the axial resolution of tomographic sequences and achieve a refined 3D reconstruction, are proposed and compared. The techniques taken into consideration are based on motion-compensated frame-interpolation concepts, which have been developed for video applications, mainly frame-rate conversion.Results: The performance of the proposed methods is quantitatively assessed by using sequences with a simulated low axial resolution obtained from the decimation of standard high-resolution computed tomography sequences. Real data with an actual low axial resolution have been used as well for a qualitative evaluation of the proposed methods.Conclusions:The experimental results demonstrate that the proposed methods enable an effective slice interpolation and that the achievable 3D models clearly benefit from the increased axial resolution.

Frame rate up-conversion in cardiac ultrasound

The temporal resolution of echocardiographic sequences affects the interpretation of cardiac motion as well as the viewing experience. Temporal resolution, however, is limited by the propagation velocity of sound in tissue. Altering the acquisition system can improve temporal resolution but normally reduces image quality. Alternatively, motion compensated frame interpolation enhances perceived temporal resolution without reducing spatial resolution. There are however significant challenges with these techniques due to the simultaneous presence of large and small motions of the cardiac structures. In this paper, we propose a new motion compensated frame interpolation algorithm based on a variational bidirectional motion estimation model. We derive the Euler-Lagrange equations characterizing the optimal dense bidirectional motion field and develop a multiscale iterative procedure to obtain the motion field and the missing intermediate frames. The performance of the proposed algorithm was evaluated on real cardiac ultrasound recordings, using both subjective and objective evaluation criteria. Experimental results show that the interpolated frames generated by the proposed algorithm do not have the ghosting or blurring artifacts that are commonly present in the existing motion compensated frame interpolation algorithms, especially near fast moving structures like the cardiac valves. Quantitatively, the proposed algorithm achieves an improvement in the average peak signal-to-noise ratio of 1.16 dB and in the structural similarity index of 3.0%.

Correlation Based Low Complex Video Frame Interpolation

Frame Interpolation is one of the main stages in video processing. Video coding standards skip some in-between frames for efficient compression and coding. At decoder the common approach to reconstruct the skipped frame using Motion Compensated Frame Interpolation (MCFI)methods. In MCFI, computational complexity is very high as calculation of Block Matching Algorithm, Motion Vectors (MV), Motion Estimation(ME) and Prediction logic of objects in motion between the frames, increases the complexity in MCFI method. A more feasible approach with minimum computational complexity using block level correlation is proposed in this paper. Errored MV at the decoder results in holes, occlusions, blurring and edge deformations in the interpolated frame. This proposal minimizes afore mentioned effects along with minimum complexity. The results are simulated in terms of peak-signal-to-noise-ratio (PSNR) and structural similarity index (SSIM).

Robust Localization of Interpolated Frames by Motion-Compensated Frame Interpolation Based on an Artifact Indicated Map and Tchebichef Moments

Motion-compensated frame interpolation (MCFI), a frame-interpolation technique to increase the motion continuity of low frame-rate video, can be utilized by counterfeiters for faking high bitrate video or splicing videos with different frame rates. For existing MCFI detectors, their performances are degraded under real-world scenarios such as H.264/AVC compression, noise, or blur. To address this issue, a robust MCFI detector is proposed to locate interpolated frames. By analyzing the distribution of residual energies within interpolated frames, we observe that there exist strong correlations between artifact regions and high residual energies. Thus, an artifact indicated map is introduced to select candidate artifact regions. Then, Tchebichef moments (TMs) are exploited to characterize the blurring effects or deformed structures among these regions. Specifically, the mean value of absolute high-order TMs of selected regions is used to model these temporal inconsistencies. Finally, a sliding window is adopted to locate interpolated frames, which are further refined by three post-processing operations. Chrominance information is also integrated with luminance information for robust identification of interpolated frames. Extensive experimental results show that compared with the state-of-the-art MCFI detectors, the proposed approach is more robust for compressed videos under various real-world scenarios.

Triple-Frame-Based Bi-Directional Motion Estimation for Motion-Compensated Frame Interpolation

In this paper, new motion estimation (ME) and motion vector refinement (MVR) methods for motion-compensated frame interpolation (MCFI) are proposed. To estimate motion vectors (MVs) with great reliability, triple-frame-based bi-directional ME (TFBME), which employs three sequential frames, is developed. In TFBME, the middle frame works as a reference to estimate MVs between two end frames, which leads to performance improvement especially when objects move linearly through three sequential frames. In addition, a modified bi-directional ME method, which employs only two successive frames, is combined with TFBME to handle cases in which objects move non-linearly through three sequential frames. In our proposed MVR, artifacts on interpolated frames are first detected by a convolutional neural network and then corrected by weighted vector median filtering. Experimental results show that the proposed MCFI method outperforms conventional methods in both subjective and objective evaluations.

A Bayer motion estimation for motion-compensated frame interpolation

We propose a Bayer ME algorithm which is used to improve the performance of Motion-Compensated Frame Interpolation (MCFI). The core of the proposed algorithm is a predictive model designed from the alternate arrangement of Bayer pattern. According to the predictive model, the Motion Vector Field (MVF) of the interpolated frame is first split into basic blocks and absent blocks, and then an improved Bilateral Motion Estimation (BME) is proposed to compute the MVs of basic blocks. Finally, with the local stationary statistics of MVF, the MV of an absent block is predicted from the MVs of its neighboring basic blocks. Experimental results show that the proposed Bayer ME algorithm can improve both objective and subjective quality of the interpolated frame with a low computational complexity.

Detection of motion compensated frame interpolation via motion-aligned temporal difference

Motion compensated frame interpolation (MCFI) is a special frame based video manipulation, which increases the temporal continuity of low frame rate videos by synthesizing new frames between successive frames. MCFI might also be used to counterfeit high frame-rate videos, which mislead users’ attraction and waste storage spaces in video-sharing websites. Existing MCFI detectors are designed to judge the absence or presence of MCFI forgery in a controllable environment of known MCFI techniques. Practical detector should consider the possibility of unknown MCFI techniques. We are motivated to propose a robust MCFI detector for more practical scenarios. Considering the effects of non-motion regions in candidate frame, the statistical moments are firstly extracted from motion-aligned frame differences (MAFD). Then, the one-class support vector machine (SVM), following a training stage capturing the properties of original frames, is exploited to judge whether the candidate frame is interpolated by MCFI or not. Finally, a special interpolated frame detection (SIFD) is designed to pick out interpolated frames, which are synthesized from two consecutive reference frames with no motion vectors (MVs) or less MVs. A series of experiments evaluated on four representative MCFI techniques have shown promising results.

Hierarchical Extended Bilateral Motion Estimation-Based Frame Rate Upconversion Using Learning-Based Linear Mapping.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

Motion-compensated frame interpolation using patch-based sparseland model

Using patch-based sparseland model, we design a novel Motion-Compensated Frame Interpolation (MCFI) method. Optical-flow estimation is first adopted to generate a reliable Motion Vector Field (MVF) from the previous frame to the following frame. Then we use patch-based bidirectional motion estimation to generate a smooth Motion Vector Felid (MVF). In the last step, we extract patches from reference frames along the motion trajectories, and perform Principle Component Analysis (PCA) to generate dictionaries that code the patches with various local structures. The sparseland model becomes the prior knowledge of the intermediate frame to fuse into the Motion Compensated Interpolation (MCI) by maximum a posteriori (MAP) criterion. By iterative numerical computing, we solve this sparseland-prior reconstruction model, and acquire a sparsity-preferred interpolated frame. Experimental results show that our method outperforms the existing algorithms in both objective and subjective picture qualities, but it also introduces a high computational complexity in the meantime.

Motion-Compensated Temporal Frame Interpolation Algorithm Based on Global Entirety Unidirectional Motion Estimation and Local Fast Bidirectional Motion Estimation

To solve the uncertainty of motion for objects displayed on liquid crystal display (LCD), the motion estimation (ME) algorithm on the basis of global entirety unidirectional motion and local fast bidirectional motion (GEU-LFB) is proposed for motion-compensated temporal frame interpolation (MCTFI). Firstly, by observing the moving objects in the scene, we will obtain a set of global motion vectors (MVs). Secondly, the fast local search based on the bidirectional ME is executed. Additionally, to make up for the defect of the bidirectional ME, a method to generate the exposure mask and the occlusion mask is proposed by cleverly using the matching criterion of absolute difference and the set of global MVs. Next, the optimal MV field is smoothed by the vector median filter. Finally, the temporal interpolation frame obtained through the weighted filtering compensation method is further improved by using masks. Overall, the experimental results show that the proposed algorithm has better performance than the existing methods in terms of objective and subjective criteria. Moreover, it can solve problems of exposure and occlusion, and get better frame interpolation for the video sequence with fast moving targets.