Motion Estimation And Motion Compensation Research Articles

IBVC: Interpolation-driven B-frame video compression

Learned B-frame video compression aims to adopt bi-directional motion estimation and motion compensation (MEMC) coding for middle frame reconstruction. However, previous learned approaches often directly extend neural P-frame codecs to B-frame relying on bi-directional optical-flow estimation or video frame interpolation. They suffer from inaccurate quantized motions and inefficient motion compensation. To address these issues, we propose a simple yet effective structure called Interpolation-driven B-frame Video Compression (IBVC). Our approach only involves two major operations: video frame interpolation and artifact reduction compression. IBVC introduces a bit-rate free MEMC based on interpolation, which avoids optical-flow quantization and additional compression distortions. Later, to reduce duplicate bit-rate consumption and focus on unaligned artifacts, a residual guided masking encoder is deployed to adaptively select the meaningful contexts with interpolated multi-scale dependencies. In addition, a conditional spatio-temporal decoder is proposed to eliminate location errors and artifacts instead of using MEMC coding in other methods. The experimental results on B-frame coding demonstrate that IBVC has significant improvements compared to the relevant state-of-the-art methods. Meanwhile, our approach can save bit rates compared with the random access (RA) configuration of H.266 (VTM). The code will be available at https://github.com/ruhig6/IBVC.

Luminance Compensation MEMC for Video Frame Interpolation

Video frame interpolation is an important technology in digital video processing, which has great impact on users’ viewing experience. In particular, in medical or industrial application scenarios, the accuracy of the frame interpolation algorithm may also influence the diagnosis results. In addition, for videos based on ionizing radiation (e.g., X-rays), each frame exposure could cause damage to human tissues by ionizing radiation. Therefore, if a frame interpolation algorithm is introduced to display the same number of frames, it only needs to sample half of the frames and halve the exposure radiation doses, which is statistically promising to reduce human cancer rate caused by ionizing radiations (e.g., medical examinations). However, since there are errors in frame interpolation caused by luminance leap, existing works are not applicable in such scenarios. To solve this problem, this paper proposes a video interpolation algorithm based on luminance compensation MEMC (LC-MEMC). Firstly, a luminance compensation method based on the electromagnetic irradiation attenuation in human tissue is introduced to improve the performance of motion estimation and motion compensation (MEMC) and reduce matching errors caused by luminance leap. Secondly, LC-MEMC proposes an improved block matching approach, including i) a new search method from basic points to local points and ii) a block matching criterion that simplifies the calculation process. LC-MEMC improves the accuracy and processing speed of video interpolation from three perspectives: adding luminance compensation, improving the search strategy and optimizing the matching degree calculation method for each search position. We evaluated LC-MEMC on collected medical videos and achieved higher accuracy, faster processing speed, and significantly better viewing experience comparing with existing methods.

Neural Reference Synthesis for Inter Frame Coding

This work proposes the neural reference synthesis (NRS) to generate high-fidelity reference block for motion estimation and motion compensation (MEMC) in inter frame coding. The NRS is comprised of two submodules: one for reconstruction enhancement and the other for reference generation. Although numerous methods have been developed in the past for these two submodules using either handcrafted rules or deep convolutional neural network (CNN) models, they basically deal with them separately, resulting in limited coding gains. By contrast, the NRS proposes to optimize them collaboratively. It first develops two CNN-based models, namely EnhNet and GenNet. The EnhNet only uses spatial correlations within the current frame for reconstruction enhancement and the GenNet is then augmented by further aggregating temporal correlations across multiple frames for reference synthesis. However, a direct concatenation of EnhNet and GenNet without considering the complex temporal reference dependency across inter frames would implicitly induce iterative CNN processing and cause the data overfitting problem, leading to visually-disturbing artifacts and oversmoothed pixels. To tackle this problem, the NRS applies a new training strategy to coordinate the EnhNet and GenNet for more robust and generalizable models, and also devises a lightweight multi-level R-D (rate-distortion) selection policy for the encoder to adaptively choose reference blocks generated from the proposed NRS model or conventional coding process. Our NRS not only offers state-of-the-art coding gains, e.g., >10% BD-Rate (Bjøntegaard Delta Rate) reduction against the High Efficiency Video Coding (HEVC) anchor for a variety of common test video sequences encoded at a wide bit range in both low-delay and random access settings, but also greatly reduces the complexity relative to existing learning-based methods by utilizing more lightweight DNNs. All models are made publicly accessible at https://github.com/IVC-Projects/NRS for reproducible research.

A motion correction approach for oral and maxillofacial cone-beam CT imaging

Patient movement affects image quality in oral and maxillofacial cone-beam computed tomography imaging. While many efforts are made to minimize the possibility of motion during a scan, relatively little attention has been given to motion correction after acquisition. We propose a novel method which can improve the image quality after an oral and maxillofacial scan. The proposed method is based on our previous work and is a retrospective motion estimation and motion compensation (ME/MC) approach that iteratively estimates and compensates for rigid pose change over time. During motion estimation, image update and motion update are performed alternately in a multi-resolution scheme to obtain the motion. We propose use of a feature-based motion update and patch-based image update in the iterative estimation process, to alleviate the effect of limited scan field of view on estimation. During motion compensation, a fine-resolution image reconstruction was performed with compensation for the estimated motion. The proposed ME/MC method was evaluated with simulations, phantom and patient studies. Two experts in dentomaxillofacial radiology assessed the diagnostic importance of the resulting motion artifact suppression. The quality of the reconstructed images was improved after motion compensation, and most of the image artifacts were eliminated. Quantitative analysis by comparison to a reference image and by calculation of a sharpness metric agreed with the qualitative observation. The results are promising, and further evaluation is required to assess the clinical value of the proposed method.

Large Motion Video Super-Resolution with Dual Subnet and Multi-Stage Communicated Upsampling

Video super-resolution (VSR) aims at restoring a video in low-resolution (LR) and improving it to higher-resolution (HR). Due to the characteristics of video tasks, it is very important that motion information among frames should be well concerned, summarized and utilized for guidance in a VSR algorithm. Especially, when a video contains large motion, conventional methods easily bring incoherent results or artifacts. In this paper, we propose a novel deep neural network with Dual Subnet and Multi-stage Communicated Upsampling (DSMC) for super-resolution of videos with large motion. We design a new module named U-shaped residual dense network with 3D convolution (U3D-RDN) for fine implicit motion estimation and motion compensation (MEMC) as well as coarse spatial feature extraction. And we present a new Multi-Stage Communicated Upsampling (MSCU) module to make full use of the intermediate results of upsampling for guiding the VSR. Moreover, a novel dual subnet is devised to aid the training of our DSMC, whose dual loss helps to reduce the solution space as well as enhance the generalization ability. Our experimental results confirm that our method achieves superior performance on videos with large motion compared to state-of-the-art methods.

An H.264/AVC High422 Profile and MPEG-2 422 Profile Encoder LSI for HDTV Broadcasting Infrastructures

An H.264/AVC encoder LSI (named “SARA”) that supports High422 profile, as well as 422 profile of MPEG-2, has been developed for HDTV broadcasting infrastructures. It contains three motion estimation and compensation (ME/MC) engines with wide search ranges of -217.75 to +199.75 horizontally, -109.75 to +145.75 vertically, which can utilize almost all H.264/AVC ME/MC coding tools, such as multiple reference frame, variable block size, quarter-pel prediction, macroblock adaptive field/frame prediction (MBAFF), spatial/temporal direct mode, and weighted prediction. Our evaluations show that it can encode fast moving scenes with 1.2dB to 1.7dB higher than the JM. It was successfully fabricated in a 90-nm technology, and integrates 140 million transistors.

Feasibility of high temporal resolution breast DCE-MRI using compressed sensing theory.

To investigate the feasibility of high temporal resolution breast DCE-MRI using compressed sensing theory. Two experiments were designed to investigate the feasibility of using reference image based compressed sensing (RICS) technique in DCE-MRI of the breast. The first experiment examined the capability of RICS to faithfully reconstruct uptake curves using undersampled data sets extracted from fully sampled clinical breast DCE-MRI data. An average approach and an approach using motion estimation and motion compensation (ME/MC) were implemented to obtain reference images and to evaluate their efficacy in reducing motion related effects. The second experiment, an in vitro phantom study, tested the feasibility of RICS for improving temporal resolution without degrading the spatial resolution. For the uptake-curve reconstruction experiment, there was a high correlation between uptake curves reconstructed from fully sampled data by Fourier transform and from undersampled data by RICS, indicating high similarity between them. The mean Pearson correlation coefficients for RICS with the ME/MC approach and RICS with the average approach were 0.977 +/- 0.023 and 0.953 +/- 0.031, respectively. The comparisons of final reconstruction results between RICS with the average approach and RICS with the ME/MC approach suggested that the latter was superior to the former in reducing motion related effects. For the in vitro experiment, compared to the fully sampled method, RICS improved the temporal resolution by an acceleration factor of 10 without degrading the spatial resolution. The preliminary study demonstrates the feasibility of RICS for faithfully reconstructing uptake curves and improving temporal resolution of breast DCE-MRI without degrading the spatial resolution.

45.4: High Frequency LVDS Interface for F‐HD 120Hz LCD TV

AbstractIn FHD 120Hz TV, frame rate converting IC with the MEMC (Motion Estimation and Motion Compensation) function is applied to make the 120Hz image and 2 cables with 4 port LVDS pairs are used to transmit FHD 120Hz image data. Therefore, The cost of FHD 120Hz TV System is higher than the cost of FHD 60Hz TV System. LGD(LG‐Display) have evaluated the new FHD 120Hz LCD panel with HF (High Frequency) LVDS which is the most cost‐effective way to make FHD 120Hz TV System. This paper will review the technical aspects of FHD 120Hz LCD panel with HF LVDS and cost and propose to adapt Next Generation Interface for the new standard of FHD 120Hz LCM (LCD Module) in the Future.

DESIGN AND IMPLEMENTATION OF WAVELET-DOMAIN VIDEO COMPRESSION USING MULTIRESOLUTION MOTION ESTIMATION AND COMPENSATION

Video compression techniques have been applied routinely in order to conserve storage space and minimize bandwidth utilization in various video applications. To reduce inter-pixel redundancies inside and between video frames, video codecs (coder/decoder or compression/decompression) are mainly characterized by applying motion estimation and motion compensation (MEMC) in combination with discrete cosine transform (DCT). However, blocking artifacts are obvious from the block-based motion estimation and transformation, especially at low bit rates. Due to the intrinsic advantages of multiresolution and scalability of discrete wavelet transform (DWT), video compression techniques have been infused with exciting prospects by the beneficial integration of motion estimation and wavelet transformation. The contributions of this paper are in (1) proposing a design architecture of the wavelet-domain encoder and decoder; (2) implementing the multiresolution motion estimation and compensation method; and in (3) realizing the dynamic entropy encoding and decoding, so as to achieve more efficient video compression than the conventional spatial domain methods for low bit rate applications without the annoying blocking artifacts.

A flexible architecture for H.263 video coding

In this paper a very flexible and efficient architecture that implements the core of a video coder according to Rec. H.263 is presented. It consists of a RISC processor that controls the scheduling of a set of specialized processors for the transforms (DCT and IDCT), quantizers (DQ and IQ), motion estimation and motion compensation (ME/MC). The architecture also includes preprocessing modules for the input video signal from the camera and interfaces for the external video memory and the H.263 bit-stream generation. The architecture has been written in synthesizable Verilog and tested using standard video sequences. It has also been prototyped into a development system based on an FPGA and a RISC.