Backward Motion Vectors Research Articles

Frame Rate Upconversion Using Optical Flow and Patch-Based Reconstruction

In this paper, we present a frame rate upconversion method using optical flow motion estimation and a patch-based reconstruction scheme. First, forward and backward motion vectors (MVs) are obtained using an optical flow algorithm, and reconstructed versions of the current and previous frames are generated by our patch-based reconstruction scheme. Using the original and reconstructed versions of the current and previous frames, two mismatch masks are obtained. Then, two versions of the middle frame are generated using a patch-based scheme, with estimated MVs and the current and previous frames. Finally, a middle mask, which identifies the mismatch areas of the two middle frames, is reconstructed. Using these three masks, the best candidates for interpolation are selected and fused to obtain the final middle frame. Due to the patch-based nature of our reconstruction scheme, most of the holes and cracks will be filled. Although there is always a probability of having holes, the size and number of such holes are much smaller than those that would be generated using pixel-based mapping. The rare holes are filled using existing hole-filling algorithms. The experimental results and a comparison of our method with existing algorithms show that our method performs better in terms of both objective and subjective quality of the final interpolated frames. The average peak signal-to-noise ratio (PSNR) improvement of our method is 1–2 dB.

Compression Algorithm Analysis of In-Situ (S)TEM Video: Towards Automatic Event Detection and Characterization

Journal Article Compression Algorithm Analysis of In-Situ (S)TEM Video: Towards Automatic Event Detection and Characterization Get access Jeremy R Teuton, Jeremy R Teuton Fundamental and Computational Sciences, Pacific Northwest National Laboratory, Richland, WA 99352, USA Search for other works by this author on: Oxford Academic Google Scholar Richard L Griswold, Richard L Griswold National Security Division, Pacific Northwest National Laboratory, Richland, WA 99352, USA Search for other works by this author on: Oxford Academic Google Scholar B Layla Mehdi, B Layla Mehdi Fundamental and Computational Sciences, Pacific Northwest National Laboratory, Richland, WA 99352, USA Search for other works by this author on: Oxford Academic Google Scholar Nigel D.Browning Nigel D.Browning Fundamental and Computational Sciences, Pacific Northwest National Laboratory, Richland, WA 99352, USA Search for other works by this author on: Oxford Academic Google Scholar Microscopy and Microanalysis, Volume 21, Issue S3, 1 August 2015, Pages 2329–2330, https://doi.org/10.1017/S1431927615012428 Published: 23 September 2015

Toward naturalistic 2D-to-3D conversion.

Natural scene statistics (NSSs) models have been developed that make it possible to impose useful perceptually relevant priors on the luminance, colors, and depth maps of natural scenes. We show that these models can be used to develop 3D content creation algorithms that can convert monocular 2D videos into statistically natural 3D-viewable videos. First, accurate depth information on key frames is obtained via human annotation. Then, both forward and backward motion vectors are estimated and compared to decide the initial depth values, and a compensation process is applied to further improve the depth initialization. Then, the luminance/chrominance and initial depth map are decomposed by a Gabor filter bank. Each subband of depth is modeled to produce a NSS prior term. The statistical color-depth priors are combined with the spatial smoothness constraint in the depth propagation target function as a prior regularizing term. The final depth map associated with each frame of the input 2D video is optimized by minimizing the target function over all subbands. In the end, stereoscopic frames are rendered from the color frames and their associated depth maps. We evaluated the quality of the generated 3D videos using both subjective and objective quality assessment methods. The experimental results obtained on various sequences show that the presented method outperforms several state-of-the-art 2D-to-3D conversion methods.

Multi-Channel Mixed-Pattern Based Frame Rate Up-Conversion Using Spatio-Temporal Motion Vector Refinement and Dual-Weighted Overlapped Block Motion Compensation

In this paper, a novel motion compensated frame rate up-conversion (MC-FRUC) algorithm is proposed to enhance the visual quality of video sequences. First of all, the multi-channel mixed pattern (MCMP) is proposed to design a block matching criterion which uses a few computations to reveal the variances of one luminance channel and two chrominance channels in a video frame. Second, in basis of the forward and backward initial motion vector fields (MVFs) estimated by a variant of 3DRS algorithm, the spatio-temporal motion vector refinement (ST-MVR) algorithm is proposed to obtain the more smoother MVF by implicitly adding the spatio-temporal smooth constraint into the process of motion vector refinement, and then the highly fault-tolerant motion vector smoothing (HFT-MVS) algorithm is proposed to prevent the emergence of outliers in MVF. Finally, in order to reduce the edge blurring and occlusions, the proposed dual-weighted overlapped block motion compensation (DW-OBMC) algorithm uses the forward and backward MVFs to jointly produce the interpolated frame. Experimental results show that the proposed algorithm can significantly improve both objective and subjective quality of the interpolated frame with a low computational complexity, and provide the better performance than the existing algorithms.

Direction-Select Motion Estimation for Motion-Compensated Frame Rate Up-Conversion

In this paper, we propose a new motion estimation algorithm to be used for motion-compensated frame rate up-conversion. The proposed algorithm independently carries out motion estimations in both forward and backward directions, and selects a more reliable one between forward and backward motion vectors by evaluating the motion vector reliability from the viewpoint of the interpolated frame. The proposed algorithm smooths and refines both the forward and backward motion vectors before selecting the reliable one. This procedure helps to select the reasonable motion estimation direction. In identifying the motion vector outliers, the proposed algorithm uses a circular range of which center is located at the mean of the eight neighboring motion vectors of the motion vector being processed. Experimental results using 1720 test images show that the proposed motion estimation algorithm improves the average peak signal-to-noise ratio and the average structural similarity of the interpolated frames by up to 5.31 dB and 0.053, respectively, compared to conventional motion estimation algorithms.

Novel True-Motion Estimation Algorithm and Its Application to Motion-Compensated Temporal Frame Interpolation

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

Improved motion vector scaling technique based on half-pixel offset compensation

It is observed that there exists a half-pixel offset in vertical direction between top and bottom fields in interlaced sequence, and this half-pixel offset has an impact on performing motion vector scaling in multipicture motion-compensated prediction. However, the impact caused by this half-pixel offset is processed inconsiderately during motion vector derivation in the up-to-date video coding standards, where temporal motion vector predictor usually derives its forward or backward motion vector by performing motion vector scaling based on temporal distance. In this paper, an improved motion vector scaling technique is proposed to compensate the impact caused by the half-pixel offset between top and bottom fields. Motion vector scaling is performed on corresponding coordinates which represent the position relationship between top and bottom fields properly. Experimental results show that the proposed motion vector scaling technique improves coding efficiency for interlaced sequence, especially for bi-predictive pictures.

B 슬라이스의 압축 효율 향상을 위한 개선된 양방향 대칭 예측 부호화 방법

H.264/AVC의 B 슬라이스의 부호화 효율 향상과 두 개의 움직임벡터를 탐색하는데 소요되는 계산량 감소를 위하여 양방향 대칭(Bi-directional Symmetric) 기법이 개발된 바 있다. 이 기법은, 전방향과 역방향 참조영상 각각에 대하여 움직임 벡터를 구하고 이 두 개를 각각 다 전송하는 종래의 양방향 예측기법과는 달리, 전방향 참조영상에 대해 움직임 벡터를 찾는 동시에 역방향 참조영상에 대한 역방향 움직임 벡터를 전방향 참조영상, 역방향 참조영상, 그리고 현재 영상간의 상대적 거리를 고려한 대칭(Symmetric) 구조로 동시에 계산하여 추정하는 방법이다. 이 기법에 따르면, 전방향 움직임 벡터가 정해지면, 역방향 움직임벡터는 이와 대칭적으로 계산하여 얻어지므로 움직임벡터 추정 복잡도를 반으로 줄이고, 전방향 움직임벡터만을 전송하도록 하여 부호화할 움직임벡터의 양도 줄일 수 있다. 그러나 이 방법은 항상 전방향 움직임 벡터를 기준으로 역방향 움직임 벡터를 계산하여 얻다 보니, 장면전환등의 경우 오히려 역방향 움직임벡터를 기준으로 전방향 움직임벡터를 추산하는 것이 더욱 효율적인 경우도 있다. 본 논문에서는 전방향 참조영상에 대한 움직임 벡터를 중심으로 역방향 움직임 예측벡터를 추정하는 방법을 일반화시켜, 역방향 움직임 벡터를 중심으로 전방향 움직임 벡터를 추산하여 사용하는 방법을 제안하고 아울러 기존 방법과 제안 방법을 율왜곡 관점에서 최적으로 선택하여 사용하는 방법을 제안한다. A bi-directional symmetric prediction technique has been developed to improve coding efficiency of B-slice and to reduce the computational complexity required to estimate two motion vectors. On the contrary to the conventional bi-directional mode which encodes both forward and backward motion vectors, it only encodes a single forward motion vector, and the missing backward motion vector is derived in a symmetric way from the forward motion vector using temporal distance between forward/backward reference frames to and from the current B picture. Since the backward motion vector is derived from the forward motion vector, it can halve the computational complexity for motion estimation, and also reduces motion vector data to encode. This technique always derives the backward motion vector from the forward motion vector, however, there are cases when the forward motion vector is better to be derived from the backward motion vector especially in scene changes. In this paper, we generalize the idea of the symmetric coding with forward motion vector coding, and propose a new symmetric coding with backward motion vector coding and adaptive selection between the conventional symmetric mode and the proposed symmetric mode based on rate-distortion optimization.

Face Tracking in the Compressed Domain

A compressed domain generic object tracking algorithm offers, in combination with a face detection algorithm, a low-computational-cost solution to the problem of detecting and locating faces in frames of compressed video sequences (such as MPEG-1 or MPEG-2). Objects such as faces can thus be tracked through a compressed video stream using motion information provided by existing forward and backward motion vectors. The described solution requires only low computational resources on CE devices and offers at one and the same time sufficiently good location rates.

Trinocular stereo sequence coding based on MPEG-2

This paper presents a trinocular stereo sequence-coding algorithm based on moving picture experts group (MPEG)-2. Trinocular stereo images are a set of three images; center, left, and right view images are taken at the same time. To reduce the redundancy of a trinocular stereo sequence, we use disparity information, as well as motion information. Three reference images are employed: two reference (previous and future) images for the forward and backward motion vectors and the center view image for the disparity vector (DV). We investigate three temporal prediction structures used for trinocular stereo sequence coding. New macroblock types and variable length code tables that include DV are designed. Experimental results with real test sequences show that the proposed algorithm gives a higher peak signal-to-noise ratio (PSNR) than MPEG-2 and MPEG-2 multiview profile.

Loss concealment using b-pictures motion information

In this paper, the motion parameters of the bidirectionally predicted pictures (B-pictures) of MPEG-1,2 are exploited for concealment of large portions of corrupted anchor pictures (and vice versa) that might arise due to channel errors or packet losses. To further enhance the quality of the concealed pictures, we propose two methods of constraining the motion vectors of the B-pictures that strengthen the tie between them and those of the anchor pictures in the same picture subgroup. In one method, the macroblock decisions on the last B-picture in each subgroup is constrained to be bidirectional if those of the other B-pictures are not, such that the derived motion vectors for the concealment of the anchor picture are always composed from the forward and backward motion vectors of the bidirectional motions. Second, the bidirectional motion vectors of the B-pictures in each subgroup is constrained such that the vectorial sum of their forward and backward motion vectors results in an accurate motion prediction of the anchor picture. The experimental results show that while the composed motion vectors improve the quality of concealment over the conventional methods by more than 3-4 dB, another 2 dB improvement can be achieved by constraining the generation of the bidirectional motion vectors.

Adaptive motion estimation technique for motion compensated interframe interpolation

We propose a new fast motion compensated interframe interpolation algorithm. First, we propose a fast algorithm that obtains backward motion vectors by using forward motion vectors received from the transmitter. It is important to predict well both the covered and uncovered regions in order to interpolate skipped frames at the receiver, therefore, backward motion vectors are needed as well as forward motion vectors. But, in the case of the conventional video communication systems, we usually receive the forward estimated motion vectors from the transmitter. To estimate the backward motion vectors used to predict the uncovered region, the conventional interframe interpolation technique requires a large amount of calculation at the receiver because it uses exhaustive motion search such as full search block matching algorithm (FSBMA). In this paper, we reduce the computational complexity to get the backward motion vectors by using the dynamic search range varied for the forward motion vectors received from the transmitter. Second, we also propose the algorithm to obtain the motion vectors which are closer to the true object motion by utilizing the local characteristics of the vector field. This algorithm makes it possible, showing that the resulting motion vectors are more adaptive to the real object motion than the conventional one. According to the simulation results, the proposed algorithm is better than the conventional one in subjective quality as well as in objective quality.

Bidirectional motion estimation via vector propagation

A low-complexity vector propagation (VP) algorithm is introduced for the estimation of bidirectional motion vector fields in image sequences. The proposed VP algorithm exploits the strong correlation between forward and backward motion vector fields in image sequences. The performance of the VP algorithm is compared to that of a bidirectional multiresolution block-matching (MRBM) motion estimation (ME) algorithm. Computer simulation results demonstrate that with the VP algorithm, the computational workload of the bidirectional ME is reduced by a factor of nearly 2. The robustness of the VP algorithm is extensively tested using computer generated image sequences and real movies for a motion picture restoration (MPR) system. It is shown that the VP algorithm is robust enough to be used in the computationally demanding MPR algorithm, in which the performance of the novel VP algorithm is close to that of a bidirectional MRBM ME algorithm. With this technique, other video processing systems that desire to take advantage of bidirectional motion estimation can now do so without an excessive increase in computing time.

Efficient motion estimation algorithmfor bidirectional prediction scheme

A novel block matching algorithm based on successive refinement using motion correlation is presented, which estimates more accurate forward and backward motion vectors for interpolative prediction as well as forward and backward predictions in bidirectionally predictive-coded pictures. Experimental results show that this algorithm achieves good performance in PSNR and subjective quality with low computational complexity.< >

Joint estimation of forward and backward motion vectors for interpolative prediction of video

A low-complexity iterative algorithm is introduced for joint estimation of forward and backward motion vectors in interpolative prediction of video. Starting from the initial values obtained by a commonly-used block matching independent search method, the motion vectors are iteratively refined until a locally optimal solution to the motion estimation problem for interpolative prediction is achieved. Each iteration consists of a series of two similar procedures. First, the backward motion vector is fixed and a new forward motion vector is searched to minimize the interpolation error. Then the forward motion vector is fixed and the backward motion vector is similarly refined by minimizing the interpolation error. This process is repeated until the interpolation error stops decreasing. Computer simulation results demonstrate that with this technique the prediction error in some scenes is significantly reduced.