Long-term Memory Motion-compensated Prediction Research Articles

Spatial-Random-Access-Enabled Video Coding for Interactive Virtual Pan/Tilt/Zoom Functionality

High-spatial-resolution videos offer the possibility of viewing an arbitrary region-of-interest (RoI) interactively. Zoom functionality enables watching high-resolution content even on displays of lower spatial resolution. If arbitrary regions corresponding to arbitrary zoom factors can be served to the user, the transmission and/or decoding of the entire high-spatial-resolution video can be avoided. Moreover, if the video content can be encoded such that arbitrary RoIs corresponding to different zoom factors can be simply extracted from the compressed bitstream, we can avoid dedicated video encoding for each user. We propose such a video coding scheme that is vital in allowing the system to scale to large numbers of remote users as well as to encode and store the content for subsequent repeated playback. Apart from generating a multi-resolution representation, our coding scheme uses P slices from H.264/AVC. We study the tradeoff in the choice of slice size. A larger slice size enables higher coding efficiency for representing the entire scene but increases the number of pixels that have to be transmitted. The optimal slice size achieves the best tradeoff and minimizes the expected transmission bitrate. Experimental results confirm the optimality of our predicted slice size for various test cases. Furthermore, we propose an improvement based on background extraction and long-term memory motion-compensated prediction. Experiments indicate up to 85% bitrate reduction while retaining efficient random access capability.

Affine multipicture motion-compensated prediction

Affine motion compensation is combined with long-term memory motion-compensated prediction. The idea is to determine several affine motion parameter sets on subareas of the image. Then, for each affine motion parameter set, a complete reference picture is warped and inserted into the multipicture buffer. Given the multipicture buffer of decoded pictures and affine warped versions thereof, block-based translational motion-compensated prediction and Lagrangian coder control are utilized. The affine motion parameters are transmitted as side information requiring additional bit rate. Hence, the utility of each reference picture and, with that, each affine motion parameter set is tested for its rate-distortion efficiency. The combination of affine and long-term memory motion-compensated prediction provides a highly efficient video compression scheme in terms of rate-distortion performance. The two incorporated multipicture concepts complement each other well providing almost additive rate-distortion gains. When warping the prior decoded picture, average bit-rate savings of 15% against TMN-10, the test model of ITU-T Recommendation H.263, are reported for the case that 20 warped reference pictures are used. When employing 20 warped reference pictures and 10 decoded reference pictures, average bit-rate savings of 24% can be obtained for a set of eight test sequences. These bit-rate savings correspond to gains in PSNR between 0.8-3 dB. For some cases, the combination of affine and long-term memory motion-compensated prediction provides more than additive gains.

Error-resilient video coding using long-term memory prediction and feedback channel

Abstract The MC–DCT-based video coding has been generally used for its efficiency and easy implementation, but it is very vulnerable in error-prone environments. In this paper, we address an error-resilient video coding using multiple reference frame, based on the long-term memory motion compensated prediction (LTMP) (IEEE Trans. Circuits Syst. Video Technol. 9 (1) (1999) 70), and an error concealment technique associated with the proposed technique. The rate-distortion optimization of T. Wiegand et al. (1999) is modified by adding dispersion measure of the temporal motion vectors in order to provide the error-resiliency and much improved error concealment performance. Also, the proposed algorithm confines the temporal error propagation using negative acknowledgement in feedback channel, in which both the corrupted area by channel errors and the error-propagated area are estimated and removed from the search region for the motion compensation. Thus, the proposed algorithm yields the similar performance to the forced intra update (FIU) in terms of the PSNR, but avoids the abrupt increase in the bitrate, resulting in more efficient network utilization than the FIU. By the computer simulations, we shall demonstrate that the proposed technique provides an acceptable performance both subjectively and objectively in error-prone environments, as compared with H.263 and LTMP, with or without the feedback messages.

Error-resilient video transmission using long-term memory motion-compensated prediction

Long-term memory prediction extends the spatial displacement vector utilized in hybrid video coding by a variable time delay, permitting the use of more than one reference frame for motion compensation. This extension leads to improved rate-distortion performance. However, motion compensation in combination with transmission errors leads to temporal error propagation that occurs when the reference frames at the coder and decoder differ. In this paper, we present a framework that incorporates an estimated error into rate-constrained motion estimation and mode decision. Experimental results with a Rayleigh fading channel show that long-term memory prediction significantly outperforms the single-frame prediction H.263-based anchor. When a feedback channel is available, the decoder can inform the encoder about successful or unsuccessful transmission events by sending positive (ACK) or negative (NACK) acknowledgments. This information is utilized for updating the error estimates at the encoder. Similar concepts, such as the ACK and NACK mode known from the H.263 standard, are unified into a general framework providing superior transmission performance.

Long-term memory motion-compensated prediction

Long-term memory motion-compensated prediction extends the spatial displacement vector utilized in block-based hybrid video coding by a variable time delay permitting the use of more frames than the previously decoded one for motion compensated prediction. The long-term memory covers several seconds of decoded frames at the encoder and decoder. The use of multiple frames for motion compensation in most cases provides significantly improved prediction gain. The variable time delay has to be transmitted as side information requiring an additional bit rate which may be prohibitive when the size of the long-term memory becomes too large. Therefore, me control the bit rate of the motion information by employing rate constrained motion estimation. Simulation results are obtained by integrating long-term memory prediction into an H.263 codec. Reconstruction PSNR improvements up to 2 dB for the Foreman sequence and 1.5 dB for the Mother-Daughter sequence are demonstrated in comparison to the TMN-2.0 H.263 coder. The PSNR improvements correspond to bit-rate savings up to 34 and 30%, respectively. Mathematical inequalities are used to speed up motion estimation while achieving full prediction gain.