Joint optical flow motion compensation and video compression using hybrid vector quantization

Abstract

Summary form only given. We propose a video coding and delivery scheme which is geared towards low bit-rate and real-time performance requirements. We use a hybrid vector quantization scheme, with finite state wavelet-based hierarchical lookup vector quantization (FSWHVQ) for coding motion vectors, which embeds the Horn and Schunck optical flow algorithm in table-lookups, and uses lattice vector quantization (LVQ) for the prediction errors. This video coding scheme is both fast (table-lookups) and accurate (dense motion field), and avoids the blocking artifacts and poor prediction which plagues block coding schemes at low bit rates. For restricted image compression/transmission scenarios like teleconferencing, for which a good training set may be available, the FSWHVQ scheme may be viewed as storing as an internal representation in its lookup tables, a valid and complete model of the problem domain. Two row-wise lookups quantize the row-wise pixel values to give two table indices, Y0 and Y1, followed by a column-wise second-level table lookup, which quantizes the column-wise indices into one index, ZO. Horn and Schunck's optical flow method applies spatial and temporal derivative masks, e.g., of size 2/spl times/2, at corresponding spatial locations of the current and previous images. These mask applications are embedded inside table-lookups. The motion field averages (u/sub av/,v/sub av/), and the previous frame, I/sub t-I/, together comprise the "state", and are updated at each frame. The prediction errors are encoded with LVQ, as the error vectors are small (due to the accurate optical flow-based motion estimation), but more randomly distributed. It is possible to handle large displacements, by using the optical flow computations on the "smooth" bands in the DWT (discrete wavelet transform) domain.