Wyner-Ziv Rate-distortion Function Research Articles

Structural Properties of the Wyner-Ziv Rate Distortion Function: Applications for Multivariate Gaussian Sources.

The main focus of this paper is the derivation of the structural properties of the test channels of Wyner's operational information rate distortion function (RDF), R¯(ΔX), for arbitrary abstract sources and, subsequently, the derivation of additional properties for a tuple of multivariate correlated, jointly independent, and identically distributed Gaussian random variables, {Xt,Yt}t=1∞, Xt:Ω→Rnx, Yt:Ω→Rny, with average mean-square error at the decoder and the side information, {Yt}t=1∞, available only at the decoder. For the tuple of multivariate correlated Gaussian sources, we construct optimal test channel realizations which achieve the informational RDF, R¯(ΔX)=▵infM(ΔX)I(X;Z|Y), where M(ΔX) is the set of auxiliary RVs Z such that PZ|X,Y=PZ|X, X^=f(Y,Z), and E{||X-X^||2}≤ΔX. We show the following fundamental structural properties: (1) Optimal test channel realizations that achieve the RDF and satisfy conditional independence, PX|X^,Y,Z=PX|X^,Y=PX|X^,EX|X^,Y,Z=EX|X^=X^. (2) Similarly, for the conditional RDF, RX|Y(ΔX), when the side information is available to both the encoder and the decoder, we show the equality R¯(ΔX)=RX|Y(ΔX). (3) We derive the water-filling solution for RX|Y(ΔX).

Rate-Distortion Bounds for Wyner–Ziv Coding With Gaussian Scale Mixture Correlation Noise

The objective of this work is the characterization of the Wyner-Ziv rate-distortion function for memoryless continuous sources, when the correlation between the sources is modeled via an additive noise channel. Modeling the distribution of the correlation noise via a Gaussian mixture, with discrete or continuous mixing variable, provides a unified signal model able to describe a wide class of distributions, useful in the context of practical applications. The Wyner-Ziv rate-distortion function associated with this signal model cannot, in general, be obtained in analytical form. This work contributes a method for its analysis, by providing computable upper and lower bounds.

Data-Processing Bounds for Scalar Lossy Source Codes With Side Information at the Decoder

In this paper, we introduce new lower bounds on the distortion of scalar fixed-rate codes for lossy compression with side information available at the receiver. These bounds are derived by presenting the relevant random variables as a Markov chain and applying generalized data-processing inequalities a la Ziv and Zakai. We show that by replacing the logarithmic function with other functions, in the data-processing theorem we formulate, we obtain new lower bounds on the distortion of scalar coding with side information at the decoder. The usefulness of these results is demonstrated for uniform sources and the convex function <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i> )= <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1-α</sup> , α > 1. The bounds in this case are shown to be better than one can obtain from the Wyner-Ziv rate-distortion function.

The Infinite-Message Limit of Two-Terminal Interactive Source Coding

A two-terminal interactive function computation problem with alternating messages is studied within the framework of distributed block source coding theory. For any finite number of messages, a single-letter characterization of the sum-rate-distortion function was established in previous works using standard information-theoretic techniques. This, however, does not provide a satisfactory characterization of the infinite-message limit, which is a new, unexplored dimension for asymptotic-analysis in distributed block source coding involving potentially an infinite number of infinitesimal-rate messages. In this paper, the infinite-message sum-rate-distortion function, viewed as a functional of the joint source pmf and the distortion levels, is characterized as the least element of a partially ordered family of functionals having certain convex-geometric properties. The new characterization does not involve evaluating the infinite-message limit of a finite-message sum-rate-distortion expression. This characterization leads to a family of lower bounds for the infinite-message sum-rate-distortion expression and a simple criterion to test the optimality of any achievable infinite-message sum-rate-distortion expression. For computing the amplewise Boolean AND function, the infinite-message minimum sum-rates are characterized in closed analytic form. These sum-rates are shown to be achievable using infinitely many infinitesimal-rate messages. The new convex-geometric characterization is used to develop an iterative algorithm for evaluating any finite-message sumrate-distortion function. It is also used to construct the first examples which demonstrate that for lossy source reproduction, two messages can strictly improve the one-message Wyner-Ziv rate-distortion function settling an unresolved question from a 1985 paper.

Theory and Practice of Rate Division in Distributed Video Coding

Wyner and Ziv characterized the rate distortion function for lossy source coding with side information at the decoder. It is well known that for the quadratic Gaussian case, the Wyner-Ziv rate-distortion function coincides with the conditional rate-distortion function. In this paper, we extend the problem to the coding of multivariate Gaussian source with multiple Gaussian side information at the decoder. The achievable region is obtained, and it is easily extended to the case that the difference between the source and the side information is multivariate Gaussian, no matter what distributions the source and the side information are. We apply this theoretical model to Distributed Video Coding (DVC) by considering the difference of the Distributed frame (D frame) and the Side-information frame (S frame) to be multivariate Gaussian distributed. This introduces rate allocation problem into DVC, which can be solved by a reverse water-filling method. Simulation results show that around 1.5-2 dB coding gain benefits from the multivariate Gaussian Wyner-Ziv coding model.

Source Coding With Limited-Look-Ahead Side Information at the Decoder

We characterize the rate distortion function for the source coding with decoder side information setting when the ith reconstruction symbol is allowed to depend only on the first i+lscr side information symbols, for some finite look-ahead lscr, in addition to the index from the encoder. For the case of causal side information, i.e., lscr=0, we find that the penalty of causality is the omission of the subtracted mutual information term in the Wyner-Ziv rate distortion function. For lscr>0, we derive a computable "infinite-letter" expression for the rate distortion function. When specialized to the near-lossless case, our results characterize the best achievable rate for the Slepian-Wolf source coding problem with finite side information looka-head, and have some surprising implications. We find that side information is useless for any fixed lscr when the joint probability mass function (PMF) of the source and side information satisfies the positivity condition P(x,y)>0 for all (x,y). More generally, the optimal rate depends on the distribution of the pair X,Y only through the distribution of X and the bipartite graph whose edges represent the pairs x,y for which P(x,y)>0. On the other hand, if side information look-ahead is allowed to grow faster than logarithmic in the block length, then H(X|Y) is achievable. Finally, we apply our approach to derive a computable expression for channel capacity when state information is available at the encoder with limited look-ahead.

Coding for the Feedback Gel'fand–Pinsker Channel and the Feedforward Wyner–Ziv Source

We consider both channel coding and source coding, with perfect past feedback/feedforward, in the presence of side information. It is first observed that feedback does not increase the capacity of the Gel'fand-Pinsker channel, nor does feedforward improve the achievable rate-distortion performance in the Wyner-Ziv problem. We then focus on the Gaussian case showing that, as in the absence of side information, feedback/feedforward allows to efficiently attain the respective performance limits. In particular, we derive schemes via variations on that of Schalkwijk and Kailath. These variants, which are as simple as their origin and require no binning, are shown to achieve, respectively, the capacity of Costa's channel, and the Wyner-Ziv rate distortion function. Finally, we consider the finite-alphabet setting and derive schemes for both the channel and the source coding problems that attain the fundamental limits, using variations on schemes of Ahlswede and Ooi and Wornell, and of Martinian and Wornell, respectively

On Successive Refinement for the Wyner–Ziv Problem

Achievable rates are characterized for successive refinement (SR) in the Wyner-Ziv scenario, namely, in the presence of correlated side information (SI) at the receivers. In this setting, the encoder is assumed to operate in two stages, where the first corresponds to relatively low rate and high distortion, and the second, comprising a refinement code on top of the first code, is aimed at reproduction at reduced distortion. Both decoders (for low rate/high distortion and for high rate/low distortion) are equipped with SI streams, correlated to the source, but unavailable to the encoder. Furthermore, it is assumed that the decoder that receives the higher rate bitstream, i.e., the additional refinement bits, accesses also SI of better quality than that of the lower resolution decoder. By better quality, we mean that the source and the SI are modeled together as a stochastically degraded joint source, where the encoded symbols, the SI at the refinement stage, and the SI at the initial (coarse) stage, form a Markov chain in this order. For a memoryless joint process (that includes the source to be encoded and its instantaneously correlated SI streams), necessary and sufficient conditions are furnished, in terms of single-letter formulas, for the achievability of a pair of rates, corresponding to two given distortion levels. Special attention is devoted to the degenerate, but important, case where the two SI streams, at the two decoders, are identical. For this case, conditions are provided for successive refinability in the sense of the existence of codes that asymptotically achieve the Wyner-Ziv rate-distortion function, simultaneously at both distortion levels. In this context, the doubly symmetric binary source (with the Hamming distortion measure) and the jointly Gaussian source (with the squared error distortion measure) are successively refinable in the Wyner-Ziv setting. It is demonstrated that a source that is not successively refinable in the ordinary sense (i.e., without SI) may become successively refinable in the presence of SI at the decoders.

The duality between information embedding and source coding with side information and some applications

Aspects of the duality between the information-embedding problem and the Wyner-Ziv (1976) problem of source coding with side information at the decoder are developed and used to establish a spectrum new results on these and related problems, with implications for a number of important applications. The single-letter characterization of the information-embedding problem is developed and related to the corresponding characterization of the Wyner-Ziv problem, both of which correspond to optimization of a common mutual information difference. Dual variables and dual Markov conditions are identified, along with the dual role of noise and distortion in the two problems. For a Gaussian context with quadratic distortion metric, a geometric interpretation of the duality is developed. From such insights, we develop a capacity-achieving information-embedding system based on nested lattices. We show the resulting encoder-decoder has precisely the same decoder-encoder structure as the corresponding Wyner-Ziv system based on nested lattices that achieves the rate-distortion limit. For a binary context with Hamming distortion metric, the information-embedding capacity is developed, along with its relationship to the corresponding Wyner-Ziv rate-distortion function. In turn, an information-embedding system for this case based on nested linear codes is constructed having an encoder-decoder that is identical to the decoder-encoder structure for the corresponding system that achieves the Wyner-Ziv rate-distortion limit. Finally, based on these results, a simple layered joint source-channel coding system is developed with a perfectly symmetric encoder-decoder structure. Its application and performance is discussed in a broadcast setting in which there is a need to control the fidelity experienced by different receivers. Among other results, we show that such systems and their multilayer extensions retain attractive optimality properties in the Gaussian-quadratic case, but not in the binary-Hamming case.

Duality between channel capacity and rate distortion with two-sided state information

We show that the duality between channel capacity and data compression is retained when state information is available to the sender, to the receiver, to both, or to neither. We present a unified theory for eight special cases of channel capacity and rate distortion with state information, which also extends existing results to arbitrary pairs of independent and identically distributed (i.i.d.) correlated state information (S/sub 1/, S/sub 2/) available at the sender and at the receiver, respectively. In particular, the resulting general formula for channel capacity C = max/sub p/(u,x|s/sub 1/) [I(U; S/sub 2/, Y) I(U; S/sub 1/)] assumes the same form as the generalized Wyner-Ziv (1976) rate distortion function R(D) = min/sub p/(u|x, s/sub 1/)p(x/spl I.cap/|u, s/sub 2/) [I(U; S/sub 1/, X) 1(U; S/sub 2/)].

On source coding with side-information-dependent distortion measures

High-resolution bounds in lossy coding of a real memoryless source are considered when side information is present. Let X be a smooth source and let Y be the side information. First we treat the case when both the encoder and the decoder have access to Y and we establish an asymptotically tight (high-resolution) formula for the conditional rate-distortion function R/sub X|Y/(D) for a class of locally quadratic distortion measures which may be functions of the side information. We then consider the case when only the decoder has access to the side information (i.e., the problem). For side-information-dependent distortion measures, we give an explicit formula which tightly approximates the Wyner-Ziv rate-distortion function R/sup WZ/(D) for small D under some assumptions on the joint distribution of X and Y. These results demonstrate that for side-information-dependent distortion measures the rate loss R/sup WZ/(D)-R/sub X|Y/(D) can be bounded away from zero in the limit of small D. This contrasts the case of distortion measures which do not depend on the side information where the rate loss vanishes as D/spl rarr/0.