Source-side Information Pairs Research Articles

Fundamental Limits of Lossless Data Compression With Side Information

The problem of lossless data compression with side information available to both the encoder and the decoder is considered. The finite-blocklength fundamental limits of the best achievable performance are defined, in two different versions of the problem: Reference-based compression, when a single side information string is used repeatedly in compressing different source messages, and pair-based compression, where a different side information string is used for each source message. General achievability and converse theorems are established for arbitrary source-side information pairs. Nonasymptotic normal approximation expansions are proved for the optimal rate in both the reference-based and pair-based settings, for memoryless sources. These are stated in terms of explicit, finite-blocklength bounds, that are tight up to third-order terms. Extensions that go significantly beyond the class of memoryless sources are obtained. The relevant source dispersion is identified and its relationship with the conditional varentropy rate is established. Interestingly, the dispersion is different in reference-based and pair-based compression, and it is proved that the reference-based dispersion is in general smaller.

Sharp Second-Order Pointwise Asymptotics for Lossless Compression with Side Information.

The problem of determining the best achievable performance of arbitrary lossless compression algorithms is examined, when correlated side information is available at both the encoder and decoder. For arbitrary source-side information pairs, the conditional information density is shown to provide a sharp asymptotic lower bound for the description lengths achieved by an arbitrary sequence of compressors. This implies that for ergodic source-side information pairs, the conditional entropy rate is the best achievable asymptotic lower bound to the rate, not just in expectation but with probability one. Under appropriate mixing conditions, a central limit theorem and a law of the iterated logarithm are proved, describing the inevitable fluctuations of the second-order asymptotically best possible rate. An idealised version of Lempel-Ziv coding with side information is shown to be universally first- and second-order asymptotically optimal, under the same conditions. These results are in part based on a new almost-sure invariance principle for the conditional information density, which may be of independent interest.

Interactive Encoding and Decoding Based on Binary LDPC Codes With Syndrome Accumulation

Interactive encoding and decoding based on binary low-density parity-check codes with syndrome accumulation (SA-LDPC-IED) is proposed and investigated. Assume that the source alphabet is <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GF</b> (2), and the side information alphabet is finite. It is first demonstrated how to convert any classical universal lossless code <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cn</i> (with block length <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> and side information available to both the encoder and decoder) into a universal SA-LDPC-IED scheme. It is then shown that with the word error probability approaching 0 subexponentially with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> , the compression rate (including both the forward and backward rates) of the resulting SA-LDPC-IED scheme is upper bounded by a functional of that of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cn</i> , which in turn approaches the compression rate of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cn</i> for each and every individual sequence pair ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xn</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">yn</i> ) and the conditional entropy rate H ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> | <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) for any stationary, ergodic source and side information ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) as the average variable node degree <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l̅</i> of the underlying LDPC code increases without bound. When applied to the class of binary source and side information ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) correlated through a binary symmetrical channel with crossover probability unknown to both the encoder and decoder, the resulting SA-LDPC-IED scheme can be further simplified, yielding even improved rate performance versus the bit error probability when <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l̅</i> is not large. Simulation results (coupled with linear time belief propagation decoding) on binary source-side information pairs confirm the theoretic analysis and further show that the SA-LDPC-IED scheme consistently outperforms the Slepian-Wolf coding scheme based on the same underlying LDPC code. As a by-product, probability bounds involving LDPC established in the course are also interesting on their own and expected to have implications on the performance of LDPC for channel coding as well.

Low-Delay Prediction- and Transform-Based Wyner–Ziv Coding

This paper studies low-delay Wyner-Ziv coding, i.e., lossy source coding with side information at the decoder, with emphasis on the extreme of zero delay. To achieve zero delay, a scalar quantizer is followed by scalar coding of quantization indices. In the fixed-length coding scenario, under high-resolution assumptions and appropriately defined decodability constraints, the optimal quantization level density is conjectured to be periodic. This conjecture, which is provable when the correlation is high, allows for a precise analysis of the rate-distortion tradeoff. The performance of variable-length coding with periodic quantization is also characterized. The results are then incorporated in predictive Wyner-Ziv coding for Gaussian sources with memory, and optimal prediction filters are numerically designed so as to strike a balance between maximally exploiting both temporal and spatial correlation and limiting the propagation of distortion due to occasional decoding errors. Finally, the zero-delay schemes are also employed in transform coding with small block lengths, where the Gaussian source and side information are transformed separately with the premise that corresponding transform coefficient pairs exhibit good spatial correlation and minimal temporal correlation. For the specific source-side information pairs studied, it is shown that transform coding, even with a small block-length, outperforms predictive coding. Performances of both predictive and transform coding are also compared with the asymptotic rate-distortion bounds.

On the Redundancy of Slepian–Wolf Coding

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In this paper, the redundancy of both variable and fixed rate Slepian–Wolf coding is considered. Given any jointly memoryless source-side information pair <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$\{(X_i, Y_i)\}_{i=1}^{\infty}$</tex></formula></emphasis> with finite alphabet, the redundancy <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$R^n(\epsilon_n)$</tex></formula></emphasis> of variable rate Slepian–Wolf coding of <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$X_1^n$</tex></formula></emphasis> with decoder only side information <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$Y_1^n$</tex></formula></emphasis> depends on both the block length <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$n$</tex></formula></emphasis> and the decoding block error probability <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$\epsilon_n$</tex></formula></emphasis>, and is defined as the difference between the minimum average compression rate of order <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$n$</tex> </formula></emphasis> variable rate Slepian–Wolf codes having the decoding block error probability less than or equal to <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$\epsilon_n$</tex></formula></emphasis>, and the conditional entropy <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$H(X\vert Y)$</tex></formula></emphasis>, where <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$H(X\vert Y)$</tex></formula></emphasis> is the conditional entropy rate of the source given the side information. The redundancy of fixed rate Slepian–Wolf coding of <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$X_1^n$</tex></formula></emphasis> with decoder only side information <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$Y_1^n$</tex> </formula></emphasis> is defined similarly and denoted by <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$R^n_F(\epsilon_n)$</tex></formula></emphasis>. It is proved that under mild assumptions about <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$\epsilon_n,$</tex></formula></emphasis> <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$R^n(\epsilon_n) = d_v \sqrt{-\log\epsilon_n/n} + o(\sqrt{-\log \epsilon_n/n})$</tex></formula></emphasis> and <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$R^n_{F}(\epsilon_n) = d_f \sqrt{- \log \epsilon_n / n} + o(\sqrt{-\log \epsilon_n/n})$</tex></formula></emphasis>, where <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$d_f$</tex> </formula></emphasis> and <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$d_v$</tex></formula></emphasis> are two constants completely determined by the joint distribution of the source-side information pair. Since <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$d_v$</tex> </formula></emphasis> is generally smaller than <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$d_f$</tex></formula></emphasis>, our results show that variable rate Slepian–Wolf coding is indeed more efficient than fixed rate Slepian–Wolf coding. </para>

Universal Multiterminal Source Coding Algorithms With Asymptotically Zero Feedback: Fixed Database Case

Consider a source network in which a finite alphabet source X = {Xi}i=0 infin is to be encoded and transmitted, and another finite alphabet source Y = {Xi}i=0 infin correlated with X is available only to the decoder as side information. Traditionally, the channel between the encoder and decoder in the source network is assumed to be one-way. This, together with the fact that the encoder does not have access to Y, implies that the encoder has to know the achievable rates before encoding. In this paper, we consider universal source coding for a feedback source network in which the channel between the encoder and decoder is two-way and asymmetric. Assume that the encoder and decoder share a random database that is independent of both X and Y. A string matching-based (variable-rate) block coding algorithm with simple progressive encoding and joint typicality decoding is first proposed for the feedback source network. The simple progressive encoder does not need to know the achievable rates at the beginning of encoding. It is proven that for any (X, Y) in a large class of sources satisfying some mixing conditions, the average number of bits per letter transmitted from the encoder to the decoder (compression rate) goes to the conditional entropy H(X | Y) of X given Y asymptotically, and at the same time the average number of bits per letter transmitted from the decoder to the encoder (feedback rate) goes to 0 asymptotically. The algorithm and the corresponding analysis results are then extended to the case where both X and Y are to be encoded separately, but decoded jointly. Finally, a universal decoding algorithm is proposed to replace the joint typicality decoding, and the resulting universal compression algorithm consisting of the simple progressive encoder and the universal decoding algorithm is further shown to be asymptotically optimal for the class of all jointly memoryless source-side information pairs (X,Y).

Kraft Inequality and Zero-Error Source Coding With Decoder Side Information

For certain source coding problems, it is well known that the Kraft inequality provides a simple sufficient condition for the existence of codes with given codeword lengths. Motivated by this fact, a sufficient condition based on the Kraft inequality can also be sought for the problem of zero-error instantaneous coding with decoder side information. More specifically, it can be envisioned that a sufficient condition for the existence of such codes with codeword lengths {l x} is that for some 0<alphales<1 SigmaxepsivFscr (2-l x)lesalpha for each clique Fscr in the characteristic graph G of the source-side information pair. In this correspondence, it is shown that (1) if the above is indeed a sufficient condition for a class G of graphs, then it is possible to come as close as 1-log2 alpha bits to the asymptotic limits for each graph in G, (2) there exist graph classes of interest for which such a can indeed be found, and finally (3) no such n can be found for the class of all graphs.