Conditional Entropy Rate Research Articles

Sharp Second-Order Pointwise Asymptotics for Lossless Compression with Side Information.

The problem of determining the best achievable performance of arbitrary lossless compression algorithms is examined, when correlated side information is available at both the encoder and decoder. For arbitrary source-side information pairs, the conditional information density is shown to provide a sharp asymptotic lower bound for the description lengths achieved by an arbitrary sequence of compressors. This implies that for ergodic source-side information pairs, the conditional entropy rate is the best achievable asymptotic lower bound to the rate, not just in expectation but with probability one. Under appropriate mixing conditions, a central limit theorem and a law of the iterated logarithm are proved, describing the inevitable fluctuations of the second-order asymptotically best possible rate. An idealised version of Lempel-Ziv coding with side information is shown to be universally first- and second-order asymptotically optimal, under the same conditions. These results are in part based on a new almost-sure invariance principle for the conditional information density, which may be of independent interest.

Interactive Encoding and Decoding Based on Binary LDPC Codes With Syndrome Accumulation

Interactive encoding and decoding based on binary low-density parity-check codes with syndrome accumulation (SA-LDPC-IED) is proposed and investigated. Assume that the source alphabet is <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GF</b> (2), and the side information alphabet is finite. It is first demonstrated how to convert any classical universal lossless code <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cn</i> (with block length <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> and side information available to both the encoder and decoder) into a universal SA-LDPC-IED scheme. It is then shown that with the word error probability approaching 0 subexponentially with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> , the compression rate (including both the forward and backward rates) of the resulting SA-LDPC-IED scheme is upper bounded by a functional of that of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cn</i> , which in turn approaches the compression rate of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cn</i> for each and every individual sequence pair ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xn</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">yn</i> ) and the conditional entropy rate H ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> | <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) for any stationary, ergodic source and side information ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) as the average variable node degree <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l̅</i> of the underlying LDPC code increases without bound. When applied to the class of binary source and side information ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) correlated through a binary symmetrical channel with crossover probability unknown to both the encoder and decoder, the resulting SA-LDPC-IED scheme can be further simplified, yielding even improved rate performance versus the bit error probability when <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l̅</i> is not large. Simulation results (coupled with linear time belief propagation decoding) on binary source-side information pairs confirm the theoretic analysis and further show that the SA-LDPC-IED scheme consistently outperforms the Slepian-Wolf coding scheme based on the same underlying LDPC code. As a by-product, probability bounds involving LDPC established in the course are also interesting on their own and expected to have implications on the performance of LDPC for channel coding as well.

Interactive Encoding and Decoding for One Way Learning: Near Lossless Recovery With Side Information at the Decoder

A source coding paradigm called interactive encoding and decoding (IED) is considered for a source network where a finite alphabet source <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> is to be encoded, and another finite alphabet source <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> correlated with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> is available only to the decoder as a helper. The optimal performance achievable asymptotically (OPAA) by IED is investigated, where the performance is measured as the average number of bits per symbol exchanged by the encoder and decoder until the decoder learns <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> with high probability. First, it is shown that for any stationary <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(X</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ), the OPAA by IED is given by the conditional entropy rate <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> | <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> given <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> . This is in contrast with noninteractive Slepian-Wolf (SW) coding, where the OPAA is shown in general to be strictly greater than <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> | <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) when <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(X</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) is not ergodic. Second, for a memoryless source pair ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X, Y</i> ), it is shown that IED approaches <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> | <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</i> ) faster than SW coding does. Finally, it is demonstrated that one can convert any classical universal data compression algorithm with side information to a universal IED algorithm for the class <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¿</i> of all stationary ergodic source pairs. In contrast, universal SW coding algorithms for the class <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¿</i> do not exist.

On the Redundancy of Slepian–Wolf Coding

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In this paper, the redundancy of both variable and fixed rate Slepian–Wolf coding is considered. Given any jointly memoryless source-side information pair <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$\{(X_i, Y_i)\}_{i=1}^{\infty}$</tex></formula></emphasis> with finite alphabet, the redundancy <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$R^n(\epsilon_n)$</tex></formula></emphasis> of variable rate Slepian–Wolf coding of <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$X_1^n$</tex></formula></emphasis> with decoder only side information <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$Y_1^n$</tex></formula></emphasis> depends on both the block length <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$n$</tex></formula></emphasis> and the decoding block error probability <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$\epsilon_n$</tex></formula></emphasis>, and is defined as the difference between the minimum average compression rate of order <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$n$</tex> </formula></emphasis> variable rate Slepian–Wolf codes having the decoding block error probability less than or equal to <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$\epsilon_n$</tex></formula></emphasis>, and the conditional entropy <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$H(X\vert Y)$</tex></formula></emphasis>, where <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$H(X\vert Y)$</tex></formula></emphasis> is the conditional entropy rate of the source given the side information. The redundancy of fixed rate Slepian–Wolf coding of <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$X_1^n$</tex></formula></emphasis> with decoder only side information <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$Y_1^n$</tex> </formula></emphasis> is defined similarly and denoted by <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$R^n_F(\epsilon_n)$</tex></formula></emphasis>. It is proved that under mild assumptions about <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$\epsilon_n,$</tex></formula></emphasis> <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$R^n(\epsilon_n) = d_v \sqrt{-\log\epsilon_n/n} + o(\sqrt{-\log \epsilon_n/n})$</tex></formula></emphasis> and <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$R^n_{F}(\epsilon_n) = d_f \sqrt{- \log \epsilon_n / n} + o(\sqrt{-\log \epsilon_n/n})$</tex></formula></emphasis>, where <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$d_f$</tex> </formula></emphasis> and <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$d_v$</tex></formula></emphasis> are two constants completely determined by the joint distribution of the source-side information pair. Since <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$d_v$</tex> </formula></emphasis> is generally smaller than <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$d_f$</tex></formula></emphasis>, our results show that variable rate Slepian–Wolf coding is indeed more efficient than fixed rate Slepian–Wolf coding. </para>

Lossy Joint Source-Channel Coding Using Raptor Codes

The straightforward application of Shannon's separation principle may entail a significant suboptimality in practical systems with limited coding delay and complexity. This is particularly evident when the lossy source code is based on entropy-coded quantization. In fact, it is well known that entropy coding is not robust to residual channel errors. In this paper, a joint source-channel coding scheme is advocated that combines the advantages and simplicity of entropy-coded quantization with the robustness of linear codes. The idea is to combine entropy coding and channel coding into a single linear encoding stage. If the channel is symmetric, the scheme can asymptotically achieve the optimal rate-distortion limit. However, its advantages are more clearly evident under finite coding delay and complexity. The sequence of quantization indices is decomposed into bitplanes, and each bitplane is independently mapped onto a sequence of channel coded symbols. The coding rate of each bitplane is chosen according to the bitplane conditional entropy rate. The use of systematic raptor encoders is proposed, in order to obtain a continuum of coding rates with a single basic encoding algorithm. Simulations show that the proposed scheme can outperform the separated baseline scheme for finite coding length and comparable complexity and, as expected, it is much more robust to channel errors in the case of channel capacity mismatch.

An Algorithm for Universal Lossless Compression With Side Information

This paper proposes a new algorithm based on the Context-Tree Weighting (CTW) method for universal compression of a finite-alphabet sequence x1 n with side information y1 n available to both the encoder and decoder. We prove that with probability one the compression ratio converges to the conditional entropy rate for jointly stationary ergodic sources. Experimental results with Markov chains and English texts show the effectiveness of the algorithm

Universal lossless data compression with side information by using a conditional MPM grammar transform

A grammar transform is a transformation that converts any data sequence to be compressed into a grammar from which the original data sequence can be fully reconstructed. In a grammar-based code, a data sequence is first converted into a grammar by a grammar transform and then losslessly encoded. Among several previously proposed grammar transforms is the multilevel pattern matching (MPM) grammar transform. In this paper, the MPM grammar transform is first extended to the case of side information known to both the encoder and decoder, yielding a conditional MPM (CMPM) grammar transform. A new simple linear-time and space complexity algorithm is then proposed to implement the MPM and CMPM grammar transforms. Based on the CMPM grammar transform, a universal lossless data compression algorithm with side information is developed, which can achieve asymptotically the conditional entropy rate of any stationary, ergodic source pair. It is shown that the algorithm's worst case redundancy/sample against the k-contest conditional empirical entropy among all individual sequences of length n is upper-bounded by c(1/logn), where c is a constant. The proposed algorithm with side information is the first in the coming family of conditional grammar-based codes, whose expected high efficiency is due to the efficiency of the corresponding unconditional codes.