Binary Markov Sources Research Articles

Noiseless coding of binary finite-order Markov sources (Corresp.)

The performance of several simple noiseless coding schemes for binary finite-order Markov sources is considered. Bounds are derived on the maximum difference between the performance achievable by the schemes and the entropy of the sources. The schemes can be modified for the case where the source statistics are unknown.

On the source matching approach for Markov sources (Corresp.)

The source matching approach is a universal noiseless source coding scheme used to approximate the solution of minimax coding. A numeric solution of the source matching approach is presented for the class of binary first-order Markov sources.

Fixed-rate universal codes for Markov sources

The existence of a fixed-rate block source code whose performance for each source in a class of Markov sources is uniformly close to the distortion-rate function of that source is investigated. Such a code is called strong universal. It is found that strong universal codes of all rates exist for the class of all binary first-order Markov sources. But for a larger alphabet or for the class of all <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> th-order Markov sources with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n &gt; 2</tex> , there is a critical rate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R\ast</tex> such that strong universal codes exist for rates greater than or equal to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R\ast</tex> , but not for rates less than <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R\ast</tex> .

Explicit bounds to R(D) for a binary symmetric Markov source

A new upper hound <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_{u}(D)</tex> and lower hound <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_{\ell}(D)</tex> are developed for the rate-distortion function of a binary symmetric Markov source with respect to the frequency of error criterion. Both hounds are explicit in the sense that they do not depend on a blocklength parameter. In the interval <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D_{c} &lt; D &lt; 1/2 = D_{max}</tex> , where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D_{c}</tex> is Gray's critical value of distortion, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_{u}(D)</tex> is convex downward and possesses the correct value and the correct slope at both endpoints. The new lower bound <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_{ell}(D)</tex> diverges from the Shannon lower bound at the same value of distortion as does the second-order Wyner-Ziv lower bound. However, it remains strictly positive for all <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D \leq 1/2</tex> and therefore eventually rises above all the Wyner-Ziv lower bounds as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</tex> approaches <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</tex> . Some generalizations suggested by the analytical and geometrical techniques employed to derive <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_{u}(D)</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_{ell}(D)</tex> are discussed.

Optimal coding with a single standard run length

The optimum single standard run lengths for a binary first-order Markov source are derived and extended to multilevel first-order Markov sources. Maximization of the compression ratio is used as the criterion of optimality. When the output symbols are block coded, the optimal single standard run length n_i for each symbol is shown to satisfy an implicit equation of the form (n_i - 1)(-\ln q_{ii}) = 1 - q_{ii}^ {n_i} , where q_{ii} is a transition probability. An expression for the overall compression ratio is derived for the binary case, and a comparison is made with enumerative source encoding. Compression ratio maxima are found by computer search for the binary independent source when the output symbols are subsequently Huffman coded, and a comparison of this scheme with ordinary run-length and source-extension coding is given.

Computation of channel capacity and rate-distortion functions

By defining mutual information as a maximum over an appropriate space, channel capacities can be defined as double maxima and rate-distortion functions as double minima. This approach yields valuable new insights regarding the computation of channel capacities and rate-distortion functions. In particular, it suggests a simple algorithm for computing channel capacity that consists of a mapping from the set of channel input probability vectors into itself such that the sequence of probability vectors generated by successive applications of the mapping converges to the vector that achieves the capacity of the given channel. Analogous algorithms then are provided for computing rate-distortion functions and constrained channel capacities. The algorithms apply both to discrete and to continuous alphabet channels or sources. In addition, a formalization of the theory of channel capacity in the presence of constraints is included. Among the examples is the calculation of close upper and lower bounds to the rate-distortion function of a binary symmetric Markov source.

Coding the binary symmetric source with a fidelity criterion (Corresp.)

A procedure is defined for coding the binary symmetric first-order Markov source with the probability of error fidelity criterion. Since the rate-distortion function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(d)</tex> for this source has not yet been formulated, the coding performance provides an apparently tight upper bound for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(d)</tex> .

Rate distortion functions for finite-state finite-alphabet Markov sources

A lower bound to the rate-distortion function R(D) of finite-alphabet sources with memory is derived for the class of balanced distortion measures. For finite-state finite-alphabet Markov sources, sufficient conditions are given for the existence of a strictly positive average distortion D_c such that R(D) equals its lower bound for 0 \leqq D \leqq D_c . The bound is evaluated for the Hamming and Lee distortion measures and is identical to the corresponding bound for memoryless sources having the same entropy and alphabet. These results are applied to yield a simple proof of the converse of the noisy-channel coding theorem for sources satisfying the sufficient conditions for equality with the lower bound and channels with memory. D_c is evaluated explicitly for the special case of the binary asymmetric Markov source.