Average Codeword Length Research Articles

Combined source/channel coding and a new coding scheme for data storage systems

Research has tended to concentrate on the separate areas of source coding (for data compression) and channel coding (for data storage), this paper proposes data compression codes that combine source and channel codings under minimum and maximum run-length constraints for storage channels. The encoding methods, decoding procedure, and average codeword lengths of the proposed codes are presented. Also, a new encoding scheme for data storage systems, one which employs the error control coding and modulation coding to make full use of the combined source/channel codes, is suggested. The suggested encoding scheme, together with the combined source/channel codes, enables the storing of more data within a given size storage system.

More on the error recovery for variable-length codes

Variable-length codes (e.g. Huffman codes) are commonly employed to minimize the average codeword length for noiseless encoding of discrete sources. Upon transmission over noisy channels, conflicting views note that such codes "tend to be self-synchronizing" and suffer from the "catastrophic effect of the error's propagation". Maxted and Robinson (1985) used a state model to describe the error recovery of the decoder. The present paper extends their results in several ways.

Codes with monotonic codeword lengths

Minimum average codeword length coding under the constraint that the codewords are monotonically nondecreasing in length is studied. Bounds on the average length of an optimal “monotonie” code are derived, and sufficient conditions are given such that algorithms for optimal alphabetic codes can be used to find the optimal “monotonic” code.

Binary prefix codes ending in a "1"

Binary prefix codes with the constraint that each codeword must end with a "1" have been recently introduced by Berger and Yeung (1990). We analyze the performance of such codes by investigating their average codeword length. In particular, we show that a very simple strategy permits the construction of a "1"-ended binary prefix code whose average codeword length is less than H+1 for any discrete source with entropy H. We also prove a tight lower bound on the optimal average codeword length in terms of H and of the minimum letter probability of the source. Finally, we discuss the problem of finding an optimum feasible code.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Huffman-type codes for infinite source distributions

A new sufficient condition is given for an infinite source distribution to share a minimum average codeword length code with the geometric distribution. Thus, some new examples of parametric families of infinite source distributions can be optimally encoded by Huffman-type codes.

Fast and efficient coding of information sources

The author considers the problem of source coding and investigates the cases of known and unknown statistics. The efficiency of the compression codes can be estimated by three characteristics: 1) the redundancy (r), defined as the maximal difference between the average codeword length and Shannon entropy in case the letters are generated by a Bernoulli source; 2) the size (in bits) of the encoder and the encoder programs (S) when implemented on a computer; and 3) the average time required for encoding and decoding of a single letter (T). He investigates S and T as a function of r when r/spl rarr/0. All known methods may be divided into two classes. The Ziv-Lempel codes and their variants fall under the first class, and the arithmetic code with the Lynch-Davisson code fall under the second one. The codes from the first class need exponential memory size S=0(exp(1/r)) for redundancy r when r/spl rarr/O. The methods from the second class have a small memory size but a low encoding speed: S=0(1/r/sup const/), T=0(log/sup const/(1/r) log log (1/r)). In this paper, the author presents a code that combines the merits of both classes; the memory size is small and the speed is high: S=0(1/r/sup const/),T=0(log/sup const/(1/r) log log (1/r)).< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Inaccuracy and the best 1:1 code

The expected codeword length LUD of the best uniquely decodable (UD) code satisfies H(P;Q) ≤ LUD < H(P;Q) + 1, where H(P;Q) is the Kerridge inaccuracy. By applying the idea of the best 1:1 code given by Leung Yan Cheong and T. Cover [IEEE Trans. Inform. TheoryIT-24, 331–338 (1978)] a relation between inaccuracy and the best 1:1 codeword length L1:1 has been obtained. Further, it is shown that the average codeword length L1:1 is shorter than the average codeword length LUD by no more than log · logn + 3. Also, the lower bounds to the exponentiated mean codeword length in terms of the inaccuracy of type α have been obtained.

Relative uniformity of sources and the comparison of optimal code costs

Conditions on the probability distributions of two sources that describe their relative uniformity are shown to be sufficient for inequalities on their minimum average codeword lengths and entropies. The Huffman coding problem and variants involving Renyi entropy are addressed. >

On the construction of statistically synchronizable codes

The problem of constructing statistically synchronizable codes over arbitrary alphabets and for any finite source is considered. It is shown how to efficiently construct a statistically synchronizable code whose average codeword length is within the least likely codeword probability from that of the Huffman code for the same source. Moreover, a method is given for constructing codes having a synchronizing codeword. The method yields synchronous codes that exhibit high synchronizing capability and low redundancy.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

A lower bound on average codeword length of variable length error-correcting codes

Codes of variable length that are capable of error correction are studied. A lower bound on the average length of such codes under the criterion of promptness is obtained. This generalizes the celebrated result due to C.E. Shannon (1948) for the noiseless case. The bound obtained provides a measure of optimality for variable-length, error-correcting codes. >

Optimum '1'-ended binary prefix codes

The problem of finding a binary prefix code of minimum average codeword length for a given finite probability distribution subject to the requirement that each codeword must end with a 1 is considered. Lower and upper bounds to the performance of the optimum code are derived; the lower bound is tight for certain probability distributions. An algorithm that generates an optimum code for any given distribution is described. >

Properties of low augmentation level T-codes

Initial investigations of the re-synchronisation properties of fifth augmentation level, variable length, self-synchronising T-codes are described. It has been found from simulation studies that within the fifth augmentation level family there are considerable differences in the resynchronisation delay statistics of individual code sets, with some code sets consistently outperforming others with the same codeword length distribution. Furthermore, the code sets with the shortest average re-synchronisation delays are generally those with the shortest average code-word length (i.e. the most efficient). The coding efficiency of these code sets has been found to be within 1% of the efficiency of a Huffman code optimised for the particular information source.

The Renyi redundancy of generalized Huffman codes

Huffman's algorithm gives optimal codes, as measured by average codeword length, and the redundancy can be measured as the difference between the average codeword length and Shannon's entropy. If the objective function is replaced by an exponentially weighted average, then a simple modification of Huffman's algorithm gives optimal codes. The redundancy can now be measured as the difference between this new average and A. Renyi's (1961) generalization of Shannon's entropy. By decreasing some of the codeword lengths in a Shannon code, the upper bound on the redundancy given in the standard proof of the noiseless source coding theorem is improved. The lower bound is improved by randomizing between codeword lengths, allowing linear programming techniques to be used on an integer programming problem. These bounds are shown to be asymptotically equal. The results are generalized to the Renyi case and are related to R.G. Gallager's (1978) bound on the redundancy of Huffman codes. >

On the average codeword length of optimal binary codes for extended sources (Corresp.)

Although optimal binary source coding using symbol blocks of increasing length must eventually yield a code having an average codeword length arbitrarily close to the source entropy, it is known that the sequence of average codeword lengths need not be nonincreasing. The sequence is, however, bounded above by the average codeword length of the source, and certain subsequences must be nondecreasing. Sufficient conditions are obtained describing sources for which a decrease in average codeword length is achieved when coding pairs of symbols. A sufficient condition specifying sources for which no such decrease is possible is also obtained.

Interval and recency rank source coding: Two on-line adaptive variable-length schemes

In the schemes presented the encoder maps each message into a codeword in a prefix-free codeword set. In interval encoding the codeword is indexed by the interval since the last previous occurrence of that message, and the codeword set must be countably infinite. In recency rank encoding the codeword is indexed by the number of distinct messages in that interval, and there must be no fewer codewords than messages. The decoder decodes each codeword on receipt. Users need not know message probabilities, but must agree on indexings, of the codeword set in an order of increasing length and of the message set in some arbitrary order. The average codeword length over a communications bout is never much larger than the value for an off-line scheme which maps the j th most frequent message in the bout into the j th shortest codeword in the given set, and is never too much larger than the value for off-line Huffman encoding of messages into the best codeword set for the bout message frequencies. Both schemes can do much better than Huffman coding when successive selections of each message type cluster much more than in the independent case.

Algorithm for obtaining a self-synchronising M-ary code enabling data compression

An algorithm for obtaining a self-synchronising M-ary code (M ⩾ 2) enabling the compression of data from a stationary discrete memoryless source is proposed. After presenting the code algorithm, its properties are analysed and the implementation of the code is described. The code proposed is compared to the Huffman code with regard to the average code-word length, the possibility of self synchronisation and the complexity of hardware realisation. Although for certain sources the code proposed is equal or nearly equal to the Huffman code regarding data compression, in general it is less efficient. However, its property of being self synchronising, and its relatively simple hardware realisation, make this code valuable for practical applications.

Synchronization of binary source codes (Corresp.)

The problem of achieving synchronization for variable-length source codes is addressed through the use of self-synchronizing binary prefix-condition codes. Although our codes are suboptimal in the sense of minimum average codeword length, they have the advantages of being generated by an explicit constructive algorithm, having minimal additional redundancy compared with optimal codes-as little as one additional bit introduced into the least likely codeword for a large class of sources-and having statistical synchronizing performance that improves on that of the optimal code in many cases.

Minimum error entropy estimation and entropic prediction filtering: An optimal predictive coding scheme

The minimum error entropy (MEE) estimation problem on discrete, finite, or countably infinite ensembles of arbitrary statistical nature is considered. A combinatorial approach is taken to solve the minimization problem. The solution is obtained in algorithmic form. The numerical complexity of the MEE algorithm is analyzed. An entropic prediction filtering (EPF) coding scheme that utilizes the MEE algorithm to minimize the per symbol error entropy is introduced. Simulation has shown that the EPF can effectively be used to reduce the average codeword length required to encode data. Other applications of the MEE algorithm in signal interpolation and extrapolation are also discussed.

Robust noiseless source coding through a game theoretical approach

Noiseless coding of a discrete source with partially known statistics is formulated as a two-person game. The payoff is the average codeword length, using Shannon codes. The code designer picks a source probability distribution for the design of the code, while an opponent picks the actual source probability distribution. It is shown that if the class of probability distributions allowed is convex, then there is a saddle point solution which is determined by the maximum entropy distribution of the convex class. The maximum entropy element is derived for three families of source probability mass functions (pmf): a) the class of c-contaminated pmf's; b) the class of pmf's for which each probability is known only through an upper and lower bound; c) the class of pmf's which is a convex hull of a finite number of known pmf's. An extension of the robust noiseless source coding problem for families of sources modeled as first-order Markov chains is discussed.

Minimax optimal universal codeword sets

In an interactive multi-user data-processing system, a user knows the probabilities of his messages and must encode them into a fixed system-wide variable-length codeword set. He needs to receive the answer to his last message before selecting the next, so his encoding is one-shot. To minimize average codeword length he encodes his messages in order of decreasing probability into codewords in order of increasing length. An algorithm is given which, for each of several measures of performance, finds the codeword set best by that measure for the worst user, and some of the minimax optimal codeword sets the algorithm has found. Some of the results hold for all user distributions: others require, e.g., that all users send exactly or at most m distinct messages, or that there is an integer k such that no user has a message of probability greater than l/k .