Noiseless Coding Research Articles

Heterogeneous Questionnaire Theory

A charging scheme based on the resolution of questions strikes a new direction from the approach of Claude Picard [13]. The relationship between questionnaire theory and noiseless coding theory is explored. Graph-theoretic methods are used to obtain results valid for codes in which words are constructed from arbitrary mixtures of alphabets, as well as arborescence questionnaires, i.e., those having representation as rooted, directed trees. A charge of log d for each resolution d question is justified. By using this charging scheme an extended noiseless coding theorem shows that the average charge for a heterogeneous questionnaire is bounded below by the Shannon entropy. This result is shown to hold for both finite and countable state spaces. Certain admissibility and essentially complete class results are obtained which indicate the structure of optimal heterogeneous questionnaires.

Noiseless coding of correlated information sources

Correlated information sequences <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\cdots ,X_{-1},X_0,X_1, \cdots</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\cdots,Y_{-1},Y_0,Y_1, \cdots</tex> are generated by repeated independent drawings of a pair of discrete random variables <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X, Y</tex> from a given bivariate distribution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P_{XY} (x,y)</tex> . We determine the minimum number of bits per character <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_X</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_Y</tex> needed to encode these sequences so that they can be faithfully reproduced under a variety of assumptions regarding the encoders and decoders. The results, some of which are not at all obvious, are presented as an admissible rate region <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\mathcal{R}</tex> in the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_X - R_Y</tex> plane. They generalize a similar and well-known result for a single information sequence, namely <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R_X \geq H (X)</tex> for faithful reproduction.

Elements of Information Theoretic Methods: Tutorial Introduction

The rudiments of information theoretic methods are introduced and companion papers dealing with solutions of some reliability problems using information theoretic approaches are preluded. Major elements of the communication system are outlined from an information processing point of view. Information is quantified following the work of Shannon. The concepts of uncertainty, self-and mutual information, and entropy are developed as seen at the encoder, channel, and decoder. Channel modeling is demonstrated using a binary symmetric channel as an example. Channel capacity is derived by maximizing transinformation. Elements of coding are described and Shannon's fundamental theorem of discrete noiseless coding is stated. The fundamental relations governing unique decipherability and irreducibility are given and demonstrated by examples. Code efficiency and redundancy are quantified and discussed as system parameters subject to tradeoff. Applications of information theoretic methods in various disciplines are discussed with emphasis on reliability and maintainability. Two unresolved reliability problem areas are identified where, potentially, information theoretic approaches may present a viable solution.

Information sources with different cost scales and the principle of conservation of entropy

The aim of this paper is to provide a mathematically rigorous and sufficiently general treatment of the basic information-theoretic problems concerning sources with symbols of different costs and noiseless coding in a general sense. The main new concepts defined in this paper are the entropy rate (entropy per unit cost) of a source with respect to a stochastic cost scale and the encoding (in particular decodable encoding) of a source in a general sense. On the basis of these concepts, we prove some general theorems on the relation of entropy rates with respect to different cost scales and on the effect of encoding to the entropy rate. In particular, the “principle of conservation of entropy” and the “noiseless coding theorem” are proved under very general conditions.

The performance of an adaptive image data compression system in the presence of noise

In the present paper the problem of image data compression is considered, and the evaluation of system performance with quantitative figures of merit as well as qualitative results is attempted. Three adaptive prediction techniques proposed by Balakrishnan are described and applied to video data for the removal of so-called redundant samples. Typical prediction efficiencies of several predictors are cited and compared. Encoding of "compressed" data for transmission in a noiseless channel is next considered. The efficiencies of several practical encoding procedures are compared with the efficiency of the ideal noiseless code. The most important results describe the performance of an image data compression system in a noisy channel. Davisson has related the sample error rate of the data reconstructed at the receiver to the channel error rate for a special case. This result is used as a quantitative basis for evaluating the effects of channel errors on the performance of a specific compression system. In addition, sample television pictures which depict the qualitative effects of channel errors are exhibited. Finally, an experimental laboratory facility is briefly described, and its use in this and future work is outlined.

Two remarks to noiseless coding

An inequality concerning Kullback's I -divergence is applied to obtain a necessary condition for the possibility of encoding symbols of the alphabet of a discrete memoryless source of entropy H by sequences of symbols of another alphabet of size D in such a way that the average code length be close to the optimum H /log D . The same idea is applied to the problem of maximizing entropy per second for unequal symbol lenghts, too.

A remark concerning noiseless coding

A remark concerning noiseless coding