Shannon Rate-distortion Function Research Articles

The Rate-Distortion Risk in Estimation From Compressed Data

Consider the problem of estimating a latent signal from a lossy compressed version of the data when the compressor is agnostic to the relation between the signal and the data. This situation arises in a host of modern applications when data is transmitted or stored prior to determining the downstream inference task. Given a bitrate constraint and a distortion measure between the data and its compressed version, let us consider the joint distribution achieving Shannon's rate-distortion (RD) function. Given an estimator and a loss function associated with the downstream inference task, define the RD risk as the expected loss under the RD-achieving distribution. We provide general conditions under which the operational risk in estimating from the compressed data is asymptotically equivalent to the RD risk. The main theoretical tools to prove this equivalence are transportation-cost inequalities in conjunction with properties of compression codes achieving Shannon's RD function. Whenever such equivalence holds, a recipe for designing estimators from datasets undergoing lossy compression without specifying the actual compression technique emerges: design the estimator to minimize the RD risk. Our conditions are simplified in the special cases of discrete memoryless or multivariate normal data. For these scenarios, we derive explicit expressions for the RD risk of several estimators and compare them to the optimal source coding performance associated with full knowledge of the relation between the latent signal and the data.

The Rate-Distortion Function and Excess-Distortion Exponent of Sparse Regression Codes With Optimal Encoding

This paper studies the performance of sparse regression codes for lossy compression with the squared-error distortion criterion. In a sparse regression code, codewords are linear combinations of subsets of columns of a design matrix. It is shown that with minimum-distance encoding, sparse regression codes achieve the Shannon rate-distortion function for i.i.d. Gaussian sources $R^*(D)$ as well as the optimal excess-distortion exponent. This completes a previous result which showed that $R^*(D)$ and the optimal exponent were achievable for distortions below a certain threshold. The proof of the rate-distortion result is based on the second moment method, a popular technique to show that a non-negative random variable $X$ is strictly positive with high probability. In our context, $X$ is the number of codewords within target distortion $D$ of the source sequence. We first identify the reason behind the failure of the standard second moment method for certain distortions, and illustrate the different failure modes via a stylized example. We then use a refinement of the second moment method to show that $R^*(D)$ is achievable for all distortion values. Finally, the refinement technique is applied to Suen's correlation inequality to prove the achievability of the optimal Gaussian excess-distortion exponent.

Lossy Compression via Sparse Linear Regression: Performance Under Minimum-Distance Encoding

We study a new class of codes for lossy compression with the squared-error distortion criterion, designed using the statistical framework of high-dimensional linear regression. Codewords are linear combinations of subsets of columns of a design matrix. Called a sparse superposition or sparse regression codebook, this structure is motivated by an analogous construction proposed recently by Barron and Joseph for communication over an Additive White Gaussian Noise channel. For independent identically distributed (i.i.d) Gaussian sources and minimum-distance encoding, we show that such a code can attain the Shannon rate-distortion function with the optimal error exponent, for all distortions below a specified value. It is also shown that sparse regression codes are robust in the following sense: a codebook designed to compress an i.i.d Gaussian source of variance σ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> with (squared-error) distortion D can compress any ergodic source of variance less than σ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> to within distortion D. Thus, the sparse regression ensemble retains many of the good covering properties of the i.i.d random Gaussian ensemble, while having a compact representation in terms of a matrix whose size is a low-order polynomial in the block-length.

Improved Upper Bounds to the Causal Quadratic Rate-Distortion Function for Gaussian Stationary Sources

We improve the existing achievable rate regions for causal and for zero-delay source coding of stationary Gaussian sources under an average mean squared error distortion measure. To begin with, we find a closed-form expression for the information-theoretic causal rate-distortion function (RDF) under such distortion measure, denoted by R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it</sup> (D), for first-order Gauss-Markov processes. R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it</sup> (D) is a lower bound to the optimal performance theoretically attainable (OPTA) by any causal source code, namely R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">op</sup> (D). We show that, for Gaussian sources, the latter can also be upper bounded as R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">op</sup> (D) ≤ R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it</sup> (D) + 0.5 log <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> (2πe) bits/sample. In order to analyze R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it</sup> (D) for arbitrary zero-mean Gaussian stationary sources, we introduce R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D), the information-theoretic causal RDF when the reconstruction error is jointly stationary with the source. Based upon R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D), we derive three closed-form upper bounds to the additive rate loss defined as R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D) - R(D), where R(D) denotes Shannon's RDF. Two of these bounds are strictly smaller than 0.5 bits/sample at all rates. These bounds differ from one another in their tightness and ease of evaluation; the tighter the bound, the more involved its evaluation. We then show that, for any source spectral density and any positive distortion D ≤ σ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , RU(D) can be realized by an additive white Gaussian noise channel surrounded by a unique set of causal pre-, post-, and feed- back niters. We show that finding such filters constitutes a convex optimization problem. In order to solve the latter, we propose an iterative optimization procedure that yields the optimal niters and is guaranteed to converge to R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D). Finally, by establishing a connection to feedback quantization, we design a causal and a zero-delay coding scheme which, for Gaussian sources, achieves an operational rate lower than R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D) +0.254 and R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D) + 0.754 bits/sample, respectively. This implies that the OPTA among all zero-delay source codes, denoted by R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">zd</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">op</sup> (D), is upper bounded as R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">zd</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">op</sup> (D) <; R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">it̅</sup> (D) + 1-254 <; R(D) + 1.754 bits/sample.

Distributed Source Coding Using Abelian Group Codes: A New Achievable Rate-Distortion Region

A distributed source coding problem with a joint distortion criterion that depends on the sources and the reconstruction is considered in this work. While the prevalent trend in information theory has been to prove achievability results using Shannon's random coding arguments, using structured random codes offer rate gains over unstructured random codes for many problems. Motivated by this, a new achievable rate-distortion region (an inner bound to the performance limit) is presented for this problem for discrete memoryless sources based on “good” structured random nested codes built over abelian groups. For certain sources and distortion functions, the new rate region is shown to be strictly bigger than the Berger-Tung rate region, which has been the best known achievable rate region for this problem till now. This is done using numerical plots. Achievable rates for single-user source coding using abelian group codes are also obtained as a corollary of the main coding theorem. It is shown that nested linear codes achieve the Shannon rate-distortion function in the arbitrary discrete memoryless case.

Rate Distortion and Denoising of Individual Data Using Kolmogorov Complexity

We examine the structure of families of distortion balls from the perspective of Kolmogorov complexity. Special attention is paid to the canonical rate-distortion function of a source word which returns the minimal Kolmogorov complexity of all distortion balls containing that word subject to a bound on their cardinality. This canonical rate-distortion function is related to the more standard algorithmic rate-distortion function for the given distortion measure. Examples are given of list distortion, Hamming distortion, and Euclidean distortion. The algorithmic rate-distortion function can behave differently from Shannon's rate-distortion function. To this end, we show that the canonical rate-distortion function can and does assume a wide class of shapes (unlike Shannon's); we relate low algorithmic mutual information to low Kolmogorov complexity (and consequently suggest that certain aspects of the mutual information formulation of Shannon's rate-distortion function behave differently than would an analogous formulation using algorithmic mutual information); we explore the notion that low Kolmogorov complexity distortion balls containing a given word capture the interesting properties of that word (which is hard to formalize in Shannon's theory) and this suggests an approach to denoising.

Low-Resolution Scalar Quantization for Gaussian Sources and Absolute Error

This correspondence considers low-resolution scalar quantization for a memoryless Gaussian source with respect to absolute error distortion. It shows that slope of the operational rate-distortion function of scalar quantization is infinite at the point D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">max</sub> where the rate becomes zero. Thus, unlike the situation for squared error distortion, or for Laplacian and exponential sources with squared or absolute error distortion, for a Gaussian source and absolute error, scalar quantization at low rates is far from the Shannon rate-distortion function, i.e., far from the performance of the best lossy coding technique

Low-resolution scalar quantization for Gaussian sources and squared error

This correspondence analyzes the low-resolution performance of entropy-constrained scalar quantization. It focuses mostly on Gaussian sources, for which it is shown that for both binary quantizers and infinite-level uniform threshold quantizers, as D approaches the source variance /spl sigma//sup 2/, the least entropy of such quantizers with mean-squared error D or less approaches zero with slope -log/sub 2/e/2/spl sigma//sup 2/. As the Shannon rate-distortion function approaches zero with the same slope, this shows that in the low-resolution region, scalar quantization with entropy coding is asymptotically as good as any coding technique.

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression. The application-specific cost is minimized subject to a constraint on the randomness of the solution, which is gradually lowered. We emphasize the intuition gained from analogy to statistical physics. Alternatively the method is derived within rate-distortion theory, where the annealing process is equivalent to computation of Shannon's rate-distortion function, and the annealing temperature is inversely proportional to the slope of the curve. The basic algorithm is extended by incorporating structural constraints to allow optimization of numerous popular structures including vector quantizers, decision trees, multilayer perceptrons, radial basis functions, and mixtures of experts.

Elasticity conditions for storage versus error exchange in question--Answering systems

It has been conjectured that error-allowance could improve dramatically the performance of data processing systems. This hypothesis is tested in the framework of question-answering (QA) systems with storage requirements as a complexity measure. Shannon's rate distortion function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> represents the minimum amount of memory a system must employ in order to achieve an average distortion less than <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</tex> (the distortion can be, for example, the average proportion of erroneous answers produced by the system). The ability of a system to convert an amount <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</tex> of distortion into memory savings is measured by the ratio <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)/R(O)</tex> . A system will be called elastic if this ratio goes to zero as the size of the dataset ensemble goes to infinity. Asymptotic bounds to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> are derived giving rise to elasticity conditions invoking the structure of the distortion matrix associated with the system. The bounds established represent a marked Improvement over former results by narrowing the gap between the necessary and sufficient conditions for elasticity. Moreover, conditions are established under which the amount of computation required for testing elasticity can be substantially reduced.

Bounds on memory versus error trade-offs in question-answering systems

A question-answering (QA) system answers queries from a given set about data sets drawn from a given ensemble. Shannon's rate-distortion function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> provides the minimum amount of memory that a QA system must employ in order to achieve an average distortion less than <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</tex> (the distortion can, for example, be the average proportion of erroneous answers produced by the system). The ability of a system to convert an amount <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</tex> of distortion into a saving in memory is measured by the ratio <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)/R(0)</tex> . A system will be called elastic if this ratio goes to zero as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</tex> , the size of the data set ensemble, tends to infinity. The asymptotic bounds to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> are derived, giving rise to elasticity conditions which involve only general system parameters.

Theoretical bounds on the complexity of inexact computations

This paper considers the reduction in algorithmic complexity that can be achieved by permitting approximate answers to computational problems. It is shown that Shannon's rate-distortion function could, under quite general conditions, provide lower bounds on the mean complexity of inexact computations. As practical examples of this approach, we show that partial sorting of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> items, insisting on matching any nonzero fraction of the terms with their correct successors, requires <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O (N \log N)</tex> comparisons. On the other hand, partial sorting in linear time is feasible (and necessary) if one permits any finite fraction of pairs to remain out of order. It is also shown that any error tolerance below 50 percent can neither reduce the state complexity of binary <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> -sequences from the zero-error value of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(N)</tex> nor reduce the combinational complexity of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> -variable Boolean functions from the zero-error level of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(2^{N}/N)</tex> .

Differential pulse-code modulation (PCM) with entropy coding

As the transmission rate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</tex> gets large, differential pulse-code modulation (PCM) when followed by entropy coding forms a source encoding system which performs within 1.53 dB of Shannon's rate distortion function which bounds the performance of any encoding system with a minimum mean-square error (mmse) fidelity criterion. This is true for any ergodic signal source. Furthermore, this source encoder introduces the same amount of uncertainty as the mmse encoder. The 1.53 dB difference between this encoder and the mmse encoder is perceptually so small that it would probably not be noticed by a human user of a high quality (signal-to-noise ratio <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(S/N) \geq 30</tex> dB) speech or television source encoding system.

The effects of a visual fidelity criterion of the encoding of images

Shannon's rate-distortion function provides a potentially useful lower bound against which to compare the rate-versus-distortion performance of practical encoding-transmission systems. However, this bound is not applicable unless one can arrive at a numerically-valued measure of distortion which is in reasonable correspondence with the subjective evaluation of the observer or interpreter. We have attempted to investigate this choice of distortion measure for monochrome still images. This investigation has considered a class of distortion measures for which it is possible to simulate the optimum (in a rate-distortion sense) encoding. Such simulation was performed at a fixed rate for various measures in the class and the results compared subjectively by observers. For several choices of transmission rate and original images, one distortion measure was fairly consistently rated as yielding the most satisfactory appearing encoded images.

A 2-cycle algorithm for source coding with a fidelity criterion

Since the birth of rate-distortion theory with the landmark paper of Shannon, research has centered on the extension of his theory to new situations and on the calculation of rate-distortion functions. Comparatively little has been done toward developing efficient algorithms for encoding information sources at rates near Shannon's rate-distortion limit. This paper reports one such algorithm, the 2-cycle algorithm, used with randomly chosen tree codes. In analyzing the 2-cycle algorithm, we present the first theoretical analysis of a realizable algorithm for tree coding of sources with a fidelity criterion. The algorithm may prove to be the first practical method of coding whose design actually derives from rate-distortion theory. Analysis proceeds by bounding methods related to difference equations and branching processes. Upper bounds on average distortion and encoding work are obtained and the latter are shown bounded as long as the coding rate exceeds Shannon's rate-distortion function. Of particular interest is the stack-searched branching process bound of Section IV. The two cycles of the algorithm are described fully in terms of expectations and probabilities, and recursions are listed to compute these quantities. Concluding the paper is numerical analysis, by theory and by simulation, with respect to the binary i.i.d, source and Hamming distortion measure. These numerical results are sufficient to optimize the algorithm over its free parameters.

Epsilon Entropy and Data Compression

This article studies efficient data transmission, or "data compression", from the standpoint of the theory of epsilon entropy. The notion of the entropy of a "data source" is defined. This quantity gives a precise measure of the amount of channel capacity necessary to describe a data source to within a given fidelity, epsilon, with probability one, when each separate "experiment" must be transmitted without storage from experiment to experiment. We also define the absolute epsilon entropy of a source, which is the amount of capacity needed when storage of experiments is allowed before transmission. The absolute epsilon entropy is shown to be equal to Shannon's rate distortion function evaluated for zero distortion, when suitable identifications are made. The main result is that the absolute epsilon entropy and the epsilon entropy have ratio close to one if either is large. Thus, very little can be saved by storing the results of independent experiments before transmission.

The computation and bounding of rate-distortion functions

Methods are given for the numerical computation of Shannon's rate-distortion function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> for certain memoryless message sources. It is first assumed that <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</tex> , the set of possible message-source outputs, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</tex> , the set of possible destination symbols, are countable. The computation of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> for this case is reduced to a minimization problem in which the variables are the destination-symbol probabilities. For arbitrary <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</tex> , upper and lower bounds on <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> are derived by partitioning <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</tex> each into a countable collection of disjoint subsets and employing the results derived previously for the case of countable <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</tex> . Conditions are then discussed under which these bounds can be made arbitrarily close to each other by choosing sufficiently fine partitions of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</tex> . Two examples are included to illustrate the results in detail.

The rate distortion function for a class of sources

For a source with a known probability distribution, α , Shannon's rate distortion function, R α (d) , specifies the minimum channel capacity required to transmit the output of the source with average distortion ≦ d . In practice, the encoding system must often be designed when only vague information is available concerning the source distribution; i.e., it is known only that the source distribution is a member of some class α . It is thus important to extend Shannon's theory to cover this case. In this paper we define a rate distortion function for a class of sources, R α (d) , and prove a coding theorem. This theorem establishes that R α (d) is the minimum channel capacity required by any system which can transmit each source in α with average distortion ≦ d , and that for a compact class of sources this rate can be approached as closely as desired. Further, we show that for a compact class, R α (d) is the sup of R α (d) over all α in α . This last result is important in that it greatly simplifies the computation of R α (d) .