Shannon Lower Bound Research Articles

A Lower Bound on the Differential Entropy of Log-Concave Random Vectors with Applications.

We derive a lower bound on the differential entropy of a log-concave random variable X in terms of the p-th absolute moment of X. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. Specifically, we study the rate-distortion function for log-concave sources and distortion measure , with , and we establish that the difference between the rate-distortion function and the Shannon lower bound is at most bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most bit, regardless of d. We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most bit. Our results generalize to the case of a random vector X with possibly dependent coordinates. Our proof technique leverages tools from convex geometry.

Entropy-Constrained Scalar Quantization with a Lossy-Compressed Bit

We consider the compression of a continuous real-valued source X using scalar quantizers and average squared error distortion D. Using lossless compression of the quantizer’s output, Gish and Pierce showed that uniform quantizing yields the smallest output entropy in the limit D → 0 , resulting in a rate penalty of 0.255 bits/sample above the Shannon Lower Bound (SLB). We present a scalar quantization scheme named lossy-bit entropy-constrained scalar quantization (Lb-ECSQ) that is able to reduce the D → 0 gap to SLB to 0.251 bits/sample by combining both lossless and binary lossy compression of the quantizer’s output. We also study the low-resolution regime and show that Lb-ECSQ significantly outperforms ECSQ in the case of 1-bit quantization.

The Shannon Lower Bound Is Asymptotically Tight

The Shannon lower bound is one of the few lower bounds on the rate-distortion function that holds for a large class of sources. In this paper, it is demonstrated that its gap to the rate-distortion function vanishes as the allowed distortion tends to zero for all sources having a finite differential entropy and whose integer part is finite. Conversely, it is demonstrated that if the integer part of the source has an infinite entropy, then its rate-distortion function is infinite for every finite distortion. Consequently, the Shannon lower bound provides an asymptotically tight bound on the rate-distortion function if, and only if, the integer part of the source has a finite entropy.

LQG optimality and separation principle for general discrete time partially observed stochastic systems over finite capacity communication channels

This paper is concerned with control of stochastic systems subject to finite communication channel capacity. Necessary conditions for reconstruction and stability of system outputs are derived using the Information Transmission theorem and the Shannon lower bound. These conditions are expressed in terms of the Shannon entropy rate and the distortion measure employed to describe reconstruction and stability. The methodology is general, and hence it is applicable to a variety of systems. The results are applied to linear partially observed stochastic Gaussian controlled systems, when the channel is an Additive White Gaussian Noise (AWGN) channel. For such systems and channels, sufficient conditions are also derived by first showing that the Shannon lower bound is exactly equal to the rate distortion function, and then designing the encoder, decoder and controller which achieve the capacity of the channel. The conditions imposed are the standard detectability and stabilizability of Linear Quadratic Gaussian (LQG) theory, while a separation principle is shown between the design of the control and communication systems, without assuming knowledge of the control sequence at the encoder/decoder.

On competitive prediction and its relation to rate-distortion theory

Consider the normalized cumulative loss of a predictor F on the sequence x/sup n/=(x/sub 1/,...,x/sub n/), denoted L/sub F/(x/sup n/). For a set of predictors G, let L(G,x/sup n/)=min/sub F/spl isin/G/L/sub F/(x/sup n/) denote the loss of the best predictor in the class on x/sup n/. Given the stochastic process X=X/sub 1/,X/sub 2/,..., we look at EL(G,X/sup n/), termed the competitive predictability of G on X/sup n/. Our interest is in the optimal predictor set of size M, i.e., the predictor set achieving min/sub |G|/spl les/M/EL(G,X/sup n/). When M is subexponential in n, simple arguments show that min/sub |G|/spl les/M/EL(G,X/sup n/) coincides, for large n, with the Bayesian envelope min/sub F/EL/sub F/(X/sup n/). We investigate the behavior, for large n, of min/sub |G|/spl les/e//sup nR/EL(G,X/sup n/), which we term the competitive predictability of X at rate R. We show that whenever X has an autoregressive representation via a predictor with an associated independent and identically distributed (i.i.d.) innovation process, its competitive predictability is given by the distortion-rate function of that innovation process. Indeed, it will be argued that by viewing G as a rate-distortion codebook and the predictors in it as codewords allowed to base the reconstruction of each symbol on the past unquantized symbols, the result can be considered as the source-coding analog of Shannon's classical result that feedback does not increase the capacity of a memoryless channel. For a general process X, we show that the competitive predictability is lower-bounded by the Shannon lower bound (SLB) on the distortion-rate function of X and upper-bounded by the distortion-rate function of any (not necessarily memoryless) innovation process through which the process X has an autoregressive representation. Thus, the competitive predictability is also precisely characterized whenever X can be autoregressively represented via an innovation process for which the SLB is tight. The error exponent, i.e., the exponential behavior of min/sub |G|/spl les/exp(nR)/Pr(L(G,X/sup n/)>d), is also characterized for processes that can be autoregressively represented with an i.i.d. innovation process.

Additive successive refinement

Rate-distortion bounds for scalable coding, and conditions under which they coincide with nonscalable bounds, have been extensively studied. These bounds have been derived for the general tree-structured refinement scheme, where reproduction at each layer is an arbitrarily complex function of all encoding indexes up to that layer. However, in most practical applications (e.g., speech coding) refinement structures such as the multistage vector quantizer are preferred due to memory limitations. We derive an achievable region for the additive successive refinement problem, and show via a converse result that the rate-distortion bound of additive refinement is above that of tree-structured refinement. Necessary and sufficient conditions for the two bounds to coincide are derived. These results easily extend to abstract alphabet sources under the condition E{d(X,a)}</spl infin/ for some letter a. For the special cases of square-error and absolute-error distortion measures, and subcritical distortion (where the Shannon lower bound (SLB) is tight), we show that successive refinement without rate loss is possible not only in the tree-structured sense, but also in the additive-coding sense. We also provide examples which are successively refinable without rate loss for all distortion values, but the optimal refinement is not additive.

Image subband coding using arithmetic coded trellis coded quantization

A method is presented for encoding memoryless sources using trellis coded quantization (TCQ) with uniform thresholds. The trellis symbols are entropy-coded using arithmetic coding. The performance of the arithmetic coded uniform threshold TCQ, for encoding the family of generalized Gaussian densities, is compared with uniform threshold quantization (UTQ) and the Shannon lower bound (SLB). At high rates, the method performs within 0.5 dB of the rate-distortion bound for the family of generalized Gaussian densities. A simple modification of the uniform codebook is shown to result in improved performance at low bit rates. The arithmetic and trellis coding method is used for encoding image subbands. Coding results for monochrome images are presented and compared with other results in the literature. Working C code that implements arithmetic coded uniform threshold TCQ can be obtained using anonymous ftp.

On the asymptotic tightness of the Shannon lower bound

New results are proved on the convergence of the Shannon (1959) lower bound to the rate distortion function as the distortion decreases to zero. The key convergence result is proved using a fundamental property of informational divergence. As a corollary, it is shown that the Shannon lower bound is asymptotically tight for norm-based distortions, when the source vector has a finite differential entropy and a finite /spl alpha/ th moment for some /spl alpha/>0, with respect to the given norm. Moreover, we derive a theorem of Linkov (1965) on the asymptotic tightness of the Shannon lower bound for general difference distortion measures with more relaxed conditions on the source density. We also show that the Shannon lower bound relative to a stationary source and single-letter difference distortion is asymptotically tight under very weak assumptions on the source distribution. >

Information and distortion in reduced-order filter design

The relation and practical relevance of information theory to the filtering problem has long been an open question. The design, evaluation, and comparison of (suboptimal) reduced-order filters by information methods is considered. First, the differences and similarities between the information theory problem and the filtering problem are delineated. Then, based on these considerations, a formulation that {\em realistically} imbeds the reduced-order filter problem in an information-theoretic framework is presented. This formulation includes a constrained version of the rate-distortion function. The Shannon lower bound is used both to derive formulas for (achievable) rose lower bounds for suboptimal filters and to prove that for thc reduced-order filter problem the given formulation specifies a useful relation between information and distortion in filtering. Theorems addressed to reduced-order filter design, evaluation, and comparison based on information are given. A two step design procedure is outlined which results in a decoupling of thc search in filter parameter space, and hence in computational savings.