Entropy-constrained Quantization Research Articles

Scalable Audio Coding Using Trellis-Based Optimized Joint Entropy Coding and Quantization

There is a considerable performance gap between the current scalable audio coding schemes and a nonscalable coder operating at the same bitrate. This suboptimality results from the independent coding of the layers in these systems. One of the aspects that plays a role in this suboptimality is the entropy coding. In practical audio coding systems including MPEG advanced audio coding (AAC), the transform domain coefficients are quantized using an entropy-constrained quantizer. In MPEG-4 scalable AAC (S-AAC), the quantization and coding are performed separately at each layer. In case of Huffman coding, the redundancy introduced by the entropy coding at each layer is larger at lower quantization resolutions. Also, the redundancy for the overall coder becomes larger as the number of layers increases. In fact, there is a tradeoff between the overall redundancy and the fine-grain scalability in which the bitrate per layer is smaller and more layers are required. In this paper, a fine-grain scalable coder for audio signals is proposed where the entropy coding of a quantizer is made scalable via joint design of entropy coding and quantization. By constructing a Huffman-like coding tree where the internal nodes can be mapped to the reconstruction points, the tree can be pruned at any internal node to control the rate-distortion (RD) performance of the encoder in a fine-grain manner. A set of metrics and a trellis-based approach is proposed to create a coding tree so that an appropriate path is generated on the RD plane. The results show the proposed method outperforms the scalable audio coding performed based on reconstruction error quantization as used in practical systems, e.g., in S-AAC.

Parametric Audio Coding With Exponentially Damped Sinusoids

Sinusoidal modeling is one of the most popular techniques for low bitrate audio coding. Usually, the sinusoidal parameters (amplitude, pulsation and phase of each sinusoidal component) are kept constant within a time segment. An alternative model, the so-called Exponentially-Damped Sinusoidal (EDS) model, includes an additional damping parameter for each sinusoidal component to better represent the signal characteristics. It was however never shown that the EDS model could be efficient for perceptual audio coding. To that aim, we propose in this paper an efficient analysis/synthesis framework with dynamic time-segmentation on transients and psychoacoustic modeling, and an asymptotically optimal entropy-constrained quantization method for the four sinusoid parameters (e.g., including damping). We then apply this coding technique to real audio excerpts for a given entropy target corresponding to a low bitrate (20 kbits/s), and compare this method with a classical sinusoidal coding scheme using a constant-amplitude sinusoidal model and the perceptually weighted Matching Pursuit algorithm. Subjective listening tests show that the EDS model is more efficient on audio samples with fast transient content, and similar to the classical model for more stationary audio samples.

Maximum Expected Rates of Block-Fading Channels with Entropy-Constrained Channel State Feedback

We obtain the maximum average data rates achievable over block-fading channels when the receiver has perfect channel state information (CSI), and only an entropy-constrained quantized approximation of this CSI is available at the transmitter. We assume channel gains in consecutive blocks are independent and identically distributed and consider a short term power constraint. Our analysis is valid for a wide variety of channel fading statistics, including Rician and Nakagami-m fading. For this situation, the problem translates into designing an optimal entropy-constrained quantizer to convey approximated CSI to the transmitter and to define a rate-adaptation policy for the latter so as to maximize average downlink data rate. A numerical procedure is presented which yields the thresholds and reconstruction points of the optimal quantizer, together with the associated maximum average downlink rates, by finding the roots of a small set of scalar functions of two scalar arguments. Utilizing this procedure, it is found that achieving the maximum downlink average capacity C requires, in some cases, time sharing between two regimes. In addition, it is found that, for an uplink entropy constraint H <; log2 (L), a quantizer with more than L cells provides only a small capacity increase, especially at high SNRs.

표본 적응 프로덕트 양자기에 기초한 격자 벡터 양자화의 엔트로피 부호화와 무기억성 가우시언 분포에 대한 성능 분석

Optimal quantizers in conducting the entropy-constrained quantization for high bit rates have the lattice structure. The quantization process is simple due to the regular structure, and various quantization algorithms are proposed depending on the lattice. Such a lattice vector quantizer (VQ) can be implemented by using the sample-adaptive product quantizer (SAPQ) and its output can also be easily entropy encoded. In this paper, the entropy encoding scheme for the lattice VQ is proposed based on SAPQ, and the performance of the proposed lattice VQ, which is based on SAPQ with the entropy coder, is asymptotically compared as the rate increases. It is shown by experiment that the gain for the memoryless Gaussian source also approaches the theoretic gain for the uniform density case.

High-Resolution Scalar Quantization With Rényi Entropy Constraint

We consider optimal scalar quantization with rth power distortion and constrained Rényi entropy of order α. For sources with absolutely continuous distributions the high rate asymptotics of the quantizer distortion has long been known for α = 0 (fixed-rate quantization) and α = 1 (entropy-constrained quantization). These results have recently been extended to quantization with Rényi entropy constraint of order α ≥ <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> +1. Here we consider the more challenging case α ∈ [-∞,0)∪(0,1) and for a large class of absolutely continuous source distributions we determine the sharp asymptotics of the optimal quantization distortion. The achievability proof is based on finding (asymptotically) optimal quantizers via the companding approach, and is thus constructive.

KLT-based adaptive entropy-constrained quantization with universal arithmetic coding

For flexible speech coding, a Karhunen-Loève Transform (KLT) based adaptive entropy-constrained quantization (KLT-AECQ) method is proposed. It is composed of backward-adaptive linear predictive coding (LPC) estimation, KLT estimation based on the time-varying LPC coefficients, scalar quantization of the speech signal in a KLT domain, and superframe-based universal arithmetic coding based on the estimated KLT statistics. To minimize the outliers both in rate and distortion, a new distortion criterion includes the penalty in the rate increase. Gain adaptive step size selection and bounded Gaussian source model also cooperate to increase the perceptual quality. KLT-AECQ does not require either any explicit codebook or a training step, thus KLT-AECQ can have an infinite number of rate-distortion operating points regardless of time-varying source statistics. For the speech signal, the conventional KLT-based classified vector quantization (KLT-CVQ) and the proposed KLT-AECQ yield signal-to-noise ratios of 17.86 and 26.22, respectively, at around 16 kbits/s. The perceptual evaluation of speech quality (PESQ) scores for each method are 3.87 and 4.04, respectively <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">.</sup>

High-Resolution Spherical Quantization of Sinusoidal Parameters

Sinusoidal coding is an often employed technique in low bit-rate audio coding. Therefore, methods for efficient quantization of sinusoidal parameters are of great importance. In this paper, we use high-resolution assumptions to derive analytical expressions for the optimal entropy-constrained unrestricted spherical quantizers for the amplitude, phase, and frequency parameters of the sinusoidal model. This is done both for the case of a single sinusoid, and for the more practically relevant case of multiple sinusoids distributed across multiple segments. To account for psychoacoustical effects of the auditory system, a perceptual distortion measure is used. The optimal quantizers minimize a high-resolution approximation of the expected perceptual distortion, while the corresponding quantization indices satisfy an entropy constraint. The quantizers turn out to be flexible and of low complexity, in the sense that they can be determined easily for varying bit rate requirements, without any sort of retraining or iterative procedures. In an objective comparison it is shown that for the squared error distortion measure, the rate-distortion performance of the proposed method is very close to that of the theoretically optimal entropy-constrained vector quantization. Furthermore, for the perceptual distortion measure, the proposed scheme is shown to objectively outperform an existing sinusoidal quantization scheme, where frequency quantization is done independently. Finally, a subjective listening test, in which the proposed scheme is compared to an existing state-of-the-art sinusoidal quantization scheme with fixed quantizers for all input signals, indicates that the proposed scheme leads to an average bit rate reduction of 20%, at the same subjective quality level as the existing scheme

Capacity and Lattice Strategies for Canceling Known Interference

We consider the generalized dirty-paper channel Y=X+S+N,E{X/sup 2/}/spl les/P/sub X/, where N is not necessarily Gaussian, and the interference S is known causally or noncausally to the transmitter. We derive worst case capacity formulas and strategies for "strong" or arbitrarily varying interference. In the causal side information (SI) case, we develop a capacity formula based on minimum noise entropy strategies. We then show that strategies associated with entropy-constrained quantizers provide lower and upper bounds on the capacity. At high signal-to-noise ratio (SNR) conditions, i.e., if N is weak relative to the power constraint P/sub X/, these bounds coincide, the optimum strategies take the form of scalar lattice quantizers, and the capacity loss due to not having S at the receiver is shown to be exactly the "shaping gain" 1/2log(2/spl pi/e/12)/spl ap/ 0.254 bit. We extend the schemes to obtain achievable rates at any SNR and to noncausal SI, by incorporating minimum mean-squared error (MMSE) scaling, and by using k-dimensional lattices. For Gaussian N, the capacity loss of this scheme is upper-bounded by 1/2log2/spl pi/eG(/spl Lambda/), where G(/spl Lambda/) is the normalized second moment of the lattice. With a proper choice of lattice, the loss goes to zero as the dimension k goes to infinity, in agreement with the results of Costa. These results provide an information-theoretic framework for the study of common communication problems such as precoding for intersymbol interference (ISI) channels and broadcast channels.

A posteriori quantization of progressive matching pursuit streams

This paper proposes a rate-distortion optimal a posteriori quantization scheme for matching pursuit (MP) coefficients. The a posteriori quantization applies to an MP expansion that has been generated offline and cannot benefit of any feedback loop to the encoder in order to compensate for the quantization noise. The redundancy of the MP dictionary provides an indicator of the relative importance of coefficients and atom indices and, subsequently, on the quantization error. It is used to define a universal upper bound on the decay of the coefficients, sorted in decreasing order of magnitude. A new quantization scheme is then derived, where this bound is used as an Oracle for the design of an optimal a posteriori quantizer. The latter turns the exponentially distributed coefficient entropy-constrained quantization problem into a simple uniform quantization problem. Using simulations with random dictionaries, we show that the proposed exponentially upper bounded quantization (EUQ) clearly outperforms classical schemes. Stepping on the ideal Oracle-based approach, a suboptimal adaptive scheme is then designed that approximates the EUQ but still outperforms competing quantization methods in terms of rate-distortion characteristics. Finally, the proposed quantization method is studied in the context of image coding. It performs similarly to state-of-the-art coding methods (and even better at low rates) while interestingly providing a progressive stream that is very easy to transcode and adapt to changing rate constraints.

Do optimal entropy-constrained quantizers have a finite or infinite number of codewords?

An entropy-constrained quantizer Q is optimal if it minimizes the expected distortion D(Q) subject to a constraint on the output entropy H(Q). We use the Lagrangian formulation to show the existence and study the structure of optimal entropy-constrained quantizers that achieve a point on the lower convex hull of the operational distortion-rate function D/sub h/(R) = inf/sub Q/{D(Q) : H(Q) /spl les/ R}. In general, an optimal entropy-constrained quantizer may have a countably infinite number of codewords. Our main results show that if the tail of the source distribution is sufficiently light (resp., heavy) with respect to the distortion measure, the Lagrangian-optimal entropy-constrained quantizer has a finite (resp., infinite) number of codewords. In particular, for the squared error distortion measure, if the tail of the source distribution is lighter than the tail of a Gaussian distribution, then the Lagrangian-optimal quantizer has only a finite number of codewords, while if the tail is heavier than that of the Gaussian, the Lagrangian-optimal quantizer has an infinite number of codewords.

A Lagrangian formulation of Zador's entropy-constrained quantization theorem

Zador's (1963, 1966) classic result for the asymptotic high-rate behavior of entropy-constrained vector quantization is recast in a Lagrangian form which better matches the Lloyd algorithm used to optimize such quantizers. The equivalence of the two formulations is shown and the result is proved for source distributions that are absolutely continuous with respect to the Lebesgue measure which satisfy an entropy condition, thereby generalizing the conditions stated by Zador under which the result holds.

On the structure of optimal entropy-constrained scalar quantizers

The nearest neighbor condition implies that when searching for a mean-square optimal fixed-rate quantizer it is enough to consider the class of regular quantizers, i.e., quantizers having convex cells and codepoints which lie inside the associated cells. In contrast, quantizer regularity can preclude optimality in entropy-constrained quantization. This can be seen by exhibiting a simple discrete scalar source for which the mean-square optimal entropy-constrained scalar quantizer (ECSQ) has disconnected (and hence nonconvex) cells at certain rates. In this work, new results concerning the structure and existence of optimal ECSQs are presented. One main result shows that for continuous sources and distortion measures of the form d(x,y)=/spl rho/(|x-y|), where /spl rho/ is a nondecreasing convex function, any finite-level ECSQ can be "regularized" so that the resulting regular quantizer has the same entropy and equal or less distortion. Regarding the existence of optimal ECSQs, we prove that under rather general conditions there exists an "almost regular" optimal ECSQ for any entropy constraint. For the squared error distortion measure and sources with piecewise-monotone and continuous densities, the existence of a regular optimal ECSQ is shown.

On optimal entropy-constrained deadzone quantization

This paper studies entropy-constrained deadzone quantizer design. It first shows that for symmetric uni-modal distributions, the entropy of the quantized output is a monotonic decreasing function of the outer bin-width of the deadzone quantizer. Based on this result, a fast algorithm is introduced for optimal entropy-constrained deadzone quantizer design. As experimental results, we give the optimal zero bin-width and outer bin-width for several generalized Gaussian distributions that are often used to model the AC coefficient distributions in image and video transform coding.

Adaptive wavelet packet based image coding with optimal entropy-constrained lattice vector quantizer (ECLVQ)

In this work, wavelet basis and source coding are jointly optimized, while specifying the source coding strategy as entropy-constrained lattice vector quantizer (ECLVQ). The presented approach differs from previous works in which the choice of wavelet basis is quasioptimal, but the quantizer set is optimally chosen.

Region adaptive subband image coding

We present a region adaptive subband image coding scheme using the statistical properties of image subbands for various subband decompositions. Motivated by analytical results obtained when the input signal to the subband decomposition is a unit step function, we analyze the energy packing properties toward the lower frequency subbands, edges, and the dependency of energy distribution on the orientation of the edges, in subband decomposed images. Based on these investigations and ideal analysis/synthesis filtering done in the frequency domain, the region adaptive subband image coding scheme extracts suitably shaped regions in each subband and then uses adaptive entropy-constrained quantizers for different regions under the assumption of a generalized Gaussian distribution for the image subbands. We also address the problem of determining an optimal subband decomposition among all possible decompositions. Experimental results show that visual degradations in the reconstructed image are negligible at a bit rate of 1.0 b/pel and reasonable quality images are obtainable at rates as low as 0.25 b/pel.

An investigation of wavelet-based image coding using an entropy-constrained quantization framework

Wavelet image decompositions generate a tree-structured set of coefficients, providing an hierarchical data-structure for representing images. A new class of previously proposed image compression algorithms has focused on new ways for exploiting dependencies between this hierarchy of wavelet coefficients using "zero-tree" data structures. This paper presents a new framework for understanding the efficiency of one specific algorithm in this class we introduced previously and dubbed the space-frequency quantization (SFQ)-based coder. It describes, at a higher level, how the SFQ-based image coder of our earlier work can be construed as a simplified attempt to design a global entropy-constrained vector quantizer (ECVQ) with two noteworthy features: (i) it uses an image-sized codebook dimension (departing from conventional small-dimensional codebooks that are applied to small image blocks); and (ii) it uses an on-line image-adaptive application of constrained ECVQ (which typically uses off-line training data in its codebook design phase). The principal insight offered by the new framework is that improved performance is achieved by more accurately characterizing the joint probabilities of arbitrary sets of wavelet coefficients. We also present an empirical statistical study of the distribution of the wavelet coefficients of high-frequency bands, which are responsible for most of the performance gain of the new class of algorithms. This study verifies that the improved performance achieved by the new class of algorithms like the SFQ-based coder can be attributed to its being designed around one conveniently structured and efficient collection of such sets, namely, the zero-tree data structure. The results of this study further inspire the design of alternative, novel data structures based on nonlinear morphological operators.

Wavelet video coding with ladder structures and entropy-constrained quantization

A subband decomposition using wavelet filters has become an effective basis for image compression. The use of a ladder structure for the subband filtering provides a fast implementation and allows the system to be useful at video rates. A system is proposed that provides good quality video for moderate motion, roughly common intermediate format-f 0-t (CIF-) sized images at bit rates under 500 kb/s, and for teleconferencing scenes at bit rates from 180 kb/s to below 128 kb/s (basic rate ISDN). The system uses subband coding, a modified version of H.261, and entropy-constrained quantization to achieve the desired bit rate and image quality.

Image coding using wavelet transforms and entropy-constrained trellis-coded quantization

The discrete wavelet transform has recently emerged as a powerful technique for decomposing images into various multi-resolution approximations. Multi-resolution decomposition schemes have proven to be very effective for high-quality, low bit-rate image coding. In this work, we investigate the use of entropy-constrained trellis-coded quantization (ECTCQ) for encoding the wavelet coefficients of both monochrome and color images. ECTCQ is known as an effective scheme for quantizing memoryless sources with low to moderate complexity, The ECTCQ approach to data compression has led to some of the most effective source codes found to date for memoryless sources. Performance comparisons are made using the classical quadrature mirror filter bank of Johnston and nine-tap spline filters that were built from biorthogonal wavelet bases. We conclude that the encoded images obtained from the system employing nine-tap spline filters are marginally superior although at the expense of additional computational burden. Excellent peak-signal-to-noise ratios are obtained for encoding monochrome and color versions of the 512x512 "Lenna" image. Comparisons with other results from the literature reveal that the proposed wavelet coder is quite competitive.

New results on optimal entropy-constrained quantization

Given h>0, the problem is considered of finding an N-level quantizer Q which is optimal in the sense of encoding a given continuously distributed random variable X with minimum expected squared error, subject to the constraint H(Q(X)) >

Adaptive buffer- instrumented entropy-coded quantizer performance for memoryless sources

Variable-length codes can be used in entropy coding the outputs of an optimum entropy-constrained quantizer. Transmitting these codes over a synchronous channel; however, requires a buffer connecting the entropy coder to the channel. In a practical application, this buffer is of finite size and hence might overflow or undertow. To alleviate this difficulty, we use an adaptive scheme in which the quantizer parameters are changed successively according to the state of the buffer. Rate-distortion performance of optimum entropy-constrained quantizers in conjunction with this adaptive scheme is studied for the class of generalized Gaussian sources. It is demonstrated through simulations that the overflow/ undertow problem can be practically eliminated at the cost of a negligible increase in average distortion. Furthermore, it is shown that the efficiency of this system is more pronounced at high rates and for more broadtailed source densities. Easily computable upper and lower bounds on the average distortion of the adaptive system are developed.