Entropy-constrained Vector Quantization Research Articles

Nonlinear Transform Coding

We review a class of methods that can be collected under the name nonlinear transform coding (NTC), which over the past few years have become competitive with the best linear transform codecs for images, and have superseded them in terms of rate-distortion performance under established perceptual quality metrics such as MS-SSIM. We assess the empirical rate-distortion performance of NTC with the help of simple example sources, for which the optimal performance of a vector quantizer is easier to estimate than with natural data sources. To this end, we introduce a novel variant of entropy-constrained vector quantization. We provide an analysis of various forms of stochastic optimization techniques for NTC models; review architectures of transforms based on artificial neural networks, as well as learned entropy models; and provide a direct comparison of a number of methods to parameterize the rate-distortion trade-off of nonlinear transforms, introducing a simplified one.

A Rigorous Approach to High-Resolution Entropy-Constrained Vector Quantization

The nonnegativity of relative entropy implies that the differential entropy of a random vector X with probability density function (pdf) $f$ is upper bounded by $- {\mathsf {E}}_{}\!\left [{\log g(\text {X})}\right]$ for any arbitrary pdf $g$ . Using this inequality with a cleverly chosen $g$ , we derive a lower bound on the asymptotic excess rate of entropy-constrained vector quantization for $d$ -dimensional sources and $r$ th-power distortion, where the asymptotic excess rate is defined as the difference between the smallest output entropy of a vector quantizer satisfying the distortion constraint and the rate-distortion function in the limit as the distortion tends to zero. Specialized to the one-dimensional case, this lower bound coincides with the asymptotic excess rate achieved by a uniform quantizer, thereby recovering the result by Gish and Pierce that uniform quantizers are asymptotically optimal as the allowed distortion tends to zero. Furthermore, in the one-dimensional case, the derivation of the lower bound reveals a necessary condition for a sequence of quantizers to be asymptotically optimal. This condition implies that any sequence of asymptotically optimal almost-regular quantizers must converge to a uniform quantizer as the distortion tends to zero. While the obtained lower bound itself is not novel, to the best of our knowledge, we present the first rigorous derivation that follows the direct approach by Gish and Pierce without resorting to heuristic high-resolution approximations commonly found in the quantization literature. Furthermore, our derivation holds for all $d$ -dimensional sources having finite differential entropy and whose integer part has finite entropy. In contrast to Gish and Pierce, we do not require additional constraints on the continuity or decay of the source pdf.

A memory efficient finite-state source coding algorithm for audio MDCT coefficients

To achieve a better trade-off between the vector dimension and the memory requirements of a vector quantizer (VQ), an entropy-constrained VQ (ECVQ) scheme with finite memory, called finite-state ECVQ (FS-ECVQ), is presented in this paper. The scheme consists of a finite-state VQ (FSVQ) and multiple component ECVQs. By utilizing the FSVQ, the inter-frame dependencies within source sequence can be effectively exploited and no side information needs to be transmitted. By employing the ECVQs, the total memory requirements of the FS-ECVQ can be efficiently decreased while the coding performance is improved. An FS-ECVQ, designed for the modified discrete cosine transform (MDCT) coefficients coding, was implemented and evaluated based on the Unified Speech and Audio Coding (USAC) scheme. Results showed that the FS-ECVQ achieved a reduction of the total memory requirements by about 11.3%, compared with the encoder in USAC final version (FINAL), while maintaining a similar coding performance.

Computation of the complexity of vector quantizers by affine modeling

Several methods for evaluation of the complexity of data compression systems and for including complexity measures in the traditional rate-distortion analysis have been published in recent works. In this work, we indicate that the relationship between rate-distortion performance and complexity for some practical coding schemes—entropy-constrained vector quantization (ECVQ) and interpolative vector quantization (IVQ)—can be represented by affine models. For the same rate-distortion performance, the complexity of an interpolative vector quantizer is known to be significantly smaller than the complexity of a full-search entropy constrained vector quantizer, and this complexity difference is a suitable illustration for the rate-distortion–complexity framework. We use high-resolution theory arguments to derive the affine models for ECVQ and IVQ. The proposed affine complexity modeling successfully predicts the cost of vector quantizers designed from data sets that were not used to generate the models.

KLT-based adaptive entropy-constrained vector quantization for the speech signals

For efficient variable-rate speech coding, Karhunen-Loeve transform based adaptive entropy-constrained vector quantization (KLT-AECVQ) is proposed. The proposed method consists of backward-adaptive linear predictive coding (LPC) analysis, KLT estimation based on LPC coefficients, and lattice vector quantization followed by Huffman coding according to KLT statistics. As different statistics in an original-signal domain can be mapped into identical statistics in a KLT domain, only a few classified Huffman codebooks are sufficient to represent KLT-domain source statistics. KLT-AECVQ with 32 Huffman codebooks has comparable rate-distortion performance with theoretically optimal AECVQ with infinite number of Huffman codebooks. KLT-AECVQ also produces superior perceptual quality to KLT-based classified vector quantization (KLTCVQ) that yielded better quality than conventional code excited linear predictive (CELP) codec. Under five-sample delay constraints, KLT-AECVQ has also three times lower complexity than CELP codec.

An Improved Encoder for Joint Source-Channel Decoder Using Conditional Entropy Constraint

When the joint source-channel (JSC) decoder is used for source coding over noisy channels, the JSC decoder may invent errors even though the received data is not corrupted by the channel noise, if the JSC decoder assumes the channel was noisy. A novel encoder algorithm has been recently proposed to improve the performance of the communications system under this situation. In this letter, we propose another algorithm based on conditional entropy-constrained vector quantizer to further improve the encoder. The algorithm proposed in this letter significantly improves the performance of the communications system when the hypothesized channel bit error rate is high.

Deterministic Annealing for Entropy-Constrained Vector Quantizer Design

Typical vector quantizer (VQ) design algorithms, such as generalized Lloyd methods, are susceptible to local minima. In addition, for entropy-constrained VQ, it is not obvious what the right number of codevectors is for any given source and target rate. One method that has shown promising results for solving both problems is deterministic annealing, where randomness is added to the encoding process, the uncertainty of which is gradually reduced until a deterministic coder is found. This method has been used for fixed-rate and entropy-constrained unstructured VQ, as well as for a variety of structurally constrained VQs. For the unstructured entropy-constrained case, we generalize the existing methods to include new cost functions, and we argue that, depending on the chosen cost function, some splitting criteria are more suitable than others. We give analytical advantages and disadvantages to several cost functions and several splitting criteria, then give performance evaluations by testing on synthetic and natural sources.

Predictive vector quantization of 3-D mesh geometry by representation of vertices in local coordinate systems

In predictive 3-D mesh geometry coding, the position of each vertex is predicted from the previously coded neighboring vertices and the resultant prediction error vectors are coded. In this work, the prediction error vectors are represented in a local coordinate system in order to cluster them around a subset of a 2-D planar subspace and thereby increase block coding efficiency. Alphabet entropy constrained vector quantization (AECVQ) of Rao and Pearlman is preferred to the previously employed minimum distortion vector quantization (MDVQ) for block coding the prediction error vectors with high coding efficiency and low implementation complexity. Estimation and compensation of the bias in the parallelogram prediction rule and partial adaptation of the AECVQ codebook to the encoded vector source by normalization using source statistics, are the other salient features of the proposed coding system. Experimental results verify the advantage of the use of the local coordinate system over the global one. The visual error of the proposed coding system is lower than the predictive coding method of Touma and Gotsman especially at low rates, and lower than the spectral coding method of Karni and Gotsman at medium-to-high rates.

High-Resolution Spherical Quantization of Sinusoidal Parameters

Sinusoidal coding is an often employed technique in low bit-rate audio coding. Therefore, methods for efficient quantization of sinusoidal parameters are of great importance. In this paper, we use high-resolution assumptions to derive analytical expressions for the optimal entropy-constrained unrestricted spherical quantizers for the amplitude, phase, and frequency parameters of the sinusoidal model. This is done both for the case of a single sinusoid, and for the more practically relevant case of multiple sinusoids distributed across multiple segments. To account for psychoacoustical effects of the auditory system, a perceptual distortion measure is used. The optimal quantizers minimize a high-resolution approximation of the expected perceptual distortion, while the corresponding quantization indices satisfy an entropy constraint. The quantizers turn out to be flexible and of low complexity, in the sense that they can be determined easily for varying bit rate requirements, without any sort of retraining or iterative procedures. In an objective comparison it is shown that for the squared error distortion measure, the rate-distortion performance of the proposed method is very close to that of the theoretically optimal entropy-constrained vector quantization. Furthermore, for the perceptual distortion measure, the proposed scheme is shown to objectively outperform an existing sinusoidal quantization scheme, where frequency quantization is done independently. Finally, a subjective listening test, in which the proposed scheme is compared to an existing state-of-the-art sinusoidal quantization scheme with fixed quantizers for all input signals, indicates that the proposed scheme leads to an average bit rate reduction of 20%, at the same subjective quality level as the existing scheme

On entropy-constrained vector quantization using gaussian mixture models

A flexible and low-complexity entropy-constrained vector quantizer (ECVQ) scheme based on Gaussian mixture models (GMMs), lattice quantization, and arithmetic coding is presented. The source is assumed to have a probability density function of a GMM. An input vector is first classified to one of the mixture components, and the Karhunen-Loeve transform of the selected mixture component is applied to the vector, followed by quantization using a lattice structured codebook. Finally, the scalar elements of the quantized vector are entropy coded sequentially using a specially designed arithmetic coder. The computational complexity of the proposed scheme is low, and independent of the coding rate in both the encoder and the decoder. Therefore, the proposed scheme serves as a lower complexity alternative to the GMM based ECVQ proposed by Gardner, Subramaniam and Rao. The performance of the proposed scheme is analyzed under a high-rate assumption, and quantified for a given GMM. The practical performance of the scheme was evaluated through simulations on both synthetic and speech line spectral frequency (LSF) vectors. For LSF quantization, the proposed scheme has a comparable performance to at rates relevant for speech coding (20-28 bits per vector) with lower computational complexity.

Lossless Image Coding Using Conditional Entropy Constrained Vector Quantization

Lossless Image Coding Using Conditional Entropy Constrained Vector Quantization

A Low-Complexity Method for Fixed-Rate Entropy-Constrained Vector Quantization

This paper describes a new approach to fixed-rate entropy-constrained vector quantization (FEVQ) for stationary memoryless sources where the structure of codewords are derived from a variable-length scalar quantizer. We formulate the quantization search operation as a zero–one integer optimization problem, and show that the resulting integer program can be closely approximated by solving a simple linear program. The result is a Lagrange formulation which adjoins the constraint on the entropy (codeword length) to the distortion. Unlike the previously known methods with a fixed Lagrange multiplier, we use an iterative algorithm to optimize the underlying objective function while updating the Lagrange multiplier until the constraint on the overall rate is satisfied. The key feature of the new method is the substantial reduction in the number of iterations, in comparison with previous related methods. In order to achieve some packing gain, we combine the process of trellis-coded quantization with that of FEVQ. This results in an iterative application of the Viterbi algorithm on the underlying trellis for selecting the Lagrange multiplier. Numerical results are presented which demonstrate substantial improvement in comparison with the alternative methods reported in the literature.

A Low-Complexity Method for Fixed-Rate Entropy-Constrained Vector Quantization

This paper describes a new approach to fixed-rate entropy-constrained vector quantization (FEVQ) for stationary memoryless sources where the structure of codewords are derived from a variable-length scalar quantizer. We formulate the quantization search operation as a zero–one integer optimization problem, and show that the resulting integer program can be closely approximated by solving a simple linear program. The result is a Lagrange formulation which adjoins the constraint on the entropy (codeword length) to the distortion. Unlike the previously known methods with a fixed Lagrange multiplier, we use an iterative algorithm to optimize the underlying objective function while updating the Lagrange multiplier until the constraint on the overall rate is satisfied. The key feature of the new method is the substantial reduction in the number of iterations, in comparison with previous related methods. In order to achieve some packing gain, we combine the process of trellis-coded quantization with that of FEVQ. This results in an iterative application of the Viterbi algorithm on the underlying trellis for selecting the Lagrange multiplier. Numerical results are presented which demonstrate substantial improvement in comparison with the alternative methods reported in the literature.

Codecell convexity in optimal entropy-constrained vector quantization

Properties of optimal entropy-constrained vector quantizers (ECVQs) are studied for the squared-error distortion measure. It is known that restricting an ECVQ to have convex codecells may preclude its optimality for some sources with discrete distribution. We show that for sources with continuous distribution, any finite-level ECVQ can be replaced by another finite-level ECVQ with convex codecells that has equal or better performance. We generalize this result to infinite-level quantizers, and also consider the problem of existence of optimal ECVQs for continuous source distributions. In particular, we show that given any entropy constraint, there exists an ECVQ with (possibly infinitely many) convex codecells that has minimum distortion among all ECVQs satisfying the constraint. These results extend analogous statements in entropy-constrained scalar quantization. They also generalize results in entropy-constrained vector quantization that were obtained via the Lagrangian formulation and, therefore, are valid only for certain values of the entropy constraint.

Mismatch in high-rate entropy-constrained vector quantization

Bucklew's (1984) high-rate vector quantizer mismatch result is extended from fixed-rate coding to variable-rate coding using a Lagrangian formulation. It is shown that if an asymptotically (high-rate) optimal sequence of variable rate codes is designed for a k-dimensional probability density function (PDF) g and then applied to another PDF f for which f/g is bounded, then the resulting mismatch or loss of performance from the optimal possible is given by the relative entropy or Kullback-Leibler (1968) divergence I(f/spl par/g). It is also shown that under the same assumptions, an asymptotically optimal code sequence for g can be converted to an asymptotically optimal code sequence for a mismatched source f by modifying only the lossless component of the code. Applications to quantizer design using uniform and Gaussian densities are described, including a high-rate analog to the Shannon rate-distortion result of Sakrison (1975) and Lapidoth (1997) showing that the Gaussian is the "worst case" for lossy compression of a source with known covariance. By coupling the mismatch result with composite quantizers, the worst case properties of uniform and Gaussian densities are extended to conditionally uniform and Gaussian densities, which provides a Lloyd clustering algorithm for fitting mixtures to general densities.

Complexity-constrained tree-structured vector quantizers

We present a new algorithm for complexity-distortion optimization in tree-structured vector quantizers (TSVQ). The algorithm allows the user to specify the average rate and computational complexity budgets R and C, measured in bits and multiplications per sample, respectively. The output is an optimal—in a sense to be specified—TSVQ satisfying the constraints. The complexity budget is lower-bounded by the complexity of a binary TSVQ and upper-bounded by the complexity of a full-search entropy-constrained vector quantizer. Experimental results for synthetic and natural sources are given.

Compressing Massive Geophysical Datasets Using Vector Quantization

This article presents a procedure for compressing massive geophysical datasets. A dataset is stratified geographically, and a penalized clustering algorithm applied to each stratum independently. The algorithm, called Monte Carlo extended ECVQ, is based on the entropy-constrained vector quantizer algorithm (ECVQ). ECVQ trades off error induced by compression against data reduction to produce a set of representative points, each of which stands for some number of input observations. Since the data are massive, a preliminary set of representatives is determined from a stratum sample, then the full stratum is clustered by assigning each observation to the nearest representative. After replacing the initial representatives by means of these clusters, the new representatives and their associated counts are a compressed version, or summary, of the original stratum data. With the initial set of representatives determined from a sample, the final summary is subject to sampling variation. A statistical model for the relationship between compressed and uncompressed data provides a framework for assessing this variability. Test data from the International Satellite Cloud Climatology Project are used to demonstrate the procedure.

A Lagrangian formulation of Zador's entropy-constrained quantization theorem

Zador's (1963, 1966) classic result for the asymptotic high-rate behavior of entropy-constrained vector quantization is recast in a Lagrangian form which better matches the Lloyd algorithm used to optimize such quantizers. The equivalence of the two formulations is shown and the result is proved for source distributions that are absolutely continuous with respect to the Lebesgue measure which satisfy an entropy condition, thereby generalizing the conditions stated by Zador under which the result holds.

Entropy-constrained learning vector quantization algorithms and their application in image compression

This paper presents entropy-constrained learning vector quantization (ECLVQ) algorithms and their application in image compression. The development of these algorithms relies on refor- mulation, which is a powerful new methodology that essentially es- tablishes a link between learning vector quantization and clustering algorithms developed using alternating optimization. ECLVQ algo- rithms are developed in this paper by reformulating entropy- constrained fuzzy clustering (ECFC) algorithms, which were devel- oped by minimizing an objective function incorporating the partition entropy and the average distortion between the feature vectors and their prototypes. The proposed algorithms allow the gradual transi- tion from a maximally fuzzy partition to a nearly crisp partition of the feature vectors during the learning process. This paper presents two alternative implementations of the proposed algorithms, which differ in terms of the strategy employed for updating the prototypes during learning. The proposed algorithms are tested and evaluated on the design of codebooks used for image data compression. © 2000 SPIE and IS&T. (S1017-9909(00)00304-4)

Conditional entropy-constrained trellis-coded RVQ with application to image coding

This paper introduces an extension of conditional entropy-constrained RVQ (CEC-RVQ) that embodies trellis-coded quantization. The method, which we call conditional entropy-constrained trellis-coded residual vector quantization (CEC-TCRVQ), quantizes a supervector (made from a large number of neighboring vectors) to better extract the two-dimensional (2-D) correlation present in real images. Simulation results indicate that CEC-TCRVQ provides 0.3-0.4 dB improvement over CEC-RVQ for the 4/spl times/4 vector case and 1.3 dB improvement for the 8/spl times/8 case.