Variable-length Quantizer Research Articles

RATQ: A Universal Fixed-Length Quantizer for Stochastic Optimization

We present Rotated Adaptive Tetra-iterated Quantizer (RATQ), a fixed-length quantizer for gradients in first order stochastic optimization. RATQ is easy to implement and involves only a Hadamard transform computation and adaptive uniform quantization with appropriately chosen dynamic ranges. For noisy gradients with almost surely bounded Euclidean norms, we establish an information theoretic lower bound for optimization accuracy using finite precision gradients and show that RATQ almost attains this lower bound. For mean square bounded noisy gradients, we use a gain-shape quantizer which separately quantizes the Euclidean norm and uses RATQ to quantize the normalized unit norm vector. We establish lower bounds for performance of any optimization procedure and shape quantizer, when used with a uniform gain quantizer. Finally, we propose an adaptive quantizer for gain which when used with RATQ for shape quantizer outperforms uniform gain quantization and is, in fact, close to optimal. As a by-product, we show that our fixed-length quantizer RATQ has almost the same performance as the optimal variable-length quantizers for distributed mean estimation. Also, we obtain an efficient quantizer for Gaussian vectors which attains a rate very close to the Gaussian rate-distortion function and is, in fact, universal for sub Gaussian input vectors.

Rate-Cost Tradeoffs in Control

Consider a control problem with a communication channel connecting the observer of a linear stochastic system to the controller. The goal of the controller is to minimize a quadratic cost function in the state variables and control signal, known as the linear quadratic regulator (LQR). We study the fundamental tradeoff between the communication rate r b/s and the expected cost b. We obtain a lower bound on a certain rate-cost function, which quantifies the minimum directed mutual information between the channel input and output that is compatible with a target LQR cost. The rate-cost function has operational significance in multiple scenarios of interest: among others, it allows us to lower-bound the minimum communication rate for fixed and variable length quantization, and for control over noisy channels. We derive an explicit lower bound to the rate-cost function, which applies to the vector, non-Gaussian, and partially observed systems, thereby extending and generalizing an earlier explicit expression for the scalar Gaussian system, due to Tatikonda et al. [S. Tatikonda, A. Sahai, and S. Mitter, “Stochastic linear control over a communication channel,” IEEE Trans. Autom. Control, vol. 49, no. 9, pp. 1549-1561, Sep. 2004.]. The bound applies as long as the differential entropy of the system noise is not -∞. It can be closely approached by a simple lattice quantization scheme that only quantizes the innovation, that is, the difference between the controller's belief about the current state and the true state. Via a separation principle between control and communication, similar results hold for causal lossy compression of additive noise Markov sources. Apart from standard dynamic programming arguments, our technical approach leverages the Shannon lower bound, develops new estimates for data compression with coding memory, and uses some recent results on high resolution variablelength vector quantization to prove that the new converse bounds are tight.

Online Variable Coding Length Product Quantization for Fast Nearest Neighbor Search in Mobile Retrieval

Quantization methods are crucial for efficient nearest neighbor search in many applications such as image, music, or product search. As mobile devices are becoming increasingly more popular, the quantization methods on mobile devices are more important, because a large portion of the search queries are becoming performed on mobile devices. One important characteristic of the communication on mobile devices is the inherent unreliability of their communication channels. In order to adapt the quality changes of the communication channels, we need to change the coding length of the quantization accordingly. The existing quantization methods use fixed-length codebooks, and it is expensive to retrain another codebook with different coding length. In this paper, we propose a novel variable length product quantization framework that consists of a set of fast universal scalar quantizers. The framework is capable of producing variable length quantization without retraining the codebook. Each data vector is transformed into a new space to reduce the correlation across dimensions. A proper number of bits is allocated to represent the scalar component in each dimension according to the given coding length. For each component, we estimate its probability density function (PDF) and design an efficient universal scalar quantizer based on the PDF and the allocated bits. To reduce distortion, we learn a Gaussian mixture model for the data. The experimental results show that, compared to state-of-the-art product quantization methods, our approach can construct the codebooks online for variable coding lengths and achieve the comparable performance.

Amplify-and-Forward Relay Networks With Variable-Length Limited Feedback

We study the channel quantization problem for amplify-and-forward (AF) relay networks and our target is to design a quantizer to minimize the outage probability. It is priorly known that any fixed-length quantizer with a finite-cardinality codebook cannot attain the same minimum outage probability as the case where all nodes in the AF relay networks have access to perfect channel state information (CSI). We propose variable-length quantizers with random infinite-cardinality codebooks for the sum and individual power constraints. We provide theoretical proofs and numerical simulations to validate that the proposed quantizers can achieve the full-CSI outage probabilities with finite average feedback rates.

Variable-Length Limited Feedback Beamforming in Multiple-Antenna Fading Channels

We study a multiple-input single-output fading channel, where we would like to minimize the channel outage probability or symbol error rate (SER) by employing beamforming via quantized channel state information at the transmitter (CSIT). We consider a variable-length limited feedback scheme where the quantized CSIT is acquired through feedback binary codewords of possibly different lengths. We design and analyze the performance of the associated variable-length quantizers (VLQs) and compare their performance with the previously studied fixed-length quantizers (FLQs). For the outage probability performance measure, we construct VLQs that can achieve the full-CSIT performance with finite rate. Moreover, as the signal-to-noise ratio P tends to infinity, we show that VLQs can achieve the full-CSIT outage probability performance with asymptotically zero feedback rate. For the SER performance measure, we show that while the SER with full-CSIT is not achievable at any finite feedback rate, the diversity and array gains with full-CSIT can be achieved using VLQs with asymptotically zero feedback rate as P → ∞. Our results show that VLQs can significantly improve upon the traditional FLQs that require infinite feedback rate to achieve the outage probability or the diversity and array gains with full-CSIT.

An efficient approach to lattice-based fixed-rate entropy-coded vector quantization

In the absence of channel noise, variable-length quantizers perform better than fixed-rate Lloyd–Max quantizers for any source with a non-uniform density function. However, channel errors can lead to a loss of synchronization resulting in a propagation of error. To avoid having variable rate, one can use a vector quantizer selected as a sub-set of high probability points in the Cartesian product of a set of scalar quantizers and represent its elements with binary code-words of the same length (quantizer shaping). We choose these elements from a lattice, resulting in a higher quantization gain in comparison to simply using the Cartesian product of a set of scalar quantizers. We introduce a class of lattices which have a low encoding complexity, and at the same time result in a noticeable quantization gain. We combine the procedure of lattice encoding with that of quantizer shaping using hierarchical dynamic programming. In addition, by devising appropriate partitioning and merging rules, we obtain sub-optimum schemes of low complexity and small performance degradation. The proposed methods show a substantial improvement in performance and/or a reduction in the complexity with respect to the best known results. Copyright © 2003 AEI.