Lossy Source Compression Research Articles

Compression by and for Deep Boltzmann Machines

We answer two questions in this work: what Deep Boltzmann Machines (DBMs) can do for compression and vise versa. We show that (1) DBMs can be applied to learn the rate distortion approaching posterior as in the Blahut-Arimoto (BA) algorithm, and to construct a lossy source compression scheme based on the Deep AutoEncoder; (2) compression can improve DBMs’ training performances via compression-based denoising algorithms. The implementation of the BA algorithm in the form of DBMs is the foundation of the two applications.

Encoding of Spatially Coupled LDGM Codes for Lossy Source Compression

It has been shown that a class of spatially coupled low-density generator-matrix (SC-LDGM) code ensembles displays distortion saturation for the lossy binary symmetric source coding problem with the belief propagation guided decimation (BPGD) algorithm, i.e., the BPGD distortion approaches the optimal expected distortion of the underlying ensemble asymptotically in code length. We investigate the distortion performance of a practical class of protograph-based SC-LDGM code ensembles and demonstrate distortion saturation numerically. Moreover, taking advantage of the convolutional structure of the SC-LDGM codes, we propose an efficient windowed encoding (WE) algorithm with two decimation techniques for lowering the WE complexity that maintain distortion performance close to the rate-distortion bound.

The Likelihood Encoder for Lossy Compression

A likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on the soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma yields simple achievability proofs for classical source coding problems. The cases of the point-to-point rate-distortion function, the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem), and the multi-terminal source coding inner bound (i.e. the Berger-Tung problem) are examined in this paper. Furthermore, a non-asymptotic analysis is used for the point-to-point case to examine the upper bound on the excess distortion provided by this method. The likelihood encoder is also related to a recent alternative technique using properties of random binning.

Incremental Grassmannian Feedback Schemes for Multi-User MIMO Systems

The communication of side information forms a key component of several effective strategies for transmitter adaptation to slowly fading channels. When the relevant side information is a subspace, the feedback scheme can be viewed as a lossy source compression scheme on the Grassmannian manifold. Memoryless vector quantization on each fading block is a viable compression scheme, but it neglects any temporal correlation between the blocks. In this paper, we propose an incremental approach to Grassmannian quantization that takes advantage of temporal correlation. The approach leverages existing codebooks for memoryless quantization schemes and employs a quantized form of geodesic interpolation. Two schemes that implement the principles of the proposed approach are presented. In the first scheme, the choice of the step size in the incremental update is adapted to a first-order GaussMarkov model for the channel, which enables the use of higher resolution codebooks. In the second scheme, a single bit is allocated to the step size, which enables adaptation of the step size to the channel realization rather than the channel statistics. This provides substantial robustness against mismatches in the model for the temporal correlation. A distinguishing feature of the proposed approach is that the direction of the geodesic interpolation is specified implicitly using a point in a conventional codebook. As a result, the approach has an inherent ability to recover autonomously from errors in the feedback path. Simulation results demonstrate that these features result in improved performance over some existing schemes in a variety of channel environments.

Lossy Source Compression Using Belief Propagation with Decimation over Multi-Edge Type Factor Graphs of LDGM Codes

A low complexity approach for binary quantization is proposed. This approach uses a multi-edge type bipartite graph of the low density generator matrix (LDGM) code with the belief propagation algorithm to encode the informations. A damping decimation is developed for the algorithm convergence. Simulation results show that our algorithm achieves near Shannon capacity performance with reduced complexity.

Lossy Source Compression of Non-Uniform Binary Source via Reinforced Belief Propagation over GQ-LDGM Codes

In this letter, we consider the lossy coding of a non-uniform binary source based on GF(q)-quantized low-density generator matrix (LDGM) codes with check degree dc=2. By quantizing the GF(q) LDGM codeword, a non-uniform binary codeword can be obtained, which is suitable for direct quantization of the non-uniform binary source. Encoding is performed by reinforced belief propagation, a variant of belief propagation. Simulation results show that the performance of our method is quite close to the theoretic rate-distortion bounds. For example, when the GF(16)-LDGM code with a rate of 0.4 and block-length of 1,500 is used to compress the non-uniform binary source with probability of 1 being 0.23, the distortion is 0.091, which is very close to the optimal theoretical value of 0.074.

Polar Codes are Optimal for Lossy Source Coding

We consider lossy source compression of a binary symmetric source using polar codes and a low-complexity successive encoding algorithm. It was recently shown by Arikan that polar codes achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a successive decoding strategy. We show the equivalent result for lossy source compression, i.e., we show that this combination achieves the rate-distortion bound for a binary symmetric source. We further show the optimality of polar codes for various multiterminal problems including the binary Wyner-Ziv and the binary Gelfand-Pinsker problems. Our results extend to general versions of these problems.

Lossy Source Compression Using Low-Density Generator Matrix Codes: Analysis and Algorithms

We study the use of low-density generator matrix (LDGM) codes for lossy compression of the Bernoulli symmetric source. First, we establish rigorous upper bounds on the average distortion achieved by check-regular ensemble of LDGM codes under optimal minimum distance source encoding. These bounds establish that the average distortion using such bounded degree families rapidly approaches the Shannon limit as the degrees are increased. Second, we propose a family of message-passing algorithms, ranging from the standard belief propagation algorithm at one extreme to a variant of survey propagation algorithm at the other. When combined with a decimation subroutine and applied to LDGM codes with suitably irregular degree distributions, we show that such a message-passing/decimation algorithm yields distortion very close to the Shannon rate-distortion bound for the binary symmetric source.

A modified belief propagation algorithm for code word quantization

Modern coding advances, including dirty paper coding and information hiding, require quantizing a given binary word to a code word. A 'good' solution would approach the rate-distortion bound in lossy source compression. Here we propose a simple variant on belief propagation which is observed to converge to a solution giving respectable rate-distortion performance. Comparisons with other recently proposed source quantization methods reveal that the proposed algorithm holds particular interest in short block-length applications, as encountered in packet-based communication systems.

Statistical mechanical approach to lossy data compression: Theory and practice

The encoder and decoder for lossy data compression of binary memoryless sources are developed on the basis of a specific-type nonmonotonic perceptron. Statistical mechanical analysis indicates that the potential ability of the perceptron-based code saturates the theoretically achievable limit in most cases although exactly performing the compression is computationally difficult. To resolve this difficulty, we provide a computationally tractable approximation algorithm using belief propagation (BP), which is a current standard algorithm of probabilistic inference. Introducing several approximations and heuristics, the BP-based algorithm exhibits performance that is close to the achievable limit in a practical time scale in optimal cases.