Lossless Source Coding Research Articles

Time-Invariant Prefix Coding for LQG Control

Motivated by control with communication constraints, in this work we develop a time-invariant data compression architecture for linear–quadratic–Gaussian (LQG) control with minimum bitrate prefix-free feedback. For any fixed control performance, the approach we propose nearly achieves known directed information (DI) lower bounds on the time-average expected codeword length. We refine the analysis of a classical achievability approach, which required quantized plant measurements to be encoded via a time-varying lossless source code. We prove that the sequence of random variables describing the quantizations has a limiting distribution and that the quantizations may be encoded with a fixed source code optimized for this distribution without added time-asymptotic redundancy. Our result follows from analyzing the long-term stochastic behavior of the system, and permits us to additionally guarantee that the time-average codeword length (as opposed to expected length) is almost surely within a few bits of the minimum DI. To our knowledge, this time-invariant achievability result is the first in the literature.

Universal Covertness for Discrete Memoryless Sources

Consider a sequence X <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> of length n emitted by a Discrete Memoryless Source (DMS) with unknown distribution p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</sub> . The objective is to construct a lossless source code that maps X <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> to a sequence ∧Y <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</sup> of length m that is indistinguishable, in terms of Kullback-Leibler divergence, from a sequence emitted by another DMS with known distribution p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</sub> . The main result is the existence of a coding scheme that performs this task with an optimal ratio m/n equal to H(X)/H(Y), the ratio of the Shannon entropies of the two distributions, as n goes to infinity. The coding scheme overcomes the challenges created by the lack of knowledge about p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</sub> by a type-based universal lossless source coding scheme that produces as output an almost uniformly distributed sequence, followed by another type-based coding scheme that jointly performs source resolvability and universal lossless source coding. The result recovers and extends previous results that either assume p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</sub> or p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</sub> uniform, or p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</sub> known. The price paid for these generalizations is the use of common randomness with vanishing rate, whose length scales as the logarithm of n. By allowing common randomness larger than the logarithm of n but still negligible compared to n, a constructive low-complexity encoding and decoding counterpart to the main result is also provided for binary sources by means of polar codes.

Universal Decoding for Asynchronous Slepian-Wolf Encoding

We consider the problem of (almost) lossless source coding of two correlated memoryless sources using separate encoders and a joint decoder, that is, Slepian-Wolf (S-W) coding. In our setting, the encoding and decoding are asynchronous, i.e., there is a certain relative delay between the two sources. Neither the source parameters nor the relative delay are known to the encoders and the decoder. Since we assume that both encoders implement standard random binning, which does not require such knowledge anyway, the focus of this work is on the decoder. Our main contribution is in proposing a universal decoder, that independent of the unknown source parameters and the relative delay, and at the same time, is asymptotically as good as the optimal maximum a posteriori probability (MAP) decoder in the sense of the random coding error exponent achieved. Consequently, the achievable rate region is also the same as if the source parameters and the delay were known to the decoder.

Lossless Source Coding in the Point-to-Point, Multiple Access, and Random Access Scenarios

This work studies point-to-point, multiple access, and random access lossless source coding in the finite-blocklength regime. In each scenario, a random coding technique is developed and used to analyze third-order coding performance. Asymptotic results include a third-order characterization of the Slepian-Wolf rate region with an improved converse that relies on a connection to composite hypothesis testing. For dependent sources, the result implies that the independent encoders used by Slepian-Wolf codes can achieve the same third-order-optimal performance as a single joint encoder. The concept of random access source coding is introduced to generalize multiple access (Slepian-Wolf) source coding to the case where encoders decide independently whether or not to participate and the set of participating encoders is unknown {\em a priori} to both the encoders and the decoder. The proposed random access source coding strategy employs rateless coding with scheduled feedback. A random coding argument proves the existence of a single deterministic code of this structure that simultaneously achieves the third-order-optimal Slepian-Wolf performance for each possible active encoder set.

Optimal Source Codes for Timely Updates

A transmitter observing a sequence of independent and identically distributed random variables seeks to keep a receiver updated about its latest observations. The receiver need not be apprised about each symbol seen by the transmitter, but needs to output a symbol at each time instant t. If at time t the receiver outputs the symbol seen by the transmitter at time U (t) ≤ t, the age of information at the receiver at time t is t - U(t). We study the design of lossless source codes that enable transmission with minimum average age at the receiver. We show that the asymptotic minimum average age can be attained up to a constant gap by the Shannon codes for a tilted version of the original pmf generating the symbols, which can be computed easily by solving an optimization problem. Furthermore, we exhibit an example with alphabet X where age of a factor O(√(log |X|)) more than that achieved by our Shannon codes for the original pmf incur an asymptotic average codes. Underlying our prescription for optimal codes is a new variational formula for integer moments of random variables, which may be of independent interest. Also, we discuss possible extensions of our formulation to randomized schemes and to the erasure channel, and include a treatment of the related problem of source coding for minimum average queuing delay.

Rate-storage regions for Extractable Source Coding with side information

This papers considers the coding of a source X with J decoders, each having access to a different side information Y(j). We define a new source coding problem called Extractable Source Coding with Side Information (ESC-SI). In this problem, the server stores one single coded description of the source, from which J descriptions can be extracted without re-encoding, depending on the side information available at the decoder. We want to minimize both the storage rate of the source on the server, and the J transmission rates from the server to the decoders. We provide the achievable storage-transmission rate regions for lossless source coding of general, non i.i.d., non-ergodic sources, and the achievable storage-transmission rate-distortion regions for lossy source coding for non i.i.d. Gaussian sources. The regions obtained for such general source models provide insightful design guidelines for practical applications.

Improved Bounds on Lossless Source Coding and Guessing Moments via Rényi Measures

This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan’s bounds, they are expressed in terms of the Arimoto-Renyi conditional entropy. Although Arikan’s bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probability are also established, characterizing the exact locus of their attainable values. The bounds on optimal guessing moments serve to improve non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints. Non-asymptotic bounds on the reliability function of discrete memoryless sources are derived as well. Relying on these techniques, lower bounds on the cumulant generating function of the codeword lengths are derived, by means of the smooth Renyi entropy, for source codes that allow decoding errors.

Achievable Moderate Deviations Asymptotics for Streaming Compression of Correlated Sources

Motivated by streaming multi-view video coding and wireless sensor networks, we consider the problem of blockwise streaming compression of a pair of correlated sources, which we term streaming Slepian-Wolf coding. We study the moderate deviations regime in which the rate pairs of a sequence of codes converge, along a straight line, to various points on the boundary of the Slepian-Wolf region at a speed slower than the inverse square root of the blocklength $n$, while the error probability decays subexponentially fast in $n$. Our main result focuses on directions of approaches to corner points of the Slepian-Wolf region. It states that for each correlated source and all corner points, there exists a non-empty subset of directions of approaches such that the moderate deviations constant (the constant of proportionality for the subexponential decay of the error probability) is enhanced (over the non-streaming case) by at least a factor of $T$, the block delay of decoding source block pairs. We specialize our main result to the setting of streaming lossless source coding and generalize this result to the setting where we have different delay requirements for each of the two source blocks. The proof of our main result involves the use of various analytical tools and amalgamates several ideas from the recent information-theoretic streaming literature. We adapt the so-called truncated memory encoding idea from Draper and Khisti (2011) and Lee, Tan, and Khisti (2016) to ensure that the effect of error accumulation is nullified in the limit of large blocklengths. We also adapt the use of the so-called minimum weighted empirical suffix entropy decoder which was used by Draper, Chang, and Sahai (2014) to derive achievable error exponents for symbolwise streaming Slepian-Wolf coding.

A Variable-to-Fixed Length Lossless Source Code Attaining Better Performance than Tunstall Code in Several Criterions

A Variable-to-Fixed Length Lossless Source Code Attaining Better Performance than Tunstall Code in Several Criterions

Linear Programming-Based Converses for Finite Blocklength Lossy Joint Source-Channel Coding

A linear programming (LP)-based framework is presented for obtaining converses for finite blocklength lossy joint source-channel coding problems. The framework applies for any loss criterion, generalizes certain previously known converses, and also extends to multi-terminal settings. The finite blocklength problem is posed equivalently as a nonconvex optimization problem and using a lift-and-project-like method, a close but tractable LP relaxation of this problem is derived. Lower bounds on the original problem are obtained by the construction of feasible points for the dual of the LP relaxation. A particular application of this approach leads to new converses, which recover and improve on the converses of Kostina and Verdu for finite blocklength lossy joint source-channel coding and lossy source coding. For finite blocklength channel coding, the LP relaxation recovers the converse of Polyanskiy, Poor and Verdu and leads to a new improvement on the converse of Wolfowitz, showing thereby that our LP relaxation is asymptotically tight with increasing blocklengths for channel coding, lossless source coding, and joint source-channel coding with the excess distortion probability as the loss criterion. Using a duality-based argument, a new converse is derived for finite blocklength joint source-channel coding for a class of source-channel pairs. Employing this converse, the LP relaxation is also shown to be tight for all blocklengths for the minimization of the expected average symbolwise Hamming distortion of a $q$ -ary uniform source over a $q$ -ary symmetric memoryless channel for any $q\in {\mathbb {N}}$ . The optimization formulation and the lift-and-project method are extended to networked settings and demonstrated by obtaining an improvement on a converse of Zhou et al. for the successive refinement problem for successively refinable source-distortion measure triplets.

Universal Weak Variable-Length Source Coding on Countably Infinite Alphabets

Motivated from the fact that universal source coding on countably infinite alphabets is not feasible, this work introduces the notion of almost lossless source coding. Analog to the weak variable-length source coding problem studied by Han (IEEE TIT, 2000, 46, 1217-1226), almost lossless source coding aims at relaxing the lossless block-wise assumption to allow an average per-letter distortion that vanishes asymptotically as the block-length tends to infinity. In this setup, we show on one hand that Shannon entropy characterizes the minimum achievable rate (similarly to the case of finite alphabet sources) while on the other that almost lossless universal source coding becomes feasible for the family of finite-entropy stationary memoryless sources with infinite alphabets. Furthermore, we study a stronger notion of almost lossless universality that demands uniform convergence of the average per-letter distortion to zero, where we establish a necessary and sufficient condition for the so-called family of envelope distributions to achieve it. Remarkably, this condition is the same necessary and sufficient condition needed for the existence of a strongly minimax (lossless) universal source code for the family of envelope distributions. Finally, we show that an almost lossless coding scheme offers faster rate of convergence for the (minimax) redundancy compared to the well-known information radius developed for the lossless case at the expense of tolerating a non-zero distortion that vanishes to zero as the block-length grows. This shows that even when lossless universality is feasible, an almost lossless scheme can offer different regimes on the rates of convergence of the (worst case) redundancy versus the (worst case) distortion.

Implementation of Polar Codes over AWGN and Binary Symmetric Channel

Objectives: Forward Error Correction (FEC) is a mechanism that has been used for years to improve the reliability of digital communications over wireless channels as they are subjected to interference, fading and noise. The goal of FEC is to add some redundant data to the input stream which facilitates the receiver to recognize and correct the errors with no need of sender to transmit the data again. In this paper we have computed bit error rate using polar codes in an Additive White Gaussian Noise Channel (AWGN) and Binary Symmetric Channel (BSC) at different block lengths. Methods/Analysis: Shannon limit states that it is the maximum amount of information that any given channel can carry. Latterly, there have been inventions in Forward Error Correction (FEC) area that allows today's systems to approach the Shannon limit. One such advancement in FEC is Polar Codes which has mathematically proved to attain the Shannon capacity for Binary Discrete Memoryless Channels using the phenomenon of channel polarization which states that by recursively combining and splitting N individual channels, some channels will become completely noiseless and some become noisy. These noiseless channels are then chosen to send information bits. Findings: We have implemented polar codes over BIAWGN and BSC and have calculated their bit error rate's at different channel ranges. The decoder used for decoding polar codes is successive cancellation decoder and it's seen that the performance is not that good at small block lengths but as the block length increases the performance improves when compared with the un-coded data stream. Applications: As polar codes are the first codes that achieve the Shannon capacity, so they seem to be the best solution for discrete memoryless channels which encounter packet failures and other noisy disturbances. Also, apart from channel encoding, polar codes found its applications in lossy and lossless source coding, coding for secrecy, communication over multiple access channels.

A new method of coding geodesics on surfaces whose fundamental regions consist of the union of elementary triangles derived from the Farey series

In this paper, a new method of coding geodesics is presented. This method generalizes the method proposed by Series and it is based on the set of hyperbolic tessellations whose corresponding fundamental regions are made up of elementary triangles derived from the Farey series, and it makes use of the concepts of continued fraction expansion and cutting sequences. These tessellations have interesting properties with applications in (lossless) source coding, topological quantum codes, design of digital signal constellations, study of second-order ordinary differential equations of the Fuchsian type besides coding of geodesics related to dynamical systems. This method promises to open up new perspectives of applications of the Farey series in digital signal processing. The union of these triangles provides a systematic procedure to obtain families of tessellations and one of the main characteristics of the Farey series is the easiness in obtaining and characterizing the triangles of the new tessellations. In applications related to hyperbolic tessellations, knowing the genus of the surface is quite important due to the fact that it is related to the characteristic of the transmission channel as well as to the algebraic–geometric code to be used in the channel encoder. We consider the analyses of the surfaces, with genus $$g=1$$ , showing that the results may be extended to any value of $$n$$ ( $$n>0$$ ), as well as to any $$g\ge 2$$ . Finally, the proposed method for coding geodesics in the hyperbolic plane is presented and the new codes are tabulated.

Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding

Tight bounds for several symmetric divergence measures are derived in terms of the total variation distance. It is shown that each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csisz\'{a}r. Another application of these bounds has been recently introduced by Yardi. et al. for channel-code detection.

Asymptotically Optimal Decision Rules for Joint Detection and Source Coding

The problem of joint detection and lossless source coding is considered. We derive asymptotically optimal decision rules for deciding whether or not a sequence of observations has emerged from a desired information source, and to compress it if has. In particular, our decision rules asymptotically minimize the cost of compression in the case that the data have been classified as desirable, subject to given constraints on the two kinds of the probability of error. In another version of this performance criterion, the constraint on the false alarm probability is replaced by a constraint on the cost of compression in the false alarm event. We then analyze the asymptotic performance of these decision rules. We also derive universal decision rules for the case where the underlying sources (under either hypothesis or both) are unknown, and training sequences from each source may or may not be available. Finally, we discuss how our framework can be extended in several directions.

Constructing Linear Encoders With Good Spectra

Linear encoders with good joint spectra are suitable candidates for optimal lossless joint source-channel coding (JSCC), where the joint spectrum is a variant of the input-output complete weight distribution and is considered good if it is close to the average joint spectrum of all linear encoders (of the same coding rate). In spite of their existence, little is known on how to construct such encoders in practice. This paper is devoted to their construction. In particular, two families of linear encoders are presented and proved to have good joint spectra. The first family is derived from Gabidulin codes, a class of maximum-rank-distance codes. The second family is constructed using a serial concatenation of an encoder of a low-density parity-check code (as outer encoder) with a low-density generator matrix encoder (as inner encoder). In addition, criteria for good linear encoders are defined for three coding applications: 1) lossless source coding; 2) channel coding; and 3) lossless JSCC. In the framework of the code-spectrum approach, these three scenarios correspond to the problems of constructing linear encoders with good kernel spectra, good image spectra, and good joint spectra, respectively. Good joint spectra imply both good kernel spectra and good image spectra, and for every linear encoder having a good kernel (respectively, image) spectrum, it is proved that there exists a linear encoder not only with the same kernel (respectively, image) but also with a good joint spectrum. Thus, a good joint spectrum is the most important feature of a linear encoder.

Delay and Redundancy in Lossless Source Coding

The penalty incurred by imposing a finite delay constraint in lossless source coding of a memoryless source is investigated. It is well known that for the so-called block-to-variable and variable-to-variable codes, the redundancy decays at best polynomially with the delay, where in this case the delay is identified with the source block length or maximal source phrase length, respectively. In stark contrast, it is shown that for sequential codes (e.g., a delay-limited arithmetic code) the redundancy can be made to decay exponentially with the delay constraint. The corresponding redundancy-delay exponent is shown to be at least as good as the Rényi entropy of order 2 of the source, but (for almost all sources) not better than a quantity depending on the minimal source symbol probability and the alphabet size.

On the Dispersions of Three Network Information Theory Problems

We analyze the dispersions of distributed lossless source coding (the Slepian-Wolf problem), the multiple-access channel and the asymmetric broadcast channel. For the two-encoder Slepian-Wolf problem, we introduce a quantity known as the entropy dispersion matrix, which is analogous to the scalar dispersions that have gained interest recently. We prove a global dispersion result that can be expressed in terms of this entropy dispersion matrix and provides intuition on the approximate rate losses at a given blocklength and error probability. To gain better intuition about the rate at which the non-asymptotic rate region converges to the Slepian-Wolf boundary, we define and characterize two operational dispersions: the local dispersion and the weighted sum-rate dispersion. The former represents the rate of convergence to a point on the Slepian-Wolf boundary while the latter represents the fastest rate for which a weighted sum of the two rates converges to its asymptotic fundamental limit. Interestingly, when we approach either of the two corner points, the local dispersion is characterized not by a univariate Gaussian but a bivariate one as well as a subset of off-diagonal elements of the aforementioned entropy dispersion matrix. Finally, we demonstrate the versatility of our achievability proof technique by providing inner bounds for the multiple-access channel and the asymmetric broadcast channel in terms of dispersion matrices. All our proofs are unified a so-called vector rate redundancy theorem which is proved using the multidimensional Berry-Esseen theorem.

Information Theory and Rate Distortion Theory for Communications and Compression

This book is very specifically targeted to problems in communications and compression by providing the fundamental principles and results in information theory and rate distortion theory for these applications and presenting methods that have proved and will prove useful in analyzing and designing real systems. The chapters contain treatments of entropy, mutual information, lossless source coding, channel capacity, and rate distortion theory; however, it is the selection, ordering, and presentation of the topics within these broad categories that is unique to this concise book. While the coverage of some standard topics is shortened or eliminated, the standard, but important, topics of the chain rules for entropy and mutual information, relative entropy, the data processing inequality, and the Markov chain condition receive a full treatment. Similarly, lossless source coding techniques presented include the Lempel-Ziv-Welch coding method. The material on rate Distortion theory and exploring fundamental limits on lossy source coding covers the often-neglected Shannon lower bound and the Shannon backward channel condition, rate distortion theory for sources with memory, and the extremely practical topic of rate distortion functions for composite sources. The target audience for the book consists of graduate students at the master's degree level and practicing engineers. It is hoped that practicing engineers can work through this book and comprehend the key results needed to understand the utility of information theory and rate distortion theory and then utilize the results presented to analyze and perhaps improve the communications and compression systems with which they are familiar.

Multiterminal Source Coding With Action-Dependent Side Information

We consider multiterminal source coding with a single encoder and multiple decoders where either the encoder or the decoders can take cost-constrained actions which affect the quality of the side information present at the decoders. For the scenario where decoders take actions, we characterize the rate-cost tradeoff region for lossless source coding, and give an achievability scheme for lossy source coding for two decoders which is optimum for a variety of special cases of interest. For the case where the encoder takes actions, we characterize the rate-cost tradeoff for a class of lossless source coding scenarios with multiple decoders. Finally, we also consider extensions to other multiterminal source coding settings with actions, and characterize the rate-distortion-cost tradeoff for a case of successive refinement with actions.