Gauss-Markov Source Research Articles

Low-Delay Analog Joint Source-Channel Coding With Deep Learning

We consider the design of low-delay joint source-channel coding (JSCC) schemes for the transmission of discrete-time analog sources over noisy channels based on deep neural networks. The design problem is addressed as optimization of an autoencoder model, and several scenarios are discussed. For point-to-point communication of independent and identically distributed (i.i.d) Gaussian sources and Gauss-Markov sources over additive-white Gaussian noise (AWGN) channels, the encoder and decoder are constructed using recurrent neural networks (RNNs). With minimum prior knowledge used for design, the performance of these RNNs-based models is optimized using fine tuning techniques during training. Sinusoidal representation networks (SIRENs)-based models are proposed and optimized for three JSCC problems namely, transmitting multivariate Gaussian sources over AWGN channels, transmitting i.i.d Gaussian sources with side information at the decoder, and for communicating correlated sources over orthogonal Gaussian channels. We show that these deep learning-based JSCC schemes perform comparably or better than state-of-the-art (SOTA) traditional schemes. The proposed scheme can extend flexibly to different pairs of source and channel dimensions. Moreover, the spontaneously learned encoder mappings exhibit structured patterns that are interpretable.

The CEO Problem With Inter-Block Memory

An $n$ -dimensional source with memory is observed by $K$ isolated encoders via parallel channels, who compress their observations to transmit to the decoder via noiseless rate-constrained links while leveraging their memory of the past. At each time instant, the decoder receives $K$ new codewords from the observers, combines them with the past received codewords, and produces a minimum-distortion estimate of the latest block of $n$ source symbols. This scenario extends the classical one-shot CEO problem to multiple rounds of communication with communicators maintaining the memory of the past. We extend the Berger-Tung inner and outer bounds to the scenario with inter-block memory, showing that the minimum asymptotically (as $n \to \infty $ ) achievable sum rate required to achieve a target distortion is bounded by minimal directed mutual information problems. For the Gauss-Markov source observed via $K$ parallel AWGN channels, we show that the inner bound is tight and solve the corresponding minimal directed mutual information problem, thereby establishing the minimum asymptotically achievable sum rate. Finally, we explicitly bound the rate loss due to a lack of communication among the observers; that bound is attained with equality in the case of identical observation channels. The general coding theorem is proved via a new nonasymptotic bound that uses stochastic likelihood coders and whose asymptotic analysis yields an extension of the Berger-Tung inner bound to the causal setting. The analysis of the Gaussian case is facilitated by reversing the channels of the observers.

Nonstationary Gauss-Markov Processes: Parameter Estimation and Dispersion

This paper provides a precise error analysis for the maximum likelihood estimate $\hat{a}_{\text{ML}}(u_1^n)$ of the parameter $a$ given samples $u_1^n = (u_1, \ldots, u_n)'$ drawn from a nonstationary Gauss-Markov process $U_i = a U_{i-1} + Z_i,~i\geq 1$, where $U_0 = 0$, $a> 1$, and $Z_i$'s are independent Gaussian random variables with zero mean and variance $\sigma^2$. We show a tight nonasymptotic exponentially decaying bound on the tail probability of the estimation error. Unlike previous works, our bound is tight already for a sample size of the order of hundreds. We apply the new estimation bound to find the dispersion for lossy compression of nonstationary Gauss-Markov sources. We show that the dispersion is given by the same integral formula that we derived previously for the asymptotically stationary Gauss-Markov sources, i.e., $|a| < 1$. New ideas in the nonstationary case include separately bounding the maximum eigenvalue (which scales exponentially) and the other eigenvalues (which are bounded by constants that depend only on $a$) of the covariance matrix of the source sequence, and new techniques in the derivation of our estimation error bound.

Dynamic signaling games with quadratic criteria under Nash and Stackelberg equilibria

This paper considers dynamic (multi-stage) signaling games involving an encoder and a decoder who have subjective models on the cost functions. We consider both Nash (simultaneous-move) and Stackelberg (leader–follower) equilibria of dynamic signaling games under quadratic criteria. For the multi-stage scalar cheap talk, we show that the final stage equilibrium is always quantized and under further conditions the equilibria for all time stages must be quantized. In contrast, the Stackelberg equilibria are always fully revealing. In the multi-stage signaling game where the transmission of a Gauss–Markov source over a memoryless Gaussian channel is considered, affine policies constitute an invariant subspace under best response maps for Nash equilibria; whereas the Stackelberg equilibria always admit linear policies for scalar sources but such policies may be non-linear for multi-dimensional sources. We obtain an explicit recursion for optimal linear encoding policies for multi-dimensional sources, and derive conditions under which Stackelberg equilibria are informative.

Zero-Delay Multiple Descriptions of Stationary Scalar Gauss-Markov Sources

In this paper, we introduce the zero-delay multiple-description problem, where an encoder constructs two descriptions and the decoders receive a subset of these descriptions. The encoder and decoders are causal and operate under the restriction of zero delay, which implies that at each time instance, the encoder must generate codewords that can be decoded by the decoders using only the current and past codewords. For the case of discrete-time stationary scalar Gauss—Markov sources and quadratic distortion constraints, we present information-theoretic lower bounds on the average sum-rate in terms of the directed and mutual information rate between the source and the decoder reproductions. Furthermore, we show that the optimum test channel is in this case Gaussian, and it can be realized by a feedback coding scheme that utilizes prediction and correlated Gaussian noises. Operational achievable results are considered in the high-rate scenario using a simple differential pulse code modulation scheme with staggered quantizers. Using this scheme, we achieve operational rates within // of the theoretical lower bounds for varying description rates.

The Dispersion of the Gauss–Markov Source

The Gauss-Markov source produces $U_i = aU_{i-1} + Z_i$ for $i\geq 1$, where $U_0 = 0$, $|a|<1$ and $Z_i\sim\mathcal{N}(0, \sigma^2)$ are i.i.d. Gaussian random variables. We consider lossy compression of a block of $n$ samples of the Gauss-Markov source under squared error distortion. We obtain the Gaussian approximation for the Gauss-Markov source with excess-distortion criterion for any distortion $d>0$, and we show that the dispersion has a reverse waterfilling representation. This is the \emph{first} finite blocklength result for lossy compression of \emph{sources with memory}. We prove that the finite blocklength rate-distortion function $R(n,d,\epsilon)$ approaches the rate-distortion function $\mathbb{R}(d)$ as $R(n,d,\epsilon) = \mathbb{R}(d) + \sqrt{\frac{V(d)}{n}}Q^{-1}(\epsilon) + o\left(\frac{1}{\sqrt{n}}\right)$, where $V(d)$ is the dispersion, $\epsilon \in (0,1)$ is the excess-distortion probability, and $Q^{-1}$ is the inverse of the $Q$-function. We give a reverse waterfilling integral representation for the dispersion $V(d)$, which parallels that of the rate-distortion functions for Gaussian processes. Remarkably, for all $0 < d\leq \frac{\sigma^2}{(1+|a|)^2}$, $R(n,d,\epsilon)$ of the Gauss-Markov source coincides with that of $Z_k$, the i.i.d. Gaussian noise driving the process, up to the second-order term. Among novel technical tools developed in this paper is a sharp approximation of the eigenvalues of the covariance matrix of $n$ samples of the Gauss-Markov source, and a construction of a typical set using the maximum likelihood estimate of the parameter $a$ based on $n$ observations.

The Time-Invariant Multidimensional Gaussian Sequential Rate-Distortion Problem Revisited

We revisit the sequential rate-distortion (SRD) tradeoff problem for vector-valued Gauss–Markov sources with mean-squared error distortion constraints. Our study is partly motivated by the question recently raised in the paper “Rate-cost tradeoffs in control” (in Proc. 54th Annu. Allerton Conf. Commun., Control, Comput. , 2016, pp. 1157–1164) regarding the correctness of its solution algorithm known in the literature. We show via a counterexample that the dynamic reverse water-filling algorithm suggested by (15) of the paper “Stochastic linear control over a communication channel” ( IEEE Trans. Autom. Control , vol. 49, pp. 1549–1561, 2004) is not applicable to this problem, and consequently, the closed-form expression of the asymptotic SRD function derived in (17) of the paper “Stochastic linear control over a communication channel” ( IEEE Trans. Autom. Control , vol. 49, pp. 1549–1561, 2004) is not correct in general. Nevertheless, we show that the multidimensional Gaussian SRD function is semidefinite representable, and thus, it is readily computable.

Zero-Delay Rate Distortion via Filtering for Vector-Valued Gaussian Sources

We deal with zero-delay source coding of a vector-valued Gauss-Markov source subject to a mean-squared error (MSE) fidelity criterion characterized by the operational zero-delay vector-valued Gaussian rate distortion function (RDF). We address this problem by considering the nonanticipative RDF (NRDF) which is a lower bound to the causal optimal performance theoretically attainable (OPTA) function and operational zero-delay RDF. We recall the realization that corresponds to the optimal "test-channel" of the Gaussian NRDF, when considering a vector Gauss-Markov source subject to a MSE distortion in the finite time horizon. Then, we introduce sufficient conditions to show existence of solution for this problem in the infinite time horizon. For the asymptotic regime, we use the asymptotic characterization of the Gaussian NRDF to provide a new equivalent realization scheme with feedback which is characterized by a resource allocation (reverse-waterfilling) problem across the dimension of the vector source. We leverage the new realization to derive a predictive coding scheme via lattice quantization with subtractive dither and joint memoryless entropy coding. This coding scheme offers an upper bound to the operational zero-delay vector-valued Gaussian RDF. When we use scalar quantization, then for "r" active dimensions of the vector Gauss-Markov source the gap between the obtained lower and theoretical upper bounds is less than or equal to 0.254r + 1 bits/vector. We further show that it is possible when we use vector quantization, and assume infinite dimensional Gauss-Markov sources to make the previous gap to be negligible, i.e., Gaussian NRDF approximates the operational zero-delay Gaussian RDF. We also extend our results to vector-valued Gaussian sources of any finite memory under mild conditions. Our theoretical framework is demonstrated with illustrative numerical experiments.

Finite-Time Nonanticipative Rate Distortion Function for Time-Varying Scalar-Valued Gauss-Markov Sources

We derive the finite-time horizon nonanticipative rate distortion function (NRDF) of time-varying scalar Gauss–Markov sources under an average mean squared-error (MSE) distortion fidelity. Further, we show that a conditionally Gaussian reproduction process realizes the optimal reproduction distribution, and this is determined from the solution of a dynamic reverse-waterfilling optimization problem. We provide an iterative algorithm that approximates the solution of the dynamic reverse-waterfilling problem. From the above results, we also obtain, as a special case, the NRDF under a per-letter or pointwise MSE distortion fidelity, and we draw connections to the classical RDF of Gaussian processes. Our results are corroborated with illustrative examples.

LQG Control With Minimum Directed Information: Semidefinite Programming Approach

We consider a discrete-time Linear-Quadratic-Gaussian (LQG) control problem in which Massey's directed information from the observed output of the plant to the control input is minimized while required control performance is attainable. This problem arises in several different contexts, including joint encoder and controller design for data-rate minimization in networked control systems. We show that the optimal control law is a Linear-Gaussian randomized policy. We also identify the state space realization of the optimal policy, which can be synthesized by an efficient algorithm based on semidefinite programming. Our structural result indicates that the filter-controller separation principle from the LQG control theory, and the sensor-filter separation principle from the zero-delay rate-distortion theory for Gauss-Markov sources hold simultaneously in the considered problem. A connection to the data-rate theorem for mean-square stability by Nair and Evans is also established.

A Truncated Prediction Framework for Streaming Over Erasure Channels

We propose a new coding technique for sequential transmission of a stream of Gauss–Markov sources over erasure channels under a zero decoding delay constraint. Our proposed scheme is a combination (hybrid) of predictive coding with truncated memory, and quantization-and-binning. We study the optimality of our proposed scheme using an information theoretic model. In our setup, the encoder observes a stream of source vectors that are spatially independent and identically distributed (i.i.d.) and temporally sampled from a first-order stationary Gauss–Markov process. The channel introduces an erasure burst of a certain maximum length $B$ , starting at an arbitrary time, not known to the transmitter. The reconstruction of each source vector at the destination must be with zero delay and satisfy a quadratic distortion constraint with an average distortion of $D$ . The decoder is not required to reconstruct those source vectors that belong to the period spanning the erasure burst and a recovery window of length $W$ following it. We study the minimum compression rate $R(B,W,D)$ in this setup. As our main result, we establish upper and lower bounds on the compression rate. The upper bound (achievability) is based on our hybrid scheme. It achieves significant gains over baseline schemes such as (leaky) predictive coding, memoryless binning, a separation-based scheme, and a group of pictures-based scheme. The lower bound is established by observing connection to a network source coding problem. The bounds simplify in the high resolution regime, where we provide explicit expressions whenever possible, and identify conditions when the proposed scheme is close to optimal. We finally discuss the interplay between the parameters of our burst erasure channel and the statistical channel models and explain how the bounds in the former model can be used to derive insights into the simulation results involving the latter. In particular, our proposed scheme outperforms the baseline schemes over the i.i.d. erasure channel and the Gilbert–Elliott channel, and achieves performance close to a lower bound in some regimes.

Hybrid digital-analog coding with bandwidth expansion for correlated Gaussian sources under Rayleigh fading

Consider communicating a correlated Gaussian source over a Rayleigh fading channel with no knowledge of the channel signal-to-noise ratio (CSNR) at the transmitter. In this case, a digital system cannot be optimal for a range of CSNRs. Analog transmission however is optimal at all CSNRs, if the source and channel are memoryless and bandwidth matched. This paper presents new hybrid digital-analog (HDA) systems for sources with memory and channels with bandwidth expansion, which outperform both digital-only and analog-only systems over a wide range of CSNRs. The digital part is either a predictive quantizer or a transform code, used to achieve a coding gain. Analog part uses linear encoding to transmit the quantization error which improves the performance under CSNR variations. The hybrid encoder is optimized to achieve the minimum AMMSE (average minimum mean square error) over the CSNR distribution. To this end, analytical expressions are derived for the AMMSE of asymptotically optimal systems. It is shown that the outage CSNR of the channel code and the analog-digital power allocation must be jointly optimized to achieve the minimum AMMSE. In the case of HDA predictive quantization, a simple algorithm is presented to solve the optimization problem. Experimental results are presented for both Gauss-Markov sources and speech signals.

Zero-Delay Sequential Transmission of Markov Sources Over Burst Erasure Channels

A setup involving zero-delay sequential transmission of a vector Markov source over a burst erasure channel is studied. A sequence of source vectors is compressed in a causal fashion at the encoder, and the resulting output is transmitted over a burst erasure channel. The destination is required to reconstruct each source vector with zero-delay, but those source sequences that are observed either during the burst erasure, or in the interval of length W following the burst erasure need not be reconstructed. The minimum achievable compression rate is called the rate-recovery function. We assume that each source vector is independent identically distributed (i.i.d.) across the spatial dimension and is sampled from a stationary, first-order Markov process across the temporal dimension. For discrete sources, the case of lossless recovery is considered, and upper and lower bounds on the rate-recovery function are established. Both these bounds can be expressed as the rate for predictive coding, plus a term that decreases at least inversely with the recovery window length W. For Gauss-Markov sources and a quadratic distortion measure, upper and lower bounds on the minimum rate are established when W = 0. These bounds are shown to coincide in the high resolution limit. Finally, another setup involving i.i.d. Gaussian sources is studied and the raterecovery function is completely characterized in this case.

Gaussian Robust Sequential and Predictive Coding

We introduce two new source coding problems: robust sequential coding and robust predictive coding. For the Gauss-Markov source model with the mean squared error distortion measure, we characterize certain supporting hyperplanes of the rate region of these two coding problems. Our investigation also reveals an information-theoretic minimax theorem and the associated extremal inequalities.

Hermitian Symmetric DFT Codes: A New Class of Complex DFT Codes

We introduce a new class of real number codes derived from DFT matrix, Hermitian symmetric DFT (HSDFT) codes. We propose a new decoding algorithm based on coding-theoretic as well as subspace based approach. Decoding of HSDFT codes requires only real arithmetic operations and smaller dimension matrices compared to the decoding of the state-of-art real BCH DFT (RBDFT) class of codes. HSDFT codes will also be shown to have more burst error correction capacity. For a Gauss-Markov source, on a binary symmetric channel at lower to moderate bit error rates (BERs), HSDFT codes show better performance than RBDFT codes, and on a Gilbert-Elliot channel HSDFT codes consistently perform better than RBDFT codes.

An efficient variable-length code construction for iterative source-channel decoding

We present a novel variable-length code (VLC) construction which exhibits an inherent error correcting capability due to the exclusive presence of codewords with even Hamming weight. Besides error robustness, the proposed code construction features a similar codeword length distribution as Golomb-Rice codes, and therefore, in particular for sources with exponentially distributed symbols, has good source compression properties at the same time. We show that in a source channel coding framework with outer source encoding, inner channel encoding with a recursive convolutional code, and iterative decoding the proposed VLC construction can lead to significant performance improvements compared to fixed-length source encoding with optimized mappings. In particular, simulation results for the AWGN channel verify that for Gauss-Markov sources a performance close to the theoretical limit can be achieved.

Hybrid digital-analog coding with bandwidth compression for gaussian source-channel pairs

Three hybrid digital-analog (HDA) systems, denoted by HDA-I, HDA* and HDA-II, for the coding of a memoryless discrete-time Gaussian source over a discrete-time additive memoryless Gaussian channel under bandwidth compression are studied. The systems employ simple linear coding in their analog component and superimpose their analog and digital signals before channel transmission. Information-theoretic upper bounds on the asymptotically optimal mean squared error distortion of the systems are obtained under both matched and mismatched channel conditions. Allocation schemes for distributing the channel input power between the analog and the digital signals are also examined. It is shown that systems HDA* and HDA-II can asymptotically achieve the optimal Shannon-limit performance under matched channel conditions. Low-complexity and low-delay versions of systems HDA-I and HDA-II are next designed and implemented without the use of error correcting codes. The parameters of these HDA systems, which employ vector quantization in conjunction with binary phase-shift keying modulation in their digital part, are optimized via an iterative algorithm similar to the design algorithm for channel-optimized vector quantizers. Both systems have low complexity and low delay, and guarantee graceful performance improvements for high CSNRs. For memoryless Gaussian sources the designed HDA-II system is shown to be superior to the HDA-I designed system. When applied to a Gauss-Markov source under Karhunen-Loeve processing, the HDA-I system is shown to provide considerably better performance.

COVQ for MAP hard-decision demodulated channels

We design a channel optimized vector quantizer (COVQ) for symbol-by-symbol maximum a posteriori (MAP) hard-decision demodulated channels. The main objective is to exploit the non-uniformity of the indices representing the quantized source via the MAP decoder and iteratively optimize the overall discrete channel (at the symbol level) jointly with the quantizer. We consider memoryless Gaussian and Gauss-Markov sources transmitted over a binary phase-shift keying modulated Rayleigh fading channel. Our scheme has less encoding computational and storage complexity (particularly for noisy channel conditions) than both conventional and soft-decision COVQ systems, which use hard-decision and soft-decision maximum likelihood demodulation, respectively. Furthermore, it provides a notable signal-to-distortion ratio gain over the former system, and in some cases it matches or outperforms the latter one.

On the Feedback Capacity of Power-Constrained Gaussian Noise Channels With Memory

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> For a stationary additive Gaussian-noise channel with a rational noise power spectrum of a finite-order <emphasis><formula formulatype="inline"> <tex>$L$</tex></formula></emphasis>, we derive two new results for the feedback capacity under an average channel input power constraint. First, we show that a very simple feedback-dependent Gauss–Markov source achieves the feedback capacity, and that Kalman–Bucy filtering is optimal for processing the feedback. Based on these results, we develop a new method for optimizing the channel inputs for achieving the Cover–Pombra block-length- <emphasis><formula formulatype="inline"><tex>$n$</tex></formula></emphasis> feedback capacity by using a dynamic programming approach that decomposes the computation into <emphasis><formula formulatype="inline"><tex>$n$</tex></formula></emphasis> sequentially identical optimization problems where each stage involves optimizing <emphasis><formula formulatype="inline"><tex>$O(L^{2})$</tex></formula></emphasis> variables. Second, we derive the explicit maximal information rate for stationary feedback-dependent sources. In general, evaluating the maximal information rate for stationary sources requires solving only a few equations by simple nonlinear programming. For first-order autoregressive and/or moving average (ARMA) noise channels, this optimization admits a closed-form maximal information rate formula. The maximal information rate for stationary sources is a lower bound on the feedback capacity, and it equals the feedback capacity if the long-standing conjecture, that stationary sources achieve the feedback capacity, holds. </para>

Channel-Optimized Quantization With Soft-Decision Demodulation for Space–Time Orthogonal Block-Coded Channels

We introduce three soft-decision demodulation channel-optimized vector quantizers (COVQs) to transmit analog sources over space–time orthogonal block (STOB)-coded flat Rayleigh fading channels with binary phase-shift keying (BPSK) modulation. One main objective is to judiciously utilize the soft information of the STOB-coded channel in the design of the vector quantizers while keeping a low system complexity. To meet this objective, we introduce a simple space–time decoding structure that consists of a space–time soft detector, followed by a linear combiner and a scalar uniform quantizer with resolution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$q$</tex> . The concatenation of the space–time encoder/modulator, fading channel, and space–time receiver can be described by a binary-input, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2^q$</tex> -output discrete memoryless channel (DMC). The scalar uniform quantizer is chosen so that the capacity of the equivalent DMC is maximized to fully exploit and capture the system's soft information by the DMC. We next determine the statistics of the DMC in closed form and use them to design three COVQ schemes with various degrees of knowledge of the channel noise power and fading coefficients at the transmitter and/or receiver. The performance of each quantization scheme is evaluated for memoryless Gaussian and Gauss–Markov sources and various STOB codes, and the benefits of each scheme is illustrated as a function of the antenna-diversity and soft-decision resolution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$q$</tex> . Comparisons to traditional coding schemes, which perform separate source and channel coding operations, are also provided.