Stationary Memoryless Research Articles

Zero-Error Capacity Regions of Noisy Networks

This paper presents the first systematic study of the zero-error capacity regions of noisy networks. First, we consider two simple such networks, each consisting of a stationary memoryless multiple access channel with two binary inputs and one discrete output. There are two users in each network. Each of the two users transmits a message through the network, and the sink(s) of the network can decode both messages with zero error. A graph is used to represent the distinguishability of the inputs of the channel, and a <i>graph set</i> is used to represent the distinguishability of the inputs of the network. We show that for two networks represented by the same graph set, their zero-error capacity regions are the same. We list all the possible graph sets for the two networks and determine the zero-error capacity regions for some of these graph sets. Based on this result, we explore a relation between graph theory and set theory, and then redefine the <i>cancellative pair of families of subsets</i>. We further extend the problem formulation to a general network called the <i>parallel network</i>, which may consist of more than one channel with multiple inputs and multiple outputs.

The “Nontriviality” of the Union Bound for Decoding of Convolutional Codes

Transmission of an infinite binary information sequence over a stationary memoryless channel is considered using a time-invariant non-catastrophic convolutional encoder and the Viterbi decoder. The conventional proofs of the “union upper bound” (and other upper bounds) for decoding performance of convolutional codes involve some mathematical inaccuracies. These proofs implicitly assume that all information symbols are necessarily decoded at some time and, moreover, that the decoded path on the trellis diagram “meets” the true path sooner or later. Although these facts look natural (at least, for a binary symmetric channel), they must be proved. The goal of this note is to substantiate these facts for a binary symmetric channel and a general stationary memoryless channel.

Coding Theorems for Asynchronous Slepian–Wolf Coding Systems

The Slepian-Wolf (SW) coding system is a source coding system with two encoders and a decoder, where these encoders independently encode source sequences from two correlated sources into codewords, and the decoder reconstructs both source sequences from the codewords. In this paper, we consider the situation in which the SW coding system is asynchronous, i.e., each encoder samples a source sequence with some unknown delay. We assume that delays are unknown but maximum and minimum values of possible delays are known to encoders and the decoder. We also assume that sources are discrete stationary memoryless and the probability mass function (PMF) of the sources is unknown but the system knows that it belongs to a certain set of PMFs. For this asynchronous SW coding system, we clarify the achievable rate region which is the set of rate pairs of encoders such that the decoding error probability vanishes as the blocklength tends to infinity. We show that this region does not always coincide with that of the synchronous SW coding system in which each encoder samples a source sequence without any delay.

The Sphere Packing Bound for Memoryless Channels

Sphere packing bounds (SPBs)—with prefactors that are polynomial in the block length—are derived for codes on two families of memoryless channels using Augustin’s method: (possibly nonstationary) memoryless channels with (possibly multiple) additive cost constraints and stationary memoryless channels with convex constraints on the composition (i.e., empirical distribution, type) of the input codewords. A variant of Gallager’s bound is derived in order to show that these sphere packing bounds are tight in terms of the exponential decay rate of the error probability with the block length under mild hypotheses.

A simple derivation of the refined sphere packing bound under certain symmetry hypotheses

A judicious application of the Berry-Esseen theorem via suitable Augustin information measures is demonstrated to be sufficient for deriving the sphere packing bound with a prefactor that is $\mathit{\Omega}\left(n^{-0.5(1-E_{sp}'(R))}\right)$ for all codes on certain families of channels (including the Gaussian channels and the nonstationary Renyi symmetric channels) and for the constant composition codes on stationary memoryless channels. The resulting nonasymptotic bounds have definite approximation error terms. As a preliminary result that might be of interest on its own, the trade-off between type I and type II error probabilities in the hypothesis testing problem with (possibly non-stationary) independent samples is determined up to some multiplicative constants, assuming that the probabilities of both types of error are decaying exponentially with the number of samples,using the Berry-Esseen theorem.

Граница сферической упаковки для каналов без памяти

Sphere packing bounds (SPBs) ---with prefactors that are polynomial in the block length--- are derived for codes on two families of memoryless channels using Augustin's method: (possibly non-stationary) memoryless channels with (possibly multiple) additive cost constraints and stationary memoryless channels with convex constraints on the composition (i.e. empirical distribution, type) of the input codewords. A variant of Gallager's bound is derived in order to show that these sphere packing bounds are tight in terms of the exponential decay rate of the error probability with the block length under mild hypotheses.

Optimum Overflow Thresholds in Variable-Length Source Coding Allowing Non-Vanishing Error Probability

The variable-length source coding problem allowing the error probability up to some constant is considered for general sources. In this problem the optimum mean codeword length of variable-length codes has already been determined. On the other hand, in this paper, we focus on the overflow (or excess codeword length) probability instead of the mean codeword length. The infimum of overflow thresholds under the constraint that both of the error probability and the overflow probability are smaller than or equal to some constant is called the optimum overflow threshold. In this paper, we first derive finite-length upper and lower bounds on these probabilities so as to analyze the optimum overflow thresholds. Then, by using these bounds we determine the general formula of the optimum overflow thresholds in both of the first-order and second-order forms. Next, we consider another expression of the derived general formula so as to reveal the relationship with the optimum coding rate in the fixed-length source coding problem. We also derive general formulas of the optimum overflow thresholds in the optimistic coding scenario. Finally, we apply general formulas derived in this paper to the case of stationary memoryless sources.

Smoothing Brascamp-Lieb Inequalities and Strong Converses of Coding Theorems

The Brascamp-Lieb inequality in functional analysis can be viewed as a measure of the “uncorrelatedness” of a joint probability distribution. We define the smooth Brascamp-Lieb (BL) divergence as the infimum of the best constant in the Brascamp-Lieb inequality under a perturbation of the joint probability distribution. An information spectrum upper bound on the smooth BL divergence is proved, using properties of the subgradient of a certain convex functional. In particular, in the i.i.d. setting, such an infimum converges to the best constant in a certain mutual information inequality. We then derive new single-shot converse bounds for the omniscient helper common randomness generation problem and the Gray-Wyner source coding problem in terms of the smooth BL divergence, where the proof relies on the functional formulation of the Brascamp-Lieb inequality. Exact second-order rates are thus obtained in the stationary memoryless and nonvanishing error setting. These offer rare instances of strong converses/second-order converses for continuous sources when the rate region involves auxiliary random variables.

Reliable Transmission of Short Packets Through Queues and Noisy Channels Under Latency and Peak-Age Violation Guarantees

This paper investigates the probability that the delay and the peak-age of information exceed a desired threshold in a point-to-point communication system with short information packets. The packets are generated according to a stationary memoryless Bernoulli process, placed in a single-server queue and then transmitted over a wireless channel. A variable-length stop-feedback coding scheme—a general strategy that encompasses simple automatic repetition request (ARQ) and more sophisticated hybrid ARQ techniques as special cases—is used by the transmitter to convey the information packets to the receiver. By leveraging finite-blocklength results, the delay violation and the peak-age violation probabilities are characterized without resorting to approximations based on larg-deviation theory as in previous literature. Numerical results illuminate the dependence of delay and peak-age violation probability on system parameters such as the frame size and the undetected error probability, and on the chosen packet-management policy. The guidelines provided by our analysis are particularly useful for the design of low-latency ultra-reliable communication systems.

Observability and stabilisability of networked control systems with limited data rates

ABSTRACTThis paper investigates the observability and stabilisability problem for linear time-invariant systems, where sensors and controllers are geographically separated and connected by a stationary memoryless digital communication channel. The limited information of the plant state is transmitted over such a channel to the controller. We focus on explaining the effect imposed by limited data rates. A new quantisation, coding and control scheme is presented to minimise the required data rate for observability and stabilisability. Different from prior research, our study shows that, the required data rate is determined by the state prediction error. Namely, the smaller prediction error requires the smaller codeword length, which leads to the smaller data rate. It is shown that, there exists a lower bound on the average data rate above which the system is observable and stabilisable. Illustrative examples are given to demonstrate the effectiveness of the proposed quantisation, coding and control scheme.

First- and Second-Order Hypothesis Testing for Mixed Memoryless Sources.

The first- and second-order optimum achievable exponents in the simple hypothesis testing problem are investigated. The optimum achievable exponent for type II error probability, under the constraint that the type I error probability is allowed asymptotically up to , is called the -optimum exponent. In this paper, we first give the second-order -optimum exponent in the case where the null hypothesis and alternative hypothesis are a mixed memoryless source and a stationary memoryless source, respectively. We next generalize this setting to the case where the alternative hypothesis is also a mixed memoryless source. Secondly, we address the first-order -optimum exponent in this setting. In addition, an extension of our results to the more general setting such as hypothesis testing with mixed general source and a relationship with the general compound hypothesis testing problem are also discussed.

Minimum information rate for observability of linear systems with stochastic time delay

ABSTRACTThis paper investigates the state estimation problem for linear time-invariant systems where sensors and controllers are geographically separated and connected via stationary memoryless uncertain digital communication channels with the information rate limitation. Such channels introduce data packet dropout, stochastic time delay, and data packet reordering. In the TCP/IP model, the transport layer provides a reliable service on top of an unreliable communication channel. However, the performance of such systems is still affected by packet reordering and stochastic time delay. In particular, we present an optimal encoding and decoding scheme to guarantee observability by employing the minimum information rate, and derive a necessary and sufficient condition on the delay distribution and information rate for observability. Illustrative examples are given to demonstrate the effectiveness of the proposed scheme.

An Information-Spectrum Approach to the Capacity Region of the Interference Channel

In this paper, a general formula for the capacity region of a general interference channel with two pairs of users is derived, which reveals that the capacity region is the union of a family of rectangles. In the region, each rectangle is determined by a pair of spectral inf-mutual information rates. The presented formula provides us with useful insights into the interference channels in spite of the difficulty of computing it. Specially, when the inputs are discrete, ergodic Markov processes and the channel is stationary memoryless, the formula can be evaluated by the BCJR (Bahl-Cocke-Jelinek-Raviv) algorithm. Also the formula suggests that considering the structure of the interference processes contributes to obtaining tighter inner bounds than the simplest one (obtained by treating the interference as noise). This is verified numerically by calculating the mutual information rates for Gaussian interference channels with embedded convolutional codes. Moreover, we present a coding scheme to approach the theoretical achievable rate pairs. Numerical results show that the decoding gains can be achieved by considering the structure of the interference.

Universal Weak Variable-Length Source Coding on Countably Infinite Alphabets

Motivated from the fact that universal source coding on countably infinite alphabets is not feasible, this work introduces the notion of almost lossless source coding. Analog to the weak variable-length source coding problem studied by Han (IEEE TIT, 2000, 46, 1217-1226), almost lossless source coding aims at relaxing the lossless block-wise assumption to allow an average per-letter distortion that vanishes asymptotically as the block-length tends to infinity. In this setup, we show on one hand that Shannon entropy characterizes the minimum achievable rate (similarly to the case of finite alphabet sources) while on the other that almost lossless universal source coding becomes feasible for the family of finite-entropy stationary memoryless sources with infinite alphabets. Furthermore, we study a stronger notion of almost lossless universality that demands uniform convergence of the average per-letter distortion to zero, where we establish a necessary and sufficient condition for the so-called family of envelope distributions to achieve it. Remarkably, this condition is the same necessary and sufficient condition needed for the existence of a strongly minimax (lossless) universal source code for the family of envelope distributions. Finally, we show that an almost lossless coding scheme offers faster rate of convergence for the (minimax) redundancy compared to the well-known information radius developed for the lossless case at the expense of tolerating a non-zero distortion that vanishes to zero as the block-length grows. This shows that even when lossless universality is feasible, an almost lossless scheme can offer different regimes on the rates of convergence of the (worst case) redundancy versus the (worst case) distortion.

The Dispersion of Nearest-Neighbor Decoding for Additive Non-Gaussian Channels

We study the second-order asymptotics of information transmission using random Gaussian codebooks and nearest neighbor decoding over a power-limited stationary memoryless additive non-Gaussian noise channel. We show that the dispersion term depends on the non-Gaussian noise only through its second and fourth moments, thus complementing the capacity result (Lapidoth, 1996), which depends only on the second moment. Furthermore, we characterize the second-order asymptotics of point-to-point codes over $K$ -sender interference networks with non-Gaussian additive noise. Specifically, we assume that each user’s codebook is Gaussian and that NN decoding is employed, i.e., that interference from the $K-1$ unintended users (Gaussian interfering signals) is treated as noise at each decoder. We show that while the first-order term in the asymptotic expansion of the maximum number of messages depends on the power of the interfering codewords only through their sum, this does not hold for the second-order term.

A Simple Proof of Polarization and Polarization for Non-Stationary Memoryless Channels

We give a simple proof of Arikan’s polarization phenomenon that uses only elementary methods. Using the same method, we show that Arikan’s construction also polarizes non-stationary memoryless channels in the same way it polarizes the stationary memoryless channels.

Almost Sure Convergence Coding Theorems of One-Shot and Multi-Shot Tunstall Codes for Stationary Memoryless Sources

Almost Sure Convergence Coding Theorems of One-Shot and Multi-Shot Tunstall Codes for Stationary Memoryless Sources

Reliability and Secrecy Functions of the Wiretap Channel Under Cost Constraint

The wiretap channel has been devised and studied first by Wyner, and subsequently extended to the case with nondegraded general wiretap channels by Csiszar and Korner. Focusing mainly on the stationary memoryless channel with cost constraint, we newly introduce the notion of reliability and secrecy functions as a fundamental tool to analyze and/or design the performance of an efficient wiretap channel system, including binary symmetric wiretap channels, Poisson wiretap channels, and Gaussian wiretap channels. Compact formulas for those functions are explicitly given for stationary memoryless wiretap channels. It is also demonstrated that, based on such a pair of reliability and secrecy functions, we can control the tradeoff between reliability and secrecy (usually conflicting), both with exponentially decreasing rates as block length \(n\) becomes large. Four ways to do so are given on the basis of rate shifting, rate exchange, concatenation, and change of cost constraint. In addition, the notion of the \(\delta \) secrecy capacity is defined and shown to attain the strongest secrecy standard among others. The maximized versus averaged secrecy measures is also discussed.

About Adaptive Coding on Countable Alphabets: Max-Stable Envelope Classes

In this paper, we study the problem of lossless universal source coding for stationary memoryless sources on countably infinite alphabets. This task is generally not achievable without restricting the class of sources over which universality is desired. Building on our prior work, we propose natural families of sources characterized by a common dominating envelope. We particularly emphasize the notion of adaptivity, which is the ability to perform as well as an oracle knowing the envelope, without actually knowing it. This is closely related to the notion of hierarchical universal source coding, but with the important difference that families of envelope classes are not discretely indexed and not necessarily nested. Our contribution is to extend the classes of envelopes over which adaptive universal source coding is possible, namely by including max-stable (heavy-tailed) envelopes which are excellent models in many applications, such as natural language modeling. We derive a minimax lower bound on the redundancy of any code on such envelope classes, including an oracle that knows the envelope. We then propose a constructive code that does not use knowledge of the envelope. The code is computationally efficient and is structured to use an expanding threshold for auto-censoring (ETAC), and we therefore dub it the ETAC-code. We prove that the ETAC-code achieves the lower bound on the minimax redundancy within a factor logarithmic in the sequence length, and can be therefore qualified as a near-adaptive code over families of heavy-tailed envelopes. For finite and light-tailed envelopes, the penalty is even less, and the same code follows closely previous results that explicitly made the light-tailed assumption. Our technical results are founded on methods from regular variation theory and concentration of measure.

Secure multiplex coding to attain the channel capacity in wiretap channels

It is known that a message can be transmitted safely against any wiretapper via a noisy channel without a secret key if the coding rate is less than the so-called secrecy capacity C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</sub> , which is usually smaller than the channel capacity C. In order to remove the loss C-C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</sub> , we propose a multiplex coding scheme with plural independent messages. In this paper, it is shown that the proposed multiplex coding scheme can attain the channel capacity as the total rate of the plural messages and the perfect secrecy for each massage. Several bounds of achievable multiplex coding rate region are derived for general wiretap channels in the sense of information-spectral methods, by extending Hayashi's proof, in which the coding of the channel resolvability is applied to wiretap channels. Furthermore, the exact region for deterministic coding is determined for stationary memoryless full-rank wiretap channels.