Single-letter Characterization Research Articles

First-Order Theory of Probabilistic Independence and Single-Letter Characterizations of Capacity Regions

First-Order Theory of Probabilistic Independence and Single-Letter Characterizations of Capacity Regions

Source Coding With a Listening-Helper

We consider the cooperative coding of a pair of correlated memoryless source and side-information sequences wherein an encoder assisted by a side-information aided listening-helper describe the source and side-information to a receiver. We obtain a single-letter characterization of the rate region in both cases: when the decoder reconstructs just the source and when it reconstructs both the source and side-information. In the quadratic Gaussian setting we determine the optimal law characterizing the rate region using a technique introduced by Courtade in combination with a conditional version of the converse for the Gaussian CEO problem.

Task-Oriented Communication for Multidevice Cooperative Edge Inference

This paper investigates task-oriented communication for multi-device cooperative edge inference, where a group of distributed low-end edge devices transmit the extracted features of local samples to a powerful edge server for inference. While cooperative edge inference can overcome the limited sensing capability of a single device, it substantially increases the communication overhead and may incur excessive latency. To enable low-latency cooperative inference, we propose a learning-based communication scheme that optimizes local feature extraction and distributed feature encoding in a task-oriented manner, i.e., to remove data redundancy and transmit information that is essential for the downstream inference task rather than reconstructing the data samples at the edge server. Specifically, we leverage Tishby’s information bottleneck (IB) principle (Tishby et al., 2000) to extract the task-relevant feature at each edge device, and adopt the distributed information bottleneck (DIB) framework of Aguerri and Zaidi, 2021, to formalize a single-letter characterization of the optimal rate-relevance tradeoff for distributed feature encoding. To admit flexible control of the communication overhead, we extend the DIB framework to a distributed deterministic information bottleneck (DDIB) objective that explicitly incorporates the representational costs of the encoded features. As the IB-based objectives are computationally prohibitive for high-dimensional data, we adopt variational approximations to make the optimization problems tractable. To compensate for the potential performance loss due to the variational approximations, we also develop a selective retransmission (SR) mechanism to identify the redundancy in the encoded features among multiple edge devices to attain additional communication overhead reduction. Extensive experiments on multi-view image classification and multi-view object recognition tasks evidence that the proposed task-oriented communication scheme achieves a better rate-relevance tradeoff than existing methods.

Communication Over Quantum Channels With Parameter Estimation

Communication over a random-parameter quantum channel when the decoder is required to reconstruct the parameter sequence is considered. We study scenarios that include either strictly-causal, causal, or non-causal channel side information (CSI) available at the encoder, and also when CSI is not available. This model can be viewed as a form of quantum metrology, and as the quantum counterpart of the classical rate-and-state channel with state estimation at the decoder. Regularized formulas for the capacity-distortion regions are derived. In the special case of measurement channels, single-letter characterizations are derived for the strictly-causal and causal settings. Furthermore, in the more general case of entanglement-breaking channels, a single-letter characterization is derived when CSI is not available. As a consequence, we obtain regularized formulas for the capacity of random-parameter quantum channels with CSI, generalizing previous results by Boche <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , 2016, on classical-quantum channels. Bosonic dirty paper coding is introduced as a consequence, where we demonstrate that the optimal coefficient is not necessarily that of minimum mean-square error estimation as in the classical setting.

Distributed information-theoretic clustering

Abstract We study a novel multi-terminal source coding setup motivated by the biclustering problem. Two separate encoders observe two i.i.d. sequences $X^n$ and $Y^n$, respectively. The goal is to find rate-limited encodings $f(x^n)$ and $g(z^n)$ that maximize the mutual information $\textrm{I}(\,{f(X^n)};{g(Y^n)})/n$. We discuss connections of this problem with hypothesis testing against independence, pattern recognition and the information bottleneck method. Improving previous cardinality bounds for the inner and outer bounds allows us to thoroughly study the special case of a binary symmetric source and to quantify the gap between the inner and the outer bound in this special case. Furthermore, we investigate a multiple description (MD) extension of the CEO problem with mutual information constraint. Surprisingly, this MD-CEO problem permits a tight single-letter characterization of the achievable region.

Communicating Correlated Sources Over MAC and Interference Channels II: Joint Source-Channel Coding

We present the second part of our work on communicating correlated sources over multiple access (MAC) and interference channels (IC). Specifically, we undertake a Shannon-theoretic study of the above scenarios and focus on characterizing sufficient conditions for lossless recoverability of the sources at the decoder(s). We enhance the fixed block-length (B-L) coding technique by incorporating the technique of inducing source correlation onto channel inputs, originally discovered by Cover, El Gamal and Salehi. In contrast to the first part, performance analysis of a joint source-channel decoder poses new challenges. We enhance the earlier developed suite of coding and analytical tools to overcome these challenges and derive (simplified) single-letter characterizations for a new set of sufficient conditions for both scenarios. For both the MAC and IC problems, the derived sufficient conditions are (i) subsumed in the current known tightest, and (ii) strictly weaker for identified examples. Lastly, we propose simple `plug-in' approaches that can further weaken the derived sufficient conditions. Our findings enable us to subsume Dueck's findings (1981) and go even further for the example considered therein.

A Neyman-Pearson Proper Way to Universal Testing of Multiple Hypotheses Formed by Groups of Distributions

The asymptotically optimal Neyman-Pearson procedures of detection for models characterized by M discrete probability distributions arranged into K, 2 ≤ K ≤ M groups considered as hypotheses are investigated. The sequence of tests based on a growing number of observations is logarithmically asymptotically optimal (LAO) when a certain part of the given error probability exponents (reliabilities) provides positives values for all other reliabilities. LAO tests sequences for some models of objects, including cases, when rejection of decision may be permitted, and when part, or all given error probabilities decrease subexponentially with an increase in the of number of experiments, are desined. For all reliabilities of such tests single-letter characterizations are obtained. A simple case with three distributions and two hypotheses is considered.

Single-Letter Characterizations for Information Erasure under Restriction on the Output Distribution

Single-Letter Characterizations for Information Erasure under Restriction on the Output Distribution

Non-Additivity in Classical-Quantum Wiretap Channels

Due to Csiszar and Koerner, the private capacity of classical wiretap channels has a single-letter characterization in terms of the private information. For quantum wiretap channels, however, it is known that regularization of the private information is necessary to reach the capacity. Here, we study hybrid classical-quantum wiretap channels in order to resolve to what extent quantum effects are needed to witness non-additivity phenomena in quantum Shannon theory. For wiretap channels with quantum inputs but classical outputs, we prove that the characterization of the capacity in terms of the private information stays single-letter. Hence, entangled input states are of no asymptotic advantage in this setting. For wiretap channels with classical inputs, we show by means of explicit examples that the private information already becomes non-additive when either one of the two receivers becomes quantum (with the other receiver staying classical). This gives non-additivity examples that are not caused by entanglement and illustrates that quantum adversaries are strictly different from classical adversaries in the wiretap model.

Privacy-Aware Distributed Hypothesis Testing.

A distributed binary hypothesis testing (HT) problem involving two parties, a remote observer and a detector, is studied. The remote observer has access to a discrete memoryless source, and communicates its observations to the detector via a rate-limited noiseless channel. The detector observes another discrete memoryless source, and performs a binary hypothesis test on the joint distribution of its own observations with those of the observer. While the goal of the observer is to maximize the type II error exponent of the test for a given type I error probability constraint, it also wants to keep a private part of its observations as oblivious to the detector as possible. Considering both equivocation and average distortion under a causal disclosure assumption as possible measures of privacy, the trade-off between the communication rate from the observer to the detector, the type II error exponent, and privacy is studied. For the general HT problem, we establish single-letter inner bounds on both the rate-error exponent-equivocation and rate-error exponent-distortion trade-offs. Subsequently, single-letter characterizations for both trade-offs are obtained (i) for testing against conditional independence of the observer’s observations from those of the detector, given some additional side information at the detector; and (ii) when the communication rate constraint over the channel is zero. Finally, we show by providing a counter-example where the strong converse which holds for distributed HT without a privacy constraint does not hold when a privacy constraint is imposed. This implies that in general, the rate-error exponent-equivocation and rate-error exponent-distortion trade-offs are not independent of the type I error probability constraint.

On the Commitment Capacity of Unfair Noisy Channels

Noisy channels are a valuable resource from a cryptographic point of view. They can be used for exchanging secret-keys as well as realizing other cryptographic primitives such as commitment and oblivious transfer. To be really useful, noisy channels have to be considered in the scenario where a cheating party has some degree of control over the channel characteristics. Damgard et al. (EUROCRYPT 1999) proposed a more realistic model where such level of control is permitted to an adversary, the so called unfair noisy channels, and proved that they can be used to obtain commitment and oblivious transfer protocols. Given that noisy channels are a precious resource for cryptographic purposes, one important question is determining the optimal rate in which they can be used. The commitment capacity has already been determined for the cases of discrete memoryless channels and Gaussian channels. In this work we address the problem of determining the commitment capacity of unfair noisy channels. We compute a single-letter characterization of the commitment capacity of unfair noisy channels. In the case where an adversary has no control over the channel (the fair case) our capacity reduces to the well-known capacity of a discrete memoryless binary symmetric channel.

An Operational Approach to Information Leakage

Given two random variables X and Y, an operational approach is undertaken to quantify the “leakage” of information from X to Y. The resulting measure L (X→Y ) is called maximal leakage, and is defined as the multiplicative increase, upon observing Y , of the probability of correctly guessing a randomized function of X, maximized over all such randomized functions. A closed-form expression for L (X→Y) is given for discrete X and Y, and it is subsequently generalized to handle a large class of random variables. The resulting properties are shown to be consistent with an axiomatic view of a leakage measure, and the definition is shown to be robust to variations in the setup. Moreover, a variant of the Shannon cipher system is studied, in which performance of an encryption scheme is measured using maximal leakage. A single-letter characterization of the optimal limit of (normalized) maximal leakage is derived and asymptotically-optimal encryption schemes are demonstrated. Furthermore, the sample complexity of estimating maximal leakage from data is characterized up to subpolynomial factors. Finally, the guessing framework used to define maximal leakage is used to give operational interpretations of commonly used leakage measures, such as Shannon capacity, maximal correlation, and local differential privacy.

Distributed Hypothesis Testing Over Discrete Memoryless Channels

A distributed binary hypothesis testing (HT) problem involving two parties, one referred to as the observer and the other as the detector is studied. The observer observes a discrete memoryless source (DMS) and communicates its observations to the detector over a discrete memoryless channel (DMC). The detector observes another DMS correlated with that at the observer, and performs a binary hypothesis test on the joint distribution of the two DMS’s using its own observed data and the information received from the observer. The trade-off between the type I error probability and the type II error-exponent of the HT is explored. Single-letter lower bounds on the optimal type II error-exponent are obtained by using two different coding schemes, a separate HT and channel coding scheme and a joint HT and channel coding scheme based on hybrid coding for the matched bandwidth case. Exact single-letter characterization of the same is established for the special case of testing against conditional independence, and it is shown to be achieved by the separate HT and channel coding scheme. An example is provided where the joint scheme achieves a strictly better performance than the separation based scheme.

Common Reconstructions in the Successive Refinement Problem With Receiver Side Information

We study a variant of the successive refinement problem with receiver side information in which the receivers require identical, matching reconstructions. We present general inner and outer bounds for the rate region for this variant of the problem and present a single-letter characterization of the admissible rate region for several classes of the joint distribution of the source and the side information. Unlike in the general successive refinement problem, the characterization of the admissible rate region in the cases when the derived inner and outer bounds match requires only one auxiliary random variable. The characterization reveals that the side information can be fully used to reduce the communication rates via binning; however, the receiver reconstruction functions can depend only on a certain Gacs–Korner common randomness between shared by the two receivers. Since the entropy of the Gacs–Korner common randomness between two random variables is a discontinuous function of joint distribution of the variables, we establish the fact that the admissible rate region for this variant of the successive refinement problem is discontinuous in the underlying distribution of the source and side information even though the problem formulation does not involve the zero-error or functional reconstruction constraints.

Reconstructing Gaussian Sources by Spatial Sampling

Consider a Gaussian memoryless multiple source (GMMS) with $m$ components with joint probability distribution known only to lie in a given class of distributions. A subset of $k \leq m$ components is sampled and compressed with the objective of reconstructing all the $m$ components within a specified level of distortion under a mean-squared error criterion. In Bayesian and nonBayesian settings, the notion of universal sampling rate-distortion function for Gaussian sources is introduced to capture the optimal tradeoffs among sampling, compression rate, and distortion level. Single-letter characterizations are provided for the universal sampling rate-distortion function. Our achievability proofs highlight the following structural property: it is optimal to compress and reconstruct first the sampled components of the GMMS alone, and then form estimates for the unsampled components based on the former.

Distributed Variational Representation Learning.

The problem of distributed representation learning is one in which multiple sources of information X1,…, XK are processed separately so as to learn as much information as possible about some ground truth Y. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, K encoders, K ≥ 2, compress their observations X1,…, XK separately in a manner such that, collectively, the produced representations preserve as much information as possible about Y. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations X1,…, XK being conditionally independent given Y). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

Universal Sampling Rate Distortion

We examine the coordinated and universal rate-efficient sampling of a subset of correlated discrete memoryless sources followed by lossy compression of the sampled sources. The goal is to reconstruct a predesignated subset of sources within a specified level of distortion. The combined sampling mechanism and rate distortion code are universal in that they are devised to perform robustly without exact knowledge of the underlying joint probability distribution of the sources. In Bayesian as well as nonBayesian settings, single-letter characterizations are provided for the universal sampling rate distortion function for fixed-set sampling, independent random sampling, and memoryless random sampling. It is illustrated how these sampling mechanisms are successively better. Our achievability proofs bring forth new schemes for joint source distribution-learning and lossy compression.

Controllable Identifier Measurements for Private Authentication With Secret Keys

The problem of secret-key based authentication under a privacy constraint on the source sequence is considered. The identifier measurements during authentication are assumed to be controllable via a cost-constrained "action" sequence. Single-letter characterizations of the optimal trade-off among the secret-key rate, storage rate, privacy-leakage rate, and action cost are given for the four problems where noisy or noiseless measurements of the source are enrolled to generate or embed secret keys. The results are relevant for several user-authentication scenarios including physical and biometric authentications with multiple measurements. Our results include, as special cases, new results for secret-key generation and embedding with action-dependent side information without any privacy constraint on the enrolled source sequence.

Capacity Regions of Two-Receiver Broadcast Erasure Channels With Feedback and Memory

The two-receiver broadcast packet erasure channel with feedback and memory is studied. Memory is modeled using a finite-state Markov chain representing a channel state. Two scenarios are considered: 1) when the transmitter has causal knowledge of the channel state (i.e., the state is visible) and 2) when the channel state is unknown at the transmitter, but observations of it are available at the transmitter through feedback (i.e., the state is hidden). In both scenarios, matching outer and inner bounds on the rates of communication are derived and the capacity region is determined. It is shown that similar results carry over to channels with memory and delayed feedback and memoryless compound channels with feedback. When the state is visible, the capacity region has a single-letter characterization and is in terms of a linear program. Two optimal coding schemes are devised that use feedback to keep track of the sent/received packets via a network of queues: a probabilistic scheme and a deterministic backpressure-like algorithm. The former bases its decisions solely on the past channel state information and the latter follows a max-weight queue-based policy. The performance of the algorithms is analyzed using the frameworks of rate stability in networks of queues, max-flow min-cut duality in networks, and finite-horizon Lyapunov drift analysis. When the state is hidden, the capacity region does not have a single-letter characterization and is, in this sense, uncomputable. Approximations of the capacity region are provided and two optimal coding algorithms are outlined. The first algorithm is a probabilistic coding scheme that bases its decisions on the past $L$ acknowledgments and its achievable rate region approaches the capacity region exponentially fast in $L$ . The second algorithm is a backpressure-like algorithm that performs optimally in the long run.

On the Capacity of Cloud Radio Access Networks With Oblivious Relaying

We study the transmission over a network in which users send information to a remote destination through relay nodes that are connected to the destination via finite-capacity error-free links, i.e., a cloud radio access network. The relays are constrained to operate without knowledge of the users’ codebooks, i.e., they perform oblivious processing. The destination, or central processor, however, is informed about the users’ codebooks. We establish a single-letter characterization of the capacity region of this model for a class of discrete memoryless channels in which the outputs at the relay nodes are independent given the users’ inputs. We show that both relaying a-la Cover–El Gamal, i.e., compress-and-forward with joint decompression and decoding, and “noisy network coding” are optimal. The proof of the converse part establishes, and utilizes, connections with the Chief Executive Officer source coding problem under logarithmic loss distortion measure. Extensions to general discrete memoryless channels are also investigated. In this case, we establish the inner and outer bounds on the capacity region. For memoryless Gaussian channels within the studied class of channels, we characterize the capacity region when the users are constrained to time-share among Gaussian codebooks. Furthermore, we also discuss the suboptimality of separate decompression and decoding and the role of time sharing.