Data Processing Inequality Research Articles

Information-Theoretic Lower Bounds for Distributed Function Computation

We derive information-theoretic converses (i.e., lower bounds) for the minimum time required by any algorithm for distributed function computation over a network of point-to-point channels with finite capacity, where each node of the network initially has a random observation and aims to compute a common function of all observations to a given accuracy with a given confidence by exchanging messages with its neighbors. We obtain the lower bounds on computation time by examining the conditional mutual information between the actual function value and its estimate at an arbitrary node, given the observations in an arbitrary subset of nodes containing that node. The main contributions include the following. First, a lower bound on the conditional mutual information via so-called small ball probabilities, which captures the dependence of the computation time on the joint distribution of the observations at the nodes, the structure of the function, and the accuracy requirement. For linear functions, the small ball probability can be expressed by Levy concentration functions of sums of independent random variables, for which tight estimates are available that lead to strict improvements over existing lower bounds on computation time. Second, an upper bound on the conditional mutual information via strong data processing inequalities, which complements and strengthens existing cutset-capacity upper bounds. Finally, a multi-cutset analysis that quantifies the loss (dissipation) of the information needed for computation as it flows across a succession of cutsets in the network. This analysis is based on reducing a general network to a line network with bidirectional links and self-links, and the results highlight the dependence of the computation time on the diameter of the network, a fundamental parameter that is missing from most of the existing lower bounds on computation time.

Sum-Rate Upper Bounds for the Gaussian X Channel with Limited Receiver Cooperation

In this paper, we propose sum-rate upper bounds in terms of the sum of 3 rates and the sum of 4 rates for the two-user Gaussian X channel with limited receiver cooperation. These upper bounds are derived by using the Fano's inequality, the data processing inequality and genie-aided techniques. Then, we explore the generalized degrees of freedom (GDoF) of the proposed upper bounds with a symmetric channel setting. By setting a certain set of parameters, the received results show that our results are identical to the existing results for both non-cooperation and limited receiver cooperation cases. Moreover, we obtain that the GDoF computed from our proposedupper bounds are greater than the GDoF of those in the non-cooperation case.

Information-Theoretic Lower Bounds on Bayes Risk in Decentralized Estimation

We derive lower bounds on the Bayes risk in decentralized estimation, where the estimator does not have direct access to the random samples generated conditionally on the random parameter of interest, but only to the data received from local processors that observe the samples. The received data are subject to communication constraints, due to quantization and the noise in the communication channels from the processors to the estimator. We first derive general lower bounds on the Bayes risk using information-theoretic quantities, such as mutual information, information density, small ball probability, and differential entropy. We then apply these lower bounds to the decentralized case, using strong data processing inequalities to quantify the contraction of information due to communication constraints. We treat the cases of a single processor and of multiple processors, where the samples observed by different processors may be conditionally dependent given the parameter, for noninteractive and interactive communication protocols. Our results recover and improve recent lower bounds on the Bayes risk and the minimax risk for certain decentralized estimation problems, where previously only conditionally independent sample sets and noiseless channels have been considered. Moreover, our results provide a general way to quantify the degradation of estimation performance caused by distributing resources to multiple processors, which is only discussed for specific examples in existing works.

Monotonicity of the Quantum Relative Entropy Under Positive Maps

We prove that the quantum relative entropy decreases monotonically under the application of any positive trace-preserving linear map, for underlying separable Hilbert spaces. This answers in the affirmative a natural question that has been open for a long time, as monotonicity had previously only been shown to hold under additional assumptions, such as complete positivity or Schwarz-positivity of the adjoint map. The first step in our proof is to show monotonicity of the sandwiched Renyi divergences under positive trace-preserving maps, extending a proof of the data processing inequality by Beigi [J. Math. Phys. 54, 122202 (2013)] that is based on complex interpolation techniques. Our result calls into question several measures of non-Markovianity that have been proposed, as these would assess all positive trace-preserving time evolutions as Markovian.

Preservation of a quantum Rényi relative entropy implies existence of a recovery map

It is known that a necessary and sufficient condition for equality in the data processing inequality (DPI) for the quantum relative entropy is the existence of a recovery map. We show that equality in DPI for a sandwiched Rényi relative α-entropy with is also equivalent to this property. For the proof, we use an interpolating family of Lp-norms with respect to a state.

Data processing for the sandwiched R\xe9nyi divergence: a condition for equality

The alpha -sandwiched Rényi divergence satisfies the data processing inequality, i.e. monotonicity under quantum operations, for alpha ge 1/2. In this article, we derive a necessary and sufficient algebraic condition for equality in the data processing inequality for the alpha -sandwiched Rényi divergence for all alpha ge 1/2. For the range alpha in [1/2,1), our result provides the only condition for equality obtained thus far. To prove our result, we first consider the special case of partial trace and derive a condition for equality based on the original proof of the data processing inequality by Frank and Lieb (J Math Phys 54(12):122201, 2013) using a strict convexity/concavity argument. We then generalize to arbitrary quantum operations via the Stinespring Representation Theorem. As applications of our condition for equality in the data processing inequality, we deduce conditions for equality in various entropic inequalities. We formulate a Rényi version of the Araki–Lieb inequality and analyze the case of equality, generalizing a result by Carlen and Lieb (Lett Math Phys 101(1):1–11, 2012) about equality in the original Araki–Lieb inequality. Furthermore, we prove a general lower bound on a Rényi version of the entanglement of formation and observe that it is attained by states saturating the Rényi version of the Araki–Lieb inequality. Finally, we prove that the known upper bound on the entanglement fidelity in terms of the usual fidelity is saturated only by pure states.

Fluctuation sensitivity of a transcriptional signaling cascade.

The internal biochemical state of a cell is regulated by a vast transcriptional network that kinetically correlates the concentrations of numerous proteins. Fluctuations in protein concentration that encode crucial information about this changing state must compete with fluctuations caused by the noisy cellular environment in order to successfully transmit information across the network. Oftentimes, one protein must regulate another through a sequence of intermediaries, and conventional wisdom, derived from the data processing inequality of information theory, leads us to expect that longer sequences should lose more information to noise. Using the metric of mutual information to characterize the fluctuation sensitivity of transcriptional signaling cascades, we find, counter to this expectation, that longer chains of regulatory interactions can instead lead to enhanced informational efficiency. We derive an analytic expression for the mutual information from a generalized chemical kinetics model that we reduce to simple, mass-action kinetics by linearizing for small fluctuations about the basal biological steady state, and we find that at long times this expression depends only on a simple ratio of protein production to destruction rates and the length of the cascade. We place bounds on the values of these parameters by requiring that the mutual information be at least one bit-otherwise, any received signal would be indistinguishable from noise-and we find not only that nature has devised a way to circumvent the data processing inequality, but that it must be circumvented to attain this one-bit threshold. We demonstrate how this result places informational and biochemical efficiency at odds with one another by correlating high transcription factor binding affinities with low informational output, and we conclude with an analysis of the validity of our assumptions and propose how they might be tested experimentally.

Reconstructing directed gene regulatory network by only gene expression data.

BackgroundAccurately identifying gene regulatory network is an important task in understanding in vivo biological activities. The inference of such networks is often accomplished through the use of gene expression data. Many methods have been developed to evaluate gene expression dependencies between transcription factor and its target genes, and some methods also eliminate transitive interactions. The regulatory (or edge) direction is undetermined if the target gene is also a transcription factor. Some methods predict the regulatory directions in the gene regulatory networks by locating the eQTL single nucleotide polymorphism, or by observing the gene expression changes when knocking out/down the candidate transcript factors; regrettably, these additional data are usually unavailable, especially for the samples deriving from human tissues.ResultsIn this study, we propose the Context Based Dependency Network (CBDN), a method that is able to infer gene regulatory networks with the regulatory directions from gene expression data only. To determine the regulatory direction, CBDN computes the influence of source to target by evaluating the magnitude changes of expression dependencies between the target gene and the others with conditioning on the source gene. CBDN extends the data processing inequality by involving the dependency direction to distinguish between direct and transitive relationship between genes. We also define two types of important regulators which can influence a majority of the genes in the network directly or indirectly. CBDN can detect both of these two types of important regulators by averaging the influence functions of candidate regulator to the other genes. In our experiments with simulated and real data, even with the regulatory direction taken into account, CBDN outperforms the state-of-the-art approaches for inferring gene regulatory network. CBDN identifies the important regulators in the predicted network: 1. TYROBP influences a batch of genes that are related to Alzheimer’s disease; 2. ZNF329 and RB1 significantly regulate those ‘mesenchymal’ gene expression signature genes for brain tumors.ConclusionBy merely leveraging gene expression data, CBDN can efficiently infer the existence of gene-gene interactions as well as their regulatory directions. The constructed networks are helpful in the identification of important regulators for complex diseases.

Communication and Randomness Lower Bounds for Secure Computation

In secure multiparty computation (MPC), mutually distrusting users collaborate to compute a function of their private data without revealing any additional information about their data to other users. While it is known that information theoretically secure MPC is possible among $n$ users (connected by secure and noiseless links and have access to private randomness) against the collusion of less than $n/2$ users in the honest-but-curious model, relatively less is known about the communication and randomness complexity of secure computation. In this work, we employ information theoretic techniques to obtain lower bounds on the amount of communication and randomness required for secure MPC. We restrict ourselves to a concrete interactive setting involving 3 users under which all functions are securely computable against corruption of a single user in the honest-but-curious model. We derive lower bounds for both the perfect security case (i.e., zero-error and no leakage of information) and asymptotic security (where the probability of error and information leakage vanish as block-length goes to $\infty$). Our techniques include the use of a data processing inequality for residual information (i.e., the gap between mutual information and G\'acs-K\"orner common information), a new information inequality for 3-user protocols, and the idea of distribution switching. Our lower bounds are shown to be tight for various functions of interest. In particular, we show concrete functions which have "communication-ideal" protocols, i.e., which achieve the minimum communication simultaneously on all links in the network, and also use minimum amount of randomness. Also, we obtain the first explicit example of a function that incurs a higher communication cost than the input length in the secure computation model of "Feige, Kilian, and Naor [STOC, 1994]", who had shown that such functions exist.

Approximate reversibility in the context of entropy gain, information gain, and complete positivity

There are several inequalities in physics which limit how well we can process physical systems to achieve some intended goal, including the second law of thermodynamics, entropy bounds in quantum information theory, and the uncertainty principle of quantum mechanics. Recent results provide physically meaningful enhancements of these limiting statements, determining how well one can attempt to reverse an irreversible process. In this paper, we apply and extend these results to give strong enhancements to several entropy inequalities, having to do with entropy gain, information gain, entropic disturbance, and complete positivity of open quantum systems dynamics. Our first result is a remainder term for the entropy gain of a quantum channel. This result implies that a small increase in entropy under the action of a subunital channel is a witness to the fact that the channel's adjoint can be used as a recovery map to undo the action of the original channel. We apply this result to pure-loss, quantum-limited amplifier, and phase-insensitive quantum Gaussian channels, showing how a quantum-limited amplifier can serve as a recovery from a pure-loss channel and vice versa. Our second result regards the information gain of a quantum measurement, both without and with quantum side information. We find here that a small information gain implies that it is possible to undo the action of the original measurement if it is efficient. The result also has operational ramifications for the information-theoretic tasks known as measurement compression without and with quantum side information. Our third result shows that the loss of Holevo information caused by the action of a noisy channel on an input ensemble of quantum states is small if and only if the noise can be approximately corrected on average. We finally establish that the reduced dynamics of a system-environment interaction are approximately completely positive and trace preserving if and only if the data processing inequality holds approximately.

Strong Data Processing Inequalities and <inline-formula> <tex-math notation="LaTeX">$\Phi $ </tex-math> </inline-formula>-Sobolev Inequalities for Discrete Channels

The noisiness of a channel can be measured by comparing suitable functionals of the input and output distributions. For instance, the worst case ratio of output relative entropy to input relative entropy for all possible pairs of input distributions is bounded from above by unity, by the data processing theorem. However, for a fixed reference input distribution, this quantity may be strictly smaller than one, giving the so-called strong data processing inequalities (SDPIs). The same considerations apply to an arbitrary $\Phi $ -divergence. This paper presents a systematic study of optimal constants in the SDPIs for discrete channels, including their variational characterizations, upper and lower bounds, structural results for channels on product probability spaces, and the relationship between the SDPIs and the so-called $\Phi $ -Sobolev inequalities (another class of inequalities that can be used to quantify the noisiness of a channel by controlling entropy-like functionals of the input distribution by suitable measures of input–output correlation). Several applications to information theory, discrete probability, and statistical physics are discussed.

Power-Distortion Metrics for Path Planning Over Gaussian Sensor Networks

Path planning is an important component of autonomous mobile sensing systems. This paper studies upper and lower bounds of communication performance over Gaussian sensor networks, to drive power-distortion metrics for path planning problems. The Gaussian multiple-access channel is employed as a channel model and two source models are considered. In the first setting, the underlying source is estimated with minimum mean-squared-error, while in the second, reconstruction of a random spatial field is considered. For both problem settings, the upper and the lower bounds of sensor power-distortion curve are derived. For both settings, the upper bounds follow from the amplify-and-forward scheme and the lower bounds admit a unified derivation based on data processing inequality and tensorization property of the maximal correlation measure. Next, closed-form solutions of the optimal power allocation problems are obtained under a weighted sum-power constraint. The gap between the upper and the lower bounds is analyzed for both weighted sum and individual power constrained settings. Finally, these metrics are used to drive a path planning algorithm and the effects of power-distortion metrics, network parameters, and power optimization on the optimized path selection are analyzed.

Universal Steering Criteria.

We propose a general framework for constructing universal steering criteria that are applicable to arbitrary bipartite states and measurement settings of the steering party. The same framework is also useful for studying the joint measurement problem. Based on the data-processing inequality for an extended Rényi relative entropy, we then introduce a family of steering inequalities, which detect steering much more efficiently than those inequalities known before. As illustrations, we show unbounded violation of a steering inequality for assemblages constructed from mutually unbiased bases and establish an interesting connection between maximally steerable assemblages and complete sets of mutually unbiased bases. We also provide a single steering inequality that can detect all bipartite pure states of full Schmidt rank. In the course of study, we generalize a number of results intimately connected to data-processing inequalities, which are of independent interest.

Key Capacity for Product Sources With Application to Stationary Gaussian Processes

We show that for product sources, rate splitting is optimal for secret key agreement using limited one-way communication between two terminals. This yields an alternative proof of the tensorization property of a strong data processing inequality originally studied by Erkip and Cover and amended recently by Anantharam et al. We derive a water-filling solution of the communication-rate–key-rate tradeoff for a wide class of discrete memoryless vector Gaussian sources which subsumes the case without an eavesdropper. Moreover, we derive an explicit formula for the maximum secret key per bit of communication for all discrete memoryless vector Gaussian sources using a tensorization property and a variation on the enhanced channel technique of Weingarten et al. Finally, a one-shot information spectrum achievability bound for key generation is proved from which we characterize the communication-rate–key-rate tradeoff for stationary Gaussian processes.

Beyond complete positivity

We provide a general and consistent formulation for linear subsystem quantum dynamical maps, developed from a minimal set of postulates, primary among which is a relaxation of the usual, restrictive assumption of uncorrelated initial system-bath states. We describe the space of possibilities admitted by this formulation, namely that, far from being limited to only completely positive (CP) maps, essentially any $${\mathbb {C}}$$C-linear, Hermiticity-preserving, trace-preserving map can arise as a legitimate subsystem dynamical map from a joint unitary evolution of a system coupled to a bath. The price paid for this added generality is a trade-off between the set of admissible initial states and the allowed set of joint system-bath unitary evolutions. As an application, we present a simple example of a non-CP map constructed as a subsystem dynamical map that violates some fundamental inequalities in quantum information theory, such as the quantum data processing inequality.

Simulation of a Channel with Another Channel

In this paper, we study the problem of simulating a discrete memoryless channel (DMC) from another DMC under an average-case and an exact model. We present several achievability and infeasibility results, with tight characterizations in special cases. In particular, for the exact model, we fully characterize when a binary symmetric channel can be simulated from a binary erasure channel when there is no shared randomness. We also provide infeasibility and achievability results for the simulation of a binary channel from another binary channel in the case of no shared randomness. To do this, we use the properties of Renyi capacity of a given order. We also introduce a notion of “channel diameter” which is shown to be additive and satisfy a data processing inequality.

Dissipation of Information in Channels With Input Constraints

One of the basic tenets in information theory, the data processing inequality states that output divergence does not exceed the input divergence for any channel. For channels without input constraints, various estimates on the amount of such contraction are known, Dobrushin's coefficient for the total variation being perhaps the most well-known. This work investigates channels with average input cost constraint. It is found that while the contraction coefficient typically equals one (no contraction), the information nevertheless dissipates. A certain non-linear function, the \emph{Dobrushin curve} of the channel, is proposed to quantify the amount of dissipation. Tools for evaluating the Dobrushin curve of additive-noise channels are developed based on coupling arguments. Some basic applications in stochastic control, uniqueness of Gibbs measures and fundamental limits of noisy circuits are discussed. As an application, it shown that in the chain of $n$ power-constrained relays and Gaussian channels the end-to-end mutual information and maximal squared correlation decay as $\Theta(\frac{\log\log n}{\log n})$, which is in stark contrast with the exponential decay in chains of discrete channels. Similarly, the behavior of noisy circuits (composed of gates with bounded fan-in) and broadcasting of information on trees (of bounded degree) does not experience threshold behavior in the signal-to-noise ratio (SNR). Namely, unlike the case of discrete channels, the probability of bit error stays bounded away from $1\over 2$ regardless of the SNR.

Entropy bounds for quantum processes with initial correlations

Quantum technology is progressing towards fast quantum control over systems interacting with small environments. Hence such technologies are operating in a regime where the environment remembers the system's past, and the applicability of complete-positive trace preserving maps is no longer valid. The departure from complete positivity means many useful bounds, like entropy production, Holevo, and data processing inequality are no longer applicable to such systems. We address these issues by deriving a generalized bound for entropy valid for quantum dynamics with arbitrary system-environment correlations. We employ superchannels, which map quantum operations performed by the experimenter, represented in terms of completely positive maps, to states. Our bound has information-theoretic applications, as it generalizes the data processing inequality and the Holevo bound. We prove that both data processing inequality and the Holevo are valid even when system is correlated with the environment.

Quantum Rényi relative entropies affirm universality of thermodynamics.

We formulate a complete theory of quantum thermodynamics in the Rényi entropic formalism exploiting the Rényi relative entropies, starting from the maximum entropy principle. In establishing the first and second laws of quantum thermodynamics, we have correctly identified accessible work and heat exchange in both equilibrium and nonequilibrium cases. The free energy (internal energy minus temperature times entropy) remains unaltered, when all the entities entering this relation are suitably defined. Exploiting Rényi relative entropies we have shown that this "form invariance" holds even beyond equilibrium and has profound operational significance in isothermal process. These results reduce to the Gibbs-von Neumann results when the Rényi entropic parameter α approaches 1. Moreover, it is shown that the universality of the Carnot statement of the second law is the consequence of the form invariance of the free energy, which is in turn the consequence of maximum entropy principle. Further, the Clausius inequality, which is the precursor to the Carnot statement, is also shown to hold based on the data processing inequalities for the traditional and sandwiched Rényi relative entropies. Thus, we find that the thermodynamics of nonequilibrium state and its deviation from equilibrium together determine the thermodynamic laws. This is another important manifestation of the concepts of information theory in thermodynamics when they are extended to the quantum realm. Our work is a substantial step towards formulating a complete theory of quantum thermodynamics and corresponding resource theory.

A general framework for complete positivity

Complete positivity of quantum dynamics is often viewed as a litmus test for physicality, yet it is well known that correlated initial states need not give rise to completely positive evolutions. This observation spurred numerous investigations over the past two decades attempting to identify necessary and sufficient conditions for complete positivity. Here we describe a complete and consistent mathematical framework for the discussion and analysis of complete positivity for correlated initial states of open quantum systems. This formalism is built upon a few simple axioms and is sufficiently general to contain all prior methodologies going back to Pechakas, PRL (1994). The key observation is that initial system-bath states with the same reduced state on the system must evolve under all admissible unitary operators to system-bath states with the same reduced state on the system, in order to ensure that the induced dynamical maps on the system are well-defined. Once this consistency condition is imposed, related concepts like the assignment map and the dynamical maps are uniquely defined. In general, the dynamical maps may not be applied to arbitrary system states, but only to those in an appropriately defined physical domain. We show that the constrained nature of the problem gives rise to not one but three inequivalent types of complete positivity. Using this framework we elucidate the limitations of recent attempts to provide conditions for complete positivity using quantum discord and the quantum data-processing inequality. The problem remains open, and may require fresh perspectives and new mathematical tools. The formalism presented herein may be one step in that direction.