Data Processing Inequality Research Articles

Inferring Gene Regulatory Networks Using the Improved Markov Blanket Discovery Algorithm.

Inferring gene regulatory networks (GRNs) from microarray data can help us understand the mechanisms of life and eventually develop effective therapies. Currently, many computational methods have been used in inferring GRNs. However, owing to high-dimensional data and small samples, these methods often tend to introduce redundant regulatory relationships. Therefore, a novel network inference method based on the improved Markov blanket discovery algorithm, IMBDANET, is proposed to infer GRNs. Specifically, for each target gene, data processing inequality was applied to the Markov blanket discovery algorithm for the accurate differentiation of direct regulatory genes from indirect regulatory genes. Finally, direct regulatory genes were used in constructing GRNs, and the network structure was optimized according to the importance degree score. Experimental results on six public network datasets show that the proposed method can be effectively used to infer GRNs.

A Refutation of Finite-State Language Models through Zipf's Law for Factual Knowledge.

We present a hypothetical argument against finite-state processes in statistical language modeling that is based on semantics rather than syntax. In this theoretical model, we suppose that the semantic properties of texts in a natural language could be approximately captured by a recently introduced concept of a perigraphic process. Perigraphic processes are a class of stochastic processes that satisfy a Zipf-law accumulation of a subset of factual knowledge, which is time-independent, compressed, and effectively inferrable from the process. We show that the classes of finite-state processes and of perigraphic processes are disjoint, and we present a new simple example of perigraphic processes over a finite alphabet called Oracle processes. The disjointness result makes use of the Hilberg condition, i.e., the almost sure power-law growth of algorithmic mutual information. Using a strongly consistent estimator of the number of hidden states, we show that finite-state processes do not satisfy the Hilberg condition whereas Oracle processes satisfy the Hilberg condition via the data-processing inequality. We discuss the relevance of these mathematical results for theoretical and computational linguistics.

The α → 1 limit of the sharp quantum Rényi divergence

Fawzi and Fawzi [Quantum 5, 387 (2021)] recently defined the sharp Rényi divergence, Dα#, for α ∈ (1, ∞), as an additional quantum Rényi divergence with nice mathematical properties and applications in quantum channel discrimination and quantum communication. One of their open questions was the limit α → 1 of this divergence. By finding a new expression of the sharp divergence in terms of a minimization of the geometric Rényi divergence, we show that this limit is equal to the Belavkin–Staszewski relative entropy. Analogous minimizations of arbitrary generalized divergences lead to a new family of generalized divergences that we call kringel divergences for which we prove various properties, including the data-processing inequality.

Rényi Relative Entropies and Noncommutative $$L_p$$-Spaces II

We study an extension of the sandwiched Rényi relative entropies for normal positive functionals on a von Neumann algebra, for parameter values $$\alpha \in [1/2,1)$$ . This work is intended as a continuation of Jenčová (Ann Henri Poincaré 19:2513–2542, 2018), where the values $$\alpha >1$$ were studied. We use the Araki–Masuda divergences of Berta et al. (Ann Henri Poincaré 9:1843–1867, 2018) and treat them in the framework of Kosaki’s noncommutative $$L_p$$ -spaces. Using the variational formula, recently obtained by F. Hiai, for $$\alpha \in [1/2,1)$$ , we prove the data processing inequality with respect to positive trace preserving maps and show that for $$\alpha \in (1/2,1)$$ , equality characterizes sufficiency (reversibility) for any 2-positive trace preserving map.

Directed Data-Processing Inequalities for Systems with Feedback.

We present novel data-processing inequalities relating the mutual information and the directed information in systems with feedback. The internal deterministic blocks within such systems are restricted only to be causal mappings, but are allowed to be non-linear and time varying, and randomized by their own external random input, can yield any stochastic mapping. These randomized blocks can for example represent source encoders, decoders, or even communication channels. Moreover, the involved signals can be arbitrarily distributed. Our first main result relates mutual and directed information and can be interpreted as a law of conservation of information flow. Our second main result is a pair of data-processing inequalities (one the conditional version of the other) between nested pairs of random sequences entirely within the closed loop. Our third main result introduces and characterizes the notion of in-the-loop (ITL) transmission rate for channel coding scenarios in which the messages are internal to the loop. Interestingly, in this case the conventional notions of transmission rate associated with the entropy of the messages and of channel capacity based on maximizing the mutual information between the messages and the output turn out to be inadequate. Instead, as we show, the ITL transmission rate is the unique notion of rate for which a channel code attains zero error probability if and only if such an ITL rate does not exceed the corresponding directed information rate from messages to decoded messages. We apply our data-processing inequalities to show that the supremum of achievable (in the usual channel coding sense) ITL transmission rates is upper bounded by the supremum of the directed information rate across the communication channel. Moreover, we present an example in which this upper bound is attained. Finally, we further illustrate the applicability of our results by discussing how they make possible the generalization of two fundamental inequalities known in networked control literature.

The α-z-Bures Wasserstein divergence

In this paper, we introduce the α-z-Bures Wasserstein divergence for positive semidefinite matrices A and B asΦ(A,B)=Tr((1−α)A+αB)−Tr(Qα,z(A,B)), where Qα,z(A,B)=(A1−α2zBαzA1−α2z)z is the matrix function in the α-z-Renyi relative entropy. We show that for 0≤α≤z≤1, the quantity Φ(A,B) is a quantum divergence and satisfies the Data Processing Inequality in quantum information. We also solve the least squares problem with respect to the new divergence. In addition, we show that the matrix power mean μ(t,A,B)=((1−t)Ap+tBp)1/p satisfies the in-betweenness property with respect to the α-z-Bures Wasserstein divergence.

On Non-Interactive Simulation of Binary Random Variables

We leverage proof techniques from discrete Fourier analysis and an existing result in coding theory to derive new bounds for the problem of non-interactive simulation of binary random variables. Previous bounds in the literature were derived by applying data processing inequalities concerning maximal correlation or hypercontractivity. We show that our bounds are sharp in some regimes. Indeed, for a specific instance of the problem parameters, our main result resolves an open problem posed by E. Mossel in 2017. As by-products of our analyses, various new properties of the average distance and distance enumerator of binary block codes are established.

Quantum capacity analysis of multi-level amplitude damping channels

Evaluating capacities of quantum channels is the first purpose of quantum Shannon theory, but in most cases the task proves to be very hard. Here, we introduce the set of Multi-level Amplitude Damping quantum channels as a generalization of the standard qubit Amplitude Damping Channel to quantum systems of finite dimension d. In the special case of d = 3, by exploiting degradability, data-processing inequalities, and channel isomorphism, we compute the associated quantum and private classical capacities for a rather wide class of maps, extending the set of models whose capacity can be computed known so far. We proceed then to the evaluation of the entanglement assisted quantum and classical capacities.

Devil in the details: Mechanistic variations impact information transfer across models of transcriptional cascades.

The transcriptional network determines a cell’s internal state by regulating protein expression in response to changes in the local environment. Due to the interconnected nature of this network, information encoded in the abundance of various proteins will often propagate across chains of noisy intermediate signaling events. The data-processing inequality (DPI) leads us to expect that this intracellular game of “telephone” should degrade this type of signal, with longer chains losing successively more information to noise. However, a previous modeling effort predicted that because the steps of these signaling cascades do not truly represent independent stages of data processing, the limits of the DPI could seemingly be surpassed, and the amount of transmitted information could actually increase with chain length. What that work did not examine was whether this regime of growing information transmission was attainable by a signaling system constrained by the mechanistic details of more complex protein-binding kinetics. Here we address this knowledge gap through the lens of information theory by examining a model that explicitly accounts for the binding of each transcription factor to DNA. We analyze this model by comparing stochastic simulations of the fully nonlinear kinetics to simulations constrained by the linear response approximations that displayed a regime of growing information. Our simulations show that even when molecular binding is considered, there remains a regime wherein the transmitted information can grow with cascade length, but ends after a critical number of links determined by the kinetic parameter values. This inflection point marks where correlations decay in response to an oversaturation of binding sites, screening informative transcription factor fluctuations from further propagation down the chain where they eventually become indistinguishable from the surrounding levels of noise.

From Blackwell Dominance in Large Samples to Rényi Divergences and Back Again

We study repeated independent Blackwell experiments; standard examples include drawing multiple samples from a population, or performing a measurement in different locations. In the baseline setting of a binary state of nature, we compare experiments in terms of their informativeness in large samples. Addressing a question due to Blackwell (1951), we show that generically an experiment is more informative than another in large samples if and only if it has higher Rényi divergences. We apply our analysis to the problem of measuring the degree of dissimilarity between distributions by means of divergences. A useful property of Rényi divergences is their additivity with respect to product distributions. Our characterization of Blackwell dominance in large samples implies that every additive divergence that satisfies the data processing inequality is an integral of Rényi divergences.

Understanding Convolutional Neural Networks With Information Theory: An Initial Exploration.

A novel functional estimator for Rényi's α -entropy and its multivariate extension was recently proposed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this brief, we first show that this estimator enables straightforward measurement of information flow in realistic convolutional neural networks (CNNs) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal more inner properties concerning CNN training.

Optimal extensions of resource measures and their applications

We develop a framework to extend resource measures from one domain to a larger one. We find that all extensions of resource measures are bounded between two quantities that we call the minimal and maximal extensions. We discuss various applications of our framework. We show that any relative entropy (i.e. an additive function on pairs of quantum states that satisfies the data processing inequality) must be bounded by the min and max relative entropies. We prove that the generalized trace distance, the generalized fidelity, and the purified distance are optimal extensions. And in entanglement theory we introduce a new technique to extend pure state entanglement measures to mixed bipartite states.

Bottleneck Problems: An Information and Estimation-Theoretic View.

Information bottleneck (IB) and privacy funnel (PF) are two closely related optimization problems which have found applications in machine learning, design of privacy algorithms, capacity problems (e.g., Mrs. Gerber’s Lemma), and strong data processing inequalities, among others. In this work, we first investigate the functional properties of IB and PF through a unified theoretical framework. We then connect them to three information-theoretic coding problems, namely hypothesis testing against independence, noisy source coding, and dependence dilution. Leveraging these connections, we prove a new cardinality bound on the auxiliary variable in IB, making its computation more tractable for discrete random variables. In the second part, we introduce a general family of optimization problems, termed “bottleneck problems”, by replacing mutual information in IB and PF with other notions of mutual information, namely f-information and Arimoto’s mutual information. We then argue that, unlike IB and PF, these problems lead to easily interpretable guarantees in a variety of inference tasks with statistical constraints on accuracy and privacy. While the underlying optimization problems are non-convex, we develop a technique to evaluate bottleneck problems in closed form by equivalently expressing them in terms of lower convex or upper concave envelope of certain functions. By applying this technique to a binary case, we derive closed form expressions for several bottleneck problems.

Fisher Information Under Local Differential Privacy

We develop data processing inequalities that describe how Fisher information from statistical samples can scale with the privacy parameter ε under local differential privacy constraints. These bounds are valid under general conditions on the distribution of the score of the statistical model, and they elucidate under which conditions the dependence on ε is linear, quadratic, or exponential. We show how these inequalities imply order-optimal lower bounds for private estimation for both the Gaussian location model and discrete distribution estimation for all levels of privacy ε >0. We further apply these inequalities to sparse Bernoulli models and demonstrate privacy mechanisms and estimators with order-matching squared ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> error.

Equality conditions of data processing inequality forα-zRényi relative entropies

The α–z Rényi relative entropies are a two-parameter family of Rényi relative entropies that are quantum generalizations of the classical α-Rényi relative entropies. In the work [Adv. Math. 365, 107053 (2020)], we decided the full range of (α, z) for which the data processing inequality (DPI) is valid. In this paper, we give algebraic conditions for the equality in DPI. For the full range of parameters (α, z), we give necessary conditions and sufficient conditions. For most parameters, we give equivalent conditions. This generalizes and strengthens the results of Leditzky et al. [Lett. Math. Phys. 107, 61–80 (2017)].

Conditional Entropy and Data Processing: An Axiomatic Approach Based on Core-Concavity

This work presents an axiomatization for entropy based on an extension of concavity called core-concavity. We show that core-concavity characterizes the largest class of functions for which the data-processing inequality holds, under the assumption that conditional entropy is defined as a generalized average. Also, under the same assumption, we show that data-processing and “conditioning reduces entropy” properties are equivalent. We prove several properties of core-concave functions, including generalization of perfect secrecy and of Fano's inequality. We also show that definitions of conditional entropy based on worst-case can be retrieved as limit cases of generalized averages. A connection between statistical decision making and this axiomatic approach is also presented.

A Path-Based Partial Information Decomposition.

Based on the conceptual basis of information theory, we propose a novel mutual information measure—‘path-based mutual information’. This information measure results from the representation of a set of random variables as a probabilistic graphical model. The edges in this graph are modeled as discrete memoryless communication channels, that is, the underlying data is ergodic, stationary, and the Markov condition is assumed to be applicable. The associated multilinear stochastic maps, tensors, transform source probability mass functions into destination probability mass functions. This allows for an exact expression of the resulting tensor of a cascade of discrete memoryless communication channels in terms of the tensors of the constituting communication channels in the paths. The resulting path-based information measure gives rise to intuitive, non-negative, and additive path-based information components—redundant, unique, and synergistic information—as proposed by Williams and Beer. The path-based redundancy satisfies the axioms postulated by Williams and Beer, the identity axiom postulated by Harder, and the left monotonicity axiom postulated Bertschinger. The ordering relations between redundancies of different joint collections of sources, as captured in the redundancy lattices of Williams and Beer, follow from the data processing inequality. Although negative information components can arise, we speculate that these either result from unobserved variables, or from adding additional sources that are statistically independent from all other sources to a system containing only non-negative information components. This path-based approach illustrates that information theory provides the concepts and measures for a partial information decomposition.

Information-theoretic aspects of the generalized amplitude-damping channel

The generalized amplitude damping channel (GADC) is one of the sources of noise in superconducting-circuit-based quantum computing. It can be viewed as the qubit analogue of the bosonic thermal channel, and it thus can be used to model lossy processes in the presence of background noise for low-temperature systems. In this work, we provide an information-theoretic study of the GADC. We first determine the parameter range for which the GADC is entanglement breaking and the range for which it is anti-degradable. We then establish several upper bounds on its classical, quantum, and private capacities. These bounds are based on data-processing inequalities and the uniform continuity of information-theoretic quantities, as well as other techniques. Our upper bounds on the quantum capacity of the GADC are tighter than the known upper bound reported recently in [Rosati et al., Nat. Commun. 9, 4339 (2018)] for the entire parameter range of the GADC, thus reducing the gap between the lower and upper bounds. We also establish upper bounds on the two-way assisted quantum and private capacities of the GADC. These bounds are based on the squashed entanglement, and they are established by constructing particular squashing channels. We compare these bounds with the max-Rains information bound, the mutual information bound, and another bound based on approximate covariance. For all capacities considered, we find that a large variety of techniques are useful in establishing bounds.

Amortized channel divergence for asymptotic quantum channel discrimination

It is well known that for the discrimination of classical and quantum channels in the finite, non-asymptotic regime, adaptive strategies can give an advantage over non-adaptive strategies. However, Hayashi [IEEE Trans. Inf. Theory 55(8), 3807 (2009)] showed that in the asymptotic regime, the exponential error rate for the discrimination of classical channels is not improved in the adaptive setting. We extend this result in several ways. First, we establish the strong Stein's lemma for classical-quantum channels by showing that asymptotically the exponential error rate for classical-quantum channel discrimination is not improved by adaptive strategies. Second, we recover many other classes of channels for which adaptive strategies do not lead to an asymptotic advantage. Third, we give various converse bounds on the power of adaptive protocols for general asymptotic quantum channel discrimination. Intriguingly, it remains open whether adaptive protocols can improve the exponential error rate for quantum channel discrimination in the asymmetric Stein setting. Our proofs are based on the concept of amortized distinguishability of quantum channels, which we analyse using data-processing inequalities.

On Relations Between the Relative Entropy and χ2-Divergence, Generalizations and Applications.

This paper is focused on a study of integral relations between the relative entropy and the chi-squared divergence, which are two fundamental divergence measures in information theory and statistics, a study of the implications of these relations, their information-theoretic applications, and some generalizations pertaining to the rich class of f-divergences. Applications that are studied in this paper refer to lossless compression, the method of types and large deviations, strong data–processing inequalities, bounds on contraction coefficients and maximal correlation, and the convergence rate to stationarity of a type of discrete-time Markov chains.