Information-theoretic Inequalities Research Articles

Entropy of the Canonical Occupancy (Macro) State in the Quantum Measurement Theory.

The paper analyzes the probability distribution of the occupancy numbers and the entropy of a system at the equilibrium composed by an arbitrary number of non-interacting bosons. The probability distribution is obtained through two approaches: one involves tracing out the environment from a bosonic eigenstate of the combined environment and system of interest (the empirical approach), while the other involves tracing out the environment from the mixed state of the combined environment and system of interest (the Bayesian approach). In the thermodynamic limit, the two coincide and are equal to the multinomial distribution. Furthermore, the paper proposes to identify the physical entropy of the bosonic system with the Shannon entropy of the occupancy numbers, fixing certain contradictions that arise in the classical analysis of thermodynamic entropy. Finally, by leveraging an information-theoretic inequality between the entropy of the multinomial distribution and the entropy of the multivariate hypergeometric distribution, Bayesianism of information theory and empiricism of statistical mechanics are integrated into a common "infomechanical" framework.

An experimental study of neural estimators of the mutual information between random vectors modeling power spectrum features

Mutual information (MI) quantifies the statistical dependency between a pair of random variables and plays a central role in signal processing and data analysis. Recent advances in machine learning have enabled the estimation of MI from a dataset using the expressive power of neural networks. In this study, we conducted a comparative experimental analysis of several existing neural estimators of MI between random vectors that model power spectrum features. We explored alternative models of power spectrum features by leveraging information-theoretic data processing inequality and bijective transformations. Empirical results demonstrated that each neural estimator of MI covered in this study has its limitations. In practical applications, we recommend the collective use of existing neural estimators in a complementary manner for the problem of estimating MI between power spectrum features.

Thermal Area Law for Lattice Bosons

A physical system is said to satisfy a thermal area law if the mutual information between two adjacent regions in the Gibbs state is controlled by the area of their boundary. Lattice bosons have recently gained significant interest because they can be precisely tuned in experiments and bosonic codes can be employed in quantum error correction to circumvent classical no-go theorems. However, the proofs of many basic information-theoretic inequalities such as the thermal area law break down for bosons because their interactions are unbounded. Here, we rigorously derive a thermal area law for a class of bosonic Hamiltonians in any dimension which includes the paradigmatic Bose-Hubbard model. The main idea to go beyond bounded interactions is to introduce a quasi-free reference state with artificially decreased chemical potential by means of a double Peierls-Bogoliubov estimate.

A Model Randomization Approach to Statistical Parameter Privacy

In this article, we study a privacy filter design problem for a sequence of sensor measurements whose joint probability density function (p.d.f.) depends on a private parameter. To ensure parameter privacy, we propose a filter design framework which consists of two components: a randomizer and a nonlinear transformation. The randomizer takes the private parameter as input and randomly generates a pseudo parameter. The nonlinear mapping transforms the measurements such that the joint p.d.f. of the filter's output depends on the pseudo parameter rather than the private parameter. It also ensures that the joint p.d.f. of the filter's output belongs to the same family of distributions as that of the measurements. The design of the randomizer is formulated as an optimization problem subject to a privacy constraint, in terms of mutual information, and it is shown that the optimal randomizer is the solution of a convex optimization problem. Using information-theoretic inequalities, we show that the performance of any estimator of the private parameter, based on the output of the privacy filter, is limited by the privacy constraint. The structure of the nonlinear transformation is studied in the special cases of independent and identically distributed, Markovian, and Gauss-Markov measurements. Our results show that the privacy filter in the Gauss-Markov case can be implemented as two one-step ahead Kalman predictors and a set of minimum mean square error predictors. A numerical example on occupancy privacy in a building automation system illustrates the approach.

Information theoretic inequalities as bounds in superconformal field theory

In this paper, an information theoretic approach to bounds in superconformal field theories is proposed. It is proved that the supersymmetric Rényi entropy [Formula: see text] is a monotonically decreasing function of [Formula: see text] and [Formula: see text] is a concave function of [Formula: see text]. Under the assumption that the thermal entropy associated with the “replica trick” time circle is bounded from below by the charge at [Formula: see text], it is further proved that both [Formula: see text] and [Formula: see text] monotonically increase as functions of [Formula: see text]. Because [Formula: see text] enjoys universal relations with the Weyl anomaly coefficients in even-dimensional superconformal field theories, one therefore obtains a set of bounds on these coefficients by imposing the inequalities of [Formula: see text]. Some of the bounds coincide with Hofman–Maldacena bounds and the others are new. We also check the inequalities for examples in odd-dimensions.

A Theoretical Study on the Information Theoretic Inequalities and Fisher-Shannon Product of a Free Particle

In this article, the plane wave solution for a free particle in three dimensions is considered and the wave function is normalized in an arbitrarily large but finite cube. The momentum space wave function is obtained by taking the Fourier transform of the coordinate space wave function. The probability densities are employed to compute the numerical values of the information theoretic quantities such as Shannon information entropy (S), Fisher information entropy (I), Shannon power (J) and the Fisher–Shannon product (P) both in coordinate and momentum spaces for different values of the length (L) of the cubical box. Numerical values so found satisfy the Beckner, Bialynicki-Birula and Myceilski (BBM) inequality relation; Stam-Cramer-Rao inequalities (better known as the Fisher based uncertainty relation) and Fisher-Shannon product relation. This establishes the validity of the information theoretic inequalities in respect of the motion of a free particle.

A Randomized Filtering Strategy Against Inference Attacks on Active Steering Control Systems

In this paper, we develop a framework against inference attacks aimed at inferring the values of the controller gains of an active steering control system (ASCS). We first show that an adversary with access to the shared information by a vehicle, via a vehicular ad hoc network (VANET), can reliably infer the values of the controller gains of an ASCS. This vulnerability may expose the driver as well as the manufacturer of the ASCS to severe financial and safety risks. To protect controller gains of an ASCS against inference attacks, we propose a randomized filtering framework wherein the lateral velocity and yaw rate states of a vehicle are processed by a filter consisting of two components: a nonlinear mapping and a randomizer. The randomizer randomly generates a pair of pseudo gains which are different from the true gains of the ASCS. The nonlinear mapping performs a nonlinear transformation on the lateral velocity and yaw rate states. The nonlinear transformation is in the form of a dynamical system with a feedforward-feedback structure which allows real-time and causal implementation of the proposed privacy filter. The output of the filter is then shared via the VANET. The optimal design of randomizer is studied under a privacy constraint that determines the protection level of controller gains against inference attacks, and is in terms of mutual information. It is shown that the optimal randomizer is the solution of a convex optimization problem. By characterizing the distribution of the output of the filter, it is shown that the statistical distribution of the filter’s output depends on the pseudo gains rather than the true gains. Using information-theoretic inequalities, we analyze the inference ability of an adversary in estimating the control gains based on the output of the filter. Our analysis shows that the performance of any estimator in recovering the controller gains of an ASCS based on the output of the filter is limited by the privacy constraint. The performance of the proposed privacy filter is compared with that of an additive noise privacy mechanism. Our numerical results show that the proposed privacy filter significantly outperforms the additive noise mechanism, especially in the low distortion regime.

A note on [formula omitted]-separable codes and B2 codes

We derive a simple proof, based on information theoretic inequalities, of an upper bound on the largest rates of q-ary 2‾-separable codes that improves recent results of Wang for any q≥13. For the case q=2, we recover a result of Lindström, but with a much simpler derivation. The method easily extends to give bounds on B2 codes which, although not improving on Wang's results, use much simpler tools and might be useful for future applications.

RECENT RESULTS ON CARDINALITY ESTIMATION AND INFORMATION THEORETIC INEQUALITIES

I would like to dedicate this little exposition to Prof. Phan Dinh Dieu, one of the giants and pioneers of Mathematics in Computer Science in Vietnam. In the past 15 years or so, new and exciting connections between fundamental problems in database theory and information theory have emerged. There are several angles one can take to describe this connection. This paper takes one such angle, influenced by the author's own bias and research results. In particular, we will describe how the cardinality estimation problem -- a corner-stone problem for query optimizers -- is deeply connected to information theoretic inequalities. Furthermore, we explain how inequalities can also be used to derive a couple of classic geometric inequalities such as the Loomis-Whitney inequality. A purpose of the article is to introduce the reader to these new connections, where theory and practice meet in a wonderful way. Another objective is to point the reader to a research area with many new open questions.

Islands in cosmology

A quantum extremal island suggests that a region of spacetime is encoded in the quantum state of another system, like the encoding of the black hole interior in Hawking radiation. We study conditions for islands to appear in general spacetimes, with or without black holes. They must violate Bekenstein’s area bound in a precise sense, and the boundary of an island must satisfy several other information-theoretic inequalities. These conditions combine to impose very strong restrictions, which we apply to cosmological models. We find several examples of islands in crunching universes. In particular, in the four-dimensional FRW cosmology with radiation and a negative cosmological constant, there is an island near the turning point when the geometry begins to recollapse. In a two-dimensional model of JT gravity in de Sitter spacetime, there are islands inside crunches that are encoded at future infinity or inside bubbles of Minkowski spacetime. Finally, we discuss simple tensor network toy models for islands in cosmology and black holes.

Recovery map stability for the data processing inequality

Let be a finite dimensional von Neumann algebra and a von Neumann subalgebra of it. For states and on , let and be the corresponding states induced on . The data processing inequality implies that where is the relative entropy. Petz proved that there is equality if and only if , where is the Petz recovery map. We prove a quantitative version of Petz’s theorem. In it simplest form, our bound iswhere is the relative modular operator. Since , this yields a bound that is independent of . We also prove an analogous result with a more complicated constant in which the roles of and are interchanged on the right.Quantum information theoretic inequalities are usually much harder to prove, or differ from, their classical counterparts because classical proofs often rely on conditioning argument that do not carry over to the quantum setting. In particular, quantum conditional expectations rarely preserve expectations—something that always happens in the classical setting. We also prove a simple theorem characterizing states and subalgebras for which conditional expectations do preserve expectation with respect to , illuminating the quantum obstacle to the existence of nicely behaved conditional expectations and the origins the Petz recovery map.

Sample Complexity of Solving Non-Cooperative Games

This paper studies the complexity of solving two classes of non-cooperative games in a distributed manner, in which the players communicate with a set of system nodes over noisy communication channels. The complexity of solving each game class is defined as the minimum number of iterations required to find a Nash equilibrium (NE) of any game in that class with $\epsilon $ accuracy. First, we consider the class $\mathcal {G}$ of all $N$ -player non-cooperative games with a continuous action space that admit at least one NE. Using information-theoretic inequalities, a lower bound on the complexity of solving $\mathcal {G}$ is derived which depends on the Kolmogorov $2\epsilon $ -capacity of the constraint set and the total capacity of the communication channels. Our results indicate that the game class $\mathcal {G}$ can be solved at most exponentially fast. We next consider the class of all $N$ -player non-cooperative games with at least one NE such that the players’ utility functions satisfy a certain (differential) constraint. We derive lower bounds on the complexity of solving this game class under both Gaussian and non-Gaussian noise models. Finally, we derive upper and lower bounds on the sample complexity of a class of quadratic games. It is shown that the complexity of solving this game class scales according to $\Theta \left ({\frac {1}{\epsilon ^{2}}}\right)$ where $\epsilon $ is the accuracy parameter.

Orchestrating incentive designs to reduce adverse system-level effects of large-scale EV/PV adoption – The case of Portugal

The adoption of energy transition technologies for residential use is accelerated through incentive designs. The structure of such incentives affects technology adoption patterns, that is, the locations where new technologies are installed and used. These spatial adoption patterns influence network expansion costs and provide indication on potential cross-subsidization between population groups. While until today, most programs have been involuntarily favoring households with high-income and above-average educated population groups, incentive designs are currently under review. This paper presents a spatiotemporal technology adoption model that can predict adoption behavior of residential electric vehicle (EV) chargers and photovoltaic (PV) modules up to a predefined time horizon. A set of EV and PV adoption patterns for nine incentive design combinations are compared in order to assess potential synergies that may arise under orchestrated EV and PV adoption. Effects on adoption asymmetries are evaluated using an Information-Theoretic inequality metric. Results for Continental Portugal show that global network expansion costs can be reduced while minimizing technology adoption asymmetries, if specific incentive designs are combined.

Physical Layer Secret Key Generation in Static Environments

Two legitimate parties, referred to as Alice and Bob, wish to generate secret keys from the wireless channel in the presence of an eavesdropper, referred to as Eve, in order to use such keys for encryption and decryption. In general, the secret key rate highly depends on the coherence time of the channel. In particular, a straightforward method of generating secret keys in static environments results in ultra-low rates. In order to resolve this problem, we introduce a low-complexity method called induced randomness. In this method, Alice and Bob independently generate local randomness to be used together with the uniqueness of the wireless channel coefficients in order to enable high-rate secret key generation. In this work, two scenarios are considered: first, when Alice and Bob share a direct communication channel, and second, when Alice and Bob do not have a direct link and communicate through an untrusted relay. After exchanging the induced randomness, post-processing is done by Alice and Bob to generate highly-correlated samples that are used for the key generation. Such samples are then converted into bits, disparities between the sequences generated by Alice and Bob are mitigated, and the resulting sequences are then hashed to compensate for the information leakage to the eavesdropper and to allow consistency checking of the generated key bit sequences. We utilize semantic security measures and information-theoretic inequalities to upper bound the probability of successful eavesdropping attack in terms of the mutual information measures that can be numerically computed. Given certain reasonable system parameters this bound is numerically evaluated to be $2^{-31}$ and $2^{-10.57}$ in the first and the second scenario, respectively.

Comparison of information-theoretic products and inequalities for the lowest bound and unbound electronic states of [formula omitted] based on exact calculation

A study of information-theoretic properties of molecular bond on the example of the ground (bound) 1sσg and the lowest excited (unbound) 2pσu states of H2+ was conducted. Data about general information-theoretic measures, information products (Cramér-Rao, Fisher-Shannon information, and LMC complexity) and their inequalities in position and momentum spaces as a function of internuclear distance R=0.2–20 a.u. were obtained by an exact numerical calculation of the wave function. A systematic correlation between these information products was established. It was found that LMC complexity is most sensitive and detailed among these information products. A clear correlation between accuracy of determination of energy and information products in the simplest LCAO model compared to results of the exact calculation was observed.

The conditional entropy power inequality for quantum additive noise channels

We prove the quantum conditional entropy power inequality for quantum additive noise channels. This inequality lower bounds the quantum conditional entropy of the output of an additive noise channel in terms of the quantum conditional entropies of the input state and the noise when they are conditionally independent given the memory. We also show that this conditional entropy power inequality is optimal in the sense that we can achieve equality asymptotically by choosing a suitable sequence of Gaussian input states. We apply the conditional entropy power inequality to find an array of information-theoretic inequalities for conditional entropies which are the analogs of inequalities which have already been established in the unconditioned setting. Furthermore, we give a simple proof of the convergence rate of the quantum Ornstein-Uhlenbeck semigroup based on entropy power inequalities.

Evolution of quantum correlation undergoing decoherence using the information entropy inequality

Using the information-theoretic inequality proposed by Braunstein and Caves, we study the evolution of quantum correlation in the presence of different noisy channels. First, we experimentally prepare the high-fidelity two-bit entangled state and simulated the bit-flip and phase-shift noise using an all-optical experimental setup. Then, we study the quantum correlation evolution of a two-bit entangled system using the information difference of entropy under different noisy channels and noisy environments. It is found that the quantum correlation degradation depends on the type of noise and noisy channels. Our work provides a new essential tool for quantum information processing and measurement, and offers new insight into the evolution of quantum correlation in open systems.

On the Capacity Region of the Parallel Degraded Broadcast Channel With Three Receivers and Three-Degraded Message Sets

We consider a broadcast channel with three receivers and three-degraded message sets, i.e., the transmitter has a common message intended for all three receivers, a message intended for receivers 2 and 3, and a private message intended only for receiver 3. The messages are transmitted over a family of parallel degraded broadcast channels. In the most general case, the broadcast channel consists of the product of six parallel degraded broadcast channels, each with a different order of degradedness. We first consider an achievable rate region of Nair and El Gamal, by appropriately choosing independent input random variables and auxiliary random variables for each subchannel. We then show that the achievable rate region attains the capacity region for two different classes of such broadcast channels, one consisting of the product of five parallel degraded broadcast channels and another consisting of the product of three parallel degraded broadcast channels. To accomplish this, we make use of an information-theoretic inequality that may be proven using the Csiszar-sum identity. Next, we extend the result to the Gaussian case. We consider the aligned Gaussian MIMO broadcast channel consisting of the product of six parallel degraded Gaussian broadcast channels, where the Gaussian noise vectors for each of the users in each of the subchannels follow a degradedness order, i.e., the noise covariance matrices may be ordered in a positive semi-definite sense. We show that the Nair–El Gamal achievable rate region considered in this paper is maximized by Gaussian inputs. To prove that Gaussian inputs are optimal, we prove an extremal entropy inequality employing a new method recently introduced by Geng and Nair to prove the capacity region of the two-user Gaussian MIMO broadcast channel with common and private messages.

Holographic inequalities and entanglement of purification

We study the conjectured holographic duality between entanglement of purification and the entanglement wedge cross-section. We generalize both quantities and prove several information theoretic inequalities involving them. These include upper bounds on conditional mutual information and tripartite information, as well as a lower bound for tripartite information. These inequalities are proven both holographically and for general quantum states. In addition, we use the cyclic entropy inequalities to derive a new holographic inequality for the entanglement wedge cross-section, and provide numerical evidence that the corresponding inequality for the entanglement of purification may be true in general. Finally, we use intuition from bit threads to extend the conjecture to holographic duals of suboptimal purifications.

Mutual Information Reliability for Latent Class Analysis.

Latent class models are powerful tools in psychological and educational measurement. These models classify individuals into subgroups based on a set of manifest variables, assisting decision making in a diagnostic system. In this article, based on information theory, the authors propose a mutual information reliability (MIR) coefficient that summaries the measurement quality of latent class models, where the latent variables being measured are categorical. The proposed coefficient is analogous to a version of reliability coefficient for item response theory models and meets the general concept of measurement reliability in the Standards for Educational and Psychological Testing. The proposed coefficient can also be viewed as an extension of the McFadden's pseudo R-square coefficient, which evaluates the goodness-of-fit of logistic regression model, to latent class models. Thanks to several information-theoretic inequalities, the MIR coefficient is unitless, lies between 0 and 1, and receives good interpretation from a measurement point of view. The coefficient can be applied to both fixed and computerized adaptive testing designs. The performance of the MIR coefficient is demonstrated by simulated examples.