Asymptotic Equipartition Property Research Articles

Asymptotic Equipartition Property for a Markov Source Having Ambiguous Alphabet

Asymptotic Equipartition Property for a Markov Source Having Ambiguous Alphabet

The asymptotic equipartition property for a special Markov random field

The asymptotic equipartition property (AEP) plays an important role in establishing lossless source coding theorems and asymptotic coding theorems through the concepts of typical sets and typical sequences in information theory. In this paper, we study the generalized asymptotic equipartition property in the form of moving average for N bifurcating Markov chains indexed by an N-branch Cayley tree, which is a special case of Markov Urandom fields. Firstly, we construct a class of random variables containing a parameter with means of 1, and establish a strong limit theorem for the moving average of multivariate functions of such chains using the Borel–Cantelli lemma. Secondly, we present the strong law of large numbers for the frequencies of occurrence of states of the moving average, as well as the generalized asymptotic equipartition property for N bifurcating Markov chains indexed by an N-branch Cayley tree. As corollaries, we also generalize some known results.

Information theoretic perspective on sample complexity

The statistical supervised learning framework assumes an input–output set with a joint probability distribution that is reliably represented by the training dataset. The learning system is then required to output a prediction rule learned from the training dataset’s input–output pairs. In this work, we investigate the relationship between the sample complexity, the empirical risk and the generalization error based on the asymptotic equipartition property (AEP) (Shannon, 1948). We provide theoretical guarantees for reliable learning under the information-theoretic AEP, with respect to the generalization error and the sample size in different settings.

An Information-Theoretic Branch-and-Prune Algorithm for Discrete Phase Optimization of RIS in Massive MIMO

In this paper, we consider passive RIS-assisted multi-user communication between wireless nodes to improve the blocked line-of-sight (LOS) link performance. The wireless nodes are assumed to be equipped with Massive Multiple-Input Multiple-Output antennas, hybrid precoder, combiner, and low-resolution analog-to-digital converters (ADCs). We first derive the expression for the Cramer-Rao lower bound (CRLB) of the Mean Squared Error (MSE) of the received and combined signal at the intended receiver under interference. By appropriate design of the hybrid precoder, combiner, and RIS phase settings, it can be shown that the MSE achieves the CRLB. We further show that minimizing the MSE w.r.t. the phase settings of the RIS is equivalent to maximizing the throughput and energy efficiency of the system. We then propose a novel Information-Directed Branch-and-Prune (IDBP) algorithm to derive the phase settings of the RIS. We, for the first time in the literature, use an information-theoretic measure to decide on the pruning rules in a tree-search algorithm to arrive at the RIS phase-setting solution, which is vastly different compared to the traditional branch-and-bound algorithm that uses bounds of the cost function to define the pruning rules. In addition, we provide the theoretical guarantees of the near-optimality of the RIS phase-setting solution thus obtained using the Asymptotic Equipartition property. This also ensures near-optimal throughput and MSE performance.

Overcoming entropic limitations on asymptotic state transformations through probabilistic protocols

The quantum relative entropy is known to play a key role in determining the asymptotic convertibility of quantum states in general resource-theoretic settings, often constituting the unique monotone that is relevant in the asymptotic regime. We show that this is no longer the case when one allows stochastic protocols that may only succeed with some probability, in which case the quantum relative entropy is insufficient to characterize the rates of asymptotic state transformations, and a new entropic quantity based on a regularization of the Hilbert projective metric comes into play. Such a scenario is motivated by a setting where the cost associated with transformations of quantum states, typically taken to be the number of copies of a given state, is instead identified with the size of the quantum memory needed to realize the protocol. Our approach allows for constructing transformation protocols that achieve strictly higher rates than those imposed by the relative entropy. Focusing on the task of resource distillation, we give broadly applicable strong converse bounds on the asymptotic rates of probabilistic distillation protocols, and show them to be tight in relevant settings such as entanglement distillation with nonentangling operations. This generalizes and extends previously known limitations that are only applicable to deterministic protocols. Our methods are based on recent results for probabilistic one-shot transformations as well as a new asymptotic equipartition property for the projective relative entropy.

An information theory approach to network evolution models

Abstract A novel Markovian network evolution model is introduced and analysed by means of information theory. It will be proved that the model, called network evolution chain, is a stationary and ergodic stochastic process. Therefore, the asymptotic equipartition property can be applied to it. The model’s entropy rate and typical sequences are also explored. Extracting particular information from the network and methods to simulate network evolution in the continuous time domain are discussed. Additionally, the Erdős–Rényi network evolution chain is introduced as a subset of our model with the additional property of its stationary distribution matching the Erdős–Rényi random graph model. The stationary distributions of nodes and graphs are calculated for this subset alongside its entropy rate. The simulation results at the end of the article back up the proved theorems and calculated values.

Renewal Model for Dependent Binary Sequences

We suggest to construct infinite stochastic binary sequences by associating one of the two symbols of the sequence with the renewal times of an underlying renewal process. Focusing on stationary binary sequences corresponding to delayed renewal processes, we investigate correlations and the ability of the model to implement a prescribed autocovariance structure, showing that a large variety of subexponential decay of correlations can be accounted for. In particular, robustness and efficiency of the method are tested by generating binary sequences with polynomial and stretched-exponential decay of correlations. Moreover, to justify the maximum entropy principle for model selection, an asymptotic equipartition property for typical sequences that naturally leads to the Shannon entropy of the waiting time distribution is demonstrated. To support the comparison of the theory with data, a law of large numbers and a central limit theorem are established for the time average of general observables.

The refined quantum extremal surface prescription from the asymptotic equipartition property

Information-theoretic ideas have provided numerous insights in the progress of fundamental physics, especially in our pursuit of quantum gravity. In particular, the holographic entanglement entropy is a very useful tool in studying AdS/CFT, and its efficacy is manifested in the recent black hole page curve calculation. On the other hand, the one-shot information-theoretic entropies, such as the smooth min/max-entropies, are less discussed in AdS/CFT. They are however more fundamental entropy measures from the quantum information perspective and should also play pivotal roles in holography. We combine the technical methods from both quantum information and quantum gravity to put this idea on firm grounds. In particular, we study the quantum extremal surface (QES) prescription that was recently revised to highlight the significance of one-shot entropies in characterizing the QES phase transition. Motivated by the asymptotic equipartition property (AEP), we derive the refined quantum extremal surface prescription for fixed-area states via a novel AEP replica trick, demonstrating the synergy between quantum information and quantum gravity. We further prove that, when restricted to pure bulk marginal states, such corrections do not occur for the higher Rényi entropies of a boundary subregion in fixed-area states, meaning they always have sharp QES transitions. Our path integral derivation suggests that the refinement applies beyond AdS/CFT, and we confirm it in a black hole toy model by showing that the Page curve, for a black hole in a superposition of two radiation stages, receives a large correction that is consistent with the refined QES prescription.

Large deviations and information theory for sub-critical signal-to-interference-plus-noise ratio random network models

The article obtains large deviation asymptotic for sub-critical communication networks modelled as signal-interference-noise-ratio(SINR) random networks. To achieve this, we define the empirical power measure and the empirical connectivity measure, as well as prove joint large deviation principles(LDPs) for the two empirical measures on two different scales. Using the joint LDPs, we prove an asymptotic equipartition property(AEP) for wireless telecommunication networks modelled as the subcritical SINR random networks. Further, we prove a local large deviation principle(LLDP) for the sub-critical SINR random network. From the LLDPs, we prove the large deviation principle, and a classical McMillan Theorem for the stochastic SINR model processes. Note that, the LDPs for the empirical measures of this stochastic SINR random network model were derived on spaces of measures equipped with the τ- topology, and the LLDPs were deduced in the space of SINR model process without any topological limitations. We motivate the study by describing a possible anomaly detection test for SINR random networks.

Large deviations, asymptotic equipartition property for super-critical SINR random networks

In this article, we obtain large deviation asymptotic for supercritical social or communication networks modelled as signal-interference-noise ratio (SINR) graphs. To do this, we define the empirical power measure and the empirical connectivity measure, and prove joint large deviation principles (LDPs) for the two empirical measures on two different scales, i.e. λ and λ 2 aλ , where λ is the intensity measure of the Poisson point process (PPP) which defines the SINR random network and aλ real-valued sequence such that λ 2 aλ → ∞ as λ → ∞Using these joint LDPs we prove an asymptotic equipartition property for the super-critical networks modelled as the SINR random networks. Furthermore, we prove a local large deviation principle (LLDP) for the SINR random network. From the LLDP we prove a large deviation principle, and a classical McMillian theorem for the SINR random network processes. Note that, for a given empirical power measure and typical empirical connectivity measure, qπ ⊗ π, we can deduce from the LLDP a bound on the cardinality of the space of SINR networks to be approximately equal to , where the connectivity probability of the network, , satisfies and qπ ⊗ π is the typical behavior of the empirical connectivity measure. Observe, the LDPs for the empirical measures of SINR random networks were obtained on spaces of measures equipped with the τ-topology, and the LLDPs were obtained in the space of SINR network process without any topological restrictions.

Entropy of a quantum channel

The von Neumann entropy of a quantum state is a central concept in physics and information theory, having a number of compelling physical interpretations. There is a certain perspective that the most fundamental notion in quantum mechanics is that of a quantum channel, as quantum states, unitary evolutions, measurements, and discarding of quantum systems can each be regarded as certain kinds of quantum channels. Thus, an important goal is to define a consistent and meaningful notion of the entropy of a quantum channel. Motivated by the fact that the entropy of a state $\rho$ can be formulated as the difference of the number of physical qubits and the "relative entropy distance" between $\rho$ and the maximally mixed state, here we define the entropy of a channel $\mathcal{N}$ as the difference of the number of physical qubits of the channel output with the "relative entropy distance" between $\mathcal{N}$ and the completely depolarizing channel. We prove that this definition satisfies all of the axioms, recently put forward in [Gour, IEEE Trans. Inf. Theory 65, 5880 (2019)], required for a channel entropy function. The task of quantum channel merging, in which the goal is for the receiver to merge his share of the channel with the environment's share, gives a compelling operational interpretation of the entropy of a channel. The entropy of a channel can be negative for certain channels, but this negativity has an operational interpretation in terms of the channel merging protocol. We define Renyi and min-entropies of a channel and prove that they satisfy the axioms required for a channel entropy function. Among other results, we also prove that a smoothed version of the min-entropy of a channel satisfies the asymptotic equipartition property.

On typical encodings of multivariate ergodic sources

We show that the typical coordinate-wise encoding of multivariate ergodic source into prescribed alphabets has the entropy profile close to the convolution of the entropy profile of the source and the modular polymatroid that is determined by the cardinalities of the output alphabets. We show that the proportion of the exceptional encodings that are not close to the convolution goes to zero doubly exponentially. The result holds for a class of multivariate sources that satisfy asymptotic equipartition property described via the mean fluctuation of the information functions. This class covers asymptotically mean stationary processes with ergodic mean, ergodic processes, irreducible Markov chains with an arbitrary initial distribution. We also proved that typical encodings yield the asymptotic equipartition property for the output variables. These asymptotic results are based on an explicit lower bound of the proportion of encodings that transform a multivariate random variable into a variable with the entropy profile close to the suitable convolution.

Numerical finite-key analysis of quantum key distribution

Quantum key distribution (QKD) allows for secure communications safe against attacks by quantum computers. QKD protocols are performed by sending a sizeable, but finite, number of quantum signals between the distant parties involved. Many QKD experiments, however, predict their achievable key rates using asymptotic formulas, which assume the transmission of an infinite number of signals, partly because QKD proofs with finite transmissions (and finite-key lengths) can be difficult. Here we develop a robust numerical approach for calculating the key rates for QKD protocols in the finite-key regime in terms of two semi-definite programs (SDPs). The first uses the relation between conditional smooth min-entropy and quantum relative entropy through the quantum asymptotic equipartition property, and the second uses the relation between the smooth min-entropy and quantum fidelity. The numerical programs are formulated under the assumption of collective attacks from the eavesdropper and can be promoted to withstand coherent attacks using the postselection technique. We then solve these SDPs using convex optimization solvers and obtain numerical calculations of finite-key rates for several protocols difficult to analyze analytically, such as BB84 with unequal detector efficiencies, B92, and twin-field QKD. Our numerical approach democratizes the composable security proofs for QKD protocols where the derived keys can be used as an input to another cryptosystem.

Local large deviation principle, large deviation principle and information theory for the signal-to-interference-plus-noise ratio graph models

Given devices space D, an intensity measure λm ϵ (0, ∞), a transition kernel Q from the space D to positive real numbers ℝ+, a path-loss function (which depends on the Euclidean distance between the devices and a positive constant a), we define a Marked Poisson Point process (MPPP). For a given MPPP and technical constants , we define a Marked Signal-to- Interference and Noise Ratio (SINR) graph, and associate with it two empirical measures; the empirical marked measure and the empirical connectivity measure. For a class of marked SINR graphs, we prove a joint large deviation principle(LDP) for these empirical measures, with speed λ in the τ-topology. From the joint large deviation principle for the empirical marked measure and the empirical connectivity measure, we obtain an Asymptotic Equipartition Property(AEP) for network structured data modelled as a marked SINR graph. Specifically, we show that for large dense marked SINR graph one requires approximately about λ 2 H(Q × Q)/log 2 bits to transmit the information contained in the network with high probability, where H(Q × Q) is a properly defined entropy for the exponential transition kernel with parameter c. Further, we prove a local large deviation principle (LLDP) for the class of marked SINR graphs on D, where , with speed λ from a spectral potential point. From the LLDP we derive a conditional LDP for the marked SINR graphs. Note that, while the joint LDP is established in the τ-topology, the LLDP assume no topological restriction on the space of marked SINR graphs. Observe also that all our rate functions are expressed in terms of the relative entropy or the kullback action or divergence function of the marked SINR on the devices space D.

Entropy Accumulation

We ask the question whether entropy accumulates, in the sense that the operationally relevant total uncertainty about an n-partite system A = (A_1, ldots A_n) corresponds to the sum of the entropies of its parts A_i. The Asymptotic Equipartition Property implies that this is indeed the case to first order in n—under the assumption that the parts A_i are identical and independent of each other. Here we show that entropy accumulation occurs more generally, i.e., without an independence assumption, provided one quantifies the uncertainty about the individual systems A_i by the von Neumann entropy of suitably chosen conditional states. The analysis of a large system can hence be reduced to the study of its parts. This is relevant for applications. In device-independent cryptography, for instance, the approach yields essentially optimal security bounds valid for general attacks, as shown by Arnon-Friedman et al. (SIAM J Comput 48(1):181–225, 2019).

Subcarrier wave continuous variable quantum key distribution with discrete modulation: mathematical model and finite-key analysis

In this paper we report a continuous-variable quantum key distribution protocol using multimode coherent states generated on subcarrier frequencies of the optical spectrum. We propose a coherent detection scheme where power from a carrier wave is used as a local oscillator. We compose a mathematical model of the proposed scheme and perform its security analysis in the finite-size regime using fully quantum asymptotic equipartition property technique. We calculate a lower bound on the secret key rate for the system under the assumption that the quantum channel noise is negligible compared to detector dark counts, and an eavesdropper is restricted to collective attacks. Our calculation shows that the current realistic system implementation would allow distributing secret keys over channels with losses up to 9 dB.

Quantum Channel Simulation and the Channel’s Smooth Max-Information

We study the general framework of quantum channel simulation, that is, the ability of a quantum channel to simulate another one using different classes of codes. First, we show that the minimum error of simulation and the one-shot quantum simulation cost under no-signalling assisted codes are given by semidefinite programs. Second, we introduce the channel's smooth max-information, which can be seen as a one-shot generalization of the mutual information of a quantum channel. We provide an exact operational interpretation of the channel's smooth max-information as the one-shot quantum simulation cost under no-signalling assisted codes, which significantly simplifies the study of channel simulation and provides insights and bounds for the case under entanglement-assisted codes. Third, we derive the asymptotic equipartition property of the channel's smooth max-information; i.e., it converges to the quantum mutual information of the channel in the independent and identically distributed asymptotic limit. This implies the quantum reverse Shannon theorem in the presence of no-signalling correlations. Finally, we explore the simulation cost of various quantum channels.

Generalizations of Fano's Inequality for Conditional Information Measures via Majorization Theory.

Fano’s inequality is one of the most elementary, ubiquitous, and important tools in information theory. Using majorization theory, Fano’s inequality is generalized to a broad class of information measures, which contains those of Shannon and Rényi. When specialized to these measures, it recovers and generalizes the classical inequalities. Key to the derivation is the construction of an appropriate conditional distribution inducing a desired marginal distribution on a countably infinite alphabet. The construction is based on the infinite-dimensional version of Birkhoff’s theorem proven by Révész [Acta Math. Hungar. 1962, 3, 188–198], and the constraint of maintaining a desired marginal distribution is similar to coupling in probability theory. Using our Fano-type inequalities for Shannon’s and Rényi’s information measures, we also investigate the asymptotic behavior of the sequence of Shannon’s and Rényi’s equivocations when the error probabilities vanish. This asymptotic behavior provides a novel characterization of the asymptotic equipartition property (AEP) via Fano’s inequality.

How entropic regression beats the outliers problem in nonlinear system identification.

In this work, we developed a nonlinear System Identification (SID) method that we called Entropic Regression. Our method adopts an information-theoretic measure for the data-driven discovery of the underlying dynamics. Our method shows robustness toward noise and outliers, and it outperforms many of the current state-of-the-art methods. Moreover, the method of Entropic Regression overcomes many of the major limitations of the current methods such as sloppy parameters, diverse scale, and SID in high-dimensional systems such as complex networks. The use of information-theoretic measures in entropic regression has unique advantages, due to the Asymptotic Equipartition Property of probability distributions, that outliers and other low-occurrence events are conveniently and intrinsically de-emphasized as not-typical, by definition. We provide a numerical comparison with the current state-of-the-art methods in sparse regression, and we apply the methods to different chaotic systems such as the Lorenz System, the Kuramoto-Sivashinsky equations, and the Double-Well Potential.

A class of small deviation theorems for functionals of random fields on double Cayley tree in random environment

ABSTRACT In this paper, we introduce the definition of tree-indexed Markov chains in random environment with countable state space, and then study the realization of Markov chain indexed by a tree in random environment. Meanwhile, we establish a class of small deviation theorems for functional of random fields on double Cayley tree in random environment. Finally, We prove the asymptotic equipartition property (AEP) for tree indexed Markov chains in Markovian environment with finite state space.