Information-theoretic Characterization Research Articles

An Information-Theoretic Characterization of MIMO-FAS: Optimization, Diversity-Multiplexing Tradeoff and q-Outage Capacity

An Information-Theoretic Characterization of MIMO-FAS: Optimization, Diversity-Multiplexing Tradeoff and <i>q</i>-Outage Capacity

Anomaly Detection Based on Compressed Data: An Information Theoretic Characterization

Anomaly Detection Based on Compressed Data: An Information Theoretic Characterization

Information-Theoretic Characterizations of Generalization Error for the Gibbs Algorithm

Information-Theoretic Characterizations of Generalization Error for the Gibbs Algorithm

The Shannon Information Capacity of an Arbitrary Radiating Surface: An Electromagnetic Approach

Utilizing a cross-disciplinary approach, we explore Shannon information-theoretic characterizations of the information capacity limits of generic electromagnetic (EM) surfaces intended for possible use in wireless communication links. Our principal task is to first formulate at a general and rigorous level the EM theory of the information that can be extracted from the Maxwellian fields radiated by an arbitrarily shaped continuous surface. This is then followed by a detailed derivation and illustration of practical physics-informed algorithms for computing approximations of the Shannon capacity of surfaces with any given geometry operating in Gaussian channels. Our formalism can address both near- and far-field information capacity scenarios, with a mathematical treatment that includes a complete characterization of the source-field polarization structure, mutual coupling, and interactions.

On the Computational Aspect of Coded Caching With Uncoded Prefetching

Coded caching is the distribution of content across a communication system using techniques from coding theory in order to create multicasting opportunities among the users receiving the content. This enables a multiplicative improvement over the classic uncoded caching with respect to the transmission rates required in the delivery phase of the content. Since its introduction, coded caching has eliceted significant research interest as a result of which several different schemes have been proposed over the last few years. This work focuses on the fundamental case of coded caching with uncoded prefetching. In this case, the users’ caches are filled with uncoded content during a prefetching phase in order to best serve the request made by each user during a subsequent delivery phase. This important case has recently received a complete information-theoretic characterization. However, reaching the information-theoretic optimality imposes a significant computational imbalance among the users. To mitigate this imbalance, we perform a complete computational analysis of the two major forms of coded caching with uncoded prefetching, namely centralized and decentralized. Furthermore, we propose a new information-theoretically optimal method for the delivery phase that achieves a significant computational improvement compared to the state of the art.

An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation

A new source model, which consists of an intrinsic state part and an extrinsic observation part, is proposed and its information-theoretic characterization, namely its rate-distortion function, is defined and analyzed. Such a source model is motivated by the recent surge of interest in the semantic aspect of information: the intrinsic state corresponds to the semantic feature of the source, which in general is not observable but can only be inferred from the extrinsic observation. There are two distortion measures, one between the intrinsic state and its reproduction, and the other between the extrinsic observation and its reproduction. Under a given code rate, the tradeoff between these two distortion measures is characterized by the rate-distortion function, which is solved via the indirect rate-distortion theory and is termed as the semantic rate-distortion function of the source. As an application of the general model and its analysis, the case of Gaussian extrinsic observation is studied, assuming a linear relationship between the intrinsic state and the extrinsic observation, under a quadratic distortion structure. The semantic rate-distortion function is shown to be the solution of a convex programming programming problem with respect to an error covariance matrix, and a reverse water-filling type of solution is provided when the model further satisfies a diagonalizability condition.

Sufficiently Informative and Relevant Features: An Information-Theoretic and Fourier-Based Characterization

A fundamental challenge in learning is the presence of nonlinear redundancies and dependencies in the data. To address this, we propose a Fourier-based approach to characterize feature redundancies, in unsupervised learning, and feature-label dependencies, in the supervised variant of the problem. We first develop a novel Fourier expansion for functions (more generally stochastic mappings) of correlated binary random variables. This is a generalization of the standard Fourier expansion on the Boolean cube beyond product probability spaces. As an important application of this analysis, we investigate learning with feature subset selection. In the unsupervised variant of this problem, we characterize feature redundancies via the Shannon entropy and group the features into sufficiently informative and redundant. Then, we make a connection to the proposed Fourier expansion and derive an upper bound on the joint entropy. Based on that, we propose a measure to quantify feature redundancies and present an unsupervised learning algorithm. We test our method on various real-world and synthetic datasets and demonstrate improvements on conventional unsupervised feature selection techniques. Then, we investigate the supervised feature subset selection and reformulate it in the Fourier domain. Bridging the Bayesian error rate with the Fourier coefficients, we demonstrate that the Fourier expansion provides a powerful tool to characterize nonlinear feature-label dependencies. Further, we introduce a computationally efficient measure for selecting relevant features. Via a theoretical analysis, we show that our proposed measure finds provably <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">asymptotically optimal</i> feature subsets. Lastly, we present an algorithm based on this measure and via numerical experiments demonstrate its improvements on various supervised feature selection algorithms.

Quantum Markov monogamy inequalities

Markovianity lies at the heart of communication problems. This in turn makes the information-theoretic characterization of Markov processes worthwhile. Data processing inequalities are ubiquitous in this sense, assigning necessary conditions for all Markov processes. We address here the problem of the information-theoretic analysis of constraints on Markov processes in the quantum regime. We show the existence of a novel class of quantum data processing inequalities called here quantum Markov monogamy inequalities. This new class of necessary conditions on quantum Markov processes is inspired by its counterpart for classical Markov processes, and thus providing a strong link between classical and quantum constraints on Markovianity. We go on to construct a family of multitime quantum Markov monogamy inequalities, based on the process tensor formalism and that exploits multitime correlations. We then show, by means of an explicit example, that the Markov monogamy inequalities can be stronger than the usual quantum data processing inequalities.

Information Theory as a Bridge Between Language Function and Language Form

Formal and functional theories of language seem disparate, because formal theories answer the question of what a language is, while functional theories answer the question of what functions it serves. We argue that information theory provides a bridge between these two approaches,viaa principle of minimization of complexity under constraints. Synthesizing recent work, we show how information-theoretic characterizations of functional complexity lead directly to mathematical descriptions of the forms of possible languages, in terms of solutions to constrained optimization problems. We show how certain linguistic descriptive formalisms can be recovered as solutions to such problems. Furthermore, we argue that information theory lets us define complexity in a way which has minimal dependence on the choice of theory or descriptive formalism. We illustrate this principle using recently-obtained results on universals of word and morpheme order.

Gomory-Hu Trees Over Wireless

The Gomory-Hu tree is a popular optimization algorithm that enables to efficiently find a min-cut (or equivalently, max-flow) for every pair of nodes in a graph. However, graphs cannot capture broadcasting and interference in wireless: over wireless networks we need to resort to the information-theoretical cut-set to bound the max-flow. Leveraging the submodularity of mutual information, we show that the Gomory-Hu algorithm can be used to efficiently find information-theoretic rate characterizations such as the capacity or an approximation to the capacity, over a number of network scenarios including wireless Gaussian networks and deterministic relay networks.

Causal Adversarial Channels With Feedback Snooping

With the advent of 5G and technologies such as cloud computing, Internet-of-Things (IoT), etc, future communication networks will consist of a large number of heterogeneous devices connected together. A critical aspect will be ensuring that communication is not only fast and reliable, but also resilient to malicious attack. As networks increasingly adopt zero-trust principles in their security frameworks, it is important to consider such attacks from a zero-trust perspective. Motivated by this, we consider the problem of communicating a message reliably across a binary erasure channel (BEC(<inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula>)) or a binary symmetric channel (BSC(<inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula>)) against an adversary actively injecting additional erasures or flips at the channel&#x2019;s input. The adversary can corrupt up to a fraction <inline-formula> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> of the transmitted bits and knows the transmission scheme agreed upon by the communicating terminals. Further, he has the capability to causally snoop in on both the transmitter and the receiver in real time, i.e., if <inline-formula> <tex-math notation="LaTeX">${\mathbf {x}}=(x_{1},x_{2},\ldots, x_{n})$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">${\mathbf {y}}=(y_{1},y_{2},\ldots, y_{n})$ </tex-math></inline-formula> denote the transmitted and the received codewords respectively, at each time <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>, he knows <inline-formula> <tex-math notation="LaTeX">$(x_{1},x_{2},\ldots, x_{k})$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$(y_{1},y_{2},\ldots, y_{k-1})$ </tex-math></inline-formula>. Using his side-information, the adversary is free to employ any attack that respects his constraint on the number of corruptible bits. We prove an information-theoretic tight capacity characterization as a function of <inline-formula> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula> for (i) the erasure adversary with a BEC(<inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula>) and (ii) the bit-flip adversary with a BSC(<inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula>). A unique feature of our models is the compounding of stochastic and adversarial noise sources. Our analysis reveals the worst-case adversarial attacks for both models and proves the existence of coding schemes that achieve rates equal to the capacity for any adversarial attack. In the case of bit-flips, we show that, interestingly, when <inline-formula> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> is below a certain threshold (that depends on <inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula>), the adversary is no worse than an i.i.d. memory-less noise source.

Information-theoretic characterization of the complete genotype-phenotype map of a complex pre-biotic world

How information is encoded in bio-molecular sequences is difficult to quantify since such an analysis usually requires sampling an exponentially large genetic space. Here we show how information theory reveals both robust and compressed encodings in the largest complete genotype-phenotype map (over 5 trillion sequences) obtained to date.

Embedding optimization reveals long-lasting history dependence in neural spiking activity.

Information processing can leave distinct footprints on the statistics of neural spiking. For example, efficient coding minimizes the statistical dependencies on the spiking history, while temporal integration of information may require the maintenance of information over different timescales. To investigate these footprints, we developed a novel approach to quantify history dependence within the spiking of a single neuron, using the mutual information between the entire past and current spiking. This measure captures how much past information is necessary to predict current spiking. In contrast, classical time-lagged measures of temporal dependence like the autocorrelation capture how long-potentially redundant-past information can still be read out. Strikingly, we find for model neurons that our method disentangles the strength and timescale of history dependence, whereas the two are mixed in classical approaches. When applying the method to experimental data, which are necessarily of limited size, a reliable estimation of mutual information is only possible for a coarse temporal binning of past spiking, a so-called past embedding. To still account for the vastly different spiking statistics and potentially long history dependence of living neurons, we developed an embedding-optimization approach that does not only vary the number and size, but also an exponential stretching of past bins. For extra-cellular spike recordings, we found that the strength and timescale of history dependence indeed can vary independently across experimental preparations. While hippocampus indicated strong and long history dependence, in visual cortex it was weak and short, while in vitro the history dependence was strong but short. This work enables an information-theoretic characterization of history dependence in recorded spike trains, which captures a footprint of information processing that is beyond time-lagged measures of temporal dependence. To facilitate the application of the method, we provide practical guidelines and a toolbox.

Information-theoretic characterization of eye-tracking signals with relation to cognitive tasks

Eye tracking is being increasingly used as a more powerful diagnosis instrument when compared with traditional pen-and-paper tests in psychopedagogy and psychology. This technology may significantly improve neurocognitive assessments in gathering indirect latent information about the subjects’ performance. However, the meaning and implications of these data are far from being fully understood. In this work, we present a comprehensive study of eye tracking time series in terms of statistical complexity measures. We registered the eye tracking movements of several subjects solving the two parts of the commonly applied Trail Making Test (TMT-A and TMT-B) and studied their Shannon entropy, disequilibrium, statistical complexity, and Fisher information with respect to three different probability distributions. The results show that these quantifiers reveal information about different features of the gaze depending on the distribution considered. As a meaningful result, we found that Fisher information in the position distribution reflects the difficulties encountered by the subject when solving the task. Such a characterization may be of interest to understand the underlying cognitive tasks performed by the subjects, and, additionally, it can serve as a source of valuable parameters to quantitatively assess how and why the subjects budget their attention, providing psychologists and psychopedagogues with more refined neuropsychological evaluation features and tools.

Quantum complementarity through entropic certainty principles

We approach the physical implications of the non-commutative nature of Complementary Observable Algebras (COA) from an information theoretic perspective. In particular, we derive a general \textit{entropic certainty principle} stating that the sum of two relative entropies, naturally related to the COA, is equal to the so-called algebraic index of the associated inclusion. Uncertainty relations then arise by monotonicity of the relative entropies that participate in the underlying entropic certainty. Examples and applications are described in quantum field theories with global symmetries, where the COA are formed by the charge-anticharge local operators (intertwiners) and the unitary representations of the symmetry group (twists), and in theories with local symmetries, where the COA are formed by Wilson and 't Hooft loops. In general, the entropic certainty principle naturally captures the physics of order/disorder parameters, a feature that makes it a generic handle for the information theoretic characterization of quantum phases.

Information-theoretic aspects of Werner states

In a seminal study of quantum states with Einstein–Podolsky–Rosen correlations (entanglement) admitting a hidden-variable model (Werner, 1989), Werner introduced the dichotomy of entanglement/separability and devised a family of highly symmetric states, now termed the Werner states, some of which exhibit entanglement but no Bell nonlocality. It turns out that the Werner states have a rich structure of correlations and constitute a paradigm which has played an innovative role in both theoretical and experimental explorations of quantum information. Given the theoretical significance and wide applications of the Werner states, here we first give a concise review of information contents of the Werner states, and then present an information-theoretic characterization of them in terms of the Wigner–Yanase skew information: The Werner states are identified as the states with the minimum quantum uncertainty with respect to a natural family of observables (i.e., the generators of the diagonal unitary group). For this purpose, we introduce a measure of quantum uncertainty which is of independent interest in studying asymmetry, coherence, and uncertainty, and reveal its fundamental properties. We further identify the Bell triplet states as the opposite states of the Werner states in the sense that they have the maximal amount of quantum uncertainty. Analogously, we provide a similar characterization of the isotropic states as the minimum quantum uncertainty states with respect to a closely related family of operators.

Tilting Together: An Information-Theoretic Characterization of Behavioral Roles in Rhythmic Dyadic Interaction.

Every joint collaborative physical activity performed by a group of people, e.g., carrying a table, typically leads to the emergence of spatiotemporal coordination of individual motor behavior. Such interpersonal coordination can arise solely based on the observation of the partners' and/or the object's movements, without the presence of verbal communication. In this paper, we investigate how the social coupling between two individuals in a collaborative task translates into measured objective and subjective performance indicators recorded in two different studies. We analyse the trends in the dyadic interrelationship based on the information-theoretic measure of transfer entropy and identify emerging leader-follower roles. In our experimental paradigm, the actions of the pair of subjects are continuously and seamlessly fused, resulting in a joint control of an object simulated on a tablet computer. Subjects need to synchronize their movements with a 90° phase difference in order to keep the object (a ball) rotating precisely on a predefined circular or elliptic trajectory on a tablet device. Results demonstrate how the identification of causal dependencies in this social interaction task could reveal specific trends in human behavior and provide insights into the emergence of social sensorimotor contingencies.

Quantifying Information Content in Multispectral Remote-Sensing Images Based on Image Transforms and Geostatistical Modelling

Quantifying information content in remote-sensing images is fundamental for information-theoretic characterization of remote sensing information processes, with the images being usually information sources. Information-theoretic methods, being complementary to conventional statistical methods, enable images and their derivatives to be described and analyzed in terms of information as defined in information theory rather than data per se. However, accurately quantifying images’ information content is nontrivial, as information redundancy due to spectral and spatial dependence needs to be properly handled. There has been little systematic research on this, hampering wide applications of information theory. This paper seeks to fill this important research niche by proposing a strategy for quantifying information content in multispectral images based on information theory, geostatistics, and image transformations, by which interband spectral dependence, intraband spatial dependence, and additive noise inherent to multispectral images are effectively dealt with. Specifically, to handle spectral dependence, independent component analysis (ICA) is performed to transform a multispectral image into one with statistically independent image bands (not spectral bands of the original image). The ICA-transformed image is further normal-transformed to facilitate computation of information content based on entropy formulas for Gaussian distributions. Normal transform facilitates straightforward incorporation of spatial dependence in entropy computation for the aforementioned double-transformed image bands with inter-pixel spatial correlation modeled via variograms. Experiments were undertaken using Landsat ETM+ and TM image subsets featuring different dominant land cover types (i.e., built-up, agricultural, and hilly). The experimental results confirm that the proposed methods provide more objective estimates of information content than otherwise when spectral dependence, spatial dependence, or non-normality is not accommodated properly. The differences in information content between image subsets obtained with ETM+ and TM were found to be about 3.6 bits/pixel, indicating the former’s greater information content. The proposed methods can be adapted for information-theoretic analyses of remote sensing information processes.

Information Theoretic Characterization of Uncertainty Distinguishes Surprise From Accuracy Signals in the Brain.

Uncertainty presents a problem for both human and machine decision-making. While utility maximization has traditionally been viewed as the motive force behind choice behavior, it has been theorized that uncertainty minimization may supersede reward motivation. Beyond reward, decisions are guided by belief, i.e., confidence-weighted expectations. Evidence challenging a belief evokes surprise, which signals a deviation from expectation (stimulus-bound surprise) but also provides an information gain. To support the theory that uncertainty minimization is an essential drive for the brain, we probe the neural trace of uncertainty-related decision variables, namely confidence, surprise, and information gain, in a discrete decision with a deterministic outcome. Confidence and surprise were elicited with a gambling task administered in a functional magnetic resonance imaging experiment, where agents start with a uniform probability distribution, transition to a non-uniform probabilistic state, and end in a fully certain state. After controlling for reward expectation, we find confidence, taken as the negative entropy of a trial, correlates with a response in the hippocampus and temporal lobe. Stimulus-bound surprise, taken as Shannon information, correlates with responses in the insula and striatum. In addition, we also find a neural response to a measure of information gain captured by a confidence error, a quantity we dub accuracy. BOLD responses to accuracy were found in the cerebellum and precuneus, after controlling for reward prediction errors and stimulus-bound surprise at the same time point. Our results suggest that, even absent an overt need for learning, the human brain expends energy on information gain and uncertainty minimization.

Thermal Quantum Spacetime

The intersection of thermodynamics, quantum theory and gravity has revealed many profound insights, all the while posing new puzzles. In this article, we discuss an extension of equilibrium statistical mechanics and thermodynamics potentially compatible with a key feature of general relativity, background independence; and we subsequently use it in a candidate quantum gravity system, thus providing a preliminary formulation of a thermal quantum spacetime. Specifically, we emphasise an information-theoretic characterisation of generalised Gibbs equilibrium that is shown to be particularly suited to background independent settings, and in which the status of entropy is elevated to being more fundamental than energy. We also shed light on its intimate connections with the thermal time hypothesis. Based on this, we outline a framework for statistical mechanics of quantum gravity degrees of freedom of combinatorial and algebraic type, and apply it in several examples. In particular, we provide a quantum statistical basis for the origin of covariant group field theories, shown to arise as effective statistical field theories of the underlying quanta of space in a certain class of generalised Gibbs states.