Information-theoretic Foundation Research Articles

Securing Angularly Dispersive Terahertz Links With Coding

With the large bandwidths available in the terahertz regime, directional transmissions can exhibit angular dispersion, i.e., frequency-dependent radiation direction. Unfortunately, angular dispersion introduces new security threats as increased bandwidth necessarily yields a larger signal footprint in the spatial domain and potentially benefits an eavesdropper. This paper is the first study of secure transmission strategies on angularly dispersive links. Based on information theoretic foundations, we propose a transmission strategy that channelizes the wideband transmission in frequency, and performs secure coding across frequency channels. With model-driven evaluations and over-the-air experiments, we show that the proposed method exploits the properties of angular dispersion to realize secure wideband transmissions, despite the increased signal footprint and even for practical irregular beams with side lobes and asymmetry. In contrast, without the proposed cross-channel coding strategy, angularly dispersive links can suffer from significant security degradation when bandwidth increases. In addition, we find that the security degradation due to bandwidth increment for angularly dispersive links is secondary compared to other factors including the selected secrecy rate or the directivity of the link. Nonetheless, we find that a higher angular dispersion level, i.e., a larger angular spread with the same bandwidth, results in a higher security degradation as bandwidth increases.

The information-theoretic foundation of thermodynamic work extraction

In this paper I demonstrate a novel distinction between work and heat, in terms of the distinguishability of states. Specifically, I show that if it is possible to extract different amounts of work deterministically from a physical system prepared in any one of a set of states, then those states must be distinguishable from one another. This result is formulated independently of scale and of particular dynamical laws; it also provides a novel connection between thermodynamics and information theory, established via the law of conservation of energy. This connection, unlike the well-known one between information and the second law, is exact, i.e., it does not depend on coarse-graining or ensemble approximation. Albeit compatible with these conclusions, existing thermodynamics approaches cannot provide a result of such generality, because they are scale-dependent (relying on ensembles or coarse-graining) or tied to particular dynamical laws. This result provides a foundation for thermodynamics that is both broader and more secure than existing ones, with implications for the theory of von Neumann’s universal constructor.

Editorial Issue on “Information Theoretic Foundations of Future Communication Systems”

Information theory, starting with Shannon’s groundbreaking work, has fundamentally shaped the way communication systems are designed and operated. Information theoretic principles form the underpinnings of modern communication networks. This issue explores how new advances in information theory can impact future communication systems. Several papers address issues at the heart of next generation wireless and wired networks: Multiple access, including access by a massive number of devices, multi-hop, large antenna arrays, communication security, and timeliness of information. Others consider new applications such as joint communication and sensing, communication for learning and inference, wireless imaging, and new storage mediums such as DNA, thereby providing the information theoretic foundations of modalities beyond human-to-human communications.

(Re)discovering Laws of Music Theory Using Information Lattice Learning

Information lattice learning (ILL) is a novel framework for knowledge discovery based on group-theoretic and information-theoretic foundations, which can rediscover the rules of music as known in the canon of music theory and also discover new rules that have remained unexamined. Such probabilistic rules are further demonstrated to be human-interpretable. ILL itself is a rediscovery and generalization of Shannon's lattice theory of information, where probability measures are not given but are learned from training data. This article explains the basics of the ILL framework, including both how to construct a lattice-structured abstraction universe that specifies the structural possibilities of rules, and how to find the most informative rules by performing statistical learning through an iterative student-teacher algorithmic architecture that optimizes information functionals. The ILL framework is finally shown to support both pedagogy and novel patterns of music co-creativity.

On Secure Degrees of Freedom of the MIMO Interference Channel With Local Output Feedback

This article studies the problem of Sum-secure Degrees of Freedom (SDoF) of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(M,M,N,N)$ </tex-math></inline-formula> multiple-input–multiple-output (MIMO) interference channel with local output feedback, so as to build an information-theoretic foundation and provide practical transmission schemes for 6G-enabled Vehicles-to-Vehicles (V2V). For this problem, we propose two novel transmission schemes, i.e., the interference decoding scheme and the interference alignment scheme, and thus establish a sum-SDoF lower bound. In particular, to optimize the phase duration, we analyze the security and decoding constraints and formulate a linear-fractional optimization problem. Furthermore, we show that the derived sum-SDoF lower bound is the sum-SDoF for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M \le N/2$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N=M$ </tex-math></inline-formula> , and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2N \le M$ </tex-math></inline-formula> antenna configurations, and reveal that for a fixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> , the optimal <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> to maximize the sum-SDoF is not less than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2N$ </tex-math></inline-formula> . Through simulations, we examine the secure sum-rate performance of proposed transmission schemes and reveal that using local output feedback can lead to a higher secure sum-rate than that by using delayed channel state information at the transmitter (CSIT).

Information Theory, Developmental Psychology, and the Baldwin Effect.

As part of the extended evolutionary synthesis, there has recently been a new emphasis on the effects of biological development on genetic inheritance and variation. The exciting new directions taken by those in the community have by a pre-history filled with related ideas that were never given a rigorous foundation or combined coherently. Part of the historical background of the extended synthesis is the work of James Mark Baldwin on his so-called “Baldwin Effect.” Many variant re-interpretations of his work obscure the original meaning of the Baldwin Effect. This paper emphasizes a new approach to the Baldwin Effect, focusing on his work in developmental psychology and how that would impact evolution. We propose a novel population genetics model of the Baldwin Effect. First, the impact of a kind of learning process motivated by motor babbling, in the developmental psychology literature, on evolution; second, that Information-theoretic phenotype reshaping speeds up evolution compared to populations without this kind of learning. The basic idea behind the model is to allow the organism to apply abstraction to his initial phenotype to situate it within one of a few different classes of phenotypes in the local neighborhood of a fitness maximum. The reshaping of the phenotype space thereby allows the organism to reach a nearby fitness maximum. By so doing, valleys in the fitness landscape are leveled out, making a rugged fitness landscape into a set of mesas and plateaus with increasing height. Using this model we can show the first sizeable speed-up for the Baldwin Effect compared to ordinary population genetics. We also introduce an information-theoretic foundation for the Baldwin Effect, which may be of independent interest.

Understanding quantum phenomena and quantum theories

In this paper I examine the extent to which recent work in information-theoretic foundations of quantum mechanics can be thought to facilitate understanding, either of quantum phenomena or quantum theory. To do so I utilize the modal view of understanding phenomena. I extend this view to develop an analysis of understanding of theories. The extended modal view of understanding provides a unified view of recent work in information-theoretic foundations of quantum mechanics and explains how it facilitates understanding.

Information Recovery in Shuffled Graphs via Graph Matching

While many multiple graph inference methodologies operate under the implicit assumption that an explicit vertex correspondence is known across the vertex sets of the graphs, in practice these correspondences may only be partially or errorfully known. Herein, we provide an information theoretic foundation for understanding the practical impact that errorfully observed vertex correspondences can have on subsequent inference, and the capacity of graph matching methods to recover the lost vertex alignment and inferential performance. Working in the correlated stochastic blockmodel setting, we establish a duality between the loss of mutual information due to an errorfully observed vertex correspondence and the ability of graph matching algorithms to recover the true correspondence across graphs. In the process, we establish a phase transition for graph matchability in terms of the correlation across graphs, and we conjecture the analogous phase transition for the relative information loss due to shuffling vertex labels. We demonstrate the practical effect that graph shuffling—and matching—can have on subsequent inference, with examples from two sample graph hypothesis testing and joint spectral graph clustering.

The Active Inference Approach to Ecological Perception: General Information Dynamics for Natural and Artificial Embodied Cognition

The emerging neurocomputational vision of humans as embodied, ecologically embedded, social agents—who shape and are shaped by their environment—offers a golden opportunity to revisit and revise ideas about the physical and information-theoretic underpinnings of life, mind, and consciousness itself. In particular, the active inference framework (AIF) makes it possible to bridge connections from computational neuroscience and robotics/AI to ecological psychology and phenomenology, revealing common underpinnings and overcoming key limitations. AIF opposes the mechanistic to the reductive, while staying fully grounded in a naturalistic and information-theoretic foundation, using the principle of free energy minimization. The latter provides a theoretical basis for a unified treatment of particles, organisms, and interactive machines, spanning from the inorganic to organic, non-life to life, and natural to artificial agents. We provide a brief introduction to AIF, then explore its implications for evolutionary theory, ecological psychology, embodied phenomenology, and robotics/AI research. We conclude the paper by considering implications for machine consciousness.

Maximum entropy estimation of environmental parameter distributions

This presentation regards origins and information theoretic foundations of maximum-entropy methods for estimating environmental parameter distributions. Environmental parameter distributions determine uncertainty and confidence levels in transmission loss inputs to mission planning and sonar tactical decision making tools. Given the scale of the oceans and their stochastic-dynamic nature, the sample base for estimation is usually limited relative to the required spatial and temporal resolution. Effective use of estimated environmental distributions for planning and decision making in this context requires well defined and meaningful methods of extrapolative estimation. The principle of maximum-entropy or minimal-presumption provides an important component of such methods. A complete definition of a maximum entropy method requires (1) definition of an application specific information or information base measure that counts the relative number of physically distinct possibilities in measurable subsets of the state space, and (2) a way of imposing minimally informative extrapolation on a combination of data samples, statistical hypotheses, and mathematical presumptions regarding things like smoothness of the underlying distribution. These components of distribution estimation and their interrelationships are discussed.

The Road to Novelty III: Symmetry Bridges

The polar information concept introduced in Part II, which consists of decision information (Bits & Bytes) and symmetry information (CoIn), will be positioned in Part III in historical-philosophical perspective against the background of the deliberations by D. Hume, I. Kant, E. Husserl, K. Popper, J. Habermas and K. Mainzer. In doing so, we assess the relevance of the concept with respect to the standpoints of these philosophers on the acquisition of knowledge (towards the New). As a first substantial result, we ascertain a convergence of approaches regarding symmetry-centred cognition (CoIn) in the historic works of Hume, Kant and Husserl. This emerging convergence was later on obstructed to lasting effect by Popper’s rejection of the ‘primacy of repetition’. As Mainzer shows, the concept of symmetry enjoys a significant renaissance in the natural sciences; however, as also evident from Mainzer’s work, its relevance for thought and understanding remains hidden from the specific perspective of these very sciences. It is only with more recent insights from cognitive science regarding categorisation, the use of metaphors, and the formation of analogies (G. Lakoff, S. Pinker, D. Hofstadter, E. Sander) that the discussion on the nature of insight is once again developing towards our extended information concept.Following the discussion of various approaches to understanding insight, some of them historic and some of them recent, a contrasting comparison of these approaches yields an insight that reaches beyond them. This insight points towards a core of essence that is common to all these approaches. The core materialises as the limit of convergence of these approaches on ever higher levels of abstraction and is provided by our information-theoretic CoIn foundation. On this ultimate level of abstraction, the double nature of the economic principle also re-emerges, which already loomed in Part II. Using these theoretical foundations, we are finally able to prove a fundamental asymmetry between the worlds of decisions and insights. This asymmetry implies that there can be no symmetry bridge between those two worlds and, because the limits to algorithmization are equivalent to the limits of the decision access, neither the decision approach nor any algorithmization can serve to access Novelty in the sense of access to cognition. We call this result the bridge theorem.

Network convergence: theory, architectures, and applications

With ever increasing demand for highly correlated video services in wireless networks, network information sources often carry vast redundancy in information contents, as exemplified by popular multimedia contents requested online by a large number of mobile users. The consequence is the waste of two precious resources - spectrum and energy. A good strategy is network convergence whereby commonly requested contents are transmitted via broadcast channels while personal contents are unicast by cellular networks. This article studies crucial issues arising from network convergence including its information-theoretic foundation, architectures, and feasible schemes for various application scenarios.

A conceptual holding model for veterinary applications.

Spatial references are required when geographical information systems (GIS) are used for the collection, storage and management of data. In the veterinary domain, the spatial component of a holding (of animals) is usually defined by coordinates, and no other relevant information needs to be interpreted or used for manipulation of the data in the GIS environment provided. Users trying to integrate or reuse spatial data organised in such a way, frequently face the problem of data incompatibility and inconsistency. The root of the problem lies in differences with respect to syntax as well as variations in the semantic, spatial and temporal representations of the geographic features. To overcome these problems and to facilitate the inter-operability of different GIS, spatial data must be defined according to a \"schema\" that includes the definition, acquisition, analysis, access, presentation and transfer of such data between different users and systems. We propose an application \"schema\" of holdings for GIS applications in the veterinary domain according to the European directive framework (directive 2007/2/EC--INSPIRE). The conceptual model put forward has been developed at two specific levels to produce the essential and the abstract model, respectively. The former establishes the conceptual linkage of the system design to the real world, while the latter describes how the system or software works. The result is an application \"schema\" that formalises and unifies the information-theoretic foundations of how to spatially represent a holding in order to ensure straightforward information-sharing within the veterinary community.

An information-theoretic approach for detecting communities in networks

Detecting communities in a network can be helpful to comprehend its structure and understand its function. The detecting-communities approach, in essence, is a model of classification and belongs to the scope of social methodology. In this paper, we view the community description of a network as a lossy compression of that network’s information, and develop an information-theoretic foundation accordingly for the concept of community in networks. We present an optimization model and its algorithm to identify the communities by finding an optimal compression of the network. We also illustrate the availability of this approach by an artificial example, compare its accuracy and algorithm complexity with the other classical approaches of this field by a series of simulated networks with different parameters, and demonstrate its application in a real-world network. The tests show that the proposed method is a good one for detecting the communities and finding the proper community number in the unweighted and weighted networks.

Information content per photon versus image fidelity in three-dimensional photon-counting integral imaging

Photon-counting integral imaging has been introduced recently, and its applications in three-dimensional (3D) object sensing, visualization, recognition, and classification under photon-starved conditions have been demonstrated. This paper sheds light on the underlying information-theoretic foundation behind the ability of photon-counting integral imaging in performing complex tasks with far fewer photons than conventional imaging systems. A metric for photon-information content is formulated in the context of 3D photon-counting imaging, and its properties are investigated. It is shown that there is an inherent trade-off between imaging fidelity, measured by the entropy-normalized mutual information associated with a given imaging system, and the amount of information in each photon used in the imaging process, as represented by the photon-number-normalized mutual information. The dependence of this trade-off on photon statistics, correlation in the 3D image, and the signal-to-noise ratio of the photon-detection system is also investigated.

Log-concavity, ultra-log-concavity, and a maximum entropy property of discrete compound Poisson measures

Sufficient conditions are developed, under which the compound Poisson distribution has maximal entropy within a natural class of probability measures on the nonnegative integers. Recently, one of the authors [O.T. Johnson, Log-concavity and the maximum entropy property of the Poisson distribution, Stochastic Process. Appl., 117(6) (2007) 791–802] used a semigroup approach to show that the Poisson has maximal entropy among all ultra-log-concave distributions with fixed mean. We show via a non-trivial extension of this semigroup approach that the natural analog of the Poisson maximum entropy property remains valid if the compound Poisson distributions under consideration are log-concave, but that it fails in general. A parallel maximum entropy result is established for the family of compound binomial measures. Sufficient conditions for compound distributions to be log-concave are discussed and applications to combinatorics are examined; new bounds are derived on the entropy of the cardinality of a random independent set in a claw-free graph, and a connection is drawn to Mason’s conjecture for matroids. The present results are primarily motivated by the desire to provide an information-theoretic foundation for compound Poisson approximation and associated limit theorems, analogous to the corresponding developments for the central limit theorem and for Poisson approximation. Our results also demonstrate new links between some probabilistic methods and the combinatorial notions of log-concavity and ultra-log-concavity, and they add to the growing body of work exploring the applications of maximum entropy characterizations to problems in discrete mathematics.

Understanding molecular recognition and epitope prediction from Information Theoretic approach

Background Cellular immunity is dependent on T-cell recognition of peptide/major histocompatibility complex (MHC) and is a critical molecular recognition component [1]. A large class of bioinformatics tools facilitates the identification of T-cell epitopes to specific MHC alleles. However, not all peptide residues contribute equally or are relevant to binding due to polymorphism of genes encoding MHC, making development of statistical methods difficult. Information Theory has proved to be one of the most universal mathematical theories that governs virtually all processes [2]. The success of this approach in analyzing a huge range of engineering, technological and natural processes is impressive. In Molecular Biology the applications have been very successful at the sequence level, many sequence comparison and binding site identification methods now boasts a sound information theoretic foundation.

Information Theoretic Foundations of Adaptive Coded Modulation

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In wireless/mobile communications, terminals adapt their rate and transmit power or, more in general, their coding and modulation scheme, depending on the time-varying channel conditions. This paper presents, in a tutorial form, the information theoretic framework underlying such “adaptive modulation” techniques. First, we review fading channel models, channel state information assumptions, and related capacity results. Then, we treat the case of input power constraint, where the optimal input distribution is Gaussian. Finally, we address the case of discrete modulations. In order to treat the latter, we make use of the recently developed method of “mercury-waterfilling,” based on the relationship between mutual information and minimum mean-square error (MMSE) estimation of the channel input from the channel output. While the traditional design of adaptive modulation schemes based on uncoded bit-error rate (BER) involves the optimization over a discrete set of signal constellations, when powerful (i.e., capacity approaching) coding schemes are used the corresponding adaptive coded modulation design becomes surprisingly simple. The regime of very powerful coding is justified by the use of modern coding schemes, such as turbo codes and low-density parity-check codes, able to perform close to channel capacity at very small BER. </para>

An information-theoretic framework for resolving community structure in complex networks.

To understand the structure of a large-scale biological, social, or technological network, it can be helpful to decompose the network into smaller subunits or modules. In this article, we develop an information-theoretic foundation for the concept of modularity in networks. We identify the modules of which the network is composed by finding an optimal compression of its topology, capitalizing on regularities in its structure. We explain the advantages of this approach and illustrate them by partitioning a number of real-world and model networks.

Hvorfor er sproget så svært at forstå, når det er så let at forstå?

In an attempt to unravel the intricate relationships between syntax and meaning, this paper clears a space between entrenched 'formalist' and 'functionalist' positions from which to defend a view of language as a 2nd order representational system. Building on information-theoretic foundations, it introduces a new first clause on top of the two clauses of Dretske’s (1995: xiii) Representational Thesis, thus - All linguistic facts are mental facts (new) - All mental facts are representational facts - All representational facts are facts about informational functions Concentrating on language interpretation, it argues that the proper place to look for the functionality of language is as a property of the Language Faculty as standardly conceived. It distinguishes 1°-representations (of situations) from 2°-representations (of linguistic structure), claiming that the input to the cognitive interface in language interpretation is a 1°-representation converted from a sequence of 2°-representations at an interface created by a default, abstract representation of the utterance situation. It argues for a purely representational, as opposed to derivational, approach. The syntactic area discussed is that of argument structure and selection, with special reference to the status of hidden arguments and their place in syntactic representations. It is argued that argument selection is independent of, and prior to, assignment of category, case and thematic roles to constituents, and is purely a matter of the selectional properties of predicates with respect to a parameter of optionality. Four constituent types are recognized, two argumental (Complements and Arguments), two non-argumental (Satellites and Parasites); two obligatory (Complements and Parasites),two optional (Arguments and Satellites).