Information-theoretic Principles Research Articles

(Almost-)Quantum Bell Inequalities and Device-Independent Applications

Investigations of the boundary of the quantum correlation set have gained increased attention in recent years. This is done through the derivation of quantum Bell inequalities, which are related to Tsirelson's problem and have significant applications in device-independent (DI) information processing. However, determining quantum Bell inequalities is a notoriously difficult task and only isolated examples are known. In this paper, we present families of (almost-)quantum Bell inequalities and highlight four foundational and DI applications. Firstly, it is known that quantum correlations on the non-signaling boundary are of crucial importance in the task of DI randomness extraction from weak sources. In the practical Bell scenario of two players with two k-outcome measurements, we derive quantum Bell inequalities that demonstrate a separation between the quantum boundary and certain portions of the boundaries of the no-signaling polytope of dimension up to 4k−8, extending previous results from nonlocality distillation and the collapse of communication complexity. Secondly as an immediate by-product, we give a general proof of Aumann’s Agreement theorem for quantum systems as well as the almost-quantum correlations, which implies Aumann’s agreement theorem is a reasonable physical principle in the context of epistemics to pick out both quantum theory and almost-quantum correlations from general no-signaling theories. Thirdly, we present a family of quantum Bell inequalities in the two players with m binary measurements scenarios, that we prove serve to self-test the two-qubit singlet and the corresponding 2m measurements. Interestingly, this claim generalizes the result for m=2 discovered by Tsirelson-Landau-Masanes and shows an improvement over the state-of-the-art Device-Independent Randomness-Amplification (DIRA). Lastly, we use our quantum Bell inequalities to derive the general form of the principle of no advantage in nonlocal computation, which is an information-theoretic principle that serves to characterize the quantum correlation set.

Defending the quantum reconstruction program

The program of reconstructing quantum theory based on information-theoretic principles enjoys much popularity in the foundations of physics. Surprisingly, this endeavor has only received very little attention in philosophy. Here I argue that this should change. This is because, on the one hand, reconstructions can help us to better understand quantum mechanics, and, on the other hand, reconstructions are themselves in need of interpretation. My overall objective, thus, is to motivate the reconstruction program and to show why philosophers should care. My specific aims are threefold. (i) Clarify the relationship between reconstructing and interpreting quantum mechanics, (ii) show how the informational reconstruction of quantum theory puts pressure on standard realist interpretations, (iii) defend the quantum reconstruction program against possible objections.

Quantum Reconstructions as Stepping Stones Toward ψ-Doxastic Interpretations?

In quantum foundations, there is growing interest in the program of reconstructing the quantum formalism from clear physical principles. These reconstructions are formulated in an operational framework, deriving the formalism from information-theoretic principles. It has been recognized that this project is in tension with standard ψ-ontic interpretations. This paper presupposes that the quantum reconstruction program (QRP) (i) is a worthwhile project and (ii) puts pressure on ψ-ontic interpretations. Where does this leave us? Prima facie, it seems that ψ-epistemic interpretations perfectly fit the spirit of information-based reconstructions. However, ψ-epistemic interpretations, understood as saying that the wave functions represents one’s knowledge about a physical system, recently have been challenged on technical and conceptual grounds. More importantly, for some researchers working on reconstructions, the lesson of successful reconstructions is that the wave function does not represent objective facts about the world. Since knowledge is a factive concept, this speaks against epistemic interpretations. In this paper, I discuss whether ψ-doxastic interpretations constitute a reasonable alternative. My thesis is that if we want to engage QRP with ψ-doxastic interpretations, then we should aim at a reconstruction that is spelled out in non-factive experiential terms.

Variational Distillation for Multi-View Learning.

Information Bottleneck (IB) provides an information-theoretic principle for multi-view learning by revealing the various components contained in each viewpoint. This highlights the necessity to capture their distinct roles to achieve view-invariance and predictive representations but remains under-explored due to the technical intractability of modeling and organizing innumerable mutual information (MI) terms. Recent studies show that sufficiency and consistency play such key roles in multi-view representation learning, and could be preserved via a variational distillation framework. But when it generalizes to arbitrary viewpoints, such strategy fails as the mutual information terms of consistency become complicated. This paper presents Multi-View Variational Distillation (MV 2D), tackling the above limitations for generalized multi-view learning. Uniquely, MV 2D can recognize useful consistent information and prioritize diverse components by their generalization ability. This guides an analytical and scalable solution to achieving both sufficiency and consistency. Additionally, by rigorously reformulating the IB objective, MV 2D tackles the difficulties in MI optimization and fully realizes the theoretical advantages of the information bottleneck principle. We extensively evaluate our model on diverse tasks to verify its effectiveness, where the considerable gains provide key insights into achieving generalized multi-view representations under a rigorous information-theoretic principle.

Bias in the arrival of variation can dominate over natural selection in Richard Dawkins's biomorphs.

Biomorphs, Richard Dawkins's iconic model of morphological evolution, are traditionally used to demonstrate the power of natural selection to generate biological order from random mutations. Here we show that biomorphs can also be used to illustrate how developmental bias shapes adaptive evolutionary outcomes. In particular, we find that biomorphs exhibit phenotype bias, a type of developmental bias where certain phenotypes can be many orders of magnitude more likely than others to appear through random mutations. Moreover, this bias exhibits a strong preference for simpler phenotypes with low descriptional complexity. Such bias towards simplicity is formalised by an information-theoretic principle that can be intuitively understood from a picture of evolution randomly searching in the space of algorithms. By using population genetics simulations, we demonstrate how moderately adaptive phenotypic variation that appears more frequently upon random mutations can fix at the expense of more highly adaptive biomorph phenotypes that are less frequent. This result, as well as many other patterns found in the structure of variation for the biomorphs, such as high mutational robustness and a positive correlation between phenotype evolvability and robustness, closely resemble findings in molecular genotype-phenotype maps. Many of these patterns can be explained with an analytic model based on constrained and unconstrained sections of the genome. We postulate that the phenotype bias towards simplicity and other patterns biomorphs share with molecular genotype-phenotype maps may hold more widely for developmental systems.

Pathway analysis through mutual information.

In pathway analysis, we aim to establish a connection between the activity of a particular biological pathway and a difference in phenotype. There are many available methods to perform pathway analysis, many of them rely on an upstream differential expression analysis, and many model the relations between the abundances of the analytes in a pathway as linear relationships. Here, we propose a new method for pathway analysis, MIPath, that relies on information theoretical principles and, therefore, does not model the association between pathway activity and phenotype, resulting in relatively few assumptions. For this, we construct a graph of the data points for each pathway using a nearest-neighbor approach and score the association between the structure of this graph and the phenotype of these same samples using Mutual Information while adjusting for the effects of random chance in each score. The initial nearest neighbor approach evades individual gene-level comparisons, hence making the method scalable and less vulnerable to missing values. These properties make our method particularly useful for single-cell data. We benchmarked our method on several single-cell datasets, comparing it to established and new methods, and found that it produces robust, reproducible, and meaningful scores. Source code is available at https://github.com/statisticalbiotechnology/mipath, or through Python Package Index as "mipathway."

Generative and Discriminative Learning for Lung X-Ray Analysis Based on Probabilistic Component Analysis.

The paper presents a hybrid generative/discriminative classification method aimed at identifying abnormalities, such as cancer, in lung X-ray images. The proposed method involves a generative model that performs generative embedding in Probabilistic Component Analysis (PrCA). The primary goal of PrCA is to model co-existing information within a probabilistic framework, with the intent to locate the feature vector space for X-ray data based on a defined kernel structure. A kernel-based classifier, grounded in information-theoretic principles, was employed in this study. The performance of the proposed method is evaluated against nearest neighbour (NN) classifiers and support vector machine (SVM) classifiers, which use a diagonal covariance matrix and incorporate normal linear and non-linear kernels, respectively. The method is found to achieve superior accuracy, offering a viable solution to the class of problems presented. Accuracy rates achieved by the kernels in the NN and SVM models were 95.02% and 92.45%, respectively, suggesting the method's competitiveness with state-of-the-art approaches.

Information-theoretic principles in incremental language production

I apply a recently emerging perspective on the complexity of action selection, the rate-distortion theory of control, to provide a computational-level model of errors and difficulties in human language production, which is grounded in information theory and control theory. Language production is cast as the sequential selection of actions to achieve a communicative goal subject to a capacity constraint on cognitive control. In a series of calculations, simulations, corpus analyses, and comparisons to experimental data, I show that the model directly predicts some of the major known qualitative and quantitative phenomena in language production, including semantic interference and predictability effects in word choice; accessibility-based ("easy-first") production preferences in word order alternations; and the existence and distribution of disfluencies including filled pauses, corrections, and false starts. I connect the rate-distortion view to existing models of human language production, to probabilistic models of semantics and pragmatics, and to proposals for controlled language generation in the machine learning and reinforcement learning literature.

NaRnEA: An Information Theoretic Framework for Gene Set Analysis

Gene sets are being increasingly leveraged to make high-level biological inferences from transcriptomic data; however, existing gene set analysis methods rely on overly conservative, heuristic approaches for quantifying the statistical significance of gene set enrichment. We created Nonparametric analytical-Rank-based Enrichment Analysis (NaRnEA) to facilitate accurate and robust gene set analysis with an optimal null model derived using the information theoretic Principle of Maximum Entropy. By measuring the differential activity of ~2500 transcriptional regulatory proteins based on the differential expression of each protein's transcriptional targets between primary tumors and normal tissue samples in three cohorts from The Cancer Genome Atlas (TCGA), we demonstrate that NaRnEA critically improves in two widely used gene set analysis methods: Gene Set Enrichment Analysis (GSEA) and analytical-Rank-based Enrichment Analysis (aREA). We show that the NaRnEA-inferred differential protein activity is significantly correlated with differential protein abundance inferred from independent, phenotype-matched mass spectrometry data in the Clinical Proteomic Tumor Analysis Consortium (CPTAC), confirming the statistical and biological accuracy of our approach. Additionally, our analysis crucially demonstrates that the sample-shuffling empirical null models leveraged by GSEA and aREA for gene set analysis are overly conservative, a shortcoming that is avoided by the newly developed Maximum Entropy analytical null model employed by NaRnEA.

Advantages of Multicopy Nonlocality Distillation and Its Application to Minimizing Communication Complexity.

Nonlocal correlations are a central feature of quantum theory, and understanding why quantum theory has a limited amount of nonlocality is a fundamental problem. Since nonlocality also has technological applications, e.g., for device-independent cryptography, it is useful to understand it as a resource and, in particular, whether and how different types of nonlocality can be interconverted. Here we focus on nonlocality distillation which involves using several copies of a nonlocal resource to generate one with more nonlocality. We introduce several distillation schemes which distill an extended part of the set of nonlocal correlations including quantum correlations. Our schemes are based on a natural set of operational procedures known as wirings that can be applied regardless of the underlying theory. Some are sequential algorithms that repeatedly use a two-copy protocol, while others are genuine three-copy distillation protocols. In some regions we prove that genuine three-copy protocols are strictly better than two-copy protocols. By applying our new protocols we also increase the region in which nonlocal correlations are known to give rise to trivial communication complexity. This brings us closer to an understanding of the sets of nonlocal correlations that can be recovered from information-theoretic principles, which, in turn, enhances our understanding of what is special about quantum theory.

Bow-tie architectures in biological and artificial neural networks: Implications for network evolution and assay design

Modern artificial neural networks (ANNs) have long been designed on foundations of mathematics as opposed to their original foundations of biomimicry. However, the structure and function of these modern ANNs are often analogous to real-life biological networks. We propose that the ubiquitous information-theoretic principles underlying the development of ANNs are similar to the principles guiding the macro-evolution of biological networks and that insights gained from one field can be applied to the other. We generate hypotheses on the bow-tie network structure of the Janus kinase - signal transducers and activators of transcription (JAK-STAT) pathway, additionally informed by the evolutionary considerations, and carry out ANN simulation experiments to demonstrate that an increase in the network's input and output complexity does not necessarily require a more complex intermediate layer. This observation should guide novel biomarker discovery-namely, to prioritize sections of the biological networks in which information is most compressed as opposed to biomarkers representing the periphery of the network.

Hierarchically structured task-agnostic continual learning

One notable weakness of current machine learning algorithms is the poor ability of models to solve new problems without forgetting previously acquired knowledge. The Continual Learning paradigm has emerged as a protocol to systematically investigate settings where the model sequentially observes samples generated by a series of tasks. In this work, we take a task-agnostic view of continual learning and develop a hierarchical information-theoretic optimality principle that facilitates a trade-off between learning and forgetting. We derive this principle from a Bayesian perspective and show its connections to previous approaches to continual learning. Based on this principle, we propose a neural network layer, called the Mixture-of-Variational-Experts layer, that alleviates forgetting by creating a set of information processing paths through the network which is governed by a gating policy. Equipped with a diverse and specialized set of parameters, each path can be regarded as a distinct sub-network that learns to solve tasks. To improve expert allocation, we introduce diversity objectives, which we evaluate in additional ablation studies. Importantly, our approach can operate in a task-agnostic way, i.e., it does not require task-specific knowledge, as is the case with many existing continual learning algorithms. Due to the general formulation based on generic utility functions, we can apply this optimality principle to a large variety of learning problems, including supervised learning, reinforcement learning, and generative modeling. We demonstrate the competitive performance of our method on continual reinforcement learning and variants of the MNIST, CIFAR-10, and CIFAR-100 datasets.

Variable expression of linguistic laws in ape gesture: a case study from chimpanzee sexual solicitation.

Two language laws have been identified as consistent patterns shaping animal behaviour, both acting on the organizational level of communicative systems. Zipf's law of brevity describes a negative relationship between behavioural length and frequency. Menzerath's law defines a negative correlation between the number of behaviours in a sequence and average length of the behaviour composing it. Both laws have been linked with the information-theoretic principle of compression, which tends to minimize code length. We investigated their presence in a case study of male chimpanzee sexual solicitation gesture. We failed to find evidence supporting Zipf's law of brevity, but solicitation gestures followed Menzerath's law: longer sequences had shorter average gesture duration. Our results extend previous findings suggesting gesturing may be limited by individual energetic constraints. However, such patterns may only emerge in sufficiently large datasets. Chimpanzee gestural repertoires do not appear to manifest a consistent principle of compression previously described in many other close-range systems of communication. Importantly, the same signallers and signals were previously shown to adhere to these laws in subsets of the repertoire when used in play; highlighting that, in addition to selection on the signal repertoire, ape gestural expression appears shaped by factors in the immediate socio-ecological context.

Robust cross-network node classification via constrained graph mutual information

The recent methods for cross-network node classification mainly exploit graph neural networks (GNNs) as feature extractor to learn expressive graph representations across the source and target graphs. However, GNNs are vulnerable to noisy factors, such as adversarial attacks or perturbations on the node features or graph structure, which can cause a significant negative impact on their learning performance. To this end, we propose a robust graph domain adaptive learning framework RGDAL which exploits an information-theoretic principle to filter the noisy factors for cross-network node classification. Specifically, RGDAL utilizes graph convolutional network (GCN) with constrained graph mutual information and an adversarial learning component to learn noise-resistant and domain-invariant graph representations. To overcome the difficulties of estimating the mutual information for the non independent and identically distributed (non-i.i.d.) graph structured data, we design a dynamic neighborhood sampling strategy that can discretize the graph and incorporate the graph structural information for mutual information estimation. Experimental results on two real-world graph datasets demonstrate that RGDAL shows better robustness for cross-network node classification compared with the SOTA graph adaptive learning methods.

Elastic Information Bottleneck

Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed representation that preserves as much information about labels as possible. Under this principle, two different methods have been proposed, i.e., information bottleneck (IB) and deterministic information bottleneck (DIB), and have gained significant progress in explaining the representation mechanisms of deep learning algorithms. However, these theoretical and empirical successes are only valid with the assumption that training and test data are drawn from the same distribution, which is clearly not satisfied in many real-world applications. In this paper, we study their generalization abilities within a transfer learning scenario, where the target error could be decomposed into three components, i.e., source empirical error, source generalization gap (SG), and representation discrepancy (RD). Comparing IB and DIB on these terms, we prove that DIB’s SG bound is tighter than IB’s while DIB’s RD is larger than IB’s. Therefore, it is difficult to tell which one is better. To balance the trade-off between SG and the RD, we propose an elastic information bottleneck (EIB) to interpolate between the IB and DIB regularizers, which guarantees a Pareto frontier within the IB framework. Additionally, simulations and real data experiments show that EIB has the ability to achieve better domain adaptation results than IB and DIB, which validates the correctness of our theories.

Competing Mutual Information Constraints with Stochastic Competition-Based Activations for Learning Diversified Representations

This work aims to address the long-established problem of learning diversified representations. To this end, we combine information-theoretic arguments with stochastic competition-based activations, namely Stochastic Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional deep architectures commonly used in Representation Learning, that rely on non-linear activations; instead, we replace them with sets of locally and stochastically competing linear units. In this setting, each network layer yields sparse outputs, determined by the outcome of the competition between units that are organized into blocks of competitors. We adopt stochastic arguments for the competition mechanism, which perform posterior sampling to determine the winner of each block. We further endow the considered networks with the ability to infer the sub-part of the network that is essential for modeling the data at hand; we impose appropriate stick-breaking priors to this end. To further enrich the information of the emerging representations, we resort to information-theoretic principles, namely the Information Competing Process (ICP). Then, all the components are tied together under the stochastic Variational Bayes framework for inference. We perform a thorough experimental investigation for our approach using benchmark datasets on image classification. As we experimentally show, the resulting networks yield significant discriminative representation learning abilities. In addition, the introduced paradigm allows for a principled investigation mechanism of the emerging intermediate network representations.

Editorial Issue on “Information Theoretic Foundations of Future Communication Systems”

Information theory, starting with Shannon’s groundbreaking work, has fundamentally shaped the way communication systems are designed and operated. Information theoretic principles form the underpinnings of modern communication networks. This issue explores how new advances in information theory can impact future communication systems. Several papers address issues at the heart of next generation wireless and wired networks: Multiple access, including access by a massive number of devices, multi-hop, large antenna arrays, communication security, and timeliness of information. Others consider new applications such as joint communication and sensing, communication for learning and inference, wireless imaging, and new storage mediums such as DNA, thereby providing the information theoretic foundations of modalities beyond human-to-human communications.

Approaching language levels and registers in written Chinese with the Menzerath–Altmann Law

Abstract The length of language units, such as word length or sentence length, plays a critical role in register classification studies. However, in this line of work, little attention has been paid to the relationship between the lengths of language units at different levels. The Menzerath–Altmann law (MAL) reflects the functional relationship between the lengths of linguistic units at different levels, and its parameters were shown to be register-sensitive. This article focuses on two interrelated questions based on the MAL: (1) whether there are variations in the hierarchical relationships between language units at different levels and (2) whether such variations will influence register classifications. The results based on written Chinese show that (1) the MAL fittings at the ‘sentence > clause > word’ levels outperform that at the ‘clause > word > character’ levels and (2) the classifications based on two registers, i.e., Press (reportage) and Science (academic prose), demonstrate that the fitting parameters at the ‘sentence > clause > word’ levels also outperform those at the ‘clause > word > character’ levels. These indicate that the variations of hierarchical relationships between language units at different levels should be considered in register analysis. Further interpretations were given from perspectives of the information-theoretic principle and language evolution.

Increasing generalizability via the principle of minimum description length.

Traditional statistical model evaluation typically relies on goodness-of-fit testing and quantifying model complexity by counting parameters. Both of these practices may result in overfitting and have thereby contributed to the generalizability crisis. The information-theoretic principle of minimum description length addresses both of these concerns by filtering noise from the observed data and consequently increasing generalizability to unseen data.

Spatial pattern analysis of line-segment data in ecology.

The spatial analysis of linear features (lines and curves) is a challenging and rarely attempted problem in ecology. Existing methods are typically expressed in abstract mathematical formalism, making it difficult to assess their relevance and transferability into an ecological setting. We introduce a set of concrete and accessible methods to analyze the spatial patterning of line-segment data. The methods include Monte Carlo techniques based on a new generalization of Ripley's -function and a class of line-segment processes that can be used to specify parametric models: parameters are estimated using maximum likelihood and models compared using information-theoretic principles. We apply the new methods to fallen tree (dead log) data collected from two 1-ha Australian tall eucalypt forest plots. Our results show that the spatial pattern of the fallen logs is best explained by plot-level spatial heterogeneity in combination with a slope-dependent nonuniform distribution of fallen-log orientations. These methods are of a general nature and are applicable to any line-segment data. In the context of forest ecology, the integration of fallen logs as linear structural features in a landscape with the point locations of living trees, and a quantification of their interactions, can yield new insights into the functional and structural role of tree fall in forest communities and their enduring post-mortem ecological legacy as spatially distributed decomposing logs.