Information-theoretic Principles Research Articles

Communicative need in colour naming

ABSTRACT Colour naming across languages has traditionally been held to reflect the structure of colour perception. At the same time, it has often, and increasingly, been suggested that colour naming may be shaped by patterns of communicative need. However, much remains unknown about the factors involved in communicative need, how need interacts with perception, and how this interaction may shape colour naming. Here, we engage these open questions by building on general information-theoretic principles. We present a systematic evaluation of several factors that may reflect need, and that have been proposed in the literature: capacity constraints, linguistic usage, and the visual environment. Our analysis suggests that communicative need in colour naming is reflected more directly by capacity constraints and linguistic usage than it is by the statistics of the visual environment.

The relativistic causality versus no-signaling paradigm for multi-party correlations

The ubiquitous no-signaling constraints state that the probability distributions of outputs of any subset of parties in a Bell experiment are independent of remaining parties’ inputs. These constraints are considered to form ultimate limits for physical correlations and led to the fields of post-quantum cryptography, randomness generation besides identifying information-theoretic principles underlying quantum theory. Here we show that while these constraints are sufficient, they are not necessary to enforce relativistic causality in multi-party correlations, i.e., the rule that correlations do not allow casual loops. Depending on the space-time coordinates of the measurement events, causality only imposes a subset of no-signaling conditions. We first consider the n-party Bell experiment (n > 2) and identify all configurations where subsets of the constraints suffice. Secondly, we examine the implications for device-independent cryptography against an eavesdropper constrained only by relativity, detailing among other effects explicit attacks on well-known randomness amplification and key distribution protocols.

Subjectively interesting connecting trees and forests

Consider a large graph or network, and a user-provided set of query vertices between which the user wishes to explore relations. For example, a researcher may want to connect research papers in a citation network, an analyst may wish to connect organized crime suspects in a communication network, or an internet user may want to organize their bookmarks given their location in the world wide web. A natural way to do this is to connect the vertices in the form of a tree structure that is present in the graph. However, in sufficiently dense graphs, most such trees will be large or somehow trivial (e.g. involving high degree vertices) and thus not insightful. Extending previous research, we define and investigate the new problem of mining subjectively interesting trees connecting a set of query vertices in a graph, i.e., trees that are highly surprising to the specific user at hand. Using information theoretic principles, we formalize the notion of interestingness of such trees mathematically, taking in account certain prior beliefs the user has specified about the graph. A remaining problem is efficiently fitting a prior belief model. We show how this can be done for a large class of prior beliefs. Given a specified prior belief model, we then propose heuristic algorithms to find the best trees efficiently. An empirical validation of our methods on a large real graphs evaluates the different heuristics and validates the interestingness of the given trees.

Linguistic laws in chimpanzee gestural communication.

Studies testing linguistic laws outside language have provided important insights into the organization of biological systems. For example, patterns consistent with Zipf's law of abbreviation (which predicts a negative relationship between word length and frequency of use) have been found in the vocal and non-vocal behaviour of a range of animals, and patterns consistent with Menzerath's law (according to which longer sequences are made up of shorter constituents) have been found in primate vocal sequences, and in genes, proteins and genomes. Both laws have been linked to compression-the information theoretic principle of minimizing code length. Here, we present the first test of these laws in animal gestural communication. We initially did not find the negative relationship between gesture duration and frequency of use predicted by Zipf's law of abbreviation, but this relationship was seen in specific subsets of the repertoire. Furthermore, a pattern opposite to that predicted was seen in one subset of gestures-whole body signals. We found a negative correlation between number and mean duration of gestures in sequences, in line with Menzerath's law. These results provide the first evidence that compression underpins animal gestural communication, and highlight an important commonality between primate gesturing and language.

Accelerated Guided Sampling for Multistructure Model Fitting.

The performance of many robust model fitting techniques is largely dependent on the quality of the generated hypotheses. In this paper, we propose a novel guided sampling method, called accelerated guided sampling (AGS), to efficiently generate the accurate hypotheses for multistructure model fitting. Based on the observations that residual sorting can effectively reveal the data relationship (i.e., determine whether two data points belong to the same structure), and keypoint matching scores can be used to distinguish inliers from gross outliers, AGS effectively combines the benefits of residual sorting and keypoint matching scores to efficiently generate accurate hypotheses via information theoretic principles. Moreover, we reduce the computational cost of residual sorting in AGS by designing a new residual sorting strategy, which only sorts the top-ranked residuals of input data, rather than all input data. Experimental results demonstrate the effectiveness of the proposed method in computer vision tasks, such as homography matrix and fundamental matrix estimation.

A Survey on MIMO Transmission With Finite Input Signals: Technical Challenges, Advances, and Future Trends

Multiple antennas have played an essential role in spatial multiplexing and diversity transmission for a wide range of communication applications. Most advances in the design of high-speed wireless multiple-input-multiple-output (MIMO) systems have been based on information-theoretic principles that demonstrate how to efficiently transmit signals conforming to Gaussian distribution. However, although the Gaussian signal is capacity-achieving, practical systems transmit signals belonging to finite and discrete constellations. Therefore, capacity-achieving transceiver processing based on a Gaussian input signal can be quite suboptimal for practical MIMO systems with discrete constellation input signals. To address this shortcoming, this paper aims to provide a comprehensive overview of MIMO transmission design with finite input signals. It first summarizes existing fundamental results for MIMO systems with finite input signals. Next, focusing on basic point-to-point MIMO systems, it examines transmission schemes based on the three most important criteria for communication systems: mutual-information-driven designs, mean-square-error-driven designs, and diversity-driven designs. In particular, a unified framework is developed for the design of low-complexity transmission schemes applicable to massive MIMO systems in forthcoming 5G wireless networks for the first time. Furthermore, adaptive transmission designs are proposed that switch among these criteria based on channel conditions to formulate the best transmission strategy. A survey is then given of transmission designs with finite input signals for multiuser MIMO scenarios, including MIMO uplink transmission, MIMO downlink transmission, MIMO interference channel, and MIMO wiretap channel. Additionally, transmission designs with finite input signals are discussed for other multi-antenna systems. Finally, a number of technical challenges that remain unresolved at the time of writing are highlighted, and future trends in transmission design with finite input signals are discussed.

Information Theory and Cognition: A Review.

We examine how information theory has been used to study cognition over the last seven decades. After an initial burst of activity in the 1950s, the backlash that followed stopped most work in this area. The last couple of decades has seen both a revival of interest, and a more firmly grounded, experimentally justified use of information theory. We can view cognition as the process of transforming perceptions into information—where we use information in the colloquial sense of the word. This last clarification is one of the problems we run into when trying to use information theoretic principles to understand or analyze cognition. Information theory is mathematical, while cognition is a subjective phenomenon. It is relatively easy to discern a subjective connection between cognition and information; it is a different matter altogether to apply the rigor of information theory to the process of cognition. In this paper, we will look at the many ways in which people have tried to alleviate this problem. These approaches range from narrowing the focus to only quantifiable aspects of cognition or borrowing conceptual machinery from information theory to address issues of cognition. We describe applications of information theory across a range of cognition research, from neural coding to cognitive control and predictive coding.

Information equilibria, subsystem entanglement, and dynamics of the overall entropic descriptors of molecular electronic structure

Overall descriptors of the information (determinicity) and entropy (uncertainty) content of complex molecular states are reexamined. These resultant concepts combine the classical (probability) contributions of Fisher and Shannon, and the relevant nonclassical supplements due to the state phase/current. The information-theoretic principles determining equilibria in molecules and their fragments are explored and the nonadditive part of the global entropy is advocated as a descriptor of the classical index of the quantum entanglement of molecular subsystems. Affinities associated with the probability and phase fluxes are identified and the criterion of vanishing overall information-source is shown to identify the system stationary electronic states. The production of resultant density of the gradient-information is expressed in terms of the conjugate affinities (forces, perturbations) and fluxes (currents, responses). The Schrödinger dynamics of probability and phase components of molecular electronic states is used to determine the temporal evolution of the overall gradient information and complex entropy. The global sources of the resultant information/entropy descriptors are shown to be of purely nonclassical origin, thus identically vanishing in real electronic states, e.g., the nondegenerate ground state of a molecule.

Quantum reaxiomatisations and information-theoretic interpretations of quantum theory

Jeff Bub has developed an information-theoretic interpretation of quantum mechanics on the basis of the programme to reaxiomatise the theory in terms of information-theoretic principles. According to the most recent version of the interpretation, reaxiomatisation can dissolve some of the demands for explanation traditionally associated with the task of providing an interpretation for the theory. The key idea is that the real lesson we should take away from quantum mechanics is that the ‘structure of information’ is not what we thought it was. In particular a feature of the new structure is intrinsic randomness of measurement, which allegedly dissolves a significant part of the measurement problem. I argue that it is difficult to find an appropriate argument to support the claim that measurement is intrinsically random in the relevant sense.

Information theoretic-based sampling of observations

Due to the surge in the amount of data that are being collected, analysts are increasingly faced with very large data sets. Estimation of sophisticated discrete choice models (such as Mixed Logit models) based on these typically large data sets can be computationally burdensome, or even infeasible. Hitherto, analysts tried to overcome these computational burdens by reverting to less computationally demanding choice models or by taking advantage of the increase in computational resources. In this paper we take a different approach: we develop a new method called Sampling of Observations (SoO) which scales down the size of the choice data set, prior to the estimation. More specifically, based on information-theoretic principles this method extracts a subset of observations from the data which is much smaller in volume than the original data set, yet produces statistically nearly identical results. We show that this method can be used to estimate sophisticated discrete choice models based on data sets that were originally too large to conduct sophisticated choice analysis.

Information Dropout: Learning Optimal Representations Through Noisy Computation.

The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. We show that our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity. When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of optimal disentangled representations simply by enforcing a factorized prior, a fact that has been observed empirically in recent work. Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.

Optimization Models of Natural Communication

A family of information theoretic models of communication was introduced more than a decade ago to explain the origins of Zipf’s law for word frequencies. The family is a based on a combination of two information theoretic principles: maximization of mutual information between forms and meanings and minimization of form entropy. The family also sheds light on the origins of three other patterns: the principle of contrast; a related vocabulary learning bias; and the meaning-frequency law. Here two important components of the family, namely the information theoretic principles and the energy function that combines them linearly, are reviewed from the perspective of psycholinguistics, language learning, information theory and synergetic linguistics. The minimization of this linear function is linked to the problem of compression of standard information theory and might be tuned by self-organization.

Model selection with multiple regression on distance matrices leads to incorrect inferences.

In landscape genetics, model selection procedures based on Information Theoretic and Bayesian principles have been used with multiple regression on distance matrices (MRM) to test the relationship between multiple vectors of pairwise genetic, geographic, and environmental distance. Using Monte Carlo simulations, we examined the ability of model selection criteria based on Akaike’s information criterion (AIC), its small-sample correction (AICc), and the Bayesian information criterion (BIC) to reliably rank candidate models when applied with MRM while varying the sample size. The results showed a serious problem: all three criteria exhibit a systematic bias toward selecting unnecessarily complex models containing spurious random variables and erroneously suggest a high level of support for the incorrectly ranked best model. These problems effectively increased with increasing sample size. The failure of AIC, AICc, and BIC was likely driven by the inflated sample size and different sum-of-squares partitioned by MRM, and the resulting effect on delta values. Based on these findings, we strongly discourage the continued application of AIC, AICc, and BIC for model selection with MRM.

Motion Segmentation Via a Sparsity Constraint

Motion segmentation is an important task for intelligent transportation systems. In this paper, inspired by the fact that a feature point trajectory can be sparsely represented as a combination of several feature point trajectories that share coherent transformations, an efficient and effective motion segmentation method with a sparsity constraint is proposed. Specifically, we first propose an accumulated scheme to efficiently integrate motion information from all the frames of a video sequence to construct a correlation matrix. Then, a sparse affinity matrix is built on the correlation matrix by using information-theoretic principles, where the nonzero elements in the same row of the sparse affinity matrix correspond to the feature point trajectories more likely belonging to the same motion. Thereafter, a segment and merge procedure is proposed to effectively estimate the number of motions via the sparse affinity matrix. Finally, by applying spectral clustering on the sparse affinity matrix, different motions in the video sequence are accurately segmented based on the estimated number of motions. Experimental results on the Hopkins 155 and the 62-clip datasets demonstrate that the proposed method achieves superior performance compared with several state-of-the-art methods.

Physical Intelligence and Thermodynamic Computing

This paper proposes that intelligent processes can be completely explained by thermodynamic principles. They can equally be described by information-theoretic principles that, from the standpoint of the required optimizations, are functionally equivalent. The underlying theory arises from two axioms regarding distinguishability and causality. Their consequence is a theory of computation that applies to the only two kinds of physical processes possible—those that reconstruct the past and those that control the future. Dissipative physical processes fall into the first class, whereas intelligent ones comprise the second. The first kind of process is exothermic and the latter is endothermic. Similarly, the first process dumps entropy and energy to its environment, whereas the second reduces entropy while requiring energy to operate. It is shown that high intelligence efficiency and high energy efficiency are synonymous. The theory suggests the usefulness of developing a new computing paradigm called Thermodynamic Computing to engineer intelligent processes. The described engineering formalism for the design of thermodynamic computers is a hybrid combination of information theory and thermodynamics. Elements of the engineering formalism are introduced in the reverse-engineer of a cortical neuron. The cortical neuron provides perhaps the simplest and most insightful example of a thermodynamic computer possible. It can be seen as a basic building block for constructing more intelligent thermodynamic circuits.

Addressing the Need for a Model Selection Framework in Systems Biology Using Information Theory

The field of systems biology thrives upon the use of models to organize biological knowledge and make predictions of complex processes that are hard to measure. When attempting to generate model descriptions for metabolic systems, one arrives at a crossroads. A variety of mathematical explanations are available for metabolic data with varying degrees of resolution from simple to complex. Biological modelers often rely upon subjective arguments to choose one framework over another. While there is no universal rule to determine the absolute utility of a model, certain metrics founded on information theoretical principles, demonstrate promise in providing a coherent, rational, and objective basis for addressing this model selection problem in systems biology. A model seeks to capture the regularity in biological data. Models that best capture regularity in data without excessive complexity are the most useful for applications in optimization and control. To demonstrate the efficacy of such an approach, several metabolic model selection scenarios are investigated. This work develops the argument that information theoretic model selection metrics should be extended to nonnested model comparison applications in systems biology. It also makes a novel comparison of kinetic, constraint-based, and cybernetic models of metabolism based not only on model accuracy, but also model complexity. The results show the strengths of lumped hybrid cybernetic model (L-HCM) and flux balance analysis (FBA) for applications in steady state flux prediction. Also, the hybrid cybernetic model's (HCM) merit in the modeling of dynamic changes in fluxes is also established.

Statistical inference of protein structural alignments using information and compression.

Structural molecular biology depends crucially on computational techniques that compare protein three-dimensional structures and generate structural alignments (the assignment of one-to-one correspondences between subsets of amino acids based on atomic coordinates). Despite its importance, the structural alignment problem has not been formulated, much less solved, in a consistent and reliable way. To overcome these difficulties, we present here a statistical framework for the precise inference of structural alignments, built on the Bayesian and information-theoretic principle of Minimum Message Length (MML). The quality of any alignment is measured by its explanatory power-the amount of lossless compression achieved to explain the protein coordinates using that alignment. We have implemented this approach in MMLigner , the first program able to infer statistically significant structural alignments. We also demonstrate the reliability of MMLigner 's alignment results when compared with the state of the art. Importantly, MMLigner can also discover different structural alignments of comparable quality, a challenging problem for oligomers and protein complexes. Source code, binaries and an interactive web version are available at http://lcb.infotech.monash.edu.au/mmligner . arun.konagurthu@monash.edu. Supplementary data are available at Bioinformatics online.

It's a Matter of Principle: Scientific Explanation in Information-Theoretic Reconstructions of Quantum Theory

The aim of this paper is to explore the ways in which Axiomatic Reconstructions of Quantum Theory in terms of Information-Theoretic principles (ARQITs) can contribute to explaining and understanding quantum phenomena, as well as to study their explanatory limitations. This is achieved in part by offering an account of the kind of explanation that axiomatic reconstructions of quantum theory provide, and re-evaluating the epistemic status of the program in light of this explanation. As illustrative cases studies, I take Clifton's, Bub's and Halvorson's characterization theorem and Popescu's and Rohrlich's toy models, and their explanatory contribution with respect to quantum non-locality. On the one hand, I argue that ARQITs can aspire to provide genuine explanations of (some aspects of) quantum non-locality. On the other hand, I argue that such explanations cannot rule out a mechanical quantum theory.

Toward Massive, Ultrareliable, and Low-Latency Wireless Communication With Short Packets

Most of the recent advances in the design of high-speed wireless systems are based on information-theoretic principles that demonstrate how to efficiently transmit long data packets. However, the upcoming wireless systems, notably the fifth-generation (5G) system, will need to support novel traffic types that use short packets. For example, short packets represent the most common form of traffic generated by sensors and other devices involved in machine-to-machine (M2M) communications. Furthermore, there are emerging applications in which small packets are expected to carry critical information that should be received with low latency and ultrahigh reliability. Current wireless systems are not designed to support short-packet transmissions. For example, the design of current systems relies on the assumption that the metadata (control information) is of negligible size compared to the actual information payload. Hence, transmitting metadata using heuristic methods does not affect the overall system performance. However, when the packets are short, metadata may be of the same size as the payload, and the conventional methods to transmit it may be highly suboptimal. In this paper, we review recent advances in information theory, which provide the theoretical principles that govern the transmission of short packets. We then apply these principles to three exemplary scenarios (the two-way channel, the downlink broadcast channel, and the uplink random access channel), thereby illustrating how the transmission of control information can be optimized when the packets are short. The insights brought by these examples suggest that new principles are needed for the design of wireless protocols supporting short packets. These principles will have a direct impact on the system design.

No Quantum Realization of Extremal No-Signaling Boxes.

The study of quantum correlations is important for fundamental reasons as well as for quantum communication and information processing tasks. On the one hand, it is of tremendous interest to derive the correlations produced by measurements on separated composite quantum systems from within the set of all correlations obeying the no-signaling principle of relativity, by means of information-theoretic principles. On the other hand, an important ongoing research program concerns the formulation of device-independent cryptographic protocols based on quantum nonlocal correlations for the generation of secure keys, and the amplification and expansion of random bits against general no-signaling adversaries. In both these research programs, a fundamental question arises: Can any measurements on quantum states realize the correlations present in pure extremal no-signaling boxes? Here, we answer this question in full generality showing that no nontrivial (not local realistic) extremal boxes of general no-signaling theories can be realized in quantum theory. We then explore some important consequences of this fact.