Information Theoretic Concepts Research Articles

Entropy, melody, beauty: composing with information theory

ABSTRACT Within a few years of information theory’s popularisation through the writings of Claude Shannon and Norbert Wiener, its basic framework was adopted and adapted by a loose network of music theorists, composers, and aestheticians, for whom a core principle of information theory – that ‘the aesthetic content of music can be treated in terms of fluctuations between the two extremes of total randomness and total redundancy’ (Hiller and Isaacson 1959, 110) – became an article of faith. Although it was sometimes imagined to offer a neutral conceptual framework for thinking about artmaking or a scientific alternative to the vagueness of previous artistic discourses, information theory was necessarily embroiled in the aesthetic debates of its time. In this paper I examine the encounter between information theory and musical composition in the middle of the twentieth century, paying special attention to the question of how the reception of information-theoretical concepts among composers and theorists inflected pre-existing debates about the nature and function of music in modernity.

Information-theoretic applications to Hupa verbal morphology

Hupa (Na:tinixwe Mixine:whe’) is a Pacific Coast Dene language spoken in Hoopa Valley in Northern California. Like its Dene sisters, Hupa exhibits complex verbal morphology which has attracted decades of theoretical research. One approach that has yet to have been applied to these languages is information theory. Previous information-theoretic research into verbal morphology has uncovered a cross-linguistic trend of grouping predictive information closer together and finding morphemes that are more mutually-informative to the root closer to the root, which in turn reduces overall surprisal and is easier on memory constraints. However, these studies analyzed prominently suffixing languages of Afro-Eurasia. This project is the first application of these information-theoretic concepts to a Dene language to investigate if these approaches also apply to explain morpheme order in a low-resource, Indigenous American language with intricate, prominently-prefixing morphology. The results indicate similar findings to previous research. Hupa demonstrates a word-level linear morpheme order that, on average, orders most mutually-informative morphemes closest to the verb root compared to a randomized baseline. This morpheme order also resulted in an average surprisal that was more comparable to optimized morpheme orders than a randomized baseline. Morpheme-type mutual information, however, demonstrates the discrepancies between word- and templatic-level information content in Hupa, which exemplifies the word-level efficiency that Hupa shares with other languages despite the typological uniqueness of its morphological grammar.

Explainable empirical risk minimization

AbstractThe successful application of machine learning (ML) methods increasingly depends on their interpretability or explainability. Designing explainable ML (XML) systems is instrumental for ensuring transparency of automated decision-making that targets humans. The explainability of ML methods is also an essential ingredient for trustworthy artificial intelligence. A key challenge in ensuring explainability is its dependence on the specific human end user of an ML system. The users of ML methods might have vastly different background knowledge about ML principles, with some having formal training in the specific field and others having none. We use information-theoretic concepts to develop a novel measure for the subjective explainability of predictions delivered by a ML method. We construct this measure via the conditional entropy of predictions, given the user signal. Our approach allows for a wide range of user signals, ranging from responses to surveys to biophysical measurements. We use this measure of subjective explainability as a regularizer for model training. The resulting explainable empirical risk minimization (EERM) principle strives to balance subjective explainability and risk. The EERM principle is flexible and can be combined with arbitrary ML models. We present several practical implementations of EERM for linear models and decision trees. Numerical experiments demonstrate the application of EERM to weather prediction and detecting inappropriate language in social media.

Phenomenological description of the acidity of the citric acid and its deprotonated species: informational-theoretical study.

In spite of the fact that molecular acidity is a fundamental physicochemical property of molecular systems, the vast majority of theoretical studies have focused attention on monoprotic acids and on the prediction of pKa's. Polyprotic acids, represent a challenge for electronic structure calculations since the multiple acidic sites result in a vast group of species with different conformations and reactivities. In this work, Information-theoretic (IT) concepts of localizability, order and uniformity are applied to the Citric Acid and its deprotonated species through the one-electron density functionals: Shannon entropy (S), Fisher information (I) and Disequilibrium (D), respectively. We pursue the goal of characterizing the acidity of the aforementioned species with the aim to associate the IT concepts to chemical features such as the polarizability of the protonated/deprotonated species, the liability of the acidic sites, atomic electrostatic potentials, covalent bonding. IT analyses looks very promising for future studies on the acidity of specific deprotonation-sites of polyprotic acids. Density functional theory (DFT) calculations were performed with Gaussian 09 program. A sensitivity analysis of the IT-measures was performed for the citric acid and the citrate using B3LYP, B3PW91, BPW91, M05-2X, M06-2X and PBEPBE functionals with the 6-311++g(3df,2p), 6-311++g(d,p), 6.311+g(d,p) and aug-cc-pVDZ basis sets. The rest of the analysis was performed with the M05-2X/6-311+G(d,p) level of theory. Additionally, aqueous media was considered by use of the SMD solvent model. The IT-measures were calculated using a suite of programs developed in our laboratory jointly with the DGRID software package.

Obtaining Information Leakage Bounds via Approximate Model Counting

Information leaks are a significant problem in modern software systems. In recent years, information theoretic concepts, such as Shannon entropy, have been applied to quantifying information leaks in programs. One recent approach is to use symbolic execution together with model counting constraints solvers in order to quantify information leakage. There are at least two reasons for unsoundness in quantifying information leakage using this approach: 1) Symbolic execution may not be able to explore all execution paths, 2) Model counting constraints solvers may not be able to provide an exact count. We present a sound symbolic quantitative information flow analysis that bounds the information leakage both for the cases where the program behavior is not fully explored and the model counting constraint solver is unable to provide a precise model count but provides an upper and a lower bound. We implemented our approach as an extension to KLEE for computing sound bounds for information leakage in C programs.

Primordial Gravitational Wave Circuit Complexity

In this article, we investigate the various physical implications of quantum circuit complexity using the squeezed state formalism of Primordial Gravitational Waves (PGW). Recently, quantum information-theoretic concepts, such as entanglement entropy and complexity, have played a pivotal role in understanding the dynamics of quantum systems, even in diverse fields such as high-energy physics and cosmology. This paper is devoted to studying the quantum circuit complexity of PGW for various cosmological models, such as de Sitter, inflation, radiation, reheating, matter, bouncing, cyclic and black hole gas models, etc. We compute complexity measures using both Covariance and Nielsen’s wave function method for three different choices of quantum initial vacua: Motta-Allen, α and Bunch–Davies. Besides computing circuit complexity, we also compute the Von Neumann entanglement entropy. By making the comparison between complexity and entanglement entropy, we are able to probe various features regarding the dynamics of evolution for different cosmological models. Because entanglement entropy is independent of the squeezing angle, we are able to understand more details of the system using Nielsen’s measure of complexity, which is dependent on both squeezing parameter and angle. This implies that quantum complexity could indeed be a useful probe to study quantum features on a cosmological scale. Quantum complexity is also becoming a powerful technique to understand the chaotic behaviour and random fluctuations of quantum fields. Using the growth of complexity, we are able to compute the quantum Lyapunov exponent for various cosmological models and comment on its chaotic nature.

Theoretical Analysis of Deep Neural Networks in Physical Layer Communication

Recently, deep neural network (DNN)-based physical layer communication techniques have attracted considerable interest. Although their potential to enhance communication systems and superb performance have been validated by simulation experiments, little attention has been paid to the theoretical analysis. Specifically, most studies in the physical layer have tended to focus on the application of DNN models to wireless communication problems but not to theoretically understand how does a DNN work in a communication system. In this paper, we aim to quantitatively analyze why DNNs can achieve comparable performance in the physical layer comparing with traditional techniques, and also drive their cost in terms of computational complexity. To achieve this goal, we first analyze the encoding performance of a DNN-based transmitter and compare it to a traditional one. And then, we theoretically analyze the performance of DNN-based estimator and compare it with traditional estimators. Third, we investigate and validate how information is flown in a DNN-based communication system under the information theoretic concepts. Our analysis develops a concise way to open the "black box" of DNNs in physical layer communication, which can be applied to support the design of DNN-based intelligent communication techniques and help to provide explainable performance assessment.

Causal Emergence: When Distortions in a Map Obscure the Territory

We provide a critical assessment of the account of causal emergence presented in Erik Hoel’s 2017 article “When the map is better than the territory”. The account integrates causal and information theoretic concepts to explain under what circumstances there can be causal descriptions of a system at multiple scales of analysis. We show that the causal macro variables implied by this account result in interventions with significant ambiguity, and that the operations of marginalization and abstraction do not commute. Both of these are desiderata that, we argue, any account of multi-scale causal analysis should be sensitive to. The problems we highlight in Hoel’s definition of causal emergence derive from the use of various averaging steps and the introduction of a maximum entropy distribution that is extraneous to the system under investigation.

The Odyssey of Entropy: Cryptography.

After being introduced by Shannon as a measure of disorder and unavailable information, the notion of entropy has found its applications in a broad range of scientific disciplines. In this paper, we present a systematic review on the applications of entropy and related information-theoretical concepts in the design, implementation and evaluation of cryptographic schemes, algorithms, devices and systems. Moreover, we study existing trends, and establish a roadmap for future research in these areas.

Performance Evaluation of Deep Learning Models for Image Classification Over Small Datasets: Diabetic Foot Case Study

Data scarcity is a common and challenging issue when working with Artificial Intelligence solutions, especially those including Deep Learning (DL) models for tasks such as image classification. This is particularly relevant in healthcare scenarios, in which data collection requires a long-lasting process, involving specific control protocols. The performance of DL models is usually quantified by different classification metrics, which may provide biased results, due to the lack of sufficient data. In this paper, an innovative approach is proposed to evaluate the performance of DL models when labeled data is scarce. This approach, which aims to detect the poor performance provided by DL models, in spite of traditional assessing metrics indicating otherwise, is based on information theoretic concepts and motivated by the Information Bottleneck framework. This methodology has been evaluated by implementing several experimental configurations to classify samples from a plantar thermogram dataset, focused on early stage detection of diabetic foot ulcers, as a case study. The proposed network architectures exhibited high results in terms of classification metrics. However, as our approach shows, only two of those models are indeed consistent to generalize the data properly. In conclusion, a new methodology was introduced and tested to identify promising DL models for image classification over small datasets without relying exclusively on the widely employed classification metrics.

Why Do Individuals Seek Information? A Selectionist Perspective.

Several authors have proposed that mechanisms of adaptive behavior, and reinforcement learning in particular, can be explained by an innate tendency of individuals to seek information about the local environment. In this article, I argue that these approaches adhere to an essentialist view of learning that avoids the question why information seeking should be favorable in the first place. I propose a selectionist account of adaptive behavior that explains why individuals behave as if they had a tendency to seek information without resorting to essentialist explanations. I develop my argument using a formal selectionist framework for adaptive behavior, the multilevel model of behavioral selection (MLBS). The MLBS has been introduced recently as a formal theory of behavioral selection that links reinforcement learning to natural selection within a single unified model. I show that the MLBS implies an average gain in information about the availability of reinforcement. Formally, this means that behavior reaches an equilibrium state, if and only if the Fisher information of the conditional probability of reinforcement is maximized. This coincides with a reduction in the randomness of the expected environmental feedback as captured by the information theoretic concept of expected surprise (i.e., entropy). The main result is that behavioral selection maximizes the information about the expected fitness consequences of behavior, which, in turn, minimizes average surprise. In contrast to existing attempts to link adaptive behavior to information theoretic concepts (e.g., the free energy principle), neither information gain nor surprise minimization is treated as a first principle. Instead, the result is formally deduced from the MLBS and therefore constitutes a mathematical property of the more general principle of behavioral selection. Thus, if reinforcement learning is understood as a selection process, there is no need to assume an active agent with an innate tendency to seek information or minimize surprise. Instead, information gain and surprise minimization emerge naturally because it lies in the very nature of selection to produce order from randomness.

A typology of research discovery tools

There has been a proliferation of new research discovery tools that aid scientists in finding relevant publications. To obtain a general overview of this development, this article generates a conceptual typology of all possible research discovery tools by drawing from the information-theoretical concepts of redundancy/variety. Bibliometric links between scholarly publications can thus exhibit ‘redundancy’ (i.e. expectable linkages between academic works) or ‘variety’ (i.e. original co-occurrence patterns). On the redundancy-reproducing end of the typology are machines that harness extant co-citations or keyword queries, such as academic search engines and paper recommender systems. The variety end of the spectrum harbours services that enable categorial browsing or that suggest publications randomly, such as journals’ tables of contents or random paper bots. The typology has implications for understanding how the design of research discovery platforms may ultimately shape aggregate citational networks of science.

Quantum information entropy and squeezing of 𝒫𝒯-symmetric potential

The information theoretic concepts are crucial to study the quantum mechanical systems. In this paper, the information densities of [Formula: see text]-symmetric potential have been demonstrated and their properties deeply analyzed. The position space and momentum space information entropy is obtained and Bialynicki-Birula–Mycielski inequality is saturated for different parameters of the potential. Some interesting features of information entropy have been discussed. The variation in these entropies is described which gets saturated for specific values of the parameter. These have also been analyzed for the [Formula: see text]-symmetry breaking case. Further, the entropy squeezing phenomenon has been investigated in position space as well as momentum space. Interestingly, [Formula: see text] phase transition conjectures the entropy squeezing in position space and momentum space.

Combinatorial proofs of two theorems of Lutz and Stull

AbstractRecently, Lutz and Stull used methods from algorithmic information theory to prove two new Marstrand-type projection theorems, concerning subsets of Euclidean space which are not assumed to be Borel, or even analytic. One of the theorems states that if \[K \subset {\mathbb{R}^n}\] is any set with equal Hausdorff and packing dimensions, then \begin{equation} \[{\dim _{\text{H}}}{\pi _e}(K) = \min \{ {\dim _{\text{H}}}{\text{ }}K{\text{, 1}}\} \] \end{equation}for almost every \[e \in {S^{n - 1}}\]. Here \[{\pi _e}\] stands for orthogonal projection to span (\[e\]). The primary purpose of this paper is to present proofs for Lutz and Stull’s projection theorems which do not refer to information theoretic concepts. Instead, they will rely on combinatorial-geometric arguments, such as discretised versions of Kaufman’s “potential theoretic” method, the pigeonhole principle, and a lemma of Katz and Tao. A secondary purpose is to generalise Lutz and Stull’s theorems: the versions in this paper apply to orthogonal projections to m-planes in \[{\mathbb{R}^n}\], for all \[0 < m < n\].

Concept of Orbital Entanglement and Correlation in Quantum Chemistry.

A recent development in quantum chemistry has established the quantum mutual information between orbitals as a major descriptor of electronic structure. This has already facilitated remarkable improvements in numerical methods and may lead to a more comprehensive foundation for chemical bonding theory. Building on this promising development, our work provides a refined discussion of quantum information theoretical concepts by introducing the physical correlation and its separation into classical and quantum parts as distinctive quantifiers of electronic structure. In particular, we succeed in quantifying the entanglement. Intriguingly, our results for different molecules reveal that the total correlation between orbitals is mainly classical, raising questions about the general significance of entanglement in chemical bonding. Our work also shows that implementing the fundamental particle number superselection rule, so far not accounted for in quantum chemistry, removes a major part of correlation and entanglement seen previously. In that respect, realizing quantum information processing tasks with molecular systems might be more challenging than anticipated.

The great divide: drivers of polarization in the US public

Many democratic societies have become more politically polarized, with the U.S. being the main example. The origins of this phenomenon are still not well-understood and subject to debate. To provide insight into some of the mechanisms underlying political polarization, we develop a mathematical framework and employ Bayesian Markov chain Monte-Carlo (MCMC) and information-theoretic concepts to analyze empirical data on political polarization that has been collected by Pew Research Center from 1994 to 2017. Our framework can capture the evolution of polarization in the Democratic- and Republican-leaning segments of the U.S. public and allows us to identify its drivers. Our empirical and quantitative evidence suggests that political polarization in the U.S. is mainly driven by strong political/cultural initiatives in the Democratic party.

How the T cell signaling network processes information to discriminate between self and agonist ligands

T cells exhibit remarkable sensitivity and selectivity in detecting and responding to agonist peptides (p) bound to MHC molecules in a sea of self pMHC molecules. Despite much work, understanding of the underlying mechanisms of distinguishing such ligands remains incomplete. Here, we quantify T cell discriminatory capacity using channel capacity, a direct measure of the signaling network's ability to discriminate between antigen-presenting cells (APCs) displaying either self ligands or a mixture of self and agonist ligands. This metric shows how differences in information content between these two types of peptidomes are decoded by the topology and rates of kinetic proofreading signaling steps inside T cells. Using channel capacity, we constructed numerically substantiated hypotheses to explain the discriminatory role of a recently identified slow LAT Y132 phosphorylation step. Our results revealed that in addition to the number and kinetics of sequential signaling steps, a key determinant of discriminatory capability is spatial localization of a minimum number of these steps to the engaged TCR. Biochemical and imaging experiments support these findings. Our results also reveal the discriminatory role of early negative feedback and necessary amplification conferred by late positive feedback.

Sample complexity of classification with compressed input

One of the most studied problems in machine learning is finding reasonable constraints that guarantee the generalization of a learning algorithm. These constraints are usually expressed as some simplicity assumptions on the target. For instance, in the Vapnik–Chervonenkis (VC) theory the space of possible hypotheses is considered to have a limited VC dimension One way to formulate the simplicity assumption is via information theoretic concepts. In this paper, the constraint on the entropy H(X) of the input variable X is studied as a simplicity assumption. It is proven that the sample complexity to achieve an ∊-δ Probably Approximately Correct (PAC) hypothesis is bounded by 2cH(X)/∊+log1δa∊2 which is sharp up to the 1∊2 factor (a and c are constants). Moreover, it is shown that if a feature learning process is employed to learn the compressed representation from the dataset, this bound no longer exists. These findings have important implications on the Information Bottleneck (IB) theory which had been utilized to explain the generalization power of Deep Neural Networks (DNNs), but its applicability for this purpose is currently under debate by researchers. In particular, this is a rigorous proof for the previous heuristic that compressed representations are exponentially easier to be learned. However, our analysis pinpoints two factors preventing the IB, in its current form, to be applicable in studying neural networks. Firstly, the exponential dependence of sample complexity on 1/∊., which can lead to a dramatic effect on the bounds in practical applications when ∊ is small. Secondly, our analysis reveals that arguments based on input compression are inherently insufficient to explain generalization of methods like DNNs in which the features are also learned using available data.

The Big Data razor

Classic conceptions of model simplicity for machine learning are mainly based on the analysis of the structure of the model. Bayesian, Frequentist, information theoretic and expressive power concepts are the best known of them, which are reviewed in this work, along with their underlying assumptions and weaknesses. These approaches were developed before the advent of the Big Data deluge, which has overturned the importance of structural simplicity. The computational simplicity concept is presented, and it is argued that it is more encompassing and closer to actual machine learning practices than the classic ones. In order to process the huge datasets which are commonplace nowadays, the computational complexity of the learning algorithm is the decisive factor to assess the viability of a machine learning strategy, while the classic accounts of simplicity play a surrogate role. Some of the desirable features of computational simplicity derive from its reliance on the learning system concept, which integrates key aspects of machine learning that are ignored by the classic concepts. Moreover, computational simplicity is directly associated with energy efficiency. In particular, the question of whether the maximum possibly achievable predictive accuracy should be attained, no matter the economic cost of the associated energy consumption pattern, is considered.

Parametric Representation of the Entire Pressure Drop Evolution during Particulate Filter Loading

<div class="section abstract"><div class="htmlview paragraph">Improved understanding and compact descriptions of the pressure drop evolution of Particulate Filters (both for diesel and gasoline powered vehicles) are always in demand for intelligent implementations of exhaust emission system monitoring and control. In the present paper we revisit the loading process of a particulate filter focusing on a parametric description of the deep bed-to-cake transition in the light of recent progress in the understanding of soot deposit structure, growth dynamics and evolution. Combining experimental data, simulation models and information theoretic concepts we provide a closed-form representation of the entire evolution of pressure drop (from the initial clean state up to the evolving linear cake growth regime) parameterized in terms of the physical parameters of the system (filter and particle structure/geometry and flow properties). The representation is shown to be in excellent agreement with the available experimental data and it opens the door for improved on-board monitoring/control of particulate filter systems.</div></div>