Information-theoretic Quantities Research Articles

Diagnostics of Mixed-State Topological Order and Breakdown of Quantum Memory

Topological quantum memory can protect information against local errors up to finite error thresholds. Such thresholds are usually determined based on the success of decoding algorithms rather than the intrinsic properties of the mixed states describing corrupted memories. Here we provide an intrinsic characterization of the breakdown of topological quantum memory, which both gives a bound on the performance of decoding algorithms and provides examples of topologically distinct mixed states. We employ three information-theoretical quantities that can be regarded as generalizations of the diagnostics of ground-state topological order, and serve as a definition for topological order in error-corrupted mixed states. We consider the topological contribution to entanglement negativity and two other metrics based on quantum relative entropy and coherent information. In the concrete example of the two-dimensional (2D) Toric code with local bit-flip and phase errors, we map three quantities to observables in 2D classical spin models and analytically show they all undergo a transition at the same error threshold. This threshold is an upper bound on that achieved in any decoding algorithm and is indeed saturated by that in the optimal decoding algorithm for the Toric code. Published by the American Physical Society 2024

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data.

Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

To Compress or Not to Compress-Self-Supervised Learning and Information Theory: A Review.

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.

What is Quantum in probabilistic explanations of the sure-thing principle violation?

The Prisoner’s Dilemma game (PDG) is one of the simple test-beds for the probabilistic nature of the human decision-making process. Behavioral experiments have been conducted on this game for decades and show a violation of the so-called sure-thing principle, a key principle in the rational theory of decision. Quantum probabilistic models can explain this violation as a second-order interference effect, which cannot be accounted for by classical probability theory. Here, we adopt the framework of generalized probabilistic theories and approach this explanation from the viewpoint of quantum information theory to identify the source of the interference. In particular, we reformulate one of the existing quantum probabilistic models using density matrix formalism and consider different amounts of classical and quantum uncertainties for one player’s prediction about another player’s action in PDG. This enables us to demonstrate that what makes possible the explanation of the violation is the presence of quantum coherence in the player’s initial prediction and its conversion to probabilities during the dynamics. Moreover, we discuss the role of other quantum information-theoretical quantities, such as quantum entanglement, in the decision-making process. Finally, we propose a three-choice extension of the PDG to compare the predictive powers of quantum probability theory and a more general probabilistic theory that includes it as a particular case and exhibits third-order interference.

Causality and signalling of garden-path sentences.

Sheaves are mathematical objects that describe the globally compatible data associated with open sets of a topological space. Original examples of sheaves were continuous functions; later they also became powerful tools in algebraic geometry, as well as logic and set theory. More recently, sheaves have been applied to the theory of contextuality in quantum mechanics. Whenever the local data are not necessarily compatible, sheaves are replaced by the simpler setting of presheaves. In previous work, we used presheaves to model lexically ambiguous phrases in natural language and identified the order of their disambiguation. In the work presented here, we model syntactic ambiguities and study a phenomenon in human parsing called garden-pathing. It has been shown that the information-theoretic quantity known as 'surprisal' correlates with human reading times in natural language but fails to do so in garden-path sentences. We compute the degree of signalling in our presheaves using probabilities from the large language model BERT and evaluate predictions on two psycholinguistic datasets. Our degree of signalling outperforms surprisal in two ways: (i) it distinguishes between hard and easy garden-path sentences (with a [Formula: see text]-value [Formula: see text]), whereas existing work could not, (ii) its garden-path effect is larger in one of the datasets (32 ms versus 8.75 ms per word), leading to better prediction accuracies. This article is part of the theme issue 'Quantum contextuality, causality and freedom of choice'.

PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss

Abstract The ultimate performance of machine learning algorithms for classification tasks is usually measured in terms of the empirical error probability (or accuracy) using a testing dataset, although these algorithms are optimized through the minimization of a typically different—more convenient—loss function using a training set. For classification tasks, this loss function is often the negative log-loss that yields the well-known cross-entropy risk that is typically better behaved (in terms of numerical behavior) than the 0-1 loss. Conventional studies on the generalization error do not usually take into account the underlying mismatch between losses at training and testing phases. In this work, we introduce a theoretical analysis based on a pointwise probably approximately correct (PAC) approach over the generalization gap considering the mismatch of testing on the accuracy metric and training on the negative log-loss, referred to as PACMAN. Building on the fact that the resulting mismatch can be written as a likelihood ratio, concentration inequalities can be used to obtain insights into the generalization gap in terms of PAC bounds, which depend on some meaningful information-theoretic quantities. An analysis of the obtained bounds and a comparison with available results in the literature is also provided.

Confounding Factor Analysis for Vocal Fold Oscillations.

This paper provides a methodology to better understand the relationships between different aspects of vocal fold motion, which are used as features in machine learning-based approaches for detecting respiratory infections from voice recordings. The relationships are derived through a joint multivariate analysis of the vocal fold oscillations of speakers. Specifically, the multivariate setting explores the displacements and velocities of the left and right vocal folds derived from recordings of five extended vowel sounds for each speaker (/aa/, /iy/, /ey/, /uw/, and /ow/). In this multivariate setting, the differences between the bivariate and conditional interactions are analyzed by information-theoretic quantities based on transfer entropy. Incorporation of the conditional quantities reveals information regarding the confounding factors that can influence the statistical interactions among other pairs of variables. This is demonstrated on a vector autoregressive process where the analytical derivations can be carried out. As a proof of concept, the methodology is applied on a clinically curated dataset of COVID-19. The findings suggest that the interaction between the vocal fold oscillations can change according to individuals and presence of any respiratory infection, such as COVID-19. The results are important in the sense that the proposed approach can be utilized to determine the selection of appropriate features as a supplementary or early detection tool in voice-based diagnostics in future studies.

On Entropy Measures of Some Titania and Carbon Nanotubes

A nanotube is a nanometer–scale tube-like structure, it is a kind of nanoparticle, and may be large enough to serve as a pipe through which other nanoparticles can be channeled, or, depending on the material, may be used as an electrical conductor or an electrical insulator. For computing the structural information of nanotubes, the graph entropies have become the information theoretic quantities. The graph entropy measure has attracted the research community due to its potential application in discrete mathematics, biology, and chemistry. In this paper, our contribution is to explore graph entropies for structures of some nanotubes based on novel information function, which is the number of different degree vertices along with the number of edges between various degree vertices. More precisely, we computed entropies of some classes of nanotubes such as titania nanotube TNT[Formula: see text], TNT[Formula: see text] and carbon nanotubes HAC[Formula: see text] by making a relation of degree-based topological indices with the help of information function.

Information-theoretic analyses of neural data to minimize the effect of researchers' assumptions in predictive coding studies.

Studies investigating neural information processing often implicitly ask both, which processing strategy out of several alternatives is used and how this strategy is implemented in neural dynamics. A prime example are studies on predictive coding. These often ask whether confirmed predictions about inputs or prediction errors between internal predictions and inputs are passed on in a hierarchical neural system-while at the same time looking for the neural correlates of coding for errors and predictions. If we do not know exactly what a neural system predicts at any given moment, this results in a circular analysis-as has been criticized correctly. To circumvent such circular analysis, we propose to express information processing strategies (such as predictive coding) by local information-theoretic quantities, such that they can be estimated directly from neural data. We demonstrate our approach by investigating two opposing accounts of predictive coding-like processing strategies, where we quantify the building blocks of predictive coding, namely predictability of inputs and transfer of information, by local active information storage and local transfer entropy. We define testable hypotheses on the relationship of both quantities, allowing us to identify which of the assumed strategies was used. We demonstrate our approach on spiking data collected from the retinogeniculate synapse of the cat (N = 16). Applying our local information dynamics framework, we are able to show that the synapse codes for predictable rather than surprising input. To support our findings, we estimate quantities applied in the partial information decomposition framework, which allow to differentiate whether the transferred information is primarily bottom-up sensory input or information transferred conditionally on the current state of the synapse. Supporting our local information-theoretic results, we find that the synapse preferentially transfers bottom-up information.

Generalized entropies of subdivision-corona networks

The entropy of a graph is an information-theoretic quantity which expresses the complexity of a graph. Entropy functions have been used successfully to capture different aspects of graph complexity. The generalized graph entropies result from applying information measures to a graph using various schemes for defining probability distributions over the elements of the graph. In this paper, we investigate the complexity of a class of composite graphs based on subdivision graphs and corona product evaluating the generalized graph entropies, and we present explicit formulae for the complexity of subdivision-corona type product graphs.

Geometric quantum discord signals non-factorization

We propose the information-theoretic quantity of geometric quantum discord (GQD) as an indicator of the factorization properties of a given quantum system. In particular, we show how non-vanishing discord implies that the corresponding partition function does not factorize, both for generic pure states and the thermofield double state as a state with a known geometric dual in light of the AdS/CFT correspondence. Via this analysis, we give a novel interpretation to the thermomixed double state as the best purely classical approximation of the Einstein-Rosen bridge. We connect the non-vanishing of GQD with the existence of wormhole microstates.

A Bayesian approach to (online) transfer learning: Theory and algorithms

Transfer learning is a machine learning paradigm where knowledge from one problem is utilized to solve a new but related problem. While conceivable that knowledge from one task could help solve a related task, if not executed properly, transfer learning algorithms can impair the learning performance instead of improving it – commonly known as negative transfer. In this paper, we use a parametric statistical model to study transfer learning from a Bayesian perspective. Specifically, we study three variants of transfer learning problems, instantaneous, online, and time-variant transfer learning. We define an appropriate objective function for each problem and provide either exact expressions or upper bounds on the learning performance using information-theoretic quantities, which allow simple and explicit characterizations when the sample size becomes large. Furthermore, examples show that the derived bounds are accurate even for small sample sizes. The obtained bounds give valuable insights into the effect of prior knowledge on transfer learning, at least with respect to our Bayesian formulation of the transfer learning problem. In particular, we formally characterize the conditions under which negative transfer occurs. Lastly, we devise several (online) transfer learning algorithms that are amenable to practical implementations, some of which do not require the parametric assumption. We demonstrate the effectiveness of our algorithms with real data sets, focusing primarily on when the source and target data have strong similarities.

Topological analysis of information-theoretic quantities in density functional theory.

We have witnessed considerable research interest in the recent literature about the development and applications of quantities from the information-theoretic approach (ITA) in density functional theory. These ITA quantities are explicit density functionals, whose local distributions in real space are continuous and well-behaved. In this work, we further develop ITA by systematically analyzing the topological behavior of its four representative quantities, Shannon entropy, two forms of Fisher information, and relative Shannon entropy (also called information gain or Kullback-Leibler divergence). Our results from their topological analyses for 103 molecular systems provide new insights into bonding interactions and physiochemical properties, such as electrophilicity, nucleophilicity, acidity, and aromaticity. We also compare our results with those from the electron density, electron localization function, localized orbital locator, and Laplacian functions. Our results offer a new methodological approach and practical tool for applications that are especially promising for elucidating chemical bonding and reactivity propensity.

Experimental Guesswork with Quantum Side Information Using Twisted Light.

Guesswork is an information-theoretic quantity which can be seen as an alternate security criterion to entropy. Recent work has established the theoretical framework for guesswork in the presence of quantum side information, which we extend both theoretically and experimentally. We consider guesswork when the side information consists of the BB84 states and their higher-dimensional generalizations. With this side information, we compute the guesswork for two different scenarios for each dimension. We then performed a proof-of-principle experiment using Laguerre-Gauss modes to experimentally compute the guesswork for higher-dimensional generalizations of the BB84 states. We find that our experimental results agree closely with our theoretical predictions. This work shows that guesswork can be a viable security criterion in cryptographic tasks and is experimentally accessible in a number of optical setups.

Causality analysis of large-scale structures in the flow around a wall-mounted square cylinder

The aim of this work is to analyse the formation mechanisms of large-scale coherent structures in the flow around a wall-mounted square cylinder, due to their impact on pollutant transport within cities. To this end, we assess causal relations between the modes of a reduced-order model obtained by applying proper orthogonal decomposition to high-fidelity simulation data of the flow case under study. The causal relations are identified using conditional transfer entropy, which is an information-theoretical quantity that estimates the amount of information contained in the past of one variable about another. This allows for an understanding of the origins and evolution of different phenomena in the flow, with the aim of identifying the modes responsible for the formation of the main vortical structures. Our approach unveils that vortex-breaker modes are the most causal modes, in particular, over higher-order modes, and no significant causal relationships were found for vortex-generator modes. We validate this technique by determining the causal relations present in the nine-equation model of near-wall turbulence developed by Moehliset al.(New J. Phys., vol. 6, 2004, p. 56), which are in good agreement with literature results for turbulent channel flows.

Robust and Fast Measure of Information via Low-Rank Representation

The matrix-based Rényi's entropy allows us to directly quantify information measures from given data, without explicit estimation of the underlying probability distribution. This intriguing property makes it widely applied in statistical inference and machine learning tasks. However, this information theoretical quantity is not robust against noise in the data, and is computationally prohibitive in large-scale applications. To address these issues, we propose a novel measure of information, termed low-rank matrix-based Rényi's entropy, based on low-rank representations of infinitely divisible kernel matrices. The proposed entropy functional inherits the specialty of of the original definition to directly quantify information from data, but enjoys additional advantages including robustness and effective calculation. Specifically, our low-rank variant is more sensitive to informative perturbations induced by changes in underlying distributions, while being insensitive to uninformative ones caused by noises. Moreover, low-rank Rényi's entropy can be efficiently approximated by random projection and Lanczos iteration techniques, reducing the overall complexity from O(n³) to O(n²s) or even O(ns²), where n is the number of data samples and s ≪ n. We conduct large-scale experiments to evaluate the effectiveness of this new information measure, demonstrating superior results compared to matrix-based Rényi's entropy in terms of both performance and computational efficiency.

Nearest-Neighbor Sampling Based Conditional Independence Testing

The conditional randomization test (CRT) was recently proposed to test whether two random variables X and Y are conditionally independent given random variables Z. The CRT assumes that the conditional distribution of X given Z is known under the null hypothesis and then it is compared to the distribution of the observed samples of the original data. The aim of this paper is to develop a novel alternative of CRT by using nearest-neighbor sampling without assuming the exact form of the distribution of X given Z. Specifically, we utilize the computationally efficient 1-nearest-neighbor to approximate the conditional distribution that encodes the null hypothesis. Then, theoretically, we show that the distribution of the generated samples is very close to the true conditional distribution in terms of total variation distance. Furthermore, we take the classifier-based conditional mutual information estimator as our test statistic. The test statistic as an empirical fundamental information theoretic quantity is able to well capture the conditional-dependence feature. We show that our proposed test is computationally very fast, while controlling type I and II errors quite well. Finally, we demonstrate the efficiency of our proposed test in both synthetic and real data analyses.

Complexity and quenches in models with three and four spin interactions

We study information theoretic quantities in models with three and four spin interactions. These models show distinctive characteristics compared to their nearest neighbour (NN) counterparts. Here, we quantify these in terms of the Nielsen complexity (NC) in static and quench scenarios, the Fubini–Study complexity (FSC), and the entanglement entropy (EE). The models that we study have a rich phase structure, and we show how the difference in the nature of phase transitions in these, compared to ones with NN interactions, result in different behaviour of information theoretic quantities, from ones known in the literature. For example, the derivative of the NC does not diverge but shows a discontinuity near continuous phase transitions, and the FSC may be regular and continuous across such transitions. We also study multiple quench scenarios in these models and contrast these with quenches in the transverse XY model. The EE shows a novel discontinuity both at first and second order quantum phase transitions.

Geometric Structures Induced by Deformations of the Legendre Transform

The recent link discovered between generalized Legendre transforms and non-dually flat statistical manifolds suggests a fundamental reason behind the ubiquity of Rényi's divergence and entropy in a wide range of physical phenomena. However, these early findings still provide little intuition on the nature of this relationship and its implications for physical systems. Here we shed new light on the Legendre transform by revealing the consequences of its deformation via symplectic geometry and complexification. These findings reveal a novel common framework that leads to a principled and unified understanding of physical systems that are not well-described by classic information-theoretic quantities.

Phase-space quantum distributions and information theory

The local structure present in Wigner and Husimi phase-space distributions and their marginals are studied and quantified via information-theoretic quantities. Shannon, R\'enyi, and cumulative residual entropies of the Wigner and Husimi distributions are examined in the ground and excited states of a harmonic oscillator. The entropies of the Wigner function marginals are lower than the corresponding entropies of the Husimi function marginals, which illustrates how the nodal structure present in the Wigner function is lost upon consideration of the Husimi function. Shannon and cumulative residual entropies of the Wigner function yield entropies which are complex valued. Absolute values and real components of these quantities increase with quantum number, displaying a behavior which is consistent with their real-valued Husimi function counterparts. This comportment is also similar to the real-valued R\'enyi entropies of the Wigner function. The entropies of the Wigner function are observed to be lower than the corresponding Husimi function ones, in agreement with the results for the marginals. The real components of the Wigner function entropies are seen to be closer to the uncertainty relation bound compared to the corresponding Husimi function entropies. These real components are also closer to the bound when contrasted to the entropic sum of the marginal densities. The R\'enyi or collision entropy of the Wigner function sits exactly on the bound. Related statistical correlation measures show that the position-momentum correlation is larger in the Wigner function compared to the Husimi function and increases with quantum number.