Information-theoretic Quantities Research Articles

Partition function approach to non-Gaussian likelihoods: macrocanonical partitions and replicating Markov-chains

Monte-Carlo techniques are standard numerical tools for exploring non-Gaussian and multivariate likelihoods. Many variants of the original Metropolis-Hastings algorithm have been proposed to increase the sampling efficiency. Motivated by Ensemble Monte Carlo we allow the number of Markov chains to vary by exchanging particles with a reservoir, controlled by a parameter analogous to a chemical potential μ, which effectively establishes a random process that samples microstates from a macrocanonical instead of a canonical ensemble. In this paper, we develop the theory of macrocanonical sampling for statistical inference on the basis of Bayesian macrocanonical partition functions, thereby bringing to light the relations between information-theoretical quantities and thermodynamic properties. Furthermore, we propose an algorithm for macrocanonical sampling, 𝙰𝚟𝚊𝚕𝚊𝚗𝚌𝚑𝚎 𝚂𝚊𝚖𝚙𝚕𝚒𝚗𝚐, and apply it to various toy problems as well as the likelihood on the cosmological parameters Ωm and w on the basis of data from the supernova distance redshift relation.

Information theoretic measures for Lifshitz system

In this work, we have studied various mixed state information theoretic quantities for an excited state of Lifshitz spacetime in 3 + 1-dimensions. This geometry is the gravity dual to a class of 2 + 1-dimensional quantum field theories having Lifshitz symmetry. We have holographically calculated mutual information, entanglement wedge cross section, entanglement negativity and mutual complexity for strip like subsystems at the boundary. For this we have used the results of holographic entanglement entropy and complexity present in the literature. We first calculate all of these mentioned quantities for the pure state of Lifshitz spacetime. Then we have moved on to calculate all these quantities for excited state of the Lifshitz spacetime. The gravity dual of excited state of Lifshitz systems in field theory can be obtained by applying constant perturbations along the boundary direction. Further, we would like to mention that for the simplicity of calculation we are only considering results up to the first order in perturbation. The change in the obtained holographic information theoretic quantities are then related to entanglement entropy, entanglement pressure, entanglement chemical potential and charge using the stress tensor complex. These relations are analogous to the first law of entanglement thermodynamics given earlier in the literature. All the calculations are carried out for both values of dynamical scaling exponent (z) present in the Lifshitz field theory.

Information-theoretic quantities as effective descriptors of electrophilicity and nucleophilicity in density functional theory.

Electrophilicity and nucleophilicity are two vastly important chemical concepts gauging the capability of atoms in molecules to accept and donate the maximal number of electrons. In our earlier studies, we proposed to simultaneously quantify them using the Kullback-Leibler divergence from the information-theoretic approach in density functional theory. However, several issues with this scheme remain to be clarified such as its general validity, predictability, and relationship with other information-theoretic quantities. In this work, we revisit the matter with bigger datasets and deeper theoretical insights. Five information-theoretic quantities including Kullback-Leibler divergence, Hirshfeld charge, Ghost-Berkowitz-Parr entropy, and second and third orders of relative Onicescu information energy are found to be reliable and robust descriptors of electrophilicity and nucleophilicity propensities. Employing these five descriptors, we design a list of new compounds and predict their electrophilicity and nucleophilicity scales. This work should markedly improve our confidence and capability in applying information-theoretic quantities to evaluate electrophilicity and nucleophilicity propensities and henceforth pave the route for more applications of these quantities from information-theoretic approach in density functional theory in the future. All structures were fully optimized at the M06-2X/6-311 + G(d) level of DFT functional using the Gaussian 16 package (version C01) with integration grids and tight self-consistent-field convergence. The solvent effect was taken into account by using the implicit solvent model (CPCM) in the CH2Cl2 solvent, and all 3D contour surfaces of Fukui function, local temperature, and ITA (information-theoretic approach) quantities were generated by GaussView. The Multiwfn 3.8 program was used to calculate the ITA indexes and atomic charges.

Density-based quantification of steric effects: validation by Taft steric parameters from acid-catalyzed hydrolysis of esters.

The steric effect is one of the most widely used concepts for chemical understanding in publications and textbooks, yet a well-accepted formulation of this effect is still elusive. Experimentally, this concept was quantified by the acid-catalyzed hydrolysis of esters, yielding the so-called Taft steric parameter. Theoretically, we recently proposed a density-based scheme to quantify the effect from density functional theory. In this work, we directly compare these two schemes, one from theory and the other from experiment. To this end, we first establish the ester hydrolysis mechanism with multiple water molecules explicitly considered and then apply the energetic span model to represent the hydrolysis barrier height between the two schemes. Our results show that the barrier height of the reaction series is strongly correlated with both Taft steric parameters from experiment and steric quantification from theory. We also obtained strong correlations with steric potential, steric force, and steric charge from our theoretical scheme. Strong correlations with a few information-theoretic quantities are additionally unveiled. To the best of our knowledge, this is the first time in the literature that such a direct comparison between theoretical and experimental results is made. These results also suggest that our proposed two-water three-step mechanism for ester hydrolysis is effective, and our theoretical quantification of the steric effect is valid, robust, and experimentally comparable. In our view, this work should have satisfactorily addressed the issue of how the steric effect can be formulated and quantified, and thus it lays the groundwork for future applications.

Deriving measurement collapse using zeta function regularisation and speculative measurement theory

This paper shows how an application of zeta function regularisation to a physical model of quantum measurement yields a solution to the problem of wavefunction collapse. Realistic measurement dynamics based on a particle becoming non-isolated are introduced and, based on this, an outcome function is derived using the method of maximum entropy. It is shown how regularisation of an information theoretic quantity related to this outcome function leads to apparent collapse of the wavefunction. The physical principles and key assumptions that underlie this theory are discussed. Some possible experimental approaches are described.

Minimum Mean Squared Error Estimation and Mutual Information Gain

Information theoretic quantities such as entropy, entropy rate, information gain, and relative entropy are often used to understand the performance of intelligent agents in learning applications. Mean squared error has not played a role in these analyses, primarily because it is not felt to be a viable performance indicator in these scenarios. We build on a new quantity, the log ratio of entropy powers, to establish that minimum mean squared error (MMSE) estimation, prediction, and smoothing are directly connected to mutual information gain or loss in an agent learning system modeled by a Markov chain for many probability distributions of interest. Expressions for mutual information gain or loss are developed for MMSE estimation, prediction, and smoothing, and an example for fixed lag smoothing is presented.

Statistical mechanics of stochastic quantum control: d-adic Rényi circuits.

The dynamics of quantum information in many-body systems with large onsite Hilbert space dimension admits an enlightening description in terms of effective statistical mechanics models. Motivated by this fact, we reveal a connection between three separate models: the classically chaotic d-adic Rényi map with stochastic control, a quantum analog of this map for qudits, and a Potts model on a random graph. The classical model and its quantum analog share a transition between chaotic and controlled phases, driven by a randomly applied control map that attempts to order the system. In the quantum model, the control map necessitates measurements that concurrently drive a phase transition in the entanglement content of the late-time steady state. To explore the interplay of the control and entanglement transitions, we derive an effective Potts model from the quantum model and use it to probe information-theoretic quantities that witness both transitions. The entanglement transition is found to be in the bond-percolation universality class, consistent with other measurement-induced phase transitions, while the control transition is governed by a classical random walk. These two phase transitions can be made to coincide by varying a parameter in the model, producing a picture consistent with behavior observed in previous small-size numerical studies of the quantum model.

Cone-restricted information theory

The max-relative entropy and the conditional min-entropy of a quantum state plays a central role in one-shot and zero-error quantum information theory. One attractive feature of this quantity is that it can be expressed as an optimization over the cone of positive semidefinite operators. Recently, it was shown that when replacing this cone with the cone of separable operators, a new type of conditional min-entropy emerges that admits an operational interpretation in terms of communicating classical information over a quantum channel. In this work, we explore more deeply the idea of building information-theoretic quantities from different base cones and determine which results in quantum information theory rely upon the positive semidefinite cone and which can be generalized. In terms of asymptotic information processing, we find that the standard equipartition properties break down if a given cone fails to approximate the positive semidefinite cone sufficiently well. We also show that the near-equivalence of the smooth max and Hartley entropies breaks down in this setting. We present parallel results for the extended conditional min-entropy, which requires extending the notion of k-superpositive channels to superchannels. On the other hand, we show that for classical-quantum states the separable cone is sufficient to re-cover the asymptotic theory, thereby drawing a strong distinction between the fully and partial quantum settings. We also present operational uses of this framework. We show that the cone restricted min-entropy of a Choi operator captures a measure of entanglement-assisted noiseless classical communication using restricted measurements. We also introduce a novel min-entropy-like quantity that captures the conditions for when one quantum channel can be transformed into another using bistochastic pre-processing. Lastly, we relate this framework to general conic norms and their non-additivity. Throughout this work, we concretely study generalized entropies in resource theories that capture locality and resource theories of coherence/Abelian symmetries.

Diagnostics of Mixed-State Topological Order and Breakdown of Quantum Memory

Topological quantum memory can protect information against local errors up to finite error thresholds. Such thresholds are usually determined based on the success of decoding algorithms rather than the intrinsic properties of the mixed states describing corrupted memories. Here we provide an intrinsic characterization of the breakdown of topological quantum memory, which both gives a bound on the performance of decoding algorithms and provides examples of topologically distinct mixed states. We employ three information-theoretical quantities that can be regarded as generalizations of the diagnostics of ground-state topological order, and serve as a definition for topological order in error-corrupted mixed states. We consider the topological contribution to entanglement negativity and two other metrics based on quantum relative entropy and coherent information. In the concrete example of the two-dimensional (2D) Toric code with local bit-flip and phase errors, we map three quantities to observables in 2D classical spin models and analytically show they all undergo a transition at the same error threshold. This threshold is an upper bound on that achieved in any decoding algorithm and is indeed saturated by that in the optimal decoding algorithm for the Toric code. Published by the American Physical Society 2024

Information entropies with Varshni-Hellmann potential in higher dimensions

This work investigates the behavior of Shannon entropy and Fisher information for the Varshni-Hellmann potential (VHP) in one and three dimensions using the Nikiforov-Uvarov method. We employ the Greene-Aldrich approximation scheme to obtain the energy eigenvalues and normalized wavefunctions, which are then used to calculate these information-theoretic quantities. Our analysis revealed remarkably similar high-order features in both position and momentum spaces. Notably, our calculations showed enhanced accuracy in predicting particle localization within position space. Furthermore, the combined position and momentum entropies obeyed the lower and upper bounds established by the Berkner-Bialynicki-Birula-Mycieslki inequality. Additionally, for three-dimensional systems, the Stam-Cramer-Rao inequalities were fulfilled for different eigenstates with respect to the calculated Fisher information. It is observed that as the position Fisher entropy decreases, indicating a more precise measurement of position, the momentum Fisher entropy must increase. This implies that the Fisher information regarding momentum decreases, resulting in a decrease in the precision of momentum measurement. This demonstrates how position and momentum uncertainties complement each other in quantum mechanics. Exploring the balance between position and momentum Fisher entropy reveals a fundamental aspect of the uncertainty principle in quantum mechanics, highlighting the restrictions on measuring certain pairs of conjugate variables simultaneously with high precision.

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data.

Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

To Compress or Not to Compress-Self-Supervised Learning and Information Theory: A Review.

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.

What is Quantum in probabilistic explanations of the sure-thing principle violation?

The Prisoner’s Dilemma game (PDG) is one of the simple test-beds for the probabilistic nature of the human decision-making process. Behavioral experiments have been conducted on this game for decades and show a violation of the so-called sure-thing principle, a key principle in the rational theory of decision. Quantum probabilistic models can explain this violation as a second-order interference effect, which cannot be accounted for by classical probability theory. Here, we adopt the framework of generalized probabilistic theories and approach this explanation from the viewpoint of quantum information theory to identify the source of the interference. In particular, we reformulate one of the existing quantum probabilistic models using density matrix formalism and consider different amounts of classical and quantum uncertainties for one player’s prediction about another player’s action in PDG. This enables us to demonstrate that what makes possible the explanation of the violation is the presence of quantum coherence in the player’s initial prediction and its conversion to probabilities during the dynamics. Moreover, we discuss the role of other quantum information-theoretical quantities, such as quantum entanglement, in the decision-making process. Finally, we propose a three-choice extension of the PDG to compare the predictive powers of quantum probability theory and a more general probabilistic theory that includes it as a particular case and exhibits third-order interference.

Segmented strings and holography

In this paper we establish a connection between segmented strings propagating in AdSd+1 and CFTd subsystems in Minkowski spacetime characterized by quantum information theoretic quantities calculated for the vacuum state. We show that the area of the world sheet of a string segment on the AdS side can be connected to fidelity susceptibility (the real part of the quantum geometric tensor) on the CFT side. This quantity has another interpretation as the computational complexity for infinitesimally separated states corresponding to causal diamonds that are displaced in a spacelike manner according to the metric of kinematic space. These displaced causal diamonds encode information for a unique reconstruction of the string world sheet segments in a holographic manner. Dually the bulk segments are representing causally ordered sets of consecutive boundary events in boosted inertial frames or in noninertial ones proceeding with constant acceleration. For the special case of AdS3 one can also see the segmented stringy area in units of 4GL (G is Newton’s constant and L is the AdS length) as the conditional mutual information I(A,C|B) calculated for a trapezoid configuration arising from boosted spacelike intervals A, B and C. In this special case the variation of the discretized Nambu-Goto action leads to an equation for entanglement entropies in the boundary theory of the form of a Toda equation. For arbitrary d the string world sheet patches are living in the modular slices of the entanglement wedge. They seem to provide some sort of tomography of the entanglement wedge where the patches are linked together by the interpolation ansatz, i.e., the discretized version of the equations of motion for the Nambu-Goto action. Published by the American Physical Society 2024

Causality and signalling of garden-path sentences.

Sheaves are mathematical objects that describe the globally compatible data associated with open sets of a topological space. Original examples of sheaves were continuous functions; later they also became powerful tools in algebraic geometry, as well as logic and set theory. More recently, sheaves have been applied to the theory of contextuality in quantum mechanics. Whenever the local data are not necessarily compatible, sheaves are replaced by the simpler setting of presheaves. In previous work, we used presheaves to model lexically ambiguous phrases in natural language and identified the order of their disambiguation. In the work presented here, we model syntactic ambiguities and study a phenomenon in human parsing called garden-pathing. It has been shown that the information-theoretic quantity known as 'surprisal' correlates with human reading times in natural language but fails to do so in garden-path sentences. We compute the degree of signalling in our presheaves using probabilities from the large language model BERT and evaluate predictions on two psycholinguistic datasets. Our degree of signalling outperforms surprisal in two ways: (i) it distinguishes between hard and easy garden-path sentences (with a [Formula: see text]-value [Formula: see text]), whereas existing work could not, (ii) its garden-path effect is larger in one of the datasets (32 ms versus 8.75 ms per word), leading to better prediction accuracies. This article is part of the theme issue 'Quantum contextuality, causality and freedom of choice'.

PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss

Abstract The ultimate performance of machine learning algorithms for classification tasks is usually measured in terms of the empirical error probability (or accuracy) using a testing dataset, although these algorithms are optimized through the minimization of a typically different—more convenient—loss function using a training set. For classification tasks, this loss function is often the negative log-loss that yields the well-known cross-entropy risk that is typically better behaved (in terms of numerical behavior) than the 0-1 loss. Conventional studies on the generalization error do not usually take into account the underlying mismatch between losses at training and testing phases. In this work, we introduce a theoretical analysis based on a pointwise probably approximately correct (PAC) approach over the generalization gap considering the mismatch of testing on the accuracy metric and training on the negative log-loss, referred to as PACMAN. Building on the fact that the resulting mismatch can be written as a likelihood ratio, concentration inequalities can be used to obtain insights into the generalization gap in terms of PAC bounds, which depend on some meaningful information-theoretic quantities. An analysis of the obtained bounds and a comparison with available results in the literature is also provided.

Confounding Factor Analysis for Vocal Fold Oscillations.

This paper provides a methodology to better understand the relationships between different aspects of vocal fold motion, which are used as features in machine learning-based approaches for detecting respiratory infections from voice recordings. The relationships are derived through a joint multivariate analysis of the vocal fold oscillations of speakers. Specifically, the multivariate setting explores the displacements and velocities of the left and right vocal folds derived from recordings of five extended vowel sounds for each speaker (/aa/, /iy/, /ey/, /uw/, and /ow/). In this multivariate setting, the differences between the bivariate and conditional interactions are analyzed by information-theoretic quantities based on transfer entropy. Incorporation of the conditional quantities reveals information regarding the confounding factors that can influence the statistical interactions among other pairs of variables. This is demonstrated on a vector autoregressive process where the analytical derivations can be carried out. As a proof of concept, the methodology is applied on a clinically curated dataset of COVID-19. The findings suggest that the interaction between the vocal fold oscillations can change according to individuals and presence of any respiratory infection, such as COVID-19. The results are important in the sense that the proposed approach can be utilized to determine the selection of appropriate features as a supplementary or early detection tool in voice-based diagnostics in future studies.

On Entropy Measures of Some Titania and Carbon Nanotubes

A nanotube is a nanometer–scale tube-like structure, it is a kind of nanoparticle, and may be large enough to serve as a pipe through which other nanoparticles can be channeled, or, depending on the material, may be used as an electrical conductor or an electrical insulator. For computing the structural information of nanotubes, the graph entropies have become the information theoretic quantities. The graph entropy measure has attracted the research community due to its potential application in discrete mathematics, biology, and chemistry. In this paper, our contribution is to explore graph entropies for structures of some nanotubes based on novel information function, which is the number of different degree vertices along with the number of edges between various degree vertices. More precisely, we computed entropies of some classes of nanotubes such as titania nanotube TNT[Formula: see text], TNT[Formula: see text] and carbon nanotubes HAC[Formula: see text] by making a relation of degree-based topological indices with the help of information function.

Information-theoretic analyses of neural data to minimize the effect of researchers' assumptions in predictive coding studies.

Studies investigating neural information processing often implicitly ask both, which processing strategy out of several alternatives is used and how this strategy is implemented in neural dynamics. A prime example are studies on predictive coding. These often ask whether confirmed predictions about inputs or prediction errors between internal predictions and inputs are passed on in a hierarchical neural system-while at the same time looking for the neural correlates of coding for errors and predictions. If we do not know exactly what a neural system predicts at any given moment, this results in a circular analysis-as has been criticized correctly. To circumvent such circular analysis, we propose to express information processing strategies (such as predictive coding) by local information-theoretic quantities, such that they can be estimated directly from neural data. We demonstrate our approach by investigating two opposing accounts of predictive coding-like processing strategies, where we quantify the building blocks of predictive coding, namely predictability of inputs and transfer of information, by local active information storage and local transfer entropy. We define testable hypotheses on the relationship of both quantities, allowing us to identify which of the assumed strategies was used. We demonstrate our approach on spiking data collected from the retinogeniculate synapse of the cat (N = 16). Applying our local information dynamics framework, we are able to show that the synapse codes for predictable rather than surprising input. To support our findings, we estimate quantities applied in the partial information decomposition framework, which allow to differentiate whether the transferred information is primarily bottom-up sensory input or information transferred conditionally on the current state of the synapse. Supporting our local information-theoretic results, we find that the synapse preferentially transfers bottom-up information.

Generalized entropies of subdivision-corona networks

The entropy of a graph is an information-theoretic quantity which expresses the complexity of a graph. Entropy functions have been used successfully to capture different aspects of graph complexity. The generalized graph entropies result from applying information measures to a graph using various schemes for defining probability distributions over the elements of the graph. In this paper, we investigate the complexity of a class of composite graphs based on subdivision graphs and corona product evaluating the generalized graph entropies, and we present explicit formulae for the complexity of subdivision-corona type product graphs.