Information-theoretic Measures Research Articles

An information theoretic metric for measurement vulnerability to data integrity attacks on smart grids

AbstractA novel metric that describes the vulnerability of the measurements in power systems to data integrity attacks is proposed. The new metric, coined vulnerability index (VuIx), leverages information theoretic measures to assess the attack effect in terms of the fundamental limits of the disruption and detection tradeoff. The result of computing the VuIx of the measurements in the system yields an ordering of their vulnerability based on the degree of exposure to data integrity attacks. This new framework is used to assess the measurement vulnerability of IEEE 9‐bus and 30‐bus test systems and it is observed that power injection measurements are significantly more vulnerable to data integrity attacks than power flow measurements. A detailed numerical evaluation of the VuIx values for IEEE test systems is provided.

Large-scale patterns set the predictability limit of extreme events in Kolmogorov flow

Events of extreme intensity in turbulent flows from atmospheric to industrial scales have a strong social and economic impact, and hence there is a need to develop models and indicators which enable their early prediction. Part of the difficulty here stems from the intrinsic sensitivity to initial conditions of turbulent flows. Despite recent progress in understanding and predicting extreme events, the question of how far in advance they can be ideally predicted (without model error and subject only to uncertainty in the initial conditions) remains open. Here we study the predictability limit of extreme dissipation bursts in the two-dimensional Kolmogorov flow by applying information-theoretic measures to massive statistical ensembles with more than $10^7$ direct numerical simulations. We find that extreme events with similar intensity and structure can exhibit disparate predictability due to different causal origins. Specifically, we show that highly predictable extreme events evolve from distinct large-scale circulation patterns. We thus suggest that understanding all the possible routes to the formation of extreme events is necessary to assess their predictability.

Graphs with minimum degree-entropy

We continue studying extremal values of the degree-entropy, which is an information-theoretic measure defined as the Shannon entropy based on the information functional involving vertex degrees. For a graph with a given number of vertices and edges achieving the minimum entropy value, we show its unique structure. Also, a tight lower bound for the entropy in bipartite graphs with a given number of vertices and edges is proved. Our result directly derives the result of Cao et al. (2014) that for a tree with a given number of vertices, the minimum value of the entropy is attained if and only if the tree is the star.

Neural Joint Entropy Estimation.

Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics, and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos. The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information (MI). We show that these estimators are strongly consistent and demonstrate their performance in a variety of use cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to MI estimation. Next, we apply the proposed scheme to conditional MI estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy (TE) estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all of these setups.

Forgetfulness of turbulent energy cascade associated with different mechanisms

The small scales of high Reynolds number turbulence exhibit the statistical universality by forgetting about the large scales during the energy cascade. In this study, we revisit this schematic description with an information-theoretic point of view. Our objective is to understand the importance of different cascade mechanisms in this process. To this end, we compute the “information flux”, the information-theoretic measure of causality formulated by one of the authors, of the decomposed inter-scale energy transfer. For the kinetics, the ratio of the inertial-range inter-scale energy transfer between the strain self-amplification (SSA) and the vortex stretching (VS) mechanisms is 5 : 3. However, for the information-flux, the VS contribution is roughly twice as large as the SSA. We also find several deviations between the kinetics and the informatics, highlighting the nontrivial nature of information distributed among different scales and mechanics in turbulence. This result infers that the SSA is a less efficient mechanism than the VS in terms of information transfer, and contributes to losing the large-scale information.

Respiratory modulation of the heart rate: A potential biomarker of cardiorespiratory function in human

In recent decade, wearable digital devices have shown potentials for the discovery of novel biomarkers of humans’ physiology and behavior. Heart rate (HR) and respiration rate (RR) are most crucial bio-signals in humans’ digital phenotyping research. HR is a continuous and non-invasive proxy to autonomic nervous system and ample evidence pinpoints the critical role of respiratory modulation of cardiac function. In the present study, we recorded longitudinal (7 days, 4.63 ± 1.52) HR and RR of 89 freely behaving human subjects (Female: 39, age 57.28 ± 5.67, Male: 50, age 58.48 ± 6.32) and analyzed their dynamics using linear models and information theoretic measures. While HR’s linear and nonlinear characteristics were expressed within the plane of the HR–RR directed flow of information (HR→RR – RR→HR), their dynamics were determined by its RR→HR axis. More importantly, RR→HR quantified the effect of alcohol consumption on individuals’ cardiorespiratory function independent of their consumed amount of alcohol, thereby signifying the presence of this habit in their daily life activities. The present findings provided evidence for the critical role of the respiratory modulation of HR, which was previously only studied in non-human animals. These results can contribute to humans’ phenotyping research by presenting RR→HR as a digital diagnosis/prognosis marker of humans’ cardiorespiratory pathology.

Understanding and Leveraging the Learning Phases of Neural Networks

The learning dynamics of deep neural networks are not well understood. The information bottleneck (IB) theory proclaimed separate fitting and compression phases. But they have since been heavily debated. We comprehensively analyze the learning dynamics by investigating a layer's reconstruction ability of the input and prediction performance based on the evolution of parameters during training. We empirically show the existence of three phases using common datasets and architectures such as ResNet and VGG: (i) near constant reconstruction loss, (ii) decrease, and (iii) increase. We also derive an empirically grounded data model and prove the existence of phases for single-layer networks. Technically, our approach leverages classical complexity analysis. It differs from IB by relying on measuring reconstruction loss rather than information theoretic measures to relate information of intermediate layers and inputs. Our work implies a new best practice for transfer learning: We show empirically that the pre-training of a classifier should stop well before its performance is optimal.

Information Content of Inflation Expectations: A Copula-Based Model

Abstract This paper introduces a holistic framework that integrates copula modeling and information-theoretic measures to examine the information content of inflation expectations. Copulas are used to capture the dynamic dependence between inflation and expectations, accounting for extreme events and tail dependence. Information-theoretic measures are employed to quantify the information that expectations provide about inflation. Theoretical results establish a link between copula entropy and mutual information, propose a lower bound for copula entropy, and provide a practical tool for central banks to anchor expectations to achieve inflation targets. Empirical findings reveal higher uncertainty in the tails of the joint distribution and underscore the meaningful information carried by expected inflation for forecasting inflation, particularly with shorter-term expectations.

Adding causality to the information-theoretic perspective on individuality

I extend work from Krakauer et al. (2020), who propose a conception of individuality as the capacity to propagate information through time. From this conception, they develop information-theoretic measures. I identify several shortcomings with these measures—in particular, that they are associative rather than causal. I rectify this shortcoming by deriving a causal information-theoretic measure of individuality. I then illustrate how this measure can be implemented and extended in the context of evolutionary transitions in individuality.

Performance Comparison of Different HTM-Spatial Pooler Algorithms Based on Information-Theoretic Measures

Hierarchical temporal memory (HTM) is a promising unsupervised machine-learning algorithm that models key principles of neocortical computation. One of the main components of HTM is the spatial pooler (SP), which encodes binary input streams into sparse distributed representations (SDRs). In this paper, we propose an information-theoretic framework for the performance comparison of HTM-spatial pooler (SP) algorithms, specifically, for quantifying the similarities and differences between sparse distributed representations in SP algorithms. We evaluate SP's standalone performance, as well as HTM's overall performance. Our comparison of various SP algorithms using Renyi mutual information, Renyi divergence, and Henze–Penrose divergence measures reveals that the SP algorithm with learning and a logarithmic boosting function yields the most effective and useful data representation. Moreover, the most effective SP algorithm leads to superior HTM results. In addition, we utilize our proposed framework to compare HTM with other state-of-the-art sequential learning algorithms. We illustrate that HTM exhibits superior adaptability to pattern changes over time than long short term memory (LSTM), gated recurrent unit (GRU) and online sequential extreme learning machine (OS-ELM) algorithms. This superiority is evident from the lower Renyi divergence of HTM (0.23) compared to LSTM6000 (0.33), LSTM3000 (0.38), GRU (0.41), and OS-ELM (0.49). HTM also achieved the highest Renyi mutual information value of 0.79, outperforming LSTM6000 (0.73), LSTM3000 (0.71), GRU (0.68), and OS-ELM (0.62). These findings not only confirm the numerous advantages of HTM over other sequential learning algorithm, but also demonstrate the effectiveness of our proposed information-theoretic approach as a powerful framework for comparing and evaluating various learning algorithms.

Generalized decomposition of multivariate information.

Since its introduction, the partial information decomposition (PID) has emerged as a powerful, information-theoretic technique useful for studying the structure of (potentially higher-order) interactions in complex systems. Despite its utility, the applicability of the PID is restricted by the need to assign elements as either "sources" or "targets", as well as the specific structure of the mutual information itself. Here, I introduce a generalized information decomposition that relaxes the source/target distinction while still satisfying the basic intuitions about information. This approach is based on the decomposition of the Kullback-Leibler divergence, and consequently allows for the analysis of any information gained when updating from an arbitrary prior to an arbitrary posterior. As a result, any information-theoretic measure that can be written as a linear combination of Kullback-Leibler divergences admits a decomposition in the style of Williams and Beer, including the total correlation, the negentropy, and the mutual information as special cases. This paper explores how the generalized information decomposition can reveal novel insights into existing measures, as well as the nature of higher-order synergies. We show that synergistic information is intimately related to the well-known Tononi-Sporns-Edelman (TSE) complexity, and that synergistic information requires a similar integration/segregation balance as a high TSE complexity. Finally, I end with a discussion of how this approach fits into other attempts to generalize the PID and the possibilities for empirical applications.

Enhancing Exchange-Traded Fund Price Predictions: Insights from Information-Theoretic Networks and Node Embeddings.

This study presents a novel approach to predicting price fluctuations for U.S. sector index ETFs. By leveraging information-theoretic measures like mutual information and transfer entropy, we constructed threshold networks highlighting nonlinear dependencies between log returns and trading volume rate changes. We derived centrality measures and node embeddings from these networks, offering unique insights into the ETFs' dynamics. By integrating these features into gradient-boosting algorithm-based models, we significantly enhanced the predictive accuracy. Our approach offers improved forecast performance for U.S. sector index futures and adds a layer of explainability to the existing literature.

Quantum conditional mutual information of W state in non-inertial frames

Quantum conditional mutual information (QCMI) is a versatile information theoretic measure. It is used to find the amount of correlations between two qubits from the perspective of a third qubit. In this work, we characterise the QCMI of tripartite W-states when some of the qubits are in accelerated motion. Here for our investigations we consider a massless fermionic field in the single-mode approximation. We consider all possible situations with respect to the acceleration of the qubits. From our results, we observe that QCMI can either increase or decrease depending on the role of the qubit being accelerated. Finally we discuss the connection between QCMI and correlations by studying the biseparable and separable states.

Compressing the atomic cloud in a matter-wave stripe soliton

We consider an attractive quasi-one dimensional spin-orbit coupled Bose–Einstein condensate (SOC BEC) confined in a periodic potential produced by the combination of linear and nonlinear optical lattices, and study the effects of squeezing a stripe soliton by varying the inter-atomic interaction in the nonlinear lattice. It is observed that the nodes on the soliton arising entirely due to the effect of spin–orbit coupling tend to disappear as we increase the squeezing effect to finally get a stable fundamental soliton. This leads us to conclude that external pressure can reduce the effect of spin–orbit coupling in the SOC BEC and even convert the system to a traditional BEC without spin–orbit coupling. We make use of an information theoretic measure to visualize how does the atomic density distribution in the condensate respond to continual reduction in the spin–orbit coupling effect.

Multiscale distribution entropy analysis of short epileptic EEG signals

<abstract> <p>This paper proposes an information-theoretic measure for discriminating epileptic patterns in short-term electroencephalogram (EEG) recordings. Considering nonlinearity and nonstationarity in EEG signals, quantifying complexity has been preferred. To decipher abnormal epileptic EEGs, i.e., ictal and interictal EEGs, via short-term EEG recordings, a distribution entropy (DE) is used, motivated by its robustness on the signal length. In addition, to reflect the dynamic complexity inherent in EEGs, a multiscale entropy analysis is incorporated. Here, two multiscale distribution entropy (MDE) methods using the coarse-graining and moving-average procedures are presented. Using two popular epileptic EEG datasets, i.e., the Bonn and the Bern-Barcelona datasets, the performance of the proposed MDEs is verified. Experimental results show that the proposed MDEs are robust to the length of EEGs, thus reflecting complexity over multiple time scales. In addition, the proposed MDEs are consistent irrespective of the selection of short-term EEGs from the entire EEG recording. By evaluating the Man-Whitney U test and classification performance, the proposed MDEs can better discriminate epileptic EEGs than the existing methods. Moreover, the proposed MDE with the moving-average procedure performs marginally better than one with the coarse-graining. The experimental results suggest that the proposed MDEs are applicable to practical seizure detection applications.</p> </abstract>

Inferring the roles of individuals in collective systems using information-theoretic measures of influence

Inferring the roles of individuals in collective systems using information-theoretic measures of influence

Face spoofing detection using Heterogeneous Auto-Similarities of Characteristics

Facial recognition systems are widely used in various daily applications due to their high accuracy rate and ease of use. Unfortunately, their performance drops drastically when encountering spoofing attacks. In this paper, a novel pipeline for face spoofing detection based on the extraction of Heterogeneous Auto-Similarities of Characteristics (HASC) descriptor from the HSV and YCbCr color spaces is proposed. To detect print and replay spoofing attacks, the HASC descriptor encodes linear and non-linear dependencies between low-level dense features of the color face image using covariance and information-theoretic measures. The combination of covariance and information-theoretic measures makes HASC an efficient descriptor to explore how the features are interrelated in real and fake faces. The HASC texture descriptions are extracted separately from each color channel of the HSV and YCbCr color spaces, and the resulting feature vectors are concatenated into an extended feature vector to better discriminate between real and fake faces. Extensive experiments are conducted on publicly available databases namely, Replay-Attack, CASIA FASD, and MSU MFSD. The obtained results demonstrate that while being of low dimension, the HASC descriptor when fed into a binary Support Vector Machine (SVM) classifier, achieves competitive results compared to the state-of-the-art.

Statistical complexity and stochastic resonance of an underdamped bistable periodic potential system excited by Lévy noise

This paper focuses on the underdamped bistable potential system's stochastic resonance (SR) phenomenon under Lévy noise excitation. The statistical complexity of the system is analyzed under the changes of damping factor, amplitude and frequency of the periodic signal, stability index and skew parameter of Lévy noise, and the effects of the changes of above parameters on SR are investigated. The results show that the increase of the damping factor hinders the appearance of SR phenomenon; the larger amplitude and lower frequency of the periodic signal are beneficial to the appearance of SR; the larger stability index of the noise parameter, the more likely SR occurs, and the change of the skew parameter has little effect on the SR. Because the Bandt-Pompe (B-P) algorithm is used in the information-theoretic measure to model the statistical complexity of this system, the robustness of the method is analyzed by changing the embedding dimension and the length of residual time interval sequence in the B-P algorithm. Finally, the mean first passage time of the particle is calculated as a function of the noise intensity and the noise-enhanced stability effect is found.

Stoichiometry validation of supramolecular complexes with a hydrocarbon cage host by van ’t Hoff analyses

Defining chemical processes with equations is the first important step in characterizing equilibria for the assembly of supramolecular complexes, and the stoichiometry of the assembled components must be defined to generate the equation. Recently, this subject has attracted renewed interest, and statistical and/or information-theoretic measures were introduced to examine the validities of the equilibrium models used during curve fitting analyses of titration. The present study shows that these measures may not always be appropriate for credibility examinations and that further reformation of the protocols used to determine the overall stoichiometry is necessary. Hydrocarbon cage hosts and their chloroform complexes formed via weak CH-π hydrogen bonds were studied, which allowed us to introduce van ’t Hoff analyses for effective validation of the stoichiometries of supramolecular complexes. This study shows that the stoichiometries of supramolecular complexes should be carefully examined by adopting multiple measures with different origins.

Causal Analysis of Influence of the Solar Cycle and Latitudinal Solar-Wind Structure on Co-Rotation Forecasts

Studying solar-wind conditions is central to forecasting the impact of space weather on Earth. Under the assumption that the structure of this wind is constant in time and co-rotates with the Sun, solar-wind and thereby space-weather forecasts have been made quite effectively. Such co-rotation forecasts are well studied with decades of observations from STEREO and near-Earth spacecraft. Forecast accuracy is primarily determined by three factors: i) the longitudinal separation of spacecraft from Earth determines the corotation time (and hence forecast lead time) [δ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\delta$\\end{document}t] over which the solar wind must be assumed to be constant, ii) the latitudinal separation (or offset) between Earth and spacecraft [δθ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\delta\ heta$\\end{document}]] determines the degree to which the same solar wind is being encountered at both locations, and iii) the solar cycle, via the sunspot number (SSN), acts as a proxy for both how fast the solar-wind structure is evolving and how much it varies in latitude. However, the precise dependencies factoring in uncertainties are a mixture of influences from each of these factors. Furthermore, for high-precision forecasts, it is important to understand what drives the forecast accuracy and its uncertainty. Here we present a causal inference approach based on information-theoretic measures to do this. Our framework can compute not only the direct (linear and nonlinear) dependencies of the forecast mean absolute error (MAE) on SSN, Δθ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\Delta \ heta $\\end{document}, and Δt\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\Delta t$\\end{document}, but also how these individual variables combine to enhance or diminish the MAE. We provide an initial assessment of this with the potential of aiding data assimilation in the future.