Information-theoretic Tools Research Articles

Functional Hypergraphs of Stock Markets.

In stock markets, nonlinear interdependencies between various companies result in nontrivial time-varying patterns in stock prices. A network representation of these interdependencies has been successful in identifying and understanding hidden signals before major events like stock market crashes. However, these studies have revolved around the assumption that correlations are mediated in a pairwise manner, whereas, in a system as intricate as this, the interactions need not be limited to pairwise only. Here, we introduce a general methodology using information-theoretic tools to construct a higher-order representation of the stock market data, which we call functional hypergraphs. This framework enables us to examine stock market events by analyzing the following functional hypergraph quantities: Forman-Ricci curvature, von Neumann entropy, and eigenvector centrality. We compare the corresponding quantities of networks and hypergraphs to analyze the evolution of both structures and observe features like robustness towards events like crashes during the course of a time period.

Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis.

We study stochastic linear contextual bandits (CB) where the agent observes a noisy version of the true context through a noise channel with unknown channel parameters. Our objective is to design an action policy that can "approximate" that of a Bayesian oracle that has access to the reward model and the noise channel parameter. We introduce a modified Thompson sampling algorithm and analyze its Bayesian cumulative regret with respect to the oracle action policy via information-theoretic tools. For Gaussian bandits with Gaussian context noise, our information-theoretic analysis shows that under certain conditions on the prior variance, the Bayesian cumulative regret scales as O˜(mT), where m is the dimension of the feature vector and T is the time horizon. We also consider the problem setting where the agent observes the true context with some delay after receiving the reward, and show that delayed true contexts lead to lower regret. Finally, we empirically demonstrate the performance of the proposed algorithms against baselines.

Association of State Firearm Laws With Firearm Ownership and Mortality

IntroductionFirearm injury is a leading cause of death among Americans. As the right to bear arms is protected by the Second Amendment, policy-makers must consider the impact of legislation on both firearm ownership and firearm harms. Current state of knowledge in firearm research majorly examines the impact of firearm legislation on firearm injuries and fatalities alone, and it relies on correlational analyses. The few studies that consider causal effects employ counterfactual-based inference. This study introduces information-theoretic tools to explore the role of firearm laws in mitigating firearm harms while maintaining citizens’ right to bear arms MethodsWe study monthly time series from January 2000 to October 2019 for the implementation of firearm laws from RAND's State Firearm Law Database, firearm deaths by intent from the Centers for Disease Control and Prevention databases, and firearm ownership from an econometric model. We employ transfer entropy, an information-theoretic method that relies on Granger causality, to infer relationships from time series. Specifically, we examine transfer entropy from firearm restrictiveness to deaths per firearm owner, firearm ownership, and firearm deaths, independently ResultsOn a national level, we uncover a negative association from firearm restrictiveness to deaths per firearm owner and a positive association from firearm restrictiveness to firearm ownership. On a regional level, we identify a negative association from firearm restrictiveness to deaths per firearm owner in the Northeast, a negative association from firearm restrictiveness to firearm ownership in the Midwest, and a negative association from firearm restrictiveness to firearm suicides in the South ConclusionsWe present an information-theoretic approach to study relationships in firearm research. Our method provides preliminary evidence for the role of restrictive legislation in promoting safe firearm ownership. We find that firearm acquisition considerably increases following the implementation of restrictive firearm laws and, simultaneously, firearm deaths decrease. These effects vary with respect to death by intent and the geographical region laws were implemented in.

Efficiency and controllability of stochastic boolean function generation by a random network of non-linear nanoparticle junctions

Amid efforts to address energy consumption in modern computing systems, one promising approach takes advantage of random networks of non-linear nanoscale junctions formed by nanoparticles as substrates for neuromorphic computing. These networks exhibit emergent complexity and collective behaviors akin to biological neural networks, characterized by self-organization, redundancy, and non-linearity. Based on this foundation, a generalization of n-inputs devices has been proposed, where the associated weights depend on all the input values. This model, called receptron, has demonstrated its capability to generate Boolean functions as output, representing a significant breakthrough in unconventional computing methods. In this work, we characterize and present two actual implementations of this paradigm. One approach leverages the nanoscale properties of cluster-assembled Au films, while the other utilizes the recently introduced Stochastic Resistor Network (SRN) model. We first provide a concise overview of the electrical properties of these systems, emphasizing the insights gained from the SRN regarding the physical processes within real nanostructured gold films at a coarse-grained scale. Furthermore, we present evidence indicating the minimum complexity level required by the SRN model to achieve a stochastic dynamics adequate to effectively model a novel component for logic systems. To support our argument that these systems are preferable to conventional random search algorithms, we discuss quantitative criteria based on Information-theoretic tools. This suggests a practical means to steer the stochastic dynamics of the system in a controlled way, thus focusing its random exploration where it is most useful.

A quantization-coding scheme with variable data rates for cyber-physical systems under DoS attacks

This paper considers Cyber-Physical Systems (CPSs) in the presence of Denial-of-Service (DoS) attacks, and investigates the networked control problem under data rate limitations. We develop a framework for CPSs in the presence of DoS attacks, and present a state-estimate control scheme for CPSs under DoS attacks. For the different communication status, the information on the plant state is transmitted to the controller by employing a different data rate in order to mitigate the influence of DoS attacks. Sufficient conditions on the data rate for mean square observability and stabilization of CPSs are derived by applying information theoretic tools of source coding. Our result is less conservative than some existing results. An illustrative example is given to demonstrate the effectiveness of the proposed quantization-coding scheme.

Validating a data-driven framework for vehicular traffic modeling

This study presents a data-driven framework for modeling complex systems, with a specific emphasis on traffic modeling. Traditional methods in traffic modeling often rely on assumptions regarding vehicle interactions. Our approach comprises two steps: first, utilizing information- theoretic (IT) tools to identify interaction directions and candidate variables thus eliminating assumptions, and second, employing the sparse identification of nonlinear systems (SINDy) tool to establish functional relationships. We validate the framework’s efficacy using synthetic data from two distinct traffic models, while considering measurement noise. Results show that IT tools can reliably detect directions of interaction as well as instances of no interaction. SINDy proves instrumental in creating precise functional relationships and determining coefficients in tested models. The innovation of our framework lies in its ability to use data-driven approach to model traffic dynamics without relying on assumptions, thus offering applications in various complex systems beyond traffic.

Cross-frequency cortex-muscle interactions are abnormal in young people with dystonia.

Sensory processing and sensorimotor integration are abnormal in dystonia, including impaired modulation of beta-corticomuscular coherence. However, cortex-muscle interactions in either direction are rarely described, with reports limited predominantly to investigation of linear coupling, using corticomuscular coherence or Granger causality. Information-theoretic tools such as transfer entropy detect both linear and non-linear interactions between processes. This observational case-control study applies transfer entropy to determine intra- and cross-frequency cortex-muscle coupling in young people with dystonia/dystonic cerebral palsy. Fifteen children with dystonia/dystonic cerebral palsy and 13 controls, aged 12-18 years, performed a grasp task with their dominant hand. Mechanical perturbations were provided by an electromechanical tapper. Bipolar scalp EEG over contralateral sensorimotor cortex and surface EMG over first dorsal interosseous were recorded. Multi-scale wavelet transfer entropy was applied to decompose signals into functional frequency bands of oscillatory activity and to quantify intra- and cross-frequency coupling between brain and muscle. Statistical significance against the null hypothesis of zero transfer entropy was established, setting individual 95% confidence thresholds. The proportion of individuals in each group showing significant transfer entropy for each frequency combination/direction was compared using Fisher's exact test, correcting for multiple comparisons. Intra-frequency transfer entropy was detected in all participants bidirectionally in the beta (16-32 Hz) range and in most participants from EEG to EMG in the alpha (8-16 Hz) range. Cross-frequency transfer entropy across multiple frequency bands was largely similar between groups, but a specific coupling from low-frequency EMG to beta EEG was significantly reduced in dystonia [P = 0.0061 (corrected)]. The demonstration of bidirectional cortex-muscle communication in dystonia emphasizes the value of transfer entropy for exploring neural communications in neurological disorders. The novel finding of diminished coupling from low-frequency EMG to beta EEG in dystonia suggests impaired cortical feedback of proprioceptive information with a specific frequency signature that could be relevant to the origin of the excessive low-frequency drive to muscle.

Modeling a domain wall network in BiFeO3 with stochastic geometry and entropy-based similarity measure

A compact and tractable two-dimensional model to generate the topological network structure of domain walls in BiFeO3 thin films is presented in this study. Our method combines the stochastic geometry parametric model of the centroidal Voronoi tessellation optimized using the von Neumann entropy, a novel information-theoretic tool for networks. The former permits the generation of image-based stochastic artificial samples of domain wall networks, from which the network structure is subsequently extracted and converted to the graph-based representation. The von Neumann entropy, which reflects information diffusion across multiple spatiotemporal scales in heterogeneous networks, plays a central role in defining a fitness function. It allows the use of the network as a whole rather than using a subset of network descriptors to search for optimal model parameters. The optimization of the parameters is carried out by a genetic algorithm through the maximization of the fitness function and results in the desired graph-based network connectivity structure. Ground truth empirical networks are defined, and a dataset of network connectivity structures of domain walls in BiFeO3 thin films is undertaken through manual annotation. Both a versatile tool for manual network annotation of noisy images and a new automatic network extraction method for high-quality images are developed.

Entropy and the discrete central limit theorem

A strengthened version of the central limit theorem for discrete random variables is established, relying only on information-theoretic tools and elementary arguments. It is shown that the relative entropy between the standardised sum of n independent and identically distributed lattice random variables and an appropriately discretised Gaussian, vanishes as n→∞.

Timely Inference over Networks

Brief Biography: Md Kamran Chowdhury Shisher is a final-year Ph.D. candidate in the Department of Electrical and Computer Engineering at Auburn University, Auburn, AL, USA. He expects to graduate by May 2024. His Ph.D. works are advised by Yin Sun. Shisher's research interests include Information Freshness, Wireless Networks, Semantic Communications, and Machine Learning. His Ph.D. works focus on designing low-complexity optimal scheduling algorithms for remote inference systems. He also developed an information-theoretic tool to analyze the impact of freshness on the performance of real-time applications. He received his M.Sc. degree in Electrical Engineering at Auburn University in 2022. He completed his B.Sc. in Electrical and Electronic Engineering from the Bangladesh University of Engineering and Technology in 2017.

Information-Theoretic Tools to Understand Distributed Source Coding in Neuroscience

Information-Theoretic Tools to Understand Distributed Source Coding in Neuroscience

Quantifying the direct and indirect interactions for EEG signals by using detrended permutation mutual information

In this paper, we extend detrended cross-correlations analysis (DCCA) and introduce a novel method called detrended permutation mutual information (DPMI) by utilizing information-theoretic tools. The main mechanism of this paper is to use the invariance of the nonlinear monotonic transformation and robustness in permutation mutual information (PMI), which solves the limitations of DCCA only applicable to linear measurements and enhances the robustness of DPMI to noise. We compare the performance of DPMI with DCCA on multifractal binomial measures and Hénon maps for testing the effectiveness of the new method. By analyzing EEG signals obtained from children with attention deficit hyperactivity disorder (ADHD) as well as healthy children, we demonstrate that DPMI surpasses DCCA in processing physiological signals. Furthermore, we conclude that DPMI is more effective in distinguishing between different states within distinct groups. Given that EEG signals in physiological systems are associated with other channel signals, we propose a method called partial DPMI (PDPMI) that combines partial-correlation technique with DPMI. PDPMI can further reveal the direct interactions between two considered signals by eliminating potential influences from other unconsidered signals. In summary, PDPMI serves as a complementary perspective to DPMI, enhancing our understanding in the interactive structure of physiological network.

Information-Theoretic Approaches to Differential Privacy

This tutorial studies relations between differential privacy and various information-theoretic measures by using several selective articles. In particular, we present how these connections can provide new interpretations for the privacy guarantee in systems that deploy differential privacy in an information-theoretic framework. Accordingly, the tutorial delivers an extensive summary on the existing literature that makes use of information-theoretic measures and tools such as mutual information, min-entropy, Kullback-Leibler divergence, and rate-distortion function for quantification and characterization of differential privacy in various settings.

Robust Localization for Secure Navigation of UAV Formations Under GNSS Spoofing Attack

Nowadays, aerial formations are frequently employed in outdoor scenarios to cooperatively explore and monitor wide areas of interest. In these applications, the vehicles are often exposed to relevant security vulnerabilities, as, for instance, the alteration of navigation signals from an attacker with map counterfeiting (if not even hijacking) purposes. In this work, we focus on an Unmanned Aerial Vehicle (UAV) formation that monitors an area, wherein navigation spoofing attacks may occur. Letting the UAVs cooperate and exploiting the redundancy in the available sensing information, a distributed procedure is proposed to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$i)$</tex-math> </inline-formula> detect spoofing attacks, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ii)$</tex-math> </inline-formula> support the navigation in adverse conditions. The validity of the designed approach is confirmed by numerical results. Aerial vehicles for outdoor operation are generally endowed with inertial measurements, relative ranging, and GNSS sensing capability. In this work, two cascaded estimation algorithms for concurrent GNSS spoofing detection and localization in a multi-UAV scenario is proposed, to attain robust navigation in areas subject to GNSS spoofing attacks. The attack detection leverages on information theoretic tools to provide a practical threshold test by checking the multimodal measurement consistency. The localization procedures exploit a decision logic relying on measurement reliability to combine information sources that are different in nature, for UAV self-localization in both safe and under-attack conditions. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Aerial vehicles for outdoor operation are generally endowed with inertial measurements, relative ranging, and GNSS sensing capability. In this work, two cascaded estimation algorithms for concurrent GNSS spoofing detection and localization in a multi-UAV scenario is proposed, to attain robust navigation in areas subject to GNSS spoofing attacks. The attack detection leverages on information theoretic tools to provide a practical threshold test by checking the multimodal measurement consistency. The localization procedures exploit a decision logic relying on measurement reliability to combine information sources that are different in nature, for UAV self-localization in both safe and under-attack conditions.

Security of quantum key distribution from generalised entropy accumulation

The goal of quantum key distribution (QKD) is to establish a secure key between two parties connected by an insecure quantum channel. To use a QKD protocol in practice, one has to prove that a finite size key is secure against general attacks: no matter the adversary’s attack, they cannot gain useful information about the key. A much simpler task is to prove security against collective attacks, where the adversary is assumed to behave identically and independently in each round. In this work, we provide a formal framework for general QKD protocols and show that for any protocol that can be expressed in this framework, security against general attacks reduces to security against collective attacks, which in turn reduces to a numerical computation. Our proof relies on a recently developed information-theoretic tool called generalised entropy accumulation and can handle generic prepare-and-measure protocols directly without switching to an entanglement-based version.

Information-Theoretic Analysis of Minimax Excess Risk

Two main concepts studied in machine learning theory are generalization gap (difference between train and test error) and excess risk (difference between test error and the minimum possible error). While information-theoretic tools have been used extensively to study the generalization gap of learning algorithms, the information-theoretic nature of excess risk has not yet been fully investigated. In this paper, some steps are taken toward this goal. We consider the frequentist problem of minimax excess risk as a zero-sum game between the algorithm designer and the world. Then, we argue that it is desirable to modify this game in a way that the order of play can be swapped. We then prove that, under some regularity conditions, if the world and designer can play randomly the duality gap is zero and the order of play can be changed. In this case, a Bayesian problem surfaces in the dual representation. This makes it possible to utilize recent information-theoretic results on minimum excess risk in Bayesian learning to provide bounds on the minimax excess risk. We demonstrate the applicability of the results by providing information theoretic insight on two important classes of problems: classification when the hypothesis space has finite VC-dimension, and regularized least squares.

In-Network Learning: Distributed Training and Inference in Networks.

In this paper, we study distributed inference and learning over networks which can be modeled by a directed graph. A subset of the nodes observes different features, which are all relevant/required for the inference task that needs to be performed at some distant end (fusion) node. We develop a learning algorithm and an architecture that can combine the information from the observed distributed features, using the processing units available across the networks. In particular, we employ information-theoretic tools to analyze how inference propagates and fuses across a network. Based on the insights gained from this analysis, we derive a loss function that effectively balances the model's performance with the amount of information transmitted across the network. We study the design criterion of our proposed architecture and its bandwidth requirements. Furthermore, we discuss implementation aspects using neural networks in typical wireless radio access and provide experiments that illustrate benefits over state-of-the-art techniques.

Invited Review: Applications of unsupervised machine learning in livestock behavior: Case studies in recovering unanticipated behavioral patterns from precision livestock farming data streams

The objective of this article is to review fun- damental differences between model-dependent and mod- el-free approaches to data analysis, and to explore the potential advantages of more open-ended machine learn- ing approaches in recovering complex behavioral patterns from precision livestock farming data streams. Case studies using simulated data were de- signed to mimic a real-world scenario. Data from a feeding trial in an organic dairy were reanalyzed using the Live- stock Informatics Toolkit. Case studies using simulated data are used to demonstrate how incomplete information about the management system can prohibit the development of an appropriate model for information compression, allow- ing aggregation bias to mask important behavioral indi- cators of compromised welfare. These hidden behavioral patterns are then recovered using unsupervised machine learning approaches that are able to leverage the intrinsic behavioral codependencies of group-housed animals. This simulated case study is then extended to demonstrate how model-based approaches can also overlook causes of com- promised welfare when the link between environmental factors and behavioral responses is strong but nonlinear, whereas model-free information-theoretic tools can easily recover and characterize such complex dynamics. Finally, in an empirical case study with data from a commercial organic dairy, the Livestock Informatics Toolkit is used to recover from milk parlor metadata complex associations between herd age structure, levels of milk production, and order of milking. Model-free machine learning algorithms provide a more open-ended approach to knowledge discovery that require fewer up-front as- sumptions about the management system. This can yield more comprehensive insights into large precision livestock farming data sets now commonly encountered in on-farm research trials and in applied data auditing scenarios.

Machine learning a surrogate model for transmission loss on a Pekeris waveguide

The connection between ocean environmental parameters and transmission loss (TL) can be modeled using sound propagation models. We can analyze and improve these models using information theoretic tools such as the Fisher information matrix (FIM) and methods for model reduction, where we identify parameters that can be removed from a model without sacrificing accuracy. Critical to these methods for model reduction is the ability to evaluate derivatives of the model predictions of TL with respect to model parameters. Calculating derivatives of TL models using finite difference methods with sufficient accuracy is challenging due to the complexity of sound fields in an ocean environment; instead, we train a surrogate machine learning model to which we can apply automatic differentiation (AD). We propose a general method for model reduction in which a ML model is used as a surrogate model for physical sound propagation ocean models, which is then used to find potential simplifications of the original model. We demonstrate this method using the Pekeris waveguide model of the ocean, which calculates underwater acoustic TL in an ocean environment due to seafloor characteristics. [Work supported by Office of Naval Research]

Influence of polarization and the environment on wave–particle duality

Wave-particle duality is certainly one of the most curious concepts of contemporary physics, which ascribes mutually exclusive behaviors to quantum systems that cannot be observed simultaneously. In the context of two-path interferometers, these two behaviors are usually described in terms of the visibility of interference fringes and the path distinguishability. Here, we use quantum information-theoretic tools to derive quantifiers of these two properties, which account for the combined influence of path probability and polarization, and demonstrate that they satisfy a complementarity relation. We further show that the derived quantities can work as probes in the study of open quantum dynamics by revealing interesting facets of environment actions, such as: decoherence, depolarization and scattering.