Information-theoretic Formulation Research Articles

Maximum Penalized-Likelihood Structured Covariance Estimation for Imaging Extended Objects, with Application to Radio Astronomy

Image formation in radio astronomy is often posed as a problem of constructing a nonnegative function from sparse samples of its Fourier transform. We explore an alternative approach that reformulates the problem in terms of estimating the entries of a diagonal covariance matrix from Gaussian data. Maximum-likelihood estimates of the covariance cannot be readily computed analytically; hence, we investigate an iterative algorithm originally proposed by Snyder, O’Sullivan, and Miller in the context of radar imaging. The resulting maximum-likelihood estimates tend to be unacceptably rough due to the ill-posed nature of the maximum-likelihood estimation of functions from limited data, so some kind of regularization is needed. We explore penalized likelihoods based on entropy functionals, a roughness penalty proposed by Silverman, and an information-theoretic formulation of Good’s roughness penalty crafted by O’Sullivan. We also investigate algorithm variations that perform a generic smoothing step at each iteration. The results illustrate that tuning parameters allow for a tradeoff between the noise and blurriness of the reconstruction.

Making the Thermodynamic Cost of Active Inference Explicit.

When describing Active Inference Agents (AIAs), the term "energy" can have two distinct meanings. One is the energy that is utilized by the AIA (e.g., electrical energy or chemical energy). The second meaning is so-called Variational Free Energy (VFE), a statistical quantity which provides an upper bound on surprisal. In this paper, we develop an account of the former quantity-the Thermodynamic Free Energy (TFE)-and its relationship with the latter. We highlight the necessary tradeoffs between these two in a generic, quantum information-theoretic formulation, and the macroscopic consequences of those tradeoffs for the ways that organisms approach their environments. By making this tradeoff explicit, we provide a theoretical basis for the different metabolic strategies that organisms from plants to predators use to survive.

On the information-theoretic formulation of network participation

The participation coefficient is a widely used metric of the diversity of a node’s connections with respect to a modular partition of a network. An information-theoretic formulation of this concept of connection diversity, referred to here as participation entropy, has been introduced as the Shannon entropy of the distribution of module labels across a node’s connected neighbors. While diversity metrics have been studied theoretically in other literatures, including to index species diversity in ecology, many of these results have not previously been applied to networks. Here we show that the participation coefficient is a first-order approximation to participation entropy and use the desirable additive properties of entropy to develop new metrics of connection diversity with respect to multiple labelings of nodes in a network, as joint and conditional participation entropies. The information-theoretic formalism developed here allows new and more subtle types of nodal connection patterns in complex networks to be studied.

Testing the power-law hypothesis of the interconflict interval

War is an extreme form of collective human behaviour characterized by coordinated violence. We show that this nature of war is substantiated in the temporal patterns of conflict occurrence that obey power law. The focal metric is the interconflict interval (ICI), the interval between the end of a conflict in a dyad (i.e. a pair of states) and the start of the subsequent conflict in the same dyad. Using elaborate statistical tests, we confirmed that ICI samples compiled from the history of interstate conflicts from 1816 to 2014 followed a power-law distribution. We then demonstrate that the power-law properties of ICIs can be explained by a hypothetical model assuming an information-theoretic formulation of the Clausewitz thesis on war: the use of force is a means of interstate communication. Our findings help us to understand the nature of wars between regular states, the significance of which has increased since the Russian invasion of Ukraine in 2022.

Representing model uncertainties in brittle fracture simulations

This work focuses on the representation of model-form uncertainties in phase-field models of brittle fracture. Such uncertainties can arise from the choice of the degradation function for instance, and their consideration has been unaddressed to date. The stochastic modeling framework leverages recent developments related to the analysis of nonlinear dynamical systems and relies on the construction of a stochastic reduced-order model. In the latter, a POD-based reduced-order basis is randomized using Riemannian projection and retraction operators, as well as an information-theoretic formulation enabling proper concentration in the convex hull defined by a set of model proposals. The model thus obtained is mathematically admissible in the almost sure sense and involves a low-dimensional hyperparameter, the calibration of which is facilitated through the formulation of a quadratic programming problem. The relevance of the modeling approach is further assessed on one- and two-dimensional applications. It is shown that model uncertainties can be efficiently captured and propagated to macroscopic quantities of interest. An extension based on localized randomization is also proposed to handle the case where the forward simulation is highly sensitive to sample localization. This work constitutes a methodological development allowing phase-field predictions to be endowed with statistical measures of confidence, accounting for the variability induced by modeling choices.

Robust target area search using sets of probabilities

Target area search in the standard Bayesian information-theoretic formulation consists of a repetitive cycle of sensing, recursive Bayesian estimation and motion control. The paper formulates the problem of target area search in the framework of imprecise probability theory using probability sets. The rationale is that the measurement model parameters, such as the probability of detection or the probability of false alarm, are rarely known as precise values. Instead, we adopt probability intervals to express beliefs. The paper formulates the Bayes-like state estimation equations as well as the reward function for motion control. The reward function is based on an uncertainty measure which takes into account both randomness and epistemic uncertainty.

Signaling games in multiple dimensions: Geometric properties of equilibrium solutions

Signaling game problems investigate communication scenarios where encoder(s) and decoder(s) have misaligned objectives due to the fact that they either employ different cost functions or have inconsistent priors. This problem has been studied in the literature for scalar sources under various setups. In this paper, we consider multi-dimensional sources under quadratic criteria in the presence of a bias leading to a mismatch in the criteria, where we show that the generalization from the scalar setup is more than technical. We show that the Nash equilibrium solutions lead to structural richness due to the subtle geometric analysis the problem entails, with consequences in both system design, the presence of linear Nash equilibria, and an information theoretic problem formulation. We first provide a set of geometric conditions that must be satisfied in equilibrium considering any multi-dimensional source. Then, we consider independent and identically distributed sources and characterize necessary and sufficient conditions under which an informative linear Nash equilibrium exists. These conditions involve the bias vector that leads to misaligned costs. Depending on certain conditions related to the bias vector, the existence of linear Nash equilibria requires sources with a Gaussian or a symmetric density. Moreover, in the case of Gaussian sources, our results have a rate–distortion theoretic implication that achievable rates and distortions in the considered game theoretic setup can be obtained from its team theoretic counterpart.

Entropic Dynamics in a Theoretical Framework for Biosystems.

Central to an understanding of the physical nature of biosystems is an apprehension of their ability to control entropy dynamics in their environment. To achieve ongoing stability and survival, living systems must adaptively respond to incoming information signals concerning matter and energy perturbations in their biological continuum (biocontinuum). Entropy dynamics for the living system are then determined by the natural drive for reconciliation of these information divergences in the context of the constraints formed by the geometry of the biocontinuum information space. The configuration of this information geometry is determined by the inherent biological structure, processes and adaptive controls that are necessary for the stable functioning of the organism. The trajectory of this adaptive reconciliation process can be described by an information-theoretic formulation of the living system's procedure for actionable knowledge acquisition that incorporates the axiomatic inference of the Kullback principle of minimum information discrimination (a derivative of Jaynes' principle of maximal entropy). Utilizing relative information for entropic inference provides for the incorporation of a background of the adaptive constraints in biosystems within the operations of Fisher biologic replicator dynamics. This mathematical expression for entropic dynamics within the biocontinuum may then serve as a theoretical framework for the general analysis of biological phenomena.

Additive Bayesian Network Modeling with the R Package abn

The R package abn is designed to fit additive Bayesian network models to observational datasets and contains routines to score Bayesian networks based on Bayesian or information theoretic formulations of generalized linear models. It is equipped with exact search and greedy search algorithms to select the best network, and supports continuous, discrete and count data in the same model and input of prior knowledge at a structural level. The Bayesian implementation supports random effects to control for one-layer clustering. In this paper, we give an overview of the methodology and illustrate the package's functionality using a veterinary dataset concerned with respiratory diseases in commercial swine production.

Gradual (In)Compatibility of Fairness Criteria

Impossibility results show that important fairness measures (independence, separation, sufficiency) cannot be satisfied at the same time under reasonable assumptions. This paper explores whether we can satisfy and/or improve these fairness measures simultaneously to a certain degree. We introduce information-theoretic formulations of the fairness measures and define degrees of fairness based on these formulations. The information-theoretic formulations suggest unexplored theoretical relations between the three fairness measures. In the experimental part, we use the information-theoretic expressions as regularizers to obtain fairness-regularized predictors for three standard datasets. Our experiments show that a) fairness regularization directly increases fairness measures, in line with existing work, and b) some fairness regularizations indirectly increase other fairness measures, as suggested by our theoretical findings. This establishes that it is possible to increase the degree to which some fairness measures are satisfied at the same time -- some fairness measures are gradually compatible.

Information-theoretic formulation of dynamical systems: Causality, modeling, and control

The problems of causality, modeling, and control for chaotic, high-dimensional dynamical systems are formulated in the language of information theory. The central quantity of interest is the Shannon entropy, which measures the amount of information in the states of the system. Within this framework, causality is quantified by the information flux among the variables of interest in the dynamical system. Reduced-order modeling is posed as a problem related to the conservation of information in which models aim at preserving the maximum amount of relevant information from the original system. Similarly, control theory is cast in information-theoretic terms by envisioning the tandem sensor-actuator as a device reducing the unknown information of the state to be controlled. The new formulation is used to address three problems about the causality, modeling, and control of turbulence, which stands as a primary example of a chaotic, high-dimensional dynamical system. The applications include the causality of the energy transfer in the turbulent cascade, subgrid-scale modeling for large-eddy simulation, and flow control for drag reduction in wall-bounded turbulence.

Quantifying Reinforcement-Learning Agent's Autonomy, Reliance on Memory and Internalisation of the Environment.

Intuitively, the level of autonomy of an agent is related to the degree to which the agent’s goals and behaviour are decoupled from the immediate control by the environment. Here, we capitalise on a recent information-theoretic formulation of autonomy and introduce an algorithm for calculating autonomy in a limiting process of time step approaching infinity. We tackle the question of how the autonomy level of an agent changes during training. In particular, in this work, we use the partial information decomposition (PID) framework to monitor the levels of autonomy and environment internalisation of reinforcement-learning (RL) agents. We performed experiments on two environments: a grid world, in which the agent has to collect food, and a repeating-pattern environment, in which the agent has to learn to imitate a sequence of actions by memorising the sequence. PID also allows us to answer how much the agent relies on its internal memory (versus how much it relies on the observations) when transitioning to its next internal state. The experiments show that specific terms of PID strongly correlate with the obtained reward and with the agent’s behaviour against perturbations in the observations.

Asymmetric Leaky Private Information Retrieval

Information-theoretic formulations of the private information retrieval (PIR) problem have been investigated under a variety of scenarios. Symmetric private information retrieval (SPIR) is a variant where a user is able to privately retrieve one out of K messages from N non-colluding replicated databases without learning anything about the remaining K-1 messages. However, the goal of perfect privacy can be too taxing for certain applications. In this paper, we investigate if the information-theoretic capacity of SPIR (equivalently, the inverse of the minimum download cost) can be increased by relaxing both user and DB privacy definitions. Such relaxation is relevant in applications where privacy can be traded for communication efficiency. We introduce and investigate the Asymmetric Leaky PIR (AL-PIR) model with different privacy leakage budgets in each direction. For user privacy leakage, we bound the probability ratios between all possible realizations of DB queries by a function of a non-negative constant ϵ. For DB privacy, we bound the mutual information between the undesired messages, the queries, and the answers, by a function of a non-negative constant δ. We propose a general AL-PIR scheme that achieves an upper bound on the optimal download cost for arbitrary ϵ and δ. We show that the optimal download cost of AL-PIR is upper-bounded as D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">*</sup> (ϵ,δ) ≤ 1+\frac 1N-1-\frac δe <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ϵ</sup> N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K-1</sup> -1. Second, we obtain an information-theoretic lower bound on the download cost as D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">*</sup> (ϵ,δ) ≥ 1+\frac 1Ne <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ϵ</sup> -1-\frac δ(Ne <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ϵ</sup> ) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K-1</sup> -1. The gap analysis between the two bounds shows that our AL-PIR scheme is optimal when ϵ = 0, i.e., under perfect user privacy and it is optimal within a maximum multiplicative gap of \frac N-e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-ϵ</sup> N-1 for any ϵ > 0 and δ > 0.

Information-theoretic measures for nonlinear causality detection: application to social media sentiment and cryptocurrency prices

Information transfer between time series is calculated using the asymmetric information-theoretic measure known as transfer entropy. Geweke’s autoregressive formulation of Granger causality is used to compute linear transfer entropy, and Schreiber’s general, non-parametric, information-theoretic formulation is used to quantify nonlinear transfer entropy. We first validate these measures against synthetic data. Then we apply these measures to detect statistical causality between social sentiment changes and cryptocurrency returns. We validate results by performing permutation tests by shuffling the time series, and calculate the Z-score. We also investigate different approaches for partitioning in non-parametric density estimation which can improve the significance. Using these techniques on sentiment and price data over a 48-month period to August 2018, for four major cryptocurrencies, namely bitcoin (BTC), ripple (XRP), litecoin (LTC) and ethereum (ETH), we detect significant information transfer, on hourly timescales, with greater net information transfer from sentiment to price for XRP and LTC, and instead from price to sentiment for BTC and ETH. We report the scale of nonlinear statistical causality to be an order of magnitude larger than the linear case.

Entropic measurement uncertainty relations for all the infinite components of a spin vector

The information-theoretic formulation of quantum measurement uncertainty relations (MURs), based on the notion of relative entropy between measurement probabilities, is extended to the set of all the spin components for a generic spin s. For an approximate measurement of a spin vector, which gives approximate joint measurements of the spin components, we define the device information loss as the maximum loss of information per observable occurring in approximating the ideal incompatible components with the joint measurement at hand. By optimizing on the measuring device, we define the notion of minimum information loss. By using these notions, we show how to give a significant formulation of state independent MURs in the case of infinitely many target observables. The same construction works as well for finitely many observables, and we study the related MURs for two and three orthogonal spin components. The minimum information loss plays also the role of measure of incompatibility and in this respect it allows us to compare quantitatively the incompatibility of various sets of spin observables, with different number of involved components and different values of s.

Rate Selection for Wireless Random Access Networks Over Block Fading Channels

We study uncoordinated random access over fading channels where each user independently decides whether to send a packet or not to a common receiver at any given time slot. Specifically, we develop an information theoretic formulation to characterize the overall system throughput. We consider two scenarios: classical slotted ALOHA, where no multiuser detection (MUD) capability is available and slotted ALOHA with MUD. In each case, in order to maximize the system throughput, we provide methods to obtain the optimal rates and channel activity probabilities using the user distances to the receiver (or, equivalently, their average signal to noise ratios) assuming a Rayleigh block fading channel. The results demonstrate that the newly proposed optimal rate selection solutions offer significant increase in the expected system throughputs compared to the “same rate to all users” approach commonly used in the literature. In addition to the overall throughput optimization, we also address the issue of fairness among users and propose approaches guaranteeing a minimum amount of individual throughput to each user, and design systems with limited individual outage probabilities for increased energy efficiency and reduced delay.

Near Optimal Coded Data Shuffling for Distributed Learning

Data shuffling between distributed cluster of nodes is one of the critical steps in implementing large-scale learning algorithms. Randomly shuffling the data-set among a cluster of workers allows different nodes to obtain fresh data assignments at each learning epoch. This process has been shown to provide improvements in the learning process (via testing and training error). However, the statistical benefits of distributed data shuffling come at the cost of extra communication overhead from the master node to worker nodes, and can act as one of the major bottlenecks in the overall time for computation. There has been significant recent interest in devising approaches to minimize this communication overhead. One approach is to provision for extra storage at the computing nodes. The other emerging approach is to leverage coded communication to minimize the overall communication overhead. The focus of this work is to understand the fundamental tradeoff between the amount of storage and the communication overhead for distributed data shuffling. In this paper, we first present an information theoretic formulation for the data shuffling problem, accounting for the underlying problem parameters (number of workers, $K$ , number of data points, $N$ , and available storage, and $S$ per node). We then present an information theoretic lower bound on the communication overhead for data shuffling as a function of these parameters. We next present a novel coded communication scheme and show that the resulting communication overhead of the proposed scheme is within a multiplicative factor of at most $\frac {K}{K-1}$ from the lower bound (which is upper bounded by 2 for $K\geq 2$ ). Furthermore, we present new results towards closing this gap through a novel coded communication scheme, which we call the aligned coded shuffling. This scheme is inspired by the ideas of coded shuffling and interference alignment. In particular, we show that the aligned scheme achieves the optimal storage vs communication trade-off for $K , and further reduces the maximum multiplicative gap down to $\frac {K-\frac {1}{3}}{K-1}$ , for $K\geq 5$ .

Coupling in complex systems as information transfer across time scales.

Complex systems such as the human brain or the Earth's climate consist of many subsystems interacting in intricate, nonlinear ways. Moreover, variability of such systems extends over broad ranges of spatial and temporal scales and dynamical phenomena on different scales also influence each other. In order to explain how to detect cross-scale causal interactions, we review information-theoretic formulation of the Granger causality in combination with computational statistics (surrogate data method) and demonstrate how this method can be used to infer driver-response relations from amplitudes and phases of coupled nonlinear dynamical systems. Considering complex systems evolving on multiple time scales, the reviewed methodology starts with a wavelet decomposition of a multi-scale signal into quasi-oscillatory modes of a limited bandwidth, described using their instantaneous phases and amplitudes. Then statistical associations, in particular, causality relations between phases or between phases and amplitudes on different time scales are tested using the conditional mutual information. As an application, we present the analysis of cross-scale interactions and information transfer in the dynamics of the El Niño Southern Oscillation. This article is part of the theme issue 'Coupling functions: dynamical interaction mechanisms in the physical, biological and social sciences'.

Strongly Secure Multiuser Communication and Authentication With Anonymity Constraints

We consider authentication of messages sent from transmitters to a receiver over a multiple access channel, where each transmitter shares a secret key with the legitimate receiver. Additionally, there exists a computationally unbounded opponent who has access to noisy observations of the messages transmitted and can initiate impersonation or substitution attacks. We require that the legitimate receiver must be able to authenticate the messages he receives with respect to predetermined groups of transmitters, but at the same time must be kept ignorant of the transmitter’s identity of a given message in a given group. We propose an information-theoretic formulation of these anonymity constraints as well as an authentication coding scheme for which the asymptotic probability of successful attack is shown to optimally scale with the length of the secret keys shared between each transmitter and the legitimate receiver. Our results quantify the positive impact of the multiple access setting compared to the single-user setting on the probability of successful attack.

Performance analysis of physical layer security over Weibull/lognormal composite fading channel with MRC reception

With the advent of denser network and internet of things (IoT) applications for fifth generation (5G) communication, there is a need to understand and define the security limitations. This paper proposes closed-form expressions for different physical layer security (PLS) metrics for millimeter wave (mmWave) communication for the 5G systems and IoT applications. The information-theoretic formulation is considered in which licit users can impart their data over a quasi-static channel and eavesdroppers perceive their communication through another quasi-static channel, where the channel is represented as Weibull/lognormal (WL) composite fading channel in mmWave. To this end, the closed-form expressions for the probability of density function (PDF) and cumulative distribution function (CDF) are firstly derived. This set of results are then used to derive the closed-form expressions for the lower bound of secure outage probability (SOPL) and strictly positive secrecy capacity (SPSC) with maximal ratio combining (MRC) reception. In addition, average secrecy capacity (ASC) is also analyzed and expressed in closed-form. Furthermore, the derivation is extended for the performance metrics for the two eavesdroppers case to provide perspective for more complex yet practical situation.