Directed Information Theory Research Articles

Causal inference for multivariate stochastic process prediction

Numerous real world systems of major interest are modeled as sets of analog continuous stochastic processes with delayed and varying causal relationships. Yet studying their dynamic becomes often difficult, as it involves sensing, understanding and predicting a system of inter-dependent random variables in a given context and over time. In the present work we develop systematic, rigorous and efficient framework to structurally characterize and forecast such systems in a flexible manner. In particular we use a graph method based on a maximum spanning tree approach, to capture the causal dependence structure based on directed information theory. To this end we address the sparsity problem in information causality estimation in general, and we propose a new method that identifies and eliminates redundant calculations. To forecast child nodes based on their inferred causal parents we use a linear model aiming to capture the closest approximation of functional relations. We further account for dependencies using causal conditional information by adding links that improve child nodes estimation. The result is a comprehensive and flexible approach to understanding and predicting large sets of inter-dependent narrowband processes, as we demonstrate on both synthetic and real datasets.

Directed information graphs for the Granger causality of multivariate time series

In this paper, we investigate the links between (strong) Granger causality and directed information theory for multivariate time series. Based on the decomposition of conditional directed information, we propose a definition of Granger causality including instantaneous variables in the conditional set, which can avoid the spurious causality. The directed information graphs are presented to describe the Granger causality and instantaneous coupling. The structure learning of the graph models is based on the Leonenko’s k-nn estimator of the statistics and a permutation test of the significant. Finally, we demonstrate the numerical implementation of these techniques on linear and nonlinear time series.

Strong converses for group testing from finite blocklength results

We prove new strong converse results in a variety of group testing settings, generalizing a result of Baldassini, Johnson and Aldridge. These results are proved by two distinct approaches, corresponding to the non-adaptive and adaptive cases. In the non-adaptive case, we mimic the hypothesis testing argument introduced in the finite blocklength channel coding regime by Polyanskiy, Poor and Verd\'{u}. In the adaptive case, we combine a formulation based on directed information theory with ideas of Kemperman, Kesten and Wolfowitz from the problem of channel coding with feedback. In both cases, we prove results which are valid for finite sized problems, and imply capacity results in the asymptotic regime. These results are illustrated graphically for a range of models.

Causality Analysis of fMRI Data Based on the Directed Information Theory Framework

This paper aims to conduct fMRI-based causality analysis in brain connectivity by exploiting the directed information (DI) theory framework. Unlike the well-known Granger causality (GC) analysis, which relies on the linear prediction technique, the DI theory framework does not have any modeling constraints on the sequences to be evaluated and ensures estimation convergence. Moreover, it can be used to generate the GC graphs. In this paper, first, we introduce the core concepts in the DI framework. Second, we present how to conduct causality analysis using DI measures between two time series. We provide the detailed procedure on how to calculate the DI for two finite-time series. The two major steps involved here are optimal bin size selection for data digitization and probability estimation. Finally, we demonstrate the applicability of DI-based causality analysis using both the simulated data and experimental fMRI data, and compare the results with that of the GC analysis. Our analysis indicates that GC analysis is effective in detecting linear or nearly linear causal relationship, but may have difficulty in capturing nonlinear causal relationships. On the other hand, DI-based causality analysis is more effective in capturing both linear and nonlinear causal relationships. Moreover, it is observed that brain connectivity among different regions generally involves dynamic two-way information transmissions between them. Our results show that when bidirectional information flow is present, DI is more effective than GC to quantify the overall causal relationship.

Causal conditioning and instantaneous coupling in causality graphs

This study aims at providing the definitive links between Massey and Kramer’s directed information theory and Granger causality graphs, recently formalized by Eichler. This naturally leads to consider two concepts of causality that can occur in physical systems: dynamical causality and instantaneous coupling. Although it is well accepted that the former is assessed by conditional transfer entropy, the latter is often overlooked, even if it was clearly introduced and understood in seminal studies. In the bivariate case, we show that directed information is decomposed into the sum of transfer entropy and instantaneous information exchange. In the multivariate case, encountered for conditional graph modeling, such a decomposition does not hold anymore. We provide a new insight into the problem of instantaneous coupling and show how it affects the estimated structure of a graphical model that should provide a sparse representation of a complex system. This result is discussed and analyzed for the inference of causality graphs. Two synthetic examples are developed to illustrate the theoretical concepts. Practical issues are also briefly discussed on these examples and an extension of Leonenko’s k-nearest neighbors based entropy estimators is used to derive a nonparametric estimator of directed information.

Permutation Complexity and Coupling Measures in Hidden Markov Models

Recently, the duality between values (words) and orderings (permutations) has been proposed by the authors as a basis to discuss the relationship between information theoretic measures for finite-alphabet stationary stochastic processes and their permutatio nanalogues. It has been used to give a simple proof of the equality between the entropy rate and the permutation entropy rate for any finite-alphabet stationary stochastic process and to show some results on the excess entropy and the transfer entropy for finite-alphabet stationary ergodic Markov processes. In this paper, we extend our previous results to hidden Markov models and show the equalities between various information theoretic complexity and coupling measures and their permutation analogues. In particular, we show the following two results within the realm of hidden Markov models with ergodic internal processes: the two permutation analogues of the transfer entropy, the symbolic transfer entropy and the transfer entropy on rank vectors, are both equivalent to the transfer entropy if they are considered as the rates, and the directed information theory can be captured by the permutation entropy approach.

The Principle of Maximum Causal Entropy for Estimating Interacting Processes

The principle of maximum entropy provides a powerful framework for estimating joint, conditional, and marginal probability distributions. However, there are many important distributions with elements of interaction and feedback where its applicability has not been established. This paper presents the principle of maximum causal entropy-an approach based on directed information theory for estimating an unknown process based on its interactions with a known process. We demonstrate the breadth of the approach using two applications: a predictive solution for inverse optimal control in decision processes and computing equilibrium strategies in sequential games.

The Relation between Granger Causality and Directed Information Theory: A Review

This report reviews the conceptual and theoretical links between Granger causality and directed information theory. We begin with a short historical tour of Granger causality, concentrating on its closeness to information theory. The definitions of Granger causality based on prediction are recalled, and the importance of the observation set is discussed. We present the definitions based on conditional independence. The notion of instantaneous coupling is included in the definitions. The concept of Granger causality graphs is discussed. We present directed information theory from the perspective of studies of causal influences between stochastic processes. Causal conditioning appears to be the cornerstone for the relation between information theory and Granger causality. In the bivariate case, the fundamental measure is the directed information, which decomposes as the sum of the transfer entropies and a term quantifying instantaneous coupling. We show the decomposition of the mutual information into the sums of the transfer entropies and the instantaneous coupling measure, a relation known for the linear Gaussian case. We study the multivariate case, showing that the useful decomposition is blurred by instantaneous coupling. The links are further developed by studying how measures based on directed information theory naturally emerge from Granger causality inference frameworks as hypothesis testing.

Assessing functional connectivity of neural ensembles using directed information

Neurons in the brain form highly complex networks through synaptic connections. Traditionally, functional connectivity between neurons has been explored using methods such as correlations, which do not contain any notion of directionality. Recently, an information-theoretic approach based on directed information theory has been proposed as a way to infer the direction of influence. However, it is still unclear whether this new approach provides any additional insight beyond conventional correlation analyses. In this paper, we present a modified procedure for estimating directed information and provide a comparison of results obtained using correlation analyses on both simulated and experimental data. Using physiologically realistic simulations, we demonstrate that directed information can outperform correlation in determining connections between neural spike trains while also providing directionality of the relationship, which cannot be assessed using correlation. Secondly, applying our method to rodent and primate data sets, we demonstrate that directed information can accurately estimate the conduction delay in connections between different brain structures. Moreover, directed information reveals connectivity structures that are not captured by correlations. Hence, directed information provides accurate and novel insights into the functional connectivity of neural ensembles that are applicable to data from neurophysiological studies in awake behaving animals.

On directed information theory and Granger causality graphs

Directed information theory deals with communication channels with feedback. When applied to networks, a natural extension based on causal conditioning is needed. We show here that measures built from directed information theory in networks can be used to assess Granger causality graphs of stochastic processes. We show that directed information theory includes measures such as the transfer entropy, and that it is the adequate information theoretic framework needed for neuroscience applications, such as connectivity inference problems.