Multivariate Information Measures Research Articles

PMUs data based detection of oscillatory events and identification of their associated variable: Estimation of information measures approach

Information theory can be a useful tool for quantifying the perturbations in the associated state variables at the time of disturbance occurrence. The study introduces a framework for the spectral decomposition of multivariate information measures to detect initiation of low frequency oscillations (LFOs), caused due to physical events in the power grid. A frequency-specific quantification of the information is shared between a target variable and two source variables from their time series data. Initially, the approach is applied on different synthetic test signals having different oscillatory frequency modes and decay time constant. Then, approach is extended on PMUs signals. The combination of cross-spectral and information-theoretic approaches is applied for the multi-variable analysis of PMUs signals from the same bus. The interdependence among the frequency, voltage angle and voltage magnitude, corresponding to specific oscillations, manifested due to cause-effect relationships obtained in terms of statistics is estimated. The dynamics in terms of unique (interaction), redundant and synergetic information is determined with the contribution from two of these three signals as source variables to target variable (frequency/voltage angle). This provides a direct coupling to identify driver-response relationships between source variables and target variable to indicate the onset of LFOs, following physical events in power network. The extension of approach among the variables from different buses aids to identify the responsible area of event occurrence.

Functional connectivity inference from fMRI data using multivariate information measures

Shannon’s entropy or an extension of Shannon’s entropy can be used to quantify information transmission between or among variables. Mutual information is the pair-wise information that captures nonlinear relationships between variables. It is more robust than linear correlation methods. Beyond mutual information, two generalizations are defined for multivariate distributions: interaction information or co-information and total correlation or multi-mutual information. In comparison to mutual information, interaction information and total correlation are underutilized and poorly studied in applied neuroscience research. Quantifying information flow between brain regions is not explicitly explained in neuroscience by interaction information and total correlation. This article aims to clarify the distinctions between the neuroscience concepts of mutual information, interaction information, and total correlation. Additionally, we proposed a novel method for determining the interaction information between three variables using total correlation and conditional mutual information. On the other hand, how to apply it properly in practical situations. We supplied both simulation experiments and real neural studies to estimate functional connectivity in the brain with the above three higher-order information-theoretic approaches. In order to capture redundancy information for multivariate variables, we discovered that interaction information and total correlation were both robust, and it could be able to capture both well-known and yet-to-be-discovered functional brain connections.

Information decomposition in the frequency domain: a new framework to study cardiovascular and cardiorespiratory oscillations.

While cross-spectral and information-theoretic approaches are widely used for the multivariate analysis of physiological time series, their combined utilization is far less developed in the literature. This study introduces a framework for the spectral decomposition of multivariate information measures, which provides frequency-specific quantifications of the information shared between a target and two source time series and of its expansion into amounts related to how the sources contribute to the target dynamics with unique, redundant and synergistic information. The framework is illustrated in simulations of linearly interacting stochastic processes, showing how it allows us to retrieve amounts of information shared by the processes within specific frequency bands which are otherwise not detectable by time-domain information measures, as well as coupling features which are not detectable by spectral measures. Then, it is applied to the time series of heart period, systolic and diastolic arterial pressure and respiration variability measured in healthy subjects monitored in the resting supine position and during head-up tilt. We show that the spectral measures of unique, redundant and synergistic information shared by these variability series, integrated within specific frequency bands of physiological interest and reflect the mechanisms of short-term regulation of cardiovascular and cardiorespiratory oscillations and their alterations induced by the postural stress. This article is part of the theme issue 'Advanced computation in cardiovascular physiology: new challenges and opportunities'.

Measuring High-Order Interactions in Rhythmic Processes Through Multivariate Spectral Information Decomposition

Many complex systems in physics, biology and engineering are modeled as dynamical networks and described using multivariate time series analysis. Recent developments have shown that the emergent dynamics of a network system are significantly affected by interactions involving multiple network nodes which cannot be described using pairwise links. While these higher-order interactions can be probed using information-theoretic measures, a rigorous framework to describe them in the frequency domain is still lacking. This work presents an approach for the spectral decomposition of multivariate information measures, capable of identifying higher-order synergistic and redundant interactions between oscillatory processes. We show theoretically that synergy and redundancy can coexist at different frequencies among the output signals of a network system and can be detected only using the proposed spectral method. To demonstrate the broad applicability of the framework, we provide parametric and non-parametric data-efficient estimators for the spectral information measures, and employ them to describe multivariate interactions in three complex systems producing rich oscillatory dynamics, namely the human brain, a ring of electronic oscillators, and the global climate system. In these systems, we show that the use of our framework for the spectral decomposition of information measures reveals multivariate and higher-order interactions not detectable in the time domain. Our results are exemplary of how the frequency-specific analysis of multivariate dynamics can aid the implementation of assessment and control strategies in real-world network systems.

Understanding the Evolution of Tree Size Diversity within the Multivariate Nonsymmetrical Diffusion Process and Information Measures

This study focuses on the stochastic differential calculus of Itô, as an effective tool for the analysis of noise in forest growth and yield modeling. Idea of modeling state (tree size) variable in terms of univariate stochastic differential equation is exposed to a multivariate stochastic differential equation. The new developed multivariate probability density function and its marginal univariate, bivariate and trivariate distributions, and conditional univariate, bivariate and trivariate probability density functions can be applied for the modeling of tree size variables and various stand attributes such as the mean diameter, height, crown base height, crown width, volume, basal area, slenderness ratio, increments, and much more. This study introduces generalized multivariate interaction information measures based on the differential entropy to capture multivariate dependencies between state variables. The present study experimentally confirms the effectiveness of using multivariate interaction information measures to reconstruct multivariate relationships of state variables using measurements obtained from a real-world data set.

Gene Regulatory Networks from Single Cell Data for Exploring Cell Fate Decisions.

Single cell experimental techniques now allow us to quantify gene expression in up to thousands of individual cells. These data reveal the changes in transcriptional state that occur as cells progress through development and adopt specialized cell fates. In this chapter we describe in detail how to use our network inference algorithm (PIDC)-and the associated software package NetworkInference.jl-to infer functional interactions between genes from the observed gene expression patterns. We exploit the large sample sizes and inherent variability of single cell data to detect statistical dependencies between genes that indicate putative (co-)regulatory relationships, using multivariate information measures that can capture complex statistical relationships. We provide guidelines on how best to combine this analysis with other complementary methods designed to explore single cell data, and how to interpret the resulting gene regulatory network models to gain insight into the processes regulating cell differentiation.

Temporal Information Partitioning Networks (TIPNets): A process network approach to infer ecohydrologic shifts

AbstractIn an ecohydrologic system, components of atmospheric, vegetation, and root‐soil subsystems participate in forcing and feedback interactions at varying time scales and intensities. The structure of this network of complex interactions varies in terms of connectivity, strength, and time scale due to perturbations or changing conditions such as rainfall, drought, or land use. However, characterization of these interactions is difficult due to multivariate and weak dependencies in the presence of noise, nonlinearities, and limited data. We introduce a framework for Temporal Information Partitioning Networks (TIPNets), in which time‐series variables are viewed as nodes, and lagged multivariate mutual information measures are links. These links are partitioned into synergistic, unique, and redundant information components, where synergy is information provided only jointly, unique information is only provided by a single source, and redundancy is overlapping information. We construct TIPNets from 1 min weather station data over several hour time windows. From a comparison of dry, wet, and rainy conditions, we find that information strengths increase when solar radiation and surface moisture are present, and surface moisture and wind variability are redundant and synergistic influences, respectively. Over a growing season, network trends reveal patterns that vary with vegetation and rainfall patterns. The framework presented here enables us to interpret process connectivity in a multivariate context, which can lead to better inference of behavioral shifts due to perturbations in ecohydrologic systems. This work contributes to more holistic characterizations of system behavior, and can benefit a wide variety of studies of complex systems.

Conditional multivariate mutual information measures

Thông tin tương hỗ (Mutual Information-MI) giữa hai biến đã được sử dụng để phát hiện mối quan hệ giữa hai biến; khi độ đo này lớn thì sự phụ thuộc giữa hai biến cũng lớn và ngược lại. Tuy nhiên, thông tin tương hỗ lại không cho ta biết mối quan hệ giữa các biến là trực tiếp hay gián tiếp. Để phát hiện quan hệ tương hỗ là trực tiếp hay gián tiếp, chúng ta có thể sử dụng thông tin tương hỗ có điều kiện đối với biến thứ ba (Conditional Mutual Information-CMI). Trong các nghiên cứu trước đây, chúng tôi đã đề xuất các độ đo thông tin tương hỗ đa biến. Có rất nhiều độ đo thông tin tương hỗ khi số biến nhiều hơn hai, mỗi độ đo thể hiện một loại quan hệ có thể tồn tại giữa các biến. Tuy nhiên, cũng như thông tin tương hỗ của hai biến, các độ đo thông tin tương hỗ đa biến chỉ cho ta biết tồn tại hay không một mối quan hệ đa biến; nhưng không cho ta biết mối quan hệ tương hỗ đó là trực tiếp hay gián tiếp. Trong nghiên cứu này, chúng tôi đề xuất các độ đo thông tin tương hỗ đa biến có điều kiện và sử dụng chúng để phát hiện các mối quan hệ đa biến là trực tiếp hay gián tiếp thông qua biến điều kiện.

Conditional multivariate mutual information measures

Thông tin tương hỗ (Mutual Information-MI) giữa hai biến đã được sử dụng để phát hiện mối quan hệ giữa hai biến; khi độ đo này lớn thì sự phụ thuộc giữa hai biến cũng lớn và ngược lại. Tuy nhiên, thông tin tương hỗ lại không cho ta biết mối quan hệ giữa các biến là trực tiếp hay gián tiếp. Để phát hiện quan hệ tương hỗ là trực tiếp hay gián tiếp, chúng ta có thể sử dụng thông tin tương hỗ có điều kiện đối với biến thứ ba (Conditional Mutual Information-CMI). Trong các nghiên cứu trước đây, chúng tôi đã đề xuất các độ đo thông tin tương hỗ đa biến. Có rất nhiều độ đo thông tin tương hỗ khi số biến nhiều hơn hai, mỗi độ đo thể hiện một loại quan hệ có thể tồn tại giữa các biến. Tuy nhiên, cũng như thông tin tương hỗ của hai biến, các độ đo thông tin tương hỗ đa biến chỉ cho ta biết tồn tại hay không một mối quan hệ đa biến; nhưng không cho ta biết mối quan hệ tương hỗ đó là trực tiếp hay gián tiếp. Trong nghiên cứu này, chúng tôi đề xuất các độ đo thông tin tương hỗ đa biến có điều kiện và sử dụng chúng để phát hiện các mối quan hệ đa biến là trực tiếp hay gián tiếp thông qua biến điều kiện.

Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective

Information theory has long been used to quantify interactions between two variables. With the rise of complex systems research, multivariate information measures have been increasingly used to investigate interactions between groups of three or more variables, often with an emphasis on so called synergistic and redundant interactions. While bivariate information measures are commonly agreed upon, the multivariate information measures in use today have been developed by many different groups, and differ in subtle, yet significant ways. Here, we will review these multivariate information measures with special emphasis paid to their relationship to synergy and redundancy, as well as examine the differences between these measures by applying them to several simple model systems. In addition to these systems, we will illustrate the usefulness of the information measures by analyzing neural spiking data from a dissociated culture through early stages of its development. Our aim is that this work will aid other researchers as they seek the best multivariate information measure for their specific research goals and system. Finally, we have made software available online which allows the user to calculate all of the information measures discussed within this paper.

The cumulative effect of genetic markers on classification performance: Insights from simple models

The cumulative effect of markers on classification accuracy, with even slight between-population allele frequency differences, has been a mainstay of empirical genetic research. The simple theoretical model of Edwards (2003) has enabled a clear elucidation of the effect of additional markers on average classification error. The present paper describes a mathematical generalization necessary to alleviate an oversimplification, but at the same time revealing an inherent drawback of the simple model. An alternative model for explaining the effect of additional loci on classification is proposed based on an optimal classifier. In particular, cases in which the expected performance does not profit from information at additional loci are characterized in simple terms. Further insight into the capacity of classifiers to exploit the potential afforded by each additional locus comes from an information-theoretic perspective. Multilocus correlation and multivariate mutual information measures are shown to tightly correspond with the dynamics of misclassification rates, under the cumulative effect of loci.

Anatomy of a bit: Information in a time series observation

Appealing to several multivariate information measures--some familiar, some new here--we analyze the information embedded in discrete-valued stochastic time series. We dissect the uncertainty of a single observation to demonstrate how the measures' asymptotic behavior sheds structural and semantic light on the generating process's internal information dynamics. The measures scale with the length of time window, which captures both intensive (rates of growth) and subextensive components. We provide interpretations for the components, developing explicit relationships between them. We also identify the informational component shared between the past and the future that is not contained in a single observation. The existence of this component directly motivates the notion of a process's effective (internal) states and indicates why one must build models.

Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity

The human brain undertakes highly sophisticated information processing facilitated by the interaction between its sub-regions. We present a novel method for interregional connectivity analysis, using multivariate extensions to the mutual information and transfer entropy. The method allows us to identify the underlying directed information structure between brain regions, and how that structure changes according to behavioral conditions. This method is distinguished in using asymmetric, multivariate, information-theoretical analysis, which captures not only directional and non-linear relationships, but also collective interactions. Importantly, the method is able to estimate multivariate information measures with only relatively little data. We demonstrate the method to analyze functional magnetic resonance imaging time series to establish the directed information structure between brain regions involved in a visuo-motor tracking task. Importantly, this results in a tiered structure, with known movement planning regions driving visual and motor control regions. Also, we examine the changes in this structure as the difficulty of the tracking task is increased. We find that task difficulty modulates the coupling strength between regions of a cortical network involved in movement planning and between motor cortex and the cerebellum which is involved in the fine-tuning of motor control. It is likely these methods will find utility in identifying interregional structure (and experimentally induced changes in this structure) in other cognitive tasks and data modalities.