Information Theory-based Measures Research Articles

The causality measure of partial mutual information from mixed embedding (PMIME) revisited.

The measure of partial mutual information from mixed embedding (PMIME) is an information theory-based measure to accurately identify the direct and directional coupling, termed Granger causality or simply causality, between the observed variables or subsystems of a high-dimensional dynamical and complex system, without any a priori assumptions about the nature of the coupling relationship. In its core, it is a forward selection procedure that aims to iteratively identify the lag-dependence structure of a given observed variable (response) to all the other observed variables (candidate drivers). This model-free approach is capable of detecting nonlinear interactions, abundantly present in real-world complex systems, and it was shown to perform well on multivariate time series of moderately high dimension. However, the PMIME presents some inefficiencies in its performance mainly when applied on strongly stochastic (linear or nonlinear) systems as it may falsely detect non-existent relationships. Moreover, and by construction, the measure cannot extract purely synergetic relationships present in a system. In the current work, the issue of false detections is addressed by introducing an improved resampling significance test and a procedure of rechecking the identified drivers (backward revision). Regarding the inability to detect synergetic relationships, the PMIME is further enhanced by checking pairs as candidate drivers for the response variable after having considered all drivers individually. The effects of these modifications are investigated in a systematic simulation study on properly designed systems involving strong stochasticity, regressor terms with synergetic effects, and a system dimension ranging from 3 to 30. The overall results of the simulations indicate that these modifications indeed improve the performance of PMIME and alleviate to a significant degree the issues of the original algorithm. Guidelines for balancing between accuracy and computational efficiency are also given, particularly relevant for real-world applications. Finally, the measure performance is investigated in the study of futures of various government bonds and stock market indices in the period around COVID-19 pandemic.

Information theory-based direct causality measure to assess cardiac fibrillation dynamics.

Understanding the mechanism sustaining cardiac fibrillation can facilitate the personalization of treatment. Granger causality analysis can be used to determine the existence of a hierarchical fibrillation mechanism that is more amenable to ablation treatment in cardiac time-series data. Conventional Granger causality based on linear predictability may fail if the assumption is not met or given sparsely sampled, high-dimensional data. More recently developed information theory-based causality measures could potentially provide a more accurate estimate of the nonlinear coupling. However, despite their successful application to linear and nonlinear physical systems, their use is not known in the clinical field. Partial mutual information from mixed embedding (PMIME) was implemented to identify the direct coupling of cardiac electrophysiology signals. We show that PMIME requires less data and is more robust to extrinsic confounding factors. The algorithms were then extended for efficient characterization of fibrillation organization and hierarchy using clinical high-dimensional data. We show that PMIME network measures correlate well with the spatio-temporal organization of fibrillation and demonstrated that hierarchical type of fibrillation and drivers could be identified in a subset of ventricular fibrillation patients, such that regions of high hierarchy are associated with high dominant frequency.

Improving transfer entropy and partial transfer entropy for relative detection of effective connectivity strength between time series

Information theory-based measures have recently been considered in determining functional and effective connectivity. Transfer Entropy (TE) and Partial Transfer Entropy (Partial TE) are two such measures that investigate causal connectivity in bivariate and multivariate systems, respectively. The existence or absence of causal connectivity between variables is determined through statistical testing that compares TE or Partial TE values under different conditions or between the original TE or Partial TE values and surrogate data. However, using normal TE or Partial TE provides no information about coupling strength between variables. In this study, we present a repeating statistical test method for TE and Partial TE values in linear dynamic systems to determine the relative coupling strength between connectivity in the networks. In this method, segments with different time lengths (N) were considered for each variable in a system. Using the proposed method, the number of segments with statistically significant TE or partial TE values for each pair of time series was counted and analyzed as N increased. Our research results demonstrated that the proposed statistical test distinguishes stronger linear couplings with fewer segments and shorter time lengths (N) from weaker couplings. The proposed method also confirmed that Partial TE is more capable of determining direct causal connectivity than TE. Therefore, our proposed method can not only detect direct causal connections, but also determine the causal effect size of linear connectivity between time series relative to each other in a system, thus improving the performance of TE and Partial TE criteria.

Polyvalent immunization elicits a synergistic broadly neutralizing immune response to hypervariable region 1 variants of hepatitis C virus

A hepatitis C virus (HCV) vaccine is urgently needed. Vaccine development has been hindered by HCV's genetic diversity, particularly within the immunodominant hypervariable region 1 (HVR1). Here, we developed a strategy to elicit broadly neutralizing antibodies to HVR1, which had previously been considered infeasible. We first applied a unique information theory-based measure of genetic distance to evaluate phenotypic relatedness between HVR1 variants. These distances were used to model the structure of HVR1's sequence space, which was found to have five major clusters. Variants from each cluster were used to immunize mice individually, and as a pentavalent mixture. Sera obtained following immunization neutralized every variant in a diverse HCVpp panel (n = 10), including those resistant to monovalent immunization, and at higher mean titers (1/ID50 = 435) than a glycoprotein E2 (1/ID50 =205) vaccine. This synergistic immune response offers a unique approach to overcoming antigenic variability and may be applicable to other highly mutable viruses.

ApoE Modifier Alleles for Alzheimer's Disease Discovered by Information Theory Dependency Measures: MIST Software Package.

Information theory-based measures of variable dependency (previously published) have been implemented into a software package, MIST. The design of the software and its potential uses are described, and a demonstration is presented in the discovery of modifier alleles of the ApoE gene in affecting Alzheimer's disease (AD) by analyzing the UK Biobank dataset. The modifier genes uncovered overlap strongly with genes found to be associated with AD. Others include many known to influence AD. We discuss a range of uses of the dependency calculations using MIST that can uncover additional genetic effects in similar complex datasets, like higher degrees of interaction and phenotypic pleiotropy.

Chains of Spatial and Temporal Precipitation Occurrence Predictability Across the Continental U.S.

Both spatial and temporal information sources contribute to the predictability of precipitation occurrence at a given location. These sources, and the level of predictability they provide, are relevant to forecasting and understanding precipitation processes at different time scales. We use information theory-based measures to construct connected “chains of influence” of spatial extents and timescales of precipitation occurrence predictability across the continental U.S, based on gridded daily precipitation data. These regions can also be thought of as “footprints” or regions where precipitation states tend to be most synchronized. We compute these chains of precipitation influence for grid cells in the continental US, and study metrics regarding their lengths, extents, and curvature for different seasons. We find distinct geographic and seasonal patterns, particularly longer chain lengths during the summer that are indicative of larger spatial extents for storms. While synchronous, or instantaneous, relationships are strongest for grid cells in the same region, lagged relationships arise as chains reach areas farther from the original cell. While this study focuses on precipitation occurrence predictability given only information about precipitation, it could be extended to study spatial and temporal properties of other driving factors.

Kernel-Based Phase Transfer Entropy with Enhanced Feature Relevance Analysis for Brain Computer Interfaces

Neural oscillations are present in the brain at different spatial and temporal scales, and they are linked to several cognitive functions. Furthermore, the information carried by their phases is fundamental for the coordination of anatomically distributed processing in the brain. The concept of phase transfer entropy refers to an information theory-based measure of directed connectivity among neural oscillations that allows studying such distributed processes. Phase TE is commonly obtained from probability estimations carried out over data from multiple trials, which bars its use as a characterization strategy in brain–computer interfaces. In this work, we propose a novel methodology to estimate TE between single pairs of instantaneous phase time series. Our approach combines a kernel-based TE estimator defined in terms of Renyi’s α entropy, which sidesteps the need for probability distribution computation with phase time series obtained by complex filtering the neural signals. Besides, a kernel-alignment-based relevance analysis is added to highlight relevant features from effective connectivity-based representation supporting further classification stages in EEG-based brain–computer interface systems. Our proposal is tested on simulated coupled data and two publicly available databases containing EEG signals recorded under motor imagery and visual working memory paradigms. Attained results demonstrate how the introduced effective connectivity succeeds in detecting the interactions present in the data for the former, with statistically significant results around the frequencies of interest. It also reflects differences in coupling strength, is robust to realistic noise and signal mixing levels, and captures bidirectional interactions of localized frequency content. Obtained results for the motor imagery and working memory databases show that our approach, combined with the relevance analysis strategy, codes discriminant spatial and frequency-dependent patterns for the different conditions in each experimental paradigm, with classification performances that do well in comparison with those of alternative methods of similar nature.

Evaluating the sensitivity of functional connectivity measures to motion artifact in resting-state fMRI data

Functional connectivity (FC) networks are typically inferred from resting-state fMRI data using the Pearson correlation between BOLD time series from pairs of brain regions. However, alternative methods of estimating functional connectivity have not been systematically tested for their sensitivity or robustness to head motion artifact. Here, we evaluate the sensitivity of eight different functional connectivity measures to motion artifact using resting-state data from the Human Connectome Project. We report that FC estimated using full correlation has a relatively high residual distance-dependent relationship with motion compared to partial correlation, coherence, and information theory-based measures, even after implementing rigorous methods for motion artifact mitigation. This disadvantage of full correlation, however, may be offset by higher test-retest reliability, fingerprinting accuracy, and system identifiability. FC estimated by partial correlation offers the best of both worlds, with low sensitivity to motion artifact and intermediate system identifiability, with the caveat of low test-retest reliability and fingerprinting accuracy. We highlight spatial differences in the sub-networks affected by motion with different FC metrics. Further, we report that intra-network edges in the default mode and retrosplenial temporal sub-networks are highly correlated with motion in all FC methods. Our findings indicate that the method of estimating functional connectivity is an important consideration in resting-state fMRI studies and must be chosen carefully based on the parameters of the study.

Using time causal quantifiers to characterize sleep stages

Sleep plays a substantial role in daily cognitive performance, mood, and memory. The study of sleep has attracted the interest of neuroscientists, clinicians and the overall population, with an increasing number of adults suffering from insufficient amounts of sleep. Sleep is an activity composed of different stages whose temporal dynamics, cycles and interdependencies are not fully understood. Healthy body function and personal well being, however, depends on the proper unfolding and continuance of the sleep cycles. The characterization of the different sleep stages can be undertaken with the development of biomarkers derived from sleep recording. For this purpose, in this work we analyzed single-channel EEG signals from 106 healthy subjects. The signals were quantified using the permutation vector approach using five different-information theoretic measures: i) Shannon’s entropy, ii) MPR statistical complexity, iii) Fisher information, iv) Renyí Min-entropy and v) Lempel-Ziv complexity. The results show that all five information theory-based measures make it possible to quantify and classify the underlying dynamics of the different sleep stages. In addition to this, we combine these measures to show that planes containing pairs of measures, such as the plane composed of Lempel-Ziv and Shannon, have a better performance for differentiating sleep states than measures used individually for the same purpose.

Analyzing pilots’ fatigue for prolonged flight missions: Multimodal analysis approach using vigilance test and eye tracking

Performance decrement associated with pilot fatigue is considered a leading contributor to aviation accidents and fatalities. The output of prevalent pilot fatigue methodologies (both subjective & objective) either suffer from human judgement bias or require complex data processing. Moreover, studies catering to long duration flight missions have not been performed. Presently, we investigate the impact of fatigue on pilot performance for long duration of a flight mission composed of multiple take-offs and landings. We propose a new multimodal approach that integrates traditional fatigue metrics with eye tracking methodology. The effect of fatigue on the pilots’ eye movements was evaluated using information theory-based entropy measures. Results showed an increase in the fatigue level (measured by mean reaction times and the number of lapses) with increase in flight duration. The entropy measures showed that visual attention distribution and scanning strategy both became random in nature as fatigue level increased in pilots. Obtained results suggest fatigue decreases both information searching and processing capability in pilots. The proposed method can show which aspect of the pilot performance becomes impaired by fatigue and thus can be applied to evaluate fatigue onset in real time, which enables timely recovery interventions.

Identification of topology-preserving, class-relevant feature subsets using multiobjective optimization

In the current work, a multiobjective-based feature selection technique is proposed which utilizes different quality measures to evaluate the goodness of reduced feature set. Two different perspectives are incorporated in the feature selection process: (1) selected subset of features should not destroy the geometric distribution of the sample space, i.e., the neighborhood topology should be preserved in the reduced feature space; (2) selected feature subset should have minimal redundancy and high correlation with the classes. In order to capture the second goal, several information theory-based quality measures like normalized mutual information, correlation with the class attribute, information gain and entropy are utilized. In order to capture the first aspect, concepts of shared nearest-neighbor distance are utilized. Multiobjective framework is employed to optimize all these measures, individually and in different combinations to reduce the feature set. The approach is evaluated on six publicly available data sets with respect to different classifiers, and results conclusively demonstrate the potency of utilizing both types of objectives functions in reducing the feature set. Several performance metrics like accuracy, redundancy and Jaccard score are used for measuring the quality of the selected feature subset in comparison with several state-of-the-art techniques. Experimental results on several data sets illustrate that there is no universal model (optimization of a set of objective functions) which can perform well over all the data sets with respect to different quality measures. But in general optimization of all objective functions (PMCI model) consistently performs well for all the data sets.

Random field-based regional liquefaction hazard mapping — data inference and model verification using a synthetic digital soil field

Geostatistical tools and random field models have been increasingly used in recent liquefaction mapping studies. However, a systematic verification and assessment of random field models has yet to be taken, and implications of various random field-based mapping approaches are unknown. In this paper, an extremely detailed three-dimensional synthetic digital soil field is artificially generated and used as a basis for assessing and verifying various random field-based models for liquefaction mapping. Liquefaction hazard is quantified in terms of the liquefaction potential index (LPI), which is mapped over the studied field. A classical CPT-based liquefaction model is adopted to assess liquefaction potential of a soil layer. Different virtual field investigation plans are designed to assess the dependency of data inference and model performance upon the level of availability of sampling data. Model performances are assessed using three information theory-based measures. Results show that when sampling data is sufficient, all random field-based models examined capture fairly well the benchmark liquefaction potentials in the studied field. As the size of the sampling data decreases, the accuracy of predictions decreases for all models but to different degrees; the three-dimensional random field model gives the best result in this scenario. All random field-based models examined in this paper yield a slightly more conservative prediction of liquefaction potential over the studied field.

Efficient Variational Approach to Multimodal Registration of Anatomical and Functional Intra-Patient Tumorous Brain Data.

This paper addresses the functional localization of intra-patient images of the brain. Functional images of the brain (fMRI and PET) provide information about brain function and metabolism whereas anatomical images (MRI and CT) supply the localization of structures with high spatial resolution. The goal is to find the geometric correspondence between functional and anatomical images in order to complement and fuse the information provided by each imaging modality. The proposed approach is based on a variational formulation of the image registration problem in the frequency domain. It has been implemented as a C/C[Formula: see text] library which is invoked from a GUI. This interface is routinely used in the clinical setting by physicians for research purposes (Inscanner, Alicante, Spain), and may be used as well for diagnosis and surgical planning. The registration of anatomic and functional intra-patient images of the brain makes it possible to obtain a geometric correspondence which allows for the localization of the functional processes that occur in the brain. Through 18 clinical experiments, it has been demonstrated how the proposed approach outperforms popular state-of-the-art registration methods in terms of efficiency, information theory-based measures (such as mutual information) and actual registration error (distance in space of corresponding landmarks).

Transfer Zero-Entropy and Its Application for Capturing Cause and Effect Relationship Between Variables

Detection of causality is an important and challenging problem in root cause and hazard propagation analysis. It has been shown that the transfer entropy approach is a very useful tool in quantifying directional causal influence for both linear and nonlinear relationships. A key assumption for this method is that the sampled data should follow a well-defined probability distribution; yet this assumption may not hold for some industrial process data. In this paper, a new information theory-based measure, transfer 0-entropy (T0E), is proposed for causality analysis on the basis of the definitions of 0-entropy and 0-information without assuming a probability space. For the cases of more than two variables, a direct T0E (DT0E) concept is presented to detect whether there is a direct information and/or material flow pathway from one variable to another. Estimation methods for the T0E and the DT0E are addressed. The effectiveness of the proposed method is illustrated by two data sets, one based on data from a pilot scale process and a second evaluation based on data from a benchmark industrial case study.

Novel Information Theory-Based Measures for Quantifying Incongruence among Phylogenetic Trees

Phylogenies inferred from different data matrices often conflict with each other necessitating the development of measures that quantify this incongruence. Here, we introduce novel measures that use information theory to quantify the degree of conflict or incongruence among all nontrivial bipartitions present in a set of trees. The first measure, internode certainty (IC), calculates the degree of certainty for a given internode by considering the frequency of the bipartition defined by the internode (internal branch) in a given set of trees jointly with that of the most prevalent conflicting bipartition in the same tree set. The second measure, IC All (ICA), calculates the degree of certainty for a given internode by considering the frequency of the bipartition defined by the internode in a given set of trees in conjunction with that of all conflicting bipartitions in the same underlying tree set. Finally, the tree certainty (TC) and TC All (TCA) measures are the sum of IC and ICA values across all internodes of a phylogeny, respectively. IC, ICA, TC, and TCA can be calculated from different types of data that contain nontrivial bipartitions, including from bootstrap replicate trees to gene trees or individual characters. Given a set of phylogenetic trees, the IC and ICA values of a given internode reflect its specific degree of incongruence, and the TC and TCA values describe the global degree of incongruence between trees in the set. All four measures are implemented and freely available in version 8.0.0 and subsequent versions of the widely used program RAxML.

A comparative study of statistical methods used to identify dependencies between gene expression signals

One major task in molecular biology is to understand the dependency among genes to model gene regulatory networks. Pearson's correlation is the most common method used to measure dependence between gene expression signals, but it works well only when data are linearly associated. For other types of association, such as non-linear or non-functional relationships, methods based on the concepts of rank correlation and information theory-based measures are more adequate than the Pearson's correlation, but are less used in applications, most probably because of a lack of clear guidelines for their use. This work seeks to summarize the main methods (Pearson's, Spearman's and Kendall's correlations; distance correlation; Hoeffding's D: measure; Heller-Heller-Gorfine measure; mutual information and maximal information coefficient) used to identify dependency between random variables, especially gene expression data, and also to evaluate the strengths and limitations of each method. Systematic Monte Carlo simulation analyses ranging from sample size, local dependence and linear/non-linear and also non-functional relationships are shown. Moreover, comparisons in actual gene expression data are carried out. Finally, we provide a suggestive list of methods that can be used for each type of data set.

Measuring Ambiguity in HLA Typing Methods

In hematopoietic stem cell transplantation, donor selection is based primarily on matching donor and patient HLA genes. These genes are highly polymorphic and their typing can result in exact allele assignment at each gene (the resolution at which patients and donors are matched), but it can also result in a set of ambiguous assignments, depending on the typing methodology used. To facilitate rapid identification of matched donors, registries employ statistical algorithms to infer HLA alleles from ambiguous genotypes. Linkage disequilibrium information encapsulated in haplotype frequencies is used to facilitate prediction of the most likely haplotype assignment. An HLA typing with less ambiguity produces fewer high-probability haplotypes and a more reliable prediction. We estimated ambiguity for several HLA typing methods across four continental populations using an information theory-based measure, Shannon's entropy. We used allele and haplotype frequencies to calculate entropy for different sets of 1,000 subjects with simulated HLA typing. Using allele frequencies we calculated an average entropy in Caucasians of 1.65 for serology, 1.06 for allele family level, 0.49 for a 2002-era SSO kit, and 0.076 for single-pass SBT. When using haplotype frequencies in entropy calculations, we found average entropies of 0.72 for serology, 0.73 for allele family level, 0.05 for SSO, and 0.002 for single-pass SBT. Application of haplotype frequencies further reduces HLA typing ambiguity. We also estimated expected confirmatory typing mismatch rates for simulated subjects. In a hypothetical registry with all donors typed using the same method, the entropy values based on haplotype frequencies correspond to confirmatory typing mismatch rates of 1.31% for SSO versus only 0.08% for SBT. Intermediate-resolution single-pass SBT contains the least ambiguity of the methods we evaluated and therefore the most certainty in allele prediction. The presented measure objectively evaluates HLA typing methods and can help define acceptable HLA typing for donor recruitment.

Taxonomy of granular rheology from grain property networks

We construct complex networks from symbolic time series of particle properties within a dense quasistatically deforming granular assembly subjected to biaxial compression. The structure of the resulting networks embodies the evolving structural rearrangements in the granular material, in both contact forces and contact topologies. These rearrangements are usefully summarized through standard network statistics as well as building block motifs and community structures. Dense granular media respond to applied compression and shear by a process of self-organization to form two cooperatively evolving structures comprising the major load-bearing columnlike force chains, and the lateral trusslike three-cycle triangle topologies. We construct networks summarizing their individual evolution based on relationships between symbolic time series indicating a particle's chronological force chain and three-cycle membership histories. We test which particle membership histories are similar with respect to each other through the information theory-based measure of Hamming distance. The complex networks summarize the essential structural rearrangements, while the community structures within the networks partition the material into distinct zones of deformation, including interlacing subregions of failure inside the shear band. The taxonomy of granular rheology at the mesoscopic scale distills the inelastic structural rearrangements throughout loading history down to its core elements, and should lay bare an objective and physics-based formalism for thermodynamic internal variables and associated evolution laws.

Scale effects on information theory-based measures applied to streamflow patterns in two rural watersheds

Summary Understanding streamflow patterns in space and time is important for improving flood and drought forecasting, water resources management, and predictions of ecological changes. Objectives of this work include (a) to characterize the spatial and temporal patterns of streamflow using information theory-based measures at two thoroughly-monitored agricultural watersheds located in different hydroclimatic zones with similar land use, and (b) to elucidate and quantify temporal and spatial scale effects on those measures. We selected two USDA experimental watersheds to serve as case study examples, including the Little River experimental watershed (LREW) in Tifton, Georgia and the Sleepers River experimental watershed (SREW) in North Danville, Vermont. Both watersheds possess several nested sub-watersheds and more than 30 years of continuous data records of precipitation and streamflow. Information content measures (metric entropy and mean information gain) and complexity measures (effective measure complexity and fluctuation complexity) were computed based on the binary encoding of 5-year streamflow and precipitation time series data. We quantified patterns of streamflow using probabilities of joint or sequential appearances of the binary symbol sequences. Results of our analysis illustrate that information content measures of streamflow time series are much smaller than those for precipitation data, and the streamflow data also exhibit higher complexity, suggesting that the watersheds effectively act as filters of the precipitation information that leads to the observed additional complexity in streamflow measures. Correlation coefficients between the information-theory-based measures and time intervals are close to 0.9, demonstrating the significance of temporal scale effects on streamflow patterns. Moderate spatial scale effects on streamflow patterns are observed with absolute values of correlation coefficients between the measures and sub-watershed area varying from 0.2 to 0.6 in the two watersheds. We conclude that temporal effects must be evaluated and accounted for when the information theory-based methods are used for performance evaluation and comparison of hydrological models.

Accelerated and Adaptive Evolution of Yeast Sexual Adhesins

There is a recent emergence of interest in the genes involved in gametic recognition as drivers of reproductive isolation. The recent population genomic sequencing of two species of sexually primitive yeasts (Liti G, Carter DM, Moses AM, Warringer J, Parts L, James SA, Davey RP, Roberts IN, Burt A, Koufopanou V et al. [23 co-authors]. 2009. Population genomics of domestic and wild yeasts. Nature 458:337-341.) has provided data for systematic study of the roles these genes play in the early evolution of sex and speciation. Here, we discovered that among genes encoding cell surface proteins, the sexual adhesin genes have evolved significantly more rapidly than others, both within and between Saccharomyces cerevisiae and its closest relative S. paradoxus. This result was supported by analyses using the PAML pairwise model, a modified McDonald-Kreitman test, and the PAML branch model. Moreover, using a combination of a new statistic of neutrality, an information theory-based measure of evolutionary variability, and functional characterization of amino acid changes, we found that a higher proportion of amino acid changes are fixed in the sexual adhesins than in other proteins and a greater proportion of the fixed amino acid changes either between the two species or the two subgroups of S. paradoxus are functionally dissimilar or radically different. These results suggest that the accelerated evolution of sexual adhesin genes may facilitate speciation, or incipient speciation, and promote sexual selection in general.