Information-theoretical Metrics Research Articles

The unique contribution of uncertainty reduction during naturalistic language comprehension

Language comprehension is an incremental process with prediction. Delineating various mental states during such a process is critical to understanding the relationship between human cognition and the properties of language. Entropy reduction, which indicates the dynamic decrease of uncertainty as language input unfolds, has been recognized as effective in predicting neural responses during comprehension. According to the entropy reduction hypothesis (Hale, 2006), entropy reduction is related to the processing difficulty of a word, the effect of which may overlap with other well-documented information-theoretical metrics such as surprisal or next-word entropy. However, the processing difficulty was often confused with the information conveyed by a word, especially lacking neural differentiation. We propose that entropy reduction represents the cognitive neural process of information gain that can be dissociated from processing difficulty. This study characterized various information-theoretical metrics using GPT-2 and identified the unique effects of entropy reduction in predicting fMRI time series acquired during language comprehension. In addition to the effects of surprisal and entropy, entropy reduction was associated with activations in the left inferior frontal gyrus, bilateral ventromedial prefrontal cortex, insula, thalamus, basal ganglia, and middle cingulate cortex. The reduction of uncertainty, rather than its fluctuation, proved to be an effective factor in modeling neural responses. The neural substrates underlying the reduction in uncertainty might imply the brain's desire for information regardless of processing difficulty.

Quantifying evoked responses through information-theoretical measures.

Information theory is a viable candidate to advance our understanding of how the brain processes information generated in the internal or external environment. With its universal applicability, information theory enables the analysis of complex data sets, is free of requirements about the data structure, and can help infer the underlying brain mechanisms. Information-theoretical metrics such as Entropy or Mutual Information have been highly beneficial for analyzing neurophysiological recordings. However, a direct comparison of the performance of these methods with well-established metrics, such as the t-test, is rare. Here, such a comparison is carried out by evaluating the novel method of Encoded Information with Mutual Information, Gaussian Copula Mutual Information, Neural Frequency Tagging, and t-test. We do so by applying each method to event-related potentials and event-related activity in different frequency bands originating from intracranial electroencephalography recordings of humans and marmoset monkeys. Encoded Information is a novel procedure that assesses the similarity of brain responses across experimental conditions by compressing the respective signals. Such an information-based encoding is attractive whenever one is interested in detecting where in the brain condition effects are present.

A general framework for probabilistic sensitivity analysis with respect to distribution parameters

Probabilistic sensitivity analysis identifies the influential uncertain input to guide decision-making. We propose a general sensitivity framework with respect to the input distribution parameters that unifies a wide range of sensitivity measures, including information theoretical metrics such as the Fisher information. The framework is derived analytically via a constrained maximisation and the sensitivity analysis is reformulated into an eigenvalue problem. There are only two main steps to implement the sensitivity framework utilising the likelihood ratio/score function method, a Monte Carlo type sampling followed by solving an eigenvalue equation. The resulting eigenvectors then provide the directions for simultaneous variations of the input parameters and guide the focus to perturb uncertainty the most. Not only is it conceptually simple, but numerical examples demonstrate that the proposed framework also provides new sensitivity insights, such as the combined sensitivity of multiple correlated uncertainty metrics, robust sensitivity analysis with an entropic constraint, and approximation of deterministic sensitivities. Three different examples, ranging from a simple cantilever beam to an offshore marine riser, are used to demonstrate the potential applications of the proposed sensitivity framework to applied mechanics problems.

Accounting for the Effect of Noise in Satellite Soil Moisture Data on Estimates of Land–Atmosphere Coupling Using Information Theoretical Metrics

Abstract Land states can affect the atmosphere through their control of surface turbulent fluxes and the subsequent impact of those fluxes on boundary layer properties. Information theoretic (IT) metrics are ideal to study the strength and type of coupling between surface soil moisture (SM) and land surface heat fluxes (HFs) because they are nonparametric and thus appropriate for the analysis of highly complex Earth systems containing nonlinear cause-and-effect interactions that may have nonnormal distributions. Specifically, a methodology for the estimation of IT metrics from noisy time series is proposed, accounting for random errors in satellite-based SM data. Performance of the proposed method is demonstrated through synthetic tests. Efficacy of the method is greatest for estimates of entropy and mutual information involving SM; improvements to estimates of transfer entropy are significant but less stark. A global depiction of the information flow between SM and HFs is then constructed from observationally based gridded data. This is used as independent verification for two configurations of the ECMWF modeling system: unconstrained open-loop (retrospective forecasts) and constrained by data assimilation (ERA5). Compared to studies that only investigate the linear SM–HF relationships, extended regions of significant terrestrial coupling are found over the globe, as IT metrics enable detection of nonlinear dependencies. The magnitude and spatial variability of coupling strength and type from models show discrepancies with those from observations, highlighting the potential to improve SM and HF covariability within models. Although ERA5 did not perform better than the unconstrained model in very dry climates, its performance is generally superior to that of the unconstrained model across metrics.

Privacy-Preserving Distributed Processing: Metrics, Bounds and Algorithms

Privacy-preserving distributed processing has recently attracted considerable attention. It aims to design solutions for conducting signal processing tasks over networks in a decentralized fashion without violating privacy. Many algorithms can be adopted to solve this problem such as differential privacy, secure multiparty computation, and the recently proposed distributed optimization based subspace perturbation. However, how these algorithms relate to each other is not fully explored yet. In this paper, we therefore first propose information-theoretic metrics based on mutual information. Using the proposed metrics, we are able to compare and relate a number of existing well-known algorithms. We then derive a lower bound on individual privacy that gives insights on the nature of the problem. To validate the above claims, we investigate a concrete example and compare a number of state-of-the-art approaches in terms of different aspects such as output utility, individual privacy and algorithm robustness against the number of corrupted parties, using not only theoretical analysis but also numerical validation. Finally, we discuss and provide principles for designing appropriate algorithms for different applications.

Racial landscapes – a pattern-based, zoneless method for analysis and visualization of racial topography

Quantifying and effectively communicating the spatio-racial distribution of urban residencies is important for taking the measure of how the multiracial society organizes itself in an urban environment. Most currently used approaches to this problem center around the calculation of segregation metrics; as such, they pertain to only a single pattern's feature and they lack a compelling visualization component. In this paper, we propose a reimagined approach to spatio-racial analysis based on the concept of landscape and landscape analysis. This approach unites quantification and visualization components of the analysis. It also quantifies the entire racial topography, not just segregation. Key novel concepts are the racial landscape (RL) and the exposure matrix. RL is a high-resolution grid in which each cell contains only inhabitants of a single race. The exposure matrix tabulates adjacencies between neighboring cells weighted by the local density of adjacent subpopulations; it provides a concise quantification of the RL pattern. Two information-theoretical metrics, derived from the exposure matrix, quantify diversity, and segregation of the RL. Segregation is quantified from cell adjacencies without the need for subdivision of the region of interest. Thus, the entire region, as well as its arbitrary subregions, are RLs quantified by their diversities and segregations. Coloring cells in RL according to combinations of their race and local densities provides a natural visualization of racial topography which serves as an “observation” that provides check on numerical metrics. The RL method is described and its application is demonstrated on Cook County, IL. An implementation of the RL method in R package accompanies this paper.

Assessing functional propagation patterns in COVID-19.

Among the many efforts done by the scientific community to help coping with the COVID-19 pandemic, one of the most important has been the creation of models to describe its propagation, as these are expected to guide the deployment of containment and health policies. These models are commonly based on exogenous information, as e.g. mobility data, whose limitedness always compromise the reliability of obtained results. In this contribution we propose a different approach, based on extracting relationships between the evolution of the disease in different regions through information theoretical metrics. In a way similar to what is commonly done in neuroscience, propagation is understood as information transfer, and the resulting propagation patterns are represented and studied as functional networks. By applying this methodology to the dynamics of COVID-19 in several countries and regions thereof, we were able to reconstruct static and time-varying propagation graphs. We further discuss the advantages, promises and open research questions associated with this functional approach.

Information theory as a consistent framework for quantification and classification of landscape patterns

ContextQuantitative grouping of similar landscape patterns is an important part of landscape ecology due to the relationship between a pattern and an underlying ecological process. One of the priorities in landscape ecology is a development of the theoretically consistent framework for quantifying, ordering and classifying landscape patterns.ObjectiveTo demonstrate that the information theory as applied to a bivariate random variable provides a consistent framework for quantifying, ordering, and classifying landscape patterns.MethodsAfter presenting information theory in the context of landscapes, information-theoretical metrics were calculated for an exemplar set of landscapes embodying all feasible configurations of land cover patterns. Sequences and 2D parametrization of patterns in this set were performed to demonstrate the feasibility of information theory for the analysis of landscape patterns.ResultsUniversal classification of landscape into pattern configuration types was achieved by transforming landscapes into a 2D space of weakly correlated information-theoretical metrics. An ordering of landscapes by any single metric cannot produce a sequence of continuously changing patterns. In real-life patterns, diversity induces complexity—increasingly diverse patterns are increasingly complex.ConclusionsInformation theory provides a consistent, theory-based framework for the analysis of landscape patterns. Information-theoretical parametrization of landscapes offers a method for their classification.

On the Study of Analogue Network Coding for Multi-Pair, Bidirectional Relay Channels

We consider a scenario where multiple pairs of users exchange information within pair, with the help of a dedicated multi-antenna relay. The protocol integrates the idea of analogue network coding in mixing two data streams originating from the same user pair, together with the spatial multiplexing of the data streams originating from different user pairs. The key feature of the protocol is that it enables both the relay and the users to participate in interference cancellation. We propose several beamforming schemes for the multi-antenna relay and evaluate the performance using information theoretical metrics such as ergodic capacity, outage probability and diversity and multiplexing tradeoff. Analytical and simulation results justify that the ergodic capacity, outage probability and diversity and multiplexing tradeoff of the proposed beamforming schemes outperform comparable schemes.