Information-theoretic Metrics Research Articles

Information-Theoretic Compressive Measurement Design.

An information-theoretic projection design framework is proposed, of interest for feature design and compressive measurements. Both Gaussian and Poisson measurement models are considered. The gradient of a proposed information-theoretic metric (ITM) is derived, and a gradient-descent algorithm is applied in design; connections are made to the information bottleneck. The fundamental solution structure of such design is revealed in the case of a Gaussian measurement model and arbitrary input statistics. This new theoretical result reveals how ITM parameter settings impact the number of needed projection measurements, with this verified experimentally. The ITM achieves promising results on real data, for both signal recovery and classification.

Free-space optical channel estimation for physical layer security.

We present experimental data on message transmission in a free-space optical (FSO) link at an eye-safe wavelength, using a testbed consisting of one sender and two receiver terminals, where the latter two are a legitimate receiver and an eavesdropper. The testbed allows us to emulate a typical scenario of physical-layer (PHY) security such as satellite-to-ground laser communications. We estimate information-theoretic metrics including secrecy rate, secrecy outage probability, and expected code lengths for given secrecy criteria based on observed channel statistics. We then discuss operation principles of secure message transmission under realistic fading conditions, and provide a guideline on a multi-layer security architecture by combining PHY security and upper-layer (algorithmic) security.

Variational Bayesian strategies for high-dimensional, stochastic design problems

This paper is concerned with a lesser-studied problem in the context of model-based, uncertainty quantification (UQ), that of optimization/design/control under uncertainty. The solution of such problems is hindered not only by the usual difficulties encountered in UQ tasks (e.g. the high computational cost of each forward simulation, the large number of random variables) but also by the need to solve a nonlinear optimization problem involving large numbers of design variables and potentially constraints. We propose a framework that is suitable for a large class of such problems and is based on the idea of recasting them as probabilistic inference tasks. To that end, we propose a Variational Bayesian (VB) formulation and an iterative VB-Expectation-Maximization scheme that is also capable of identifying a low-dimensional set of directions in the design space, along which, the objective exhibits the largest sensitivity. We demonstrate the validity of the proposed approach in the context of two numerical examples involving $\mathcal{O}(10^3)$ random and design variables. In all cases considered the cost of the computations in terms of calls to the forward model was of the order $\mathcal{O}(10^2)$. The accuracy of the approximations provided is assessed by appropriate information-theoretic metrics.

Sequence information gain based motif analysis.

BackgroundThe detection of regulatory regions in candidate sequences is essential for the understanding of the regulation of a particular gene and the mechanisms involved. This paper proposes a novel methodology based on information theoretic metrics for finding regulatory sequences in promoter regions.ResultsThis methodology (SIGMA) has been tested on genomic sequence data for Homo sapiens and Mus musculus. SIGMA has been compared with different publicly available alternatives for motif detection, such as MEME/MAST, Biostrings (Bioconductor package), MotifRegressor, and previous work such Qresiduals projections or information theoretic based detectors. Comparative results, in the form of Receiver Operating Characteristic curves, show how, in 70 % of the studied Transcription Factor Binding Sites, the SIGMA detector has a better performance and behaves more robustly than the methods compared, while having a similar computational time. The performance of SIGMA can be explained by its parametric simplicity in the modelling of the non-linear co-variability in the binding motif positions.ConclusionsSequence Information Gain based Motif Analysis is a generalisation of a non-linear model of the cis-regulatory sequences detection based on Information Theory. This generalisation allows us to detect transcription factor binding sites with maximum performance disregarding the covariability observed in the positions of the training set of sequences. SIGMA is freely available to the public at http://b2slab.upc.edu.

Cognitive tasks during walking affect cerebral blood flow signal features in middle cerebral arteries and their correlation to gait characteristics.

Gait is a complex process involving both cognitive and sensory ability and is strongly impacted by the environment. In this paper, we propose to study of the impact of a cognitive task during gait on the cerebral blood flow velocity, the blood flow signal features and the correlation of gait and blood flow features through a dual task methodology. Both cerebral blood flow velocity and gait characteristics of eleven participants with no history of brain or gait conditions were recorded using transcranial Doppler on mid-cerebral artery while on a treadmill. The cognitive task was induced by a backward counting starting from 10,000 with decrement of 7. Central blood flow velocity raw and envelope features were extracted in both time, frequency and time-scale domain; information-theoretic metrics were also extracted and statistical significances were inspected. A similar feature extraction was performed on the stride interval signal. Statistical differences between the cognitive and baseline trials, between the left and right mid-cerebral arteries signals and the impact of the antropometric variables where studied using linear mixed models. No statistical differences were found between the left and right mid-cerebral arteries flows or the baseline and cognitive state gait features, while statistical differences for specific features were measured between cognitive and baseline states. These statistical differences found between the baseline and cognitive states show that cognitive process has an impact on the cerebral activity during walking. The state was found to have an impact on the correlation between the gait and blood flow features.

Competition between conceptual relations affects compound recognition: the role of entropy.

Previous research has suggested that the conceptual representation of a compound is based on a relational structure linking the compound's constituents. Existing accounts of the visual recognition of modifier-head or noun-noun compounds posit that the process involves the selection of a relational structure out of a set of competing relational structures associated with the same compound. In this article, we employ the information-theoretic metric of entropy to gauge relational competition and investigate its effect on the visual identification of established English compounds. The data from two lexical decision megastudies indicates that greater entropy (i.e., increased competition) in a set of conceptual relations associated with a compound is associated with longer lexical decision latencies. This finding indicates that there exists competition between potential meanings associated with the same complex word form. We provide empirical support for conceptual composition during compound word processing in a model that incorporates the effect of the integration of co-activated and competing relational information.

A User Grouping Based Opportunistic Network Coding Scheme for Multi-user Multi-relay Uplink Transmissions

In this paper, we study a User Grouping based Opportunistic Network Coding (UG-ONC) scheme for multi-user multi-relay uplink transmission scenario. In traditional opportunistic network coding (ONC) schemes, the use of network coding is often determined by the state of relays, i.e., buffer, packets waiting time, packets' successful decoding. However, whether the use of network coding is beneficial to the users is not considered. In our proposed UG-ONC scheme, whether network coding is adopted is not only determined by the state of relays, but also by the link quality between users and base station, that is, determined by the state of users. We consider two information-theoretic metrics, i.e., outage probability and throughput, to evaluate the validity and reliability of the UG-ONC scheme. And the approximate expressions of outage probability and throughput are derived. To further examine the performance of the scheme, we compare UG-ONC scheme with an existent network coding scheme. Monte Carlo simulations are presented to validate our analysis. The results show that our proposed UG-ONC scheme achieves considerable gains over the traditional network coding scheme. Our proposed UG-ONC scheme gives a new thought for designing opportunistic network coding.

A heuristic branch-and-bound based thresholding algorithm for unveiling cognitive activity from EEG data

One of the biggest challenges in the field of computational neuroscience from the perspective of complex network analysis is the measurement of dynamic local and global interactions of the brain regions during cognitive function. Graph theoretic analysis has been extensively applied to study the dynamics of functional brain networks in the recent years. The selection of appropriate thresholding methods to construct weighted/unweighted subnetworks to detect cognitive load induced changes in brain׳s electrical activity remains an open challenge in the functional brain network research. This paper reviews the application of statistical and information theoretic metrics to construct the functional brain networks, proposes a novel Branch-and-Bound based thresholding algorithm that extracts the influential subnetwork, and applies efficient computational techniques and complex network metrics to detect and quantify the cognitive activities. The empirical analyses showcase the efficiency of the proposed thresholding algorithm by highlighting the changing neuronal patterns during cognitive activity when compared to that of baseline activity. Statistical evaluation of the results further validates the efficiency of the proposed method as well. The results demonstrate the ability of the proposed algorithm in detecting subtle cognitive load induced changes in functional brain networks.

Meaning in the avian auditory cortex: neural representation of communication calls.

Understanding how the brain extracts the behavioral meaning carried by specific vocalization types that can be emitted by various vocalizers and in different conditions is a central question in auditory research. This semantic categorization is a fundamental process required for acoustic communication, and presupposes discriminative and invariance properties of the auditory system for conspecific vocalizations. Songbirds have been used extensively to study vocal learning, but the communicative function of all their vocalizations and their neural representation has yet to be examined. In this study, we first generated a library containing almost the entire zebra finch vocal repertoire, and organised communication calls along nine different categories according to their behavioral meaning. We then investigated the neural representations of these semantic categories in the primary and secondary auditory areas of six anesthetised zebra finches. To analyse how single units encode these call categories, we described neural responses in terms of their discrimination, selectivity and invariance properties. Quantitative measures for these neural properties were obtained with an optimal decoder using both spike counts and spike patterns. Information theoretic metrics show that almost half of the single units encode semantic information. Neurons achieve higher discrimination of these semantic categories by being more selective and more invariant. These results demonstrate that computations necessary for semantic categorization of meaningful vocalizations are already present in the auditory cortex, and emphasise the value of a neuro-ethological approach to understand vocal communication.

On the Study of Cognitive Bidirectional Relaying with Asymmetric Traffic Demands

AbstractIn this paper, we consider a cognitive radio network scenario, where two primary users want to exchange information with each other and meanwhile, one secondary node wishes to send messages to a cognitive base station. To meet the target quality of service (QoS) of the primary users and raise the communication opportunity of the secondary nodes, a cognitive bidirectional relaying (BDR) scheme is examined. First, system outage probabilities of conventional direct transmission and BDR schemes are presented. Next, a new system parameter called operating region is defined and calculated, which indicates in which position a secondary node can be a cognitive relay to assist the primary users. Then, a cognitive BDR scheme is proposed, giving a transmission protocol along with a time-slot splitting algorithm between the primary and secondary transmissions. Information-theoretic metric of ergodic capacity is studied for the cognitive BDR scheme to evaluate its performance. Simulation results show that with the proposed scheme, the target QoS of the primary users can be guaranteed, while increasing the communication opportunity for the secondary nodes.

Bayesian Inference of Information Theoretic Metrics of Anonymity

Information theoretic metrics is popular theory to measure anonymity. However the difficulty in getting the probability distribution of subjects hampers its practical usage. In this paper we propose a Bayesian inference method to tackle this problem. Our method makes it possible to compare the anonymity of different anonymous systems. We use this method to analyze Threshold Mix and point out different system parameters which do and do not have influence on anonymity.

Spatial climate patterns explain negligible variation in strength of compensatory density feedbacks in birds and mammals.

The use of long-term population data to separate the demographic role of climate from density-modified demographic processes has become a major topic of ecological investigation over the last two decades. Although the ecological and evolutionary mechanisms that determine the strength of density feedbacks are now well understood, the degree to which climate gradients shape those processes across taxa and broad spatial scales remains unclear. Intuitively, harsh or highly variable environmental conditions should weaken compensatory density feedbacks because populations are hypothetically unable to achieve or maintain densities at which social and trophic interactions (e.g., competition, parasitism, predation, disease) might systematically reduce population growth. Here we investigate variation in the strength of compensatory density feedback, from long-term time series of abundance over 146 species of birds and mammals, in response to spatial gradients of broad-scale temperature precipitation variables covering 97 localities in 28 countries. We use information-theoretic metrics to rank phylogenetic generalized least-squares regression models that control for sample size (time-series length) and phylogenetic non-independence. Climatic factors explained < 1% of the remaining variation in density-feedback strength across species, with the highest non-control, model-averaged effect sizes related to extreme precipitation variables. We could not link our results directly to other published studies, because ecologists use contrasting responses, predictors and statistical approaches to correlate density feedback and climate – at the expense of comparability in a macroecological context. Censuses of multiple populations within a given species, and a priori knowledge of the spatial scales at which density feedbacks interact with climate, seem to be necessary to determine cross-taxa variation in this phenomenon. Despite the availability of robust modelling tools, the appropriate data have not yet been gathered for most species, meaning that we cannot yet make any robust generalisations about how demographic feedbacks interact with climate.

Environmental Influence on the Evolution of Morphological Complexity in Machines

Whether, when, how, and why increased complexity evolves in biological populations is a longstanding open question. In this work we combine a recently developed method for evolving virtual organisms with an information-theoretic metric of morphological complexity in order to investigate how the complexity of morphologies, which are evolved for locomotion, varies across different environments. We first demonstrate that selection for locomotion results in the evolution of organisms with morphologies that increase in complexity over evolutionary time beyond what would be expected due to random chance. This provides evidence that the increase in complexity observed is a result of a driven rather than a passive trend. In subsequent experiments we demonstrate that morphologies having greater complexity evolve in complex environments, when compared to a simple environment when a cost of complexity is imposed. This suggests that in some niches, evolution may act to complexify the body plans of organisms while in other niches selection favors simpler body plans.

Information-theoretic quantification of underwater acoustic source localization performance

Metrics that historically have been applied to quantify the performance of signal processing for source localization are algorithm-dependent. For example, performance of conventional beamforming or matched-field processing is usually quantified by main-peak width and secondary-peak levels of the beam response or spatial ambiguity function, while performance of Bayesian localization may be quantified by measures of the statistical dispersion of the a posteriori pdf of source location. While algorithm-dependent performance metrics permit comparisons within a given class of signal processing algorithms, they do not provide comparability across algorithm classes. The present work identifies fundamental information-theoretic quantities that can be used as metrics to quantify the source localization performance of diverse signal processing algorithms and thus provide for performance comparisons across signal-processor classes. These quantities include conditional entropy of source location given processor output, mutual information of source location and processor output, and cross-entropy of actual and posterior source-location probability distributions. Applications of these information-theoretic metrics are illustrated in examples of Bayesian localization, conventional beamforming, and matched-field processing of a time-harmonic source in a range-independent shallow-water acoustic waveguide. The results are interpreted in the light of the data processing inequality of information theory. [Work supported by ONR.]

What is in a name? On the misuse of information theoretic dispersion measures as design complexity metrics

Many metrics of design complexity have been proposed in the literature. Of these, the most popular are information theoretic metrics, such as information content based on Suh's axiomatic theory and differential entropy based on Shannon's information theory. In this paper, we show that these metrics do not provide common sense measures of complexity, and they also do not possess proper mathematical properties. At best, they are geared towards measuring a designs goodness-of-fit rather than its complexity. At least in part, the inappropriate use of these metrics can be attributed to misuse of terms such as ‘entropy’ and ‘minimise information content’.

A Practical Method to Estimate Information Content in the Context of 4D-Var Data Assimilation

Data assimilation obtains improved estimates of the state of a physical system by dynamically combining imperfect model results with sparse and noisy observations of reality. Not all observations used in data assimilation are equally valuable. The ability to characterize the usefulness of different data points is important for analyzing the effectiveness of the dynamic data assimilation system, for data pruning, for dynamically configuring existing sensor systems, and for designing future sensor networks. This paper focuses on the four-dimensional variational (4D-Var) data assimilation framework. Metrics from information theory are used to quantify the contribution of observations to decreasing the uncertainty with which the system state is known. An important relationship between different information-theoretic metrics and the variational cost function/gradient is established under Gaussian, linear, and time-independent assumptions. This insight allows us to derive an ensemble-based computational procedure to estimate the information content of various observations in the context of 4D-Var. (A short paper summarizing the theoretical results, and showing numerical experiments with a linear system, was presented at the International Conference on Computational Science ICCS 2012.) The approach is first illustrated on a small nonlinear test problem. Next, the methodology is applied to quantify the degrees of freedom for signal of satellite observations used in a global chemical data assimilation problem. The assimilation of a subset of data points characterized by the highest information content yields an analysis comparable in quality to the one obtained using the entire data set.

Are yesterday-s information-theoretic fading models and performance metrics adequate for the analysis of today's wireless systems?

The purpose of this article is to examine some of the models commonly used to represent fading as well as the information-theoretic metrics most commonly used to evaluate performance over those models. We raise the question of whether these models and metrics remain meaningful in light of the advances that wireless systems have undergone over the last two decades. A number of weaknesses are pointed out, and ideas on possible fixes are put forth. Some of the identified weaknesses have to do with models that, over time, have become grossly inadequate. Other weaknesses have to do with changes in the operating conditions of wireless systems, and others with the coarse and asymptotic nature of some of the most popular performance metrics (diversity and multiplexing).

An Information-Theoretic Metric of System Complexity With Application to Engineering System Design

System complexity is considered a key driver of the inability of current system design practices to at times not recognize performance, cost, and schedule risks as they emerge. We present here a definition of system complexity and a quantitative metric for measuring that complexity based on information theory. We also derive sensitivity indices that indicate the fraction of complexity that can be reduced if more about certain factors of a system can become known. This information can be used as part of a resource allocation procedure aimed at reducing system complexity. Our methods incorporate Gaussian process emulators of expensive computer simulation models and account for both model inadequacy and code uncertainty. We demonstrate our methodology on a candidate design of an infantry fighting vehicle.

Trading privacy with incentives in mobile commerce: A game theoretic approach

In mobile commerce, companies provide location based services to mobile users, who report their locations with a certain level of granularity to maintain a degree of anonymity. This level of granularity depends on their perceived risk as well as the incentives they receive in the form of monetary benefits or improved mobile services. This paper formulates a quantitative model in which information theoretic metrics such as entropy, quantify the anonymity level of mobile users. The individual perceived risks of users and the benefits they obtain are defined as functions of their chosen location information granularity. The interaction between the mobile commerce company and its users is investigated using mechanism design techniques as a privacy game. The user best responses and optimal strategies for the company are derived under budgetary constraints on incentives, which are provided to users in order to convince them to share their private information at the desired level of granularity. Information limitations in the system are analyzed to capture more realistic scenarios where the companies do not have access to user utility functions. Iterative distributed algorithm and regression learning methods are investigated to design mechanisms that overcome these limitations. The results obtained are demonstrated with a numerical example and simulations based on real GPS data.

Guided evolution of in silico microbial populations in complex environments accelerates evolutionary rates through a step-wise adaptation

BackgroundDuring their lifetime, microbes are exposed to environmental variations, each with its distinct spatio-temporal dynamics. Microbial communities display a remarkable degree of phenotypic plasticity, and highly-fit individuals emerge quite rapidly during microbial adaptation to novel environments. However, there exists a high variability when it comes to adaptation potential, and while adaptation occurs rapidly in certain environmental transitions, in others organisms struggle to adapt. Here, we investigate the hypothesis that the rate of evolution can both increase or decrease, depending on the similarity and complexity of the intermediate and final environments. Elucidating such dependencies paves the way towards controlling the rate and direction of evolution, which is of interest to industrial and medical applications.ResultsOur results show that the rate of evolution can be accelerated by evolving cell populations in sequential combinations of environments that are increasingly more complex. To quantify environmental complexity, we evaluate various information-theoretic metrics, and we provide evidence that multivariate mutual information between environmental signals in a given environment correlates well with the rate of evolution in that environment, as measured in our simulations. We find that strong positive and negative correlations between the intermediate and final environments lead to the increase of evolutionary rates, when the environmental complexity increases. Horizontal Gene Transfer is shown to further augment this acceleration, under certain conditions. Interestingly, our simulations show that weak environmental correlations lead to deceleration of evolution, regardless of environmental complexity. Further analysis of network evolution provides a mechanistic explanation of this phenomenon, as exposing cells to intermediate environments can trap the population to local neighborhoods of sub-optimal fitness.