Information Theoretic Concepts Research Articles

Application of Heuristic Graph Search to Test-Point Selection for Analog Fault Dictionary Techniques

The near-optimal test-point set selection for an analog fault dictionary is formulated as a heuristic depth-first graph search problem. Then, the test point selection process becomes a graph-node-expanding process. During the process of graph expansion, the information-theoretic concepts of entropy are used to develop the criterion for how to choose an intermediate node to expand. If a graph node has already isolated those faults that are hard isolate, then the residual faults can easily be isolated. The difficulty of isolating a given fault is evaluated by the information-theoretic concept of entropy. The harder that a fault is isolated, the larger the entropy value it will have. Statistical experimental results indicate that the proposed method more accurately finds the global minimum set of test points than other methods; therefore, it is a good solution to minimize the size of the test point set.

Fundamentals of Wireless Communication (Tse, D. and Viswanath, P.) [Book review

This book provides a thorough coverage of wireless communication fundamentals, with emphasis on information-theoretic concepts. Some of the topics covered include: mathematical models for the physical channel; digital communication and the concept of diversity transmission for fading channels; cellular systems; capacity of wireless channels; multi-user capacity and opportunistic communication; and an extensive treatment of MIMO systems. The list of references is extensive and the index has been prepared with considerable care. The book would make an ideal resource for graduate students seeking a first exposure to the field.

Quantifying information leakage in process calculi

Building on simple information-theoretic concepts, we study two quantitative models of information leakage in the pi-calculus. The first model presupposes an attacker with an essentially unlimited computational power. The resulting notion of absolute leakage, measured in bits, is in agreement with secrecy as defined by Abadi and Gordon: a process has an absolute leakage of zero precisely when it satisfies secrecy. The second model assumes a restricted observation scenario, inspired by the testing equivalence framework, where the attacker can only conduct repeated success-or-failure experiments on processes. Moreover, each experiment has a cost in terms of communication effort. The resulting notion of leakage rate, measured in bits per action, is in agreement with the first model: the maximum amount of information that can be extracted by repeated experiments coincides with the absolute leakage A of the process. Moreover, the overall extraction cost is at least A / R , where R is the rate of the process. The compositionality properties of the two models are also investigated.

Active Sensing by Unmanned Aircraft Systems in Realistic Communication Environments

This paper describes an information-theoretic framework for active sensing by unmanned aircraft systems in realistic communication environments. The position dependency of wireless networked communication is modeled using the standard empirical radio model in combination with Shannon capacity or a packet erasure channel. The covariance of the extended information filter is used to model sensed information content. The two information-theoretic concepts are then combined into a single optimization metric. Decentralized control laws are developed for simplified network scenarios. Stochastic approximation and extremum seeking control techniques are used to estimate the gradient of the sampled objective function. Experimental flight results demonstrate the capabilities of the active sensing framework.

INFORMATION-THEORETIC CONCEPTS FOR THE ANALYSIS OF COMPLEX NETWORKS

In this article, we present information-theoretic concepts for analyzing complex networks. We see that the application of information-theoretic concepts to networks leads to interesting tasks and gives a possibility for understanding information processing in networks. The main contribution of this article is a method for determining the structural information content of graphs that is based on a tree decomposition. It turns out that the computational complexity of the underlying algorithm is polynomial. Finally, we present some numerical results to study the influence of the used methods on the resulting information contents.

Genome-wide efficient attribute selection for purely epistatic models via Shannon entropy

Epistasis plays an important role in the genetic architecture of common human diseases. Most complex diseases are believed to have multiple contributing loci that often have subtle patterns which make them fairly difficult to find in large data sets. Disorders that follow purely epistatic models cannot be detected by cases/control studies based on individual analysis of susceptible loci. The computational complexity of performing exhaustive searches for detecting such models in genome-wide applications is practically unfeasible. Furthermore, with ever-increasing number of both genotypes and individuals on one side, and little knowledge of complex traits on the other, it is becoming fairly difficult and time consuming to perform systematic genome-wide studies on such traits. We present and discuss a convenient framework for modelling epistasis using information theoretic concepts and algorithms inspired by such an approach. These generalised algorithms, which are especially in favour of purely epistatic models, are applied to both simulated and real data. The real data represents the genotype-phenotype values for Age-Related Macular Degeneration (AMD) disease. Many two-locus purely epistatic patterns were found for AMD. A new visualisation approach is also presented for the purpose of better illustrating epistasy for cases where the number of loci is more than two or three.

Preservation of probabilistic information flow under refinement

Information flow properties, which describe confidentiality requirements, are not generally preserved under behavior refinement. This article describes a formal framework for refinement relations between nondeterministic probabilistic processes that capture sufficient conditions to preserve information flow properties. In particular, it uses information-theoretic concepts to investigate the refinement of a probabilistic, entropy-based information flow property. The refinement relation considers the abstract and concrete models as views on the same stochastic process. Probabilistic CSP provides the semantic basis for this investigation.

Multifield

Modern unsteady (multi-)field visualizations require an effective reduction of the data to be displayed. From a huge amount of information the most informative parts have to be extracted. Instead of the fuzzy application dependent notion of feature, a new approach based on information theoretic concepts is introduced in this paper to detect important regions. This is accomplished by extending the concept of local statistical complexity from finite state cellular automata to discretized (multi-)fields. Thus, informative parts of the data can be highlighted in an application-independent, purely mathematical sense. The new measure can be applied to unsteady multifields on regular grids in any application domain. The ability to detect and visualize important parts is demonstrated using diffusion, flow, and weather simulations.

Signal propagation in proteins and relation to equilibrium fluctuations.

Elastic network (EN) models have been widely used in recent years for describing protein dynamics, based on the premise that the motions naturally accessible to native structures are relevant to biological function. We posit that equilibrium motions also determine communication mechanisms inherent to the network architecture. To this end, we explore the stochastics of a discrete-time, discrete-state Markov process of information transfer across the network of residues. We measure the communication abilities of residue pairs in terms of hit and commute times, i.e., the number of steps it takes on an average to send and receive signals. Functionally active residues are found to possess enhanced communication propensities, evidenced by their short hit times. Furthermore, secondary structural elements emerge as efficient mediators of communication. The present findings provide us with insights on the topological basis of communication in proteins and design principles for efficient signal transduction. While hit/commute times are information-theoretic concepts, a central contribution of this work is to rigorously show that they have physical origins directly relevant to the equilibrium fluctuations of residues predicted by EN models.

Globally enumerating unstable periodic orbits for observed data using symbolic dynamics

The unstable periodic orbits of a chaotic system provide an important skeleton of the dynamics in a chaotic system, but they can be difficult to find from an observed time series. We present a global method for finding periodic orbits based on their symbolic dynamics, which is made possible by several recent methods to find good partitions for symbolic dynamics from observed time series. The symbolic dynamics are approximated by a Markov chain estimated from the sequence using information-theoretical concepts. The chain has a probabilistic graph representation, and the cycles of the graph may be exhaustively enumerated with a classical deterministic algorithm, providing a global, comprehensive list of symbolic names for its periodic orbits. Once the symbolic codes of the periodic orbits are found, the partition is used to localize the orbits back in the original state space. Using the periodic orbits found, we can estimate several quantities of the attractor such as the Lyapunov exponent and topological entropy.

On minimizing errors in 3D reconstruction for stereo camera systems

Active reconstruction of 3D surfaces deals with the control of camera viewpoints to minimize error and uncertainty in the reconstructed shape of an object. In this paper we develop a mathematical relationship between the setup and focal lengths of a stereo camera system and the corresponding error in 3D reconstruction of a given surface. We explicitly model the noise in the image plane, which can be interpreted as pixel noise or as uncertainty in the localization of corresponding point features. The results can be used to plan sensor posi- tioning, e.g., using information theoretic concepts for optimal sensor data selection. The paper is structured as follows: first, we describe the setup for 3D reconstruction using triangulation in a normalized stereo camera system. Then we present a mathematical development of the reconstruction error in a simple 2D model, taking explicitly into account noise in a one dimensional image plane. We map the problem of optimal stereo positioning to an optimiza- tion problem. This will be analyzed first, to get the opti- mal focal length and the optimal baseline (translation in x direction) in a normalized 2D stereo system. Further we gradually generalize this model, firstly by rotations, and secondly by translation in x and z directions. In this simple case we can perform a partial analytical analy- sis, but there are visibility assumptions which cannot be fulfilled in real stereo systems. Therefore, we further generalize to a 3D model with visibility constraints. In this model, we cannot perform an analytical analysis; therefore, we optimize the modifiable parameters by a Monte Carlo simulation. The results will be compared with the analytical results of the simple case. We present experimental results and compare them with the theoretical predictions, and conclude this paper with prospects for future study.

GOSim--an R-package for computation of information theoretic GO similarities between terms and gene products.

BackgroundWith the increased availability of high throughput data, such as DNA microarray data, researchers are capable of producing large amounts of biological data. During the analysis of such data often there is the need to further explore the similarity of genes not only with respect to their expression, but also with respect to their functional annotation which can be obtained from Gene Ontology (GO).ResultsWe present the freely available software package GOSim, which allows to calculate the functional similarity of genes based on various information theoretic similarity concepts for GO terms. GOSim extends existing tools by providing additional lately developed functional similarity measures for genes. These can e.g. be used to cluster genes according to their biological function. Vice versa, they can also be used to evaluate the homogeneity of a given grouping of genes with respect to their GO annotation. GOSim hence provides the researcher with a flexible and powerful tool to combine knowledge stored in GO with experimental data. It can be seen as complementary to other tools that, for instance, search for significantly overrepresented GO terms within a given group of genes.ConclusionGOSim is implemented as a package for the statistical computing environment R and is distributed under GPL within the CRAN project.

An information theoretic perspective on mesoscale seasonal variations in ground-level ozone

Relatively little prior use has been made of information theory in air quality analysis. This paper explores whether basic, but formal, quantitative measures of information content might yield fresh perspectives on seasonal variations in the ground-level ozone concentration field across the lower Fraser Valley (LFV), British Columbia, Canada. I calculate Shannon entropy in daily maximum ozone concentration on a month-by-month, station-by-station basis, using 1 year of hourly measurements from 18 air quality monitoring stations. The values are then qualitatively compared with an eye to identifying spatial and seasonal patterns. The results further demonstrate the potential utility of information theoretic concepts for assessing air quality variability; yield some new insight into tropospheric ozone dynamics across the LFV; and may provide some guidance to the refinement of monitoring network configuration. Of particular note is that, although entropy and mean concentration exhibit some similarities in their respective seasonal patterns, maximum uncertainty and information content appears to occur at times and locations somewhat different from those at which highest concentrations are experienced.

Polarization transformations of multimode light fields

The concept of P-quasispin is used as a framework for developing a formalism for polarization transformation by means of lossless polarization-changing optical elements for both classical and quantum multimode light fields. As an example, a classification of transformations of this kind is presented for two-mode biphoton fields that implement the information-theoretic concepts of qutrit and ququart (quantum systems in three-and four-dimensional Hilbert states, respectively).

Entropy and Information Content of Laboratory Test Results

This article introduces the use of information theoretic concepts such as entropy, S, for the evaluation of laboratory tests results, and it offers a new measure of information, 1 - S, which tells us just how far toward certainty a laboratory test result can predict a binary outcome. The derived method is applied to the serum markers troponin I and prostate-specific antigen and to histologic grading of HER-2/neu staining, to cytologic diagnosis of cervical specimens, and to the measurement of tumor thickness in malignant melanoma. Not only do the graphic results provide insight for these tests, they also validate prior conclusions. Thus, this information theoretic approach shows promise for evaluating and understanding laboratory test results.

Entanglement and symmetry: A case study in superselection rules, reference frames, and beyond

In recent years it has become apparent that constraints on possible quantum operations, such as those constraints imposed by superselection rules (SSRs), have a profound effect on quantum information theoretic concepts like bipartite entanglement. This paper concentrates on a particular example: the constraint that applies when the parties (Alice and Bob) cannot distinguish among certain quantum objects they have. This arises naturally in the context of ensemble quantum information processing such as in liquid NMR. We discuss how a SSR for the symmetric group can be applied, and show how the extractable entanglement can be calculated analytically in certain cases, with a maximum bipartite entanglement in an ensemble of $N$ Bell-state pairs scaling as $\mathrm{log}(N)$ as $N\ensuremath{\rightarrow}\ensuremath{\infty}$. We discuss the apparent disparity with the asymptotic $(N\ensuremath{\rightarrow}\ensuremath{\infty})$ recovery of unconstrained entanglement for other sorts of superselection rules, and show that the disparity disappears when the correct notion of applying the symmetric group SSR to multiple copies is used. Next we discuss reference frames in the context of this SSR, showing the relation to the work of von Korff and Kempe [Phys. Rev. Lett. 93, 260502 (2004)]. The action of a reference frame can be regarded as the analog of activation in mixed-state entanglement. We also discuss the analog of distillation: there exist states such that one copy can act as an imperfect reference frame for another copy. Finally we present an example of a stronger operational constraint, that operations must be noncollective as well as symmetric. Even under this stronger constraint we, nevertheless, show that Bell nonlocality (and hence entanglement) can be demonstrated for an ensemble of $N$ Bell-state pairs no matter how large $N$ is. This last work is a generalization of that of Mermin [Phys. Rev. D 22, 356 (1980)].

From linear adaptive filtering to nonlinear information processing - The design and analysis of information processing systems

Recent advances in computing capabilities and the interest in new challenging signal processing problems that cannot be successfully solved using traditional techniques have sparked an interest in information-theoretic signal processing techniques. Adaptive nonlinear filters that process signals based on their information content have become a major focus of interest. The design and analysis of such nonlinear information processing systems is demonstrated in this paper. Theoretical background on necessary information theoretic concepts are provided, nonparametric sample estimators for these quantities are derived and discussed, the use of these estimators for various statistical signal processing problems have been illustrated. These include data density modeling, system identification, blind source separation, dimensionality reduction, image registration, and data clustering

Encoding of Time-varying Stimuli in Populations of Cultured Neurons

We wondered whether random populations of dissociated cultured cortical neurons, despite of their lack of structure and/or regional specialization, are capable of modulating their neural activity as the effect of a time-varying stimulation - a simulated 'sensory' afference. More specifically, we used localized low-frequency, non-periodic trains of stimuli to simulate sensory afferences, and asked how much information about the original trains of stimuli could be extracted from the neural activity recorded at the different sites. Furthermore, motivated by the results of studies performed both in vivo and in vitro on different preparations, which suggested that isolated spikes and bursts may play different roles in coding time-varying signals, we explored the amount of such 'sensory' information that could be associated to these different firing modes. Finally, we asked whether and how such 'sensory' information is transferred from the sites of stimulation (i.e., the 'sensory' areas), to the other regions of the neural populations. To do this we applied stimulus reconstruction techniques and information theoretic concepts that are typically used to investigate neural coding in sensory systems.Our main results are that (1) slow variations of the rate of stimulation are coded into isolated spikes and in the time of occurrence of bursts (but not in the bursts' temporal structure); (2) increasing the rate of stimulation has the effect of increasing the proportion of isolated spikes in the average evoked response and their importance in coding for the stimuli; and, (3) the ability to recover the time course of the pattern of stimulation is strongly related to the degree of functional connectivity between stimulation and recording sites. These observations parallel similar findings in intact nervous systems regarding the complementary roles of bursts and tonic spikes in encoding sensory information.Our results also have interesting implications in the field of neuro-robotic interfaces. In fact, the ability of populations of neurons to code information is a prerequisite for obtaining hybrid systems, in which neuronal populations are used to control external devices.

Intelligent unmanned ground vehicle navigation via information evaluation

Sensor-centric navigation of unmanned ground vehicles (UGVs) operating in rugged and expansive terrains requires the competency to evaluate the utility of sensor information such that it results in intelligent behavior of the vehicles. Highly imperfect, inconsistent information and incomplete a priori knowledge introduce uncertainty in such unmanned navigation systems. Understanding and quantifying uncertainty yields a measure of useful information that plays a critical role in several robotic navigation tasks such as sensor fusion, mapping, localization, path planning and control. In this article, within a probabilistic framework, the utility of estimation and information-theoretic concepts towards quantifying uncertainty using entropy and mutual information metrics in various contexts of UGV navigation via experimental results is demonstrated.

On Genetic Information, Diversity and Distance

General information-theoretic concepts such as f-divergence, f-information and f-entropy are applied to the genetic models where genes are characterized by randomly distributed alleles. The paper thus presents an information-theoretic background for measuring genetic distances between populations, genetic information in various observations on individuals about their alleles and, finally, genetic diversities in various populations. Genetic distances were derived as divergences between frequencies of alleles representing a gene in two different populations. Genetic information was derived as a measure of statistical association between the observations taken on individuals and the alleles of these individuals. Genetic diversities were derived from divergences and information. The concept of genetic f-information introduced in the paper seems to be new. We show that the measures of genetic distance and diversity used in the previous literature are special cases of the genetic f-divergence and f-diversity introduced in the paper and illustrated by examples. We also display intimate connections between the genetic f-information and the genetic f-divergence on one side and genetic f-diversity on the other side. The examples at the same time also illustrate practical computations and applications of the important concepts of quantitative genetics introduced in the paper. We discussed a general class of f- divergence measures that are suitable measures of genetic distance between populations characterized by concrete frequencies of alleles. We have shown that a wide class of genetic information, called f-information, can be obtained from f-divergences and that a wide class of measures of genetic diversity, called f-diversities, can be obtained from the f-divergences and f-information.