Application Of Information Theory Research Articles

Quantitative Approaches to Representing the Value of Information within the Intelligence Cycle

The authors propose that valuation of information metrics developed near the end of the intelligence cycle are appropriate supplemental metrics for national security intelligence. Existing information and decision theoretic frameworks are often either inapplicable in the context of national security intelligence or they capture affects from inputs aside from just the information or intelligence. Applied information theory looks at the syntactic transmission of information rather than assigning it a quantitative value. Information economics determines the market value of information, which is also inapplicable in a national security intelligence context. Decision analysis can use the value of information to show the expected value of perfect information (EVPI) and the expected value of imperfect information (EVII) and although this method can be used with utility theory and not just monetary objectives, it has been shown that decision makers within the intelligence community (IC) have difficulty agreeing upon how to value objectives within analysis. Additionally, it is difficult to determine how decision makers use intelligence in the decision-making process, which makes existing decision theoretic methods problematic, and might include inputs from variables besides just the intelligence.

Brain control and information transfer.

In this review, we examine the importance of having a body as essential for the brain to transfer information about the outside world to generate appropriate motor responses. We discuss the context-dependent conditioning of the motor control neural circuits and its dependence on the completion of feedback loops, which is in close agreement with the insights of Hebb and colleagues, who have stressed that for learning to occur the body must be intact and able to interact with the outside world. Finally, we apply information theory to data from published studies to evaluate the robustness of the neuronal signals obtained by bypassing the body (as used for brain-machine interfaces) versus via the body to move in the world. We show that recording from a group of neurons that bypasses the body exhibits a vastly degraded level of transfer of information as compared to that of an entire brain using the body to engage in the normal execution of behaviour. We conclude that body sensations provide more than just feedback for movements; they sustain the necessary transfer of information as animals explore their environment, thereby creating associations through learning. This work has implications for the development of brain-machine interfaces used to move external devices.

Limits on information transduction through amplitude and frequency regulation of transcription factor activity.

Signaling pathways often transmit multiple signals through a single shared transcription factor (TF) and encode signal information by differentially regulating TF dynamics. However, signal information will be lost unless it can be reliably decoded by downstream genes. To understand the limits on dynamic information transduction, we apply information theory to quantify how much gene expression information the yeast TF Msn2 can transduce to target genes in the amplitude or frequency of its activation dynamics. We find that although the amount of information transmitted by Msn2 to single target genes is limited, information transduction can be increased by modulating promoter cis-elements or by integrating information from multiple genes. By correcting for extrinsic noise, we estimate an upper bound on information transduction. Overall, we find that information transduction through amplitude and frequency regulation of Msn2 is limited to error-free transduction of signal identity, but not signal intensity information.

Unbounded number of channel uses may be required to detect quantum capacity.

Transmitting data reliably over noisy communication channels is one of the most important applications of information theory, and is well understood for channels modelled by classical physics. However, when quantum effects are involved, we do not know how to compute channel capacities. This is because the formula for the quantum capacity involves maximizing the coherent information over an unbounded number of channel uses. In fact, entanglement across channel uses can even increase the coherent information from zero to non-zero. Here we study the number of channel uses necessary to detect positive coherent information. In all previous known examples, two channel uses already sufficed. It might be that only a finite number of channel uses is always sufficient. We show that this is not the case: for any number of uses, there are channels for which the coherent information is zero, but which nonetheless have capacity.

Interpretation of mRNA splicing mutations in genetic disease: review of the literature and guidelines for information-theoretical analysis

The interpretation of genomic variants has become one of the paramount challenges in the post-genome sequencing era. In this review we summarize nearly 20 years of research on the applications of information theory (IT) to interpret coding and non-coding mutations that alter mRNA splicing in rare and common diseases. We compile and summarize the spectrum of published variants analyzed by IT, to provide a broad perspective of the distribution of deleterious natural and cryptic splice site variants detected, as well as those affecting splicing regulatory sequences. Results for natural splice site mutations can be interrogated dynamically with Splicing Mutation Calculator, a companion software program that computes changes in information content for any splice site substitution, linked to corresponding publications containing these mutations. The accuracy of IT-based analysis was assessed in the context of experimentally validated mutations. Because splice site information quantifies binding affinity, IT-based analyses can discern the differences between variants that account for the observed reduced (leaky) versus abolished mRNA splicing. We extend this principle by comparing predicted mutations in natural, cryptic, and regulatory splice sites with observed deleterious phenotypic and benign effects. Our analysis of 1727 variants revealed a number of general principles useful for ensuring portability of these analyses and accurate input and interpretation of mutations. We offer guidelines for optimal use of IT software for interpretation of mRNA splicing mutations.

Mining Informative Hydrologic Data by Using Support Vector Machines and Elucidating Mined Data according to Information Entropy

The support vector machine is used as a data mining technique to extract informative hydrologic data on the basis of a strong relationship between error tolerance and the number of support vectors. Hydrologic data of flash flood events in the Lan-Yang River basin in Taiwan were used for the case study. Various percentages (from 50% to 10%) of hydrologic data, including those for flood stage and rainfall data, were mined and used as informative data to characterize a flood hydrograph. Information on these mined hydrologic data sets was quantified using entropy indices, namely marginal entropy, joint entropy, transinformation, and conditional entropy. Analytical results obtained using the entropy indices proved that the mined informative data could be hydrologically interpreted and have a meaningful explanation based on information entropy. Estimates of marginal and joint entropies showed that, in view of flood forecasting, the flood stage was a more informative variable than rainfall. In addition, hydrologic models with variables containing more total information were preferable to variables containing less total information. Analysis results of transinformation explained that approximately 30% of information on the flood stage could be derived from the upstream flood stage and 10% to 20% from the rainfall. Elucidating the mined hydrologic data by applying information theory enabled using the entropy indices to interpret various hydrologic processes.

Conditioned reinforcement and information theory reconsidered.

The idea that stimuli might function as conditioned reinforcers because of the information they convey about primary reinforcers has a long history in the study of learning. However, formal application of information theory to conditioned reinforcement has been largely abandoned in modern theorizing because of its failures with respect to observing behavior. In this paper we show how recent advances in the application of information theory to Pavlovian conditioning offer a novel approach to conditioned reinforcement. The critical feature of this approach is that calculations of information are based on reductions of uncertainty about expected time to primary reinforcement signaled by a conditioned reinforcer. Using this approach, we show that previous failures of information theory with observing behavior can be remedied, and that the resulting framework produces predictions similar to Delay Reduction Theory in both observing-response and concurrent-chains procedures. We suggest that the similarity of these predictions might offer an analytically grounded reason for why Delay Reduction Theory has been a successful theory of conditioned reinforcement. Finally, we suggest that the approach provides a formal basis for the assertion that conditioned reinforcement results from Pavlovian conditioning and may provide an integrative approach encompassing both domains.

A Log-Det Inequality for Random Matrices

We prove a new inequality for the expectation $E\left[\log\det\left(\mathbf{W}\mathbf{Q}+\mathbf{I}\right)\right]$, where $\mathbf{Q}$ is a nonnegative definite matrix and $\mathbf{W}$ is a diagonal random matrix with identically distributed nonnegative diagonal entries. A sharp lower bound is obtained by substituting $\mathbf{Q}$ by the diagonal matrix of its eigenvalues $\mathbf{\Gamma}$. Conversely, if this inequality holds for all $\mathbf{Q}$ and $\mathbf{\Gamma}$, then the diagonal entries of $\mathbf{W}$ are necessarily identically distributed. From this general result, we derive related deterministic inequalities of Muirhead- and Rado-type. We also present some applications in information theory: We derive bounds on the capacity of parallel Gaussian fading channels with colored additive noise and bounds on the achievable rate of noncoherent Gaussian fading channels.

Shannon Goes to the Museum: Drawing Lines Across Boundaries

The DOCAM’15 theme called for an examination of the challenges ahead with our understanding of documents in a continuously changing information landscape. One such challenge has been to find specific intersecting areas of the information sciences on which authors from different disciplines might collaborate. We take ourselves as one such case study. One of us works in information science and often thinks about applications of Information Theory. One of us works on developing models for museum practice, that is, theory upon which museum work might become more intentional and robust. Thinking about Documents Unbounded has led us to align some of our recent work, and, by doing so, demonstrate that manifestations of information theory abound across the information disciplines, which have origin in and continuing relevance with the document, museum, communications, and library studies realms. In this philosophical experiment, we try to draw lines between Wood and Latham’s (2013) Object Knowledge Framework (OKF) and O’Connor, Kearns & Anderson’s (2008) notion of Question (Q), in order to make some assertions about drawing lines between disciplines.

Interpretation of mRNA splicing mutations in genetic disease: review of the literature and guidelines for information-theoretical analysis.

The interpretation of genomic variants has become one of the paramount challenges in the post-genome sequencing era. In this review we summarize nearly 20 years of research on the applications of information theory (IT) to interpret coding and non-coding mutations that alter mRNA splicing in rare and common diseases. We compile and summarize the spectrum of published variants analyzed by IT, to provide a broad perspective of the distribution of deleterious natural and cryptic splice site variants detected, as well as those affecting splicing regulatory sequences. Results for natural splice site mutations can be interrogated dynamically with Splicing Mutation Calculator, a companion software program that computes changes in information content for any splice site substitution, linked to corresponding publications containing these mutations. The accuracy of IT-based analysis was assessed in the context of experimentally validated mutations. Because splice site information quantifies binding affinity, IT-based analyses can discern the differences between variants that account for the observed reduced (leaky) versus abolished mRNA splicing. We extend this principle by comparing predicted mutations in natural, cryptic, and regulatory splice sites with observed deleterious phenotypic and benign effects. Our analysis of 1727 variants revealed a number of general principles useful for ensuring portability of these analyses and accurate input and interpretation of mutations. We offer guidelines for optimal use of IT software for interpretation of mRNA splicing mutations.

Applying Information Theory to Neuronal Networks: From Theory to Experiments

Information-theory is being increasingly used to analyze complex, self-organizing processes on networks, predominantly in analytical and numerical studies. Perhaps one of the most paradigmatic complex systems is a network of neurons, in which cognition arises from the information storage, transfer, and processing among individual neurons. In this article we review experimental techniques suitable for validating information-theoretical predictions in simple neural networks, as well as generating new hypotheses. Specifically, we focus on techniques that may be used to measure both network (microcircuit) anatomy as well as neuronal activity simultaneously. This is needed to study the role of the network structure on the emergent collective dynamics, which is one of the reasons to study the characteristics of information processing. We discuss in detail two suitable techniques, namely calcium imaging and the application of multi-electrode arrays to simple neural networks in culture, and discuss their advantages and limitations in an accessible manner for non-experts. In particular, we show that each technique induces a qualitatively different type of error on the measured mutual information. The ultimate goal of this work is to bridge the gap between theorists and experimentalists in their shared goal of understanding the behavior of networks of neurons.

Application of information theory to a three-body coarse-grained representation of proteins in the PDB: insights into the structural and evolutionary roles of residues in protein structure.

Knowledge-based methods for analyzing protein structures, such as statistical potentials, primarily consider the distances between pairs of bodies (atoms or groups of atoms). Considerations of several bodies simultaneously are generally used to characterize bonded structural elements or those in close contact with each other, but historically do not consider atoms that are not in direct contact with each other. In this report, we introduce an information-theoretic method for detecting and quantifying distance-dependent through-space multibody relationships between the sidechains of three residues. The technique introduced is capable of producing convergent and consistent results when applied to a sufficiently large database of randomly chosen, experimentally solved protein structures. The results of our study can be shown to reproduce established physico-chemical properties of residues as well as more recently discovered properties and interactions. These results offer insight into the numerous roles that residues play in protein structure, as well as relationships between residue function, protein structure, and evolution. The techniques and insights presented in this work should be useful in the future development of novel knowledge-based tools for the evaluation of protein structure.

CSAU methodology and results for an ATWS event in a BWR using information theory methods

This paper shows an application of the CSAU methodology to an ATWS in a BWR reactor, when the temperature of the suppression pool is taken as the critical safety parameter. The method combines CSAU methodology with recent techniques of information theory. In this paper we use auxiliary tools to help in the evaluation and improvement of the parameters distribution that enter in the elements II and III of CSAU based methodologies. These tools have been implemented in two FORTRAN programs: GEDIPA (Generation of the Parameter Distribution) and UNTHERCO (Uncertainty in Thermal Hydraulic Codes). The first one analyzes the information data available on a given parameter or parameters with the goal to know all the information about the probability distribution function of these parameters. The second apply information theory methods, as the maximum entropy principle (MEP) and the maximum relative entropy Principle (MREP), in order to build conservative distribution functions for the parameters from the available data. Also, the distribution function of a given parameter can be updated using the MREP principle when new information is provided. UNTHERCO performs the MONTECARLO sampling for a given set of parameters when the distribution function of these parameters is previously known. If the distribution of a parameter is unknown, then, the MEP is applied to deduce the distribution function for this parameter.

New property of a generalized Hölder’s inequality and its applications

In this paper, we present a new property of a generalized Hölder’s inequality, and then we obtain a new refinement of the generalized Hölder’s inequality. As applications, we give a generalization and refinement of the Opial–Beesack inequality by using the obtained result. Moreover, as an application in information theory, we give a refinement of Singh’s inequality which is a generalized Shannon’s inequality.

Estimating information entropy for hydrological data: One‐dimensional case

AbstractThere has been a recent resurgence of interest in the application of Information Theory to problems of system identification in the Earth and Environmental Sciences. While the concept of entropy has found increased application, little attention has yet been given to the practical problems of estimating entropy when dealing with the unique characteristics of two commonly used kinds of hydrologic data: rainfall and runoff. In this paper, we discuss four important issues of practical relevance that can bias the computation of entropy if not properly handled. The first (zero effect) arises when precipitation and ephemeral streamflow data must be viewed as arising from a discrete‐continuous hybrid distribution due to the occurrence of many zero values (e.g., days with no rain/no runoff). Second, in the widely used bin‐counting method for estimation of PDF's, significant error can be introduced if the bin width is not carefully selected. The third (measurement effect) arises due to the fact that continuously varying hydrologic variables can typically only be observed discretely to some degree of precision. The Fourth (skewness effect) arises when the distribution of a variable is significantly skewed. Here we present an approach that can deal with all four of these issues, and test them with artificially generated and real hydrological data. The results indicate that the method is accurate and robust.

An efficient algorithm for the symmetric principal minor assignment problem

We consider the problem of reconstructing a symmetric matrix from its principal minors, which has several applications in information theory and statistical modeling. We develop a theory of symmetric matrices with equal corresponding principal minors based on a simple equivalent property due to Oeding (2011) [10]. We then use this theory to provide a method for choosing a canonical representative from the class of all symmetric matrices with specified principal minors. Finally, we provide an efficient algorithm for computing this canonical representative given its principal minors as input.

Trends in information theory-based chemical structure codification

This report offers a chronological review of the most relevant applications of information theory in the codification of chemical structure information, through the so-called information indices. Basically, these are derived from the analysis of the statistical patterns of molecular structure representations, which include primitive global chemical formulae, chemical graphs, or matrix representations. Finally, new approaches that attempt to go "back to the roots" of information theory, in order to integrate other information-theoretic measures in chemical structure coding are discussed.

DIRECTED DIVERGENCE AS A MEASURE OF SIMILARITY

The development of interrelationships between divergence mea- sures and the known statistical constants provide the applications of information theory to the field of statistics. In the literature of information measures, there exist many divergence measures for discrete probability distributions whereas we need such divergence measures for continuous distributions to extend the scope of their applications. In the present communication, we have introduced divergence measures for continuous variate distributions and then proved that the divergence between the joint distribution density and the product of the marginal distribution densities is a function of the correlation coefficient which obviously implies that the divergence is also a measure of the similarity or of the dissimilarity.

The Entropy Power Inequality for Quantum Systems

When two independent analog signals, X and Y are added together giving Z=X+Y, the entropy of Z, H(Z), is not a simple function of the entropies H(X) and H(Y), but rather depends on the details of X and Y's distributions. Nevertheless, the entropy power inequality (EPI), which states that exp [2H(Z)] \geq exp[2H(X)] + exp[2H(Y)], gives a very tight restriction on the entropy of Z. This inequality has found many applications in information theory and statistics. The quantum analogue of adding two random variables is the combination of two independent bosonic modes at a beam splitter. The purpose of this work is to give a detailed outline of the proof of two separate generalizations of the entropy power inequality to the quantum regime. Our proofs are similar in spirit to standard classical proofs of the EPI, but some new quantities and ideas are needed in the quantum setting. Specifically, we find a new quantum de Bruijin identity relating entropy production under diffusion to a divergence-based quantum Fisher information. Furthermore, this Fisher information exhibits certain convexity properties in the context of beam splitters.

Estimation of Mutual Information and Conditional Entropy for Surveillance Optimization

Summary Given a suite of potential surveillance operations, we define surveillance optimization as the problem of choosing the operation that gives the minimum expected value of P90 minus P10 (i.e., P90 – P10) of a specified reservoir variable J (e.g., cumulative oil production) that will be obtained by conditioning J to the observed data. Two questions can be posed: (1) Which surveillance operation is expected to provide the greatest uncertainty reduction in J? and (2) What is the expected value of the reduction in uncertainty that would be achieved if we were to undertake each surveillance operation to collect the associated data and then history match the data obtained? In this work, we extend and apply a conceptual idea that we recently proposed for surveillance optimization to 2D and 3D waterflooding problems. Our method is based on information theory in which the mutual information between J and the random observed data vector Dobs is estimated by use of an ensemble of prior reservoir models. This mutual information reflects the strength of the relationship between J and the potential observed data and provides a qualitative answer to Question 1. Question 2 is answered by calculating the conditional entropy of J to generate an approximation of the expected value of the reduction in (P90 – P10) of J. The reliability of our method depends on obtaining a good estimate of the mutual information. We consider several ways to estimate the mutual information and suggest how a good estimate can be chosen. We validate the results of our proposed method with an exhaustive history-matching procedure. The methodology provides an approximate way to decide the data that should be collected to maximize the uncertainty reduction in a specified reservoir variable and to estimate the reduction in uncertainty that could be obtained. We expect this paper will stimulate significant research on the application of information theory and lead to practical methods and workflows for surveillance optimization.