Application Of Information Theory Research Articles

Law: An Information Technology

This article offers an historical, theoretical and practical perspective on law as an information technology. Law fundamentally concerns information — providing information to the community about the content of legal norms and, at least in its common law form, eliciting information about the world from the disputes before a court. The article first surveys law’s history as an information technology and shows how law is changed by the information technology of its day. The article then applies information theory to understand how the computer — the key technology of our day — is changing legal search and thereby the efficient form of law. Information theory focuses on the signal to noise ratio of communication. The key to progress in creating a better computerized legal search engine is to reduce the signal to noise ratio in the link between the user and the search engine. As this ratio decreases, we show that legal search translates the uncompressed form of legal information into an algorithm for predicting what the law will be in a particular situation. The ongoing improvement in legal search is transforming the optimal form of the law by changing the cost of finding it. It rebalances the weights in the classic debate between rules and standards. In particular, exponential increases in computational power make standards relatively more attractive than rules by decreasing the costs of their application. These same increases also allow us to embed information gathering processes within the law itself by creating what we call “dynamic rules.” Dynamic rules are rules that change automatically in response to changing empirical information. Such rules are already beginning to be enacted by legislatures. Since we believe that standards and dynamic rules are likely to be prevalent legal forms of the coming era, we close our article with a comparison of their relative costs and benefits.

SOME CONSTRUCTIONS OF MUTUALLY ORTHOGONAL LATIN SQUARES AND SUPERIMPOSED CODES

Superimposed codes is a special combinatorial structure that has many applications in information theory, data communication and cryptography. On the other hand, mutually orthogonal latin squares is a beautiful combinatorial object that has deep connection with design theory. In this paper, we draw a connection between these two structures. We give explicit construction of mutually orthogonal latin squares and we show a method of generating new larger superimposed codes from an existing one by using mutually orthogonal latin squares. If n denotes the number of codewords in the existing code then the new code contains n2codewords. Recursively, using this method, we can construct a very large superimposed code from a small simple code. Well-known constructions of superimposed codes are based on algebraic Reed–Solomon codes and our new construction gives a combinatorial alternative approach.

The application of information theory to biochemical signaling systems

Cell signaling can be thought of fundamentally as an information transmission problem in which chemical messengers relay information about the external environment to the decision centers within a cell. Due to the biochemical nature of cellular signal transduction networks, molecular noise will inevitably limit the fidelity of any messages received and processed by a cell's signal transduction networks, leaving it with an imperfect impression of its environment. Fortunately, Shannon's information theory provides a mathematical framework independent of network complexity that can quantify the amount of information that can be transmitted despite biochemical noise. In particular, the channel capacity can be used to measure the maximum number of stimuli a cell can distinguish based upon the noisy responses of its signaling systems. Here, we provide a primer for quantitative biologists that covers fundamental concepts of information theory, highlights several key considerations when experimentally measuring channel capacity, and describes successful examples of the application of information theoretic analysis to biological signaling.

The use of information theory in evolutionary biology

Information is a key concept in evolutionary biology. Information stored in a biological organism's genome is used to generate the organism and to maintain and control it. Information is also that which evolves. When a population adapts to a local environment, information about this environment is fixed in a representative genome. However, when an environment changes, information can be lost. At the same time, information is processed by animal brains to survive in complex environments, and the capacity for information processing also evolves. Here, I review applications of information theory to the evolution of proteins and to the evolution of information processing in simulated agents that adapt to perform a complex task.

A Markov-based reversible data hiding method based on histogram shifting

Applying information theory, this work considers an image as a stream of symbols emitted by a Markov information source. With the Markov model, a reversible data-hiding scheme based on the histogram modification technique is proposed to provide an efficient tradeoff between hiding capacity and quality of a marked image by changing the order of the Markov model. The larger the order is, the higher the capacity is but the lower the quality is, and vice versa. The experimental results show that the proposed scheme yields not only much larger hiding capacity but also smaller image distortion than other reversible data-hiding schemes reported in the literature. This work also proposes two feasible approaches to reduce the overhead yielded during the data embedding.

Performances of Amplify-and-Forward Cooperative Relay Networks with Different Topologies

When the sizes of terminals are limited, cooperative relay networks are recommended as promising virtual MIMO systems to improve the performances of wireless communication systems in literature. Accordingly, some comparisons on the performances of different systems are useful to guide the selection and deployment of cooperative relay systems in field applications. In this paper, we compare the outage probabilities of cooperative relay networks of different topologies (different numbers of antennas used at a transmitter and relays used in a system), as well as the outage probabilities of the MISO systems. We propose an aggregated channel mode, which combines the power gain obtained from the reduction of path loss through the deployment of relays, and the flat fading caused by multi-path propagation. During the evaluation of the performances of different systems, we apply information theory in the calculation of ergodic capacities and consider the requirement on wireless resources in the estimation of outage probabilities. Numerical simulation shows that (1) cooperative networks with multiple relays outperform those with multiple antennas; (2) only when the power gain is included, the performances of cooperative relay networks are comparable with their counterparts in direct transmission; (3) the performances are significantly affected by the locations of relays.

Reversed version of a generalized sharp Hölder’s inequality and its applications

In this paper, we give a reversed version of a generalized sharp Hölder’s inequality which is due to Wu. The results are then used to improve Beckenbach’s inequality and Minkowski’s inequality. Moreover, as an application in information theory, we present a refinement of Singh’s inequality which is one generalized Shannon’s inequality.

Leveraging Strength-Based Dynamic Information Flow Analysis to Enhance Data Value Prediction

Value prediction is a technique to increase parallelism by attempting to overcome serialization constraints caused by true data dependences. By predicting the outcome of an instruction before it executes, value prediction allows data dependent instructions to issue and execute speculatively, hence increasing parallelism when the prediction is correct. In case of a misprediction, the execution is redone with the corrected value. If the benefit from increased parallelism outweighs the misprediction recovery penalty, overall performance could be improved. Enhancing performance with value prediction therefore requires highly accurate prediction methods. Most existing general value prediction techniques are local, that is, future outputs of an instruction are predicted based on outputs from previous executions of the same instruction. In this article, we investigate leveraging strength-based dynamic information flow analysis to enhance data value prediction. We use dynamic information flow analysis (DIFA) to determine when a specific value predictor can perform well and even outperform other predictors. We apply information theory to mathematically prove the validity and benefits of correlating value predictors. We also introduce the concept of the linear value predictors, a new technique that predicts a new value from another one using a linear relation. We finally present a variant of stride predictor that we call update stride . We then conduct an empirical analysis using Pin , a dynamic binary instrumentation tool, and DynFlow , a dynamic information flow analysis tool, that we apply to programs from the SPECjvm2008 and Siemens benchmarks. Our empirical measurements support our mathematical theory and allow us to make important observations on the relation between predictability of data values and information flow. Our analysis and empirical results show that the values of a set of selected variables can be predicted with a very high accuracy, up to 100%. Such prediction is based on the previous history and/or the values of one or more other source variables that have strong information flow into the predicted variable. Using our selection criteria, we show that a DIFA-directed predictor outperforms hardware value prediction for all subject programs, and sometimes by a significant margin. This was observed even when using an ideal tagged hardware value prediction table that does not suffer from aliasing or capacity misses.

Information Theory and Observational Limitations in Decision Making

We introduce a general framework for formalizing and analyzing the problem faced by a Decision Maker (DM) working under information-theoretic constraints on their observational ability. The random utility model and the "hedonic utility" model of Netzer and Robson (NR) are special cases of this framework. We begin by applying information theory to our framework to derive general results concerning the expected regret of DM under observational limitations. We then turn our attention to the effects of observational limitations on choice behavior (rather than their effects on the regret values induced by that behavior). We focus on the special case of NR. First we derive two postulates assumed by NR. We then provide a simple derivation of the result of NR that a particular hedonic utility function satisfies certain optimality principles. Next we extend NR to allow a countable, rather than uncountable, set of states of the world. In particular we show how to use dynamic programming to solve for the optimal preference order of DM in this extension. We also extend NR by considering the case where more than two options are presented to DM. In particular, we show that the results of NR change in such a case, implying that the number of options being presented is a crucial aspect of choice problems.

An application of data encryption technique using random number generator

Coding theory is one of the most important and direct applications of information theory. Using a statistical description for data, information theory quantifies the number of bits needed to describe the data, which is the information entropy of the source. Information theoretic concepts apply to cryptography and cryptanalysis. Cryptography is the study of sending and receiving secret messages. With the widespread use of information technologies and the rise of digital computer networks in many areas of the world, securing the exchange of information has become a crucial task. In the present paper an innovative technique for data encryption is proposed based on the random sequence generation. The new algorithm provides data encryption at two levels and hence security against crypto analysis is achieved at relatively low computational overhead.

Sorting of Permutations by Cost-Constrained Transpositions

The problem of finding a minimum decomposition of a permutation in terms of transpositions with predetermined non-uniform and non-negative costs is addressed. Alternatively, computing the transposition distance between two permutations, where transpositions are endowed with arbitrary non-negative costs, is studied. For such cost functions, polynomial-time, constant-approximation decomposition algorithms are described. For metric-path costs, exact polynomial-time decomposition algorithms are presented. The algorithms in this paper represent a combination of Viterbi-type algorithms and graph-search techniques for minimizing the cost of individual transpositions, and dynamic programing algorithms for finding minimum cost decompositions of cycles. The presented algorithms have a myriad of applications in information theory, bioinformatics, and algebra.

Insider Trading Rules and Price Formation in Securities Markets - An Entropy Analysis of Strategic Trading

This paper addresses the issue of how insider trading rules affect price formation in securities markets and suggests the application of information theory to market microstructure theory. We analyze a variant of Kyle's (1985) setting by simply introducing a more general criterion for informational efficiency borrowed from information theory - namely maximum information transmission. The analysis shows that both the insider's optimal trading strategy and the market price of the risky security depend on the insider trading restriction. Insider trading restrictions are reported to be detrimental to the liquidity of the securities market. We find that a unique insider trading rule exists which implements semi-strong form informational efficiency of the securities market. Alternative restrictions on insider trading give rise to either underreaction or overreaction in securities prices. Too strict insider trading rules are shown to account for excess volatility in securities prices. Contrary to common notion, the uninformed investors are shown to be hurt by too restrictive insider trading rules. We conclude that loose insider trading rules are preferred by the group of investors as a whole.

Committee Specialization in U.S. State Legislatures during the 20th Century: Do Legislatures Tap the Talents of Their Members?

AbstractThe authors use an informational-theoretical perspective to examine the appointment of legislative committees, using new measures of knowledge and expertise as well as a unique database. While competing theories (distributive and partisan) do not deny that legislative committees are a source of vital knowledge and expertise necessary for legislative policy making, information theory places the wisdom and expertise needs of collective decision making—that is, specialization—on very complex topics by legislative generalists as the raison d'être for a committee system. Thus, the authors investigate one of the fundamental arguments of information theory—that committees are formed to meet the basic knowledge and expertise needs of a legislative body. The findings from U.S. state legislative data indicate that significant member differences regarding prior knowledge and expertise exist by committee jurisdiction, state, and session, reinforcing the applicability of information theory while also suggesting the impact of setting and organizational factors. The authors also explore implications of these findings.

Information Theoretic Learning: Reny's Entropy and Kernel Perspectives (Principe, J.; 2010) [Book Review

This book, derived from Jose Principe and his group's 10 years' research in information theory and statistical learning, gives a comprehensive introduction, analysis and demonstration of almost all the major components required for understanding and developing the new theme of information-theoretical learning. The basic strategy utilized by the author is to apply information theory descriptors (namely entropy and divergence, in contrast to the statistical measures of mean and covariance)as nonparametric cost functions for the design of adaptive systems, thus creating a new paradigm of information theoretic learning. And like in statistical learning, unsupervised or supervised training modes are also fully explored.

Application of information theory to feature selection in protein docking

In the era of structural genomics, the prediction of protein interactions using docking algorithms is an important goal. The success of this method critically relies on the identification of good docking solutions among a vast excess of false solutions. We have adapted the concept of mutual information (MI) from information theory to achieve a fast and quantitative screening of different structural features with respect to their ability to discriminate between physiological and nonphysiological protein interfaces. The strategy includes the discretization of each structural feature into distinct value ranges to optimize its mutual information. We have selected 11 structural features and two datasets to demonstrate that the MI is dimensionless and can be directly compared for diverse structural features and between datasets of different sizes. Conversion of the MI values into a simple scoring function revealed that those features with a higher MI are actually more powerful for the identification of good docking solutions. Thus, an MI-based approach allows the rapid screening of structural features with respect to their information content and should therefore be helpful for the design of improved scoring functions in future. In addition, the concept presented here may also be adapted to related areas that require feature selection for biomolecules or organic ligands.

Comprehensive prediction of mRNA splicing effects of BRCA1 and BRCA2 variants

Variants of uncertain significance (VUS) in the BRCA1 and BRCA2 genes potentially affecting coding sequence as well as normal splicing activity have confounded predisposition testing in breast cancer. Here, we apply information theory to analyze BRCA1/2 mRNA splicing mutations categorized as VUS. The method was validated for 31 of 36 mutations known to cause missplicing in BRCA1/2 and all 26 that do not alter splicing. All single-nucleotide variants in the Breast Cancer Information Resource (BIC; Breast Cancer Information Core Database; http://research.nhgri.nih.gov/bic; last access June 1, 2010) were then analyzed. Information analysis is similar in sensitivity to other predictive methods; however, the thermodynamic basis of the theory also enables splice-site affinity to be determined accurately, which is important for assessing mutations that render natural splice sites partially functional and competition between cryptic and natural splice sites. We report 299 of 2,071 single-nucleotide BIC mutations that are predicted to significantly weaken natural sites and/or strengthen cryptic splice sites, 171 of which are not designated as splicing mutations in the database. Splicing alterations are predicted for 68 of 690 BRCA1 and 60 of 958 BRCA2 mutations designated as VUS. These analyses should be useful in prioritizing suspected mutations for downstream expression studies and for predicting aberrantly spliced isoforms generated by these mutations.

Mutual Information Is Copula Entropy

Mutual information (MI) is a basic concept in information theory. Therefore, estimates of the MI are fundamentally important in most information theory applications. This paper provides a new way of understanding and estimating the MI using the copula function. First, the entropy of the copula, named the copula entropy, is defined as a measure of the dependence uncertainty represented by the copula function and then the MI is shown to be equivalent to the negative copula entropy. With this equivalence, the MI can be estimated by first estimating the empirical copula and then estimating the entropy of the empirical copula. Thus, the MI estimate is an estimation of the entropy, which reduces the complexity and computational requirements. Tests show that the method is more effective than the traditional method.

Joint Coding/Routing Optimization for Distributed Video Sources in Wireless Visual Sensor Networks

This paper studies a joint coding/routing optimization between network lifetime and video distortion by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding [distributed video coding and network coding (NC)] from both combinatorial optimization and information theory could make significant progress toward the performance limit and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps the total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Also, a generalized power consumption model for distributed video sources is developed, in which the coding complexity of Key frames and Wyner-Ziv frames is measured by translating specific coding behavior into energy consumption. On the basis of the distributed multiview video coding and NC-based multipath routing, the balance problem between lifetime (costs) and distortion (capacity) is modeled as an optimization formulation with a fully distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved by the gradient algorithm. The low-level optimization problem is further decomposed into a secondary master dual problem with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. The implementation of the distributed algorithm is discussed with regard to the communication overhead and dynamic network change. Simulation results validate the convergence and performance of the proposed algorithm.

Information theory in neuroscience

Information Theory started and, according to some, ended with Shannon’s seminal paper “A Mathematical Theory of Communication” (Shannon 1948). Because its significance and flexibility were quickly recognized, there were numerous attempts to apply it to diverse fields outside of its original scope. This prompted Shannon to write his famous essay “The Bandwagon” (Shannon 1956), warning against indiscriminate use of the new tool. Nevertheless, non-standard applications of Information Theory persisted. Very soon after Shannon’s initial publication (Shannon 1948), several manuscripts provided the foundations of much of the current use of information theory in neuroscience. MacKay and McCulloch (1952) applied the concept of information to propose limits of the transmission capacity of a nerve cell. This work foreshadowed future work on what can be termed “Neural Information Flow”—how much information moves through the nervous system, and the constraints that information theory imposes on the capabilities of neural systems for communication, computation and behavior. A second set of manuscripts, by Attneave (1954) and Barlow (1961), discussed information as a constraint on neural system structure and function, proposing that neural structure in sensory system is matched to statistical structure of the sensory environment, in a way to optimize information transmission. This is the main idea behind the “Structure from Information” line of research that is still very active today. A third thread, “Reliable Computation with Noisy/Faulty Elements”, started both in the information-theoretic community (Shannon and McCarthy 1956) and neuro-science (Winograd and Cowan 1963). With the advent of integrated circuits that were essentially faultless, interest began to wane. However, as IC technology continues to push towards smaller and faster computational elements (even at the expense of reliability), and as neuromorphic systems are developed with variability designed in (Merolla and Boahen 2006), this topic is gaining in popularity again in the electronics community, and neuroscientists again may have something to contribute to the discussion.

Integrated bio-inorganic hybrid systems for nano-forensics

This tutorial review describes a new class of data processing system that applies information theory at the molecular level. We also summarize the recent multidisciplinary advances in biotechnology and nanotechnology that have facilitated the development of reliable nano-level code systems. After a brief introduction of information theory, we present possible ways to adapt this concept to the molecular world. This review explains the requirements and solutions for each step necessary to apply a nano code system to real products. Finally, we introduce a designed nano code system for agricultural products as one example of the many possible applications for nano codes.