Information-theoretic Methods Research Articles

Cluster Validation Method for Determining the Number of Clusters in Categorical Sequences.

Cluster validation, which is the process of evaluating the quality of clustering results, plays an important role for practical machine learning systems. Categorical sequences, such as biological sequences in computational biology, have become common in real-world applications. Different from previous studies, which mainly focused on attribute-value data, in this paper, we work on the cluster validation problem for categorical sequences. The evaluation of sequences clustering is currently difficult due to the lack of an internal validation criterion defined with regard to the structural features hidden in sequences. To solve this problem, in this paper, a novel cluster validity index (CVI) is proposed as a function of clustering, with the intracluster structural compactness and intercluster structural separation linearly combined to measure the quality of sequence clusters. A partition-based algorithm for robust clustering of categorical sequences is also proposed, which provides the new measure with high-quality clustering results by the deterministic initialization and the elimination of noise clusters using an information theoretic method. The new clustering algorithm and the CVI are then assembled within the common model selection procedure to determine the number of clusters in categorical sequence sets. A case study on commonly used protein sequences and the experimental results on some real-world sequence sets from different domains are given to demonstrate the performance of the proposed method.

Econometric Information Recovery in Behavioral Networks

In this paper, we suggest an approach to recovering behavior-related, preference-choice network information from observational data. We model the process as a self-organized behavior based random exponential network-graph system. To address the unknown nature of the sampling model in recovering behavior related network information, we use the Cressie-Read (CR) family of divergence measures and the corresponding information theoretic entropy basis, for estimation, inference, model evaluation, and prediction. Examples are included to clarify how entropy based information theoretic methods are directly applicable to recovering the behavioral network probabilities in this fundamentally underdetermined ill posed inverse recovery problem.

Temporal Predictability of Online Behavior in Foursquare

With the widespread use of Internet technologies, online behaviors play a more and more important role in humans’ daily lives. Knowing the times when humans perform their next online activities can be quite valuable for developing better online services, which prompts us to wonder whether the times of users’ next online activities are predictable. In this paper, we investigate the temporal predictability in human online activities through exploiting the dataset from the social network Foursquare. Through discretizing the inter-event times of users’ Foursquare activities into symbols, we map each user’s inter-event time sequence to a sequence of inter-event time symbols. By applying the information-theoretic method to the sequences of inter-event time symbols, we show that for a user’s Foursquare activities, knowing the time interval between the current activity and the previous activity decreases the entropy of the time interval between the next activity and current activity, i.e., the time of the user’s next Foursquare activity is predictable. Much of the predictability is explained by the equal-interval repeat; that is, users perform consecutive Foursquare activities with approximately equal time intervals. On the other hand, the unequal-interval preference, i.e., the preference of performing Foursquare activities with a fixed time interval after another given time interval, is also an origin for predictability. Furthermore, our results reveal that the Foursquare activities on weekdays have a higher temporal predictability than those on weekends and that users’ Foursquare activity is more temporally predictable if his/her previous activity is performed in a location that he/she visits more frequently.

Efficient solutions of interactive dynamic influence diagrams using model identification

Interactive dynamic influence diagram (I-DID) is one of the graphical frameworks for sequential decision making in partially observable environment. Subject agent in I-DID maintains beliefs over not only physical states of the environment, but also over models of the other agents. Consequently, solving I-DIDs suffers from the exponential growth of models ascribed to the other agents over time. Previous methods to solve I-DIDs aim at clustering equivalent models by comparing the entire or partial policy trees of the candidate models, which is time-consuming. In this paper, we present a new method for further reducing the model space by identifying the true model of the other agent and pruning the other irrelevant models. Toward this, we use an information-theoretic method—mutual information to measure the relevance between the candidate models and the true model in terms of predicted and observed actions of the other agent. We construct a dynamic Bayesian network to learn the value of parameters needed in the computation of mutual information. This approach bounds the model space by containing only the true model of the other agent. We evaluate our approach on multiple problem domains and empirically demonstrate the efficiency in solving I-DIDs.

Lifelong Information-Driven Exploration to Complete and Refine 4-D Spatio-Temporal Maps

This letter presents an exploration method that allows mobile robots to build and maintain spatio-temporal models of changing environments. The assumption of a perpetually changing world adds a temporal dimension to the exploration problem, making spatio-temporal exploration a never-ending, life-long learning process. We address the problem by application of information-theoretic exploration methods to spatio-temporal models that represent the uncertainty of environment states as probabilistic functions of time. This allows to predict the potential information gain to be obtained by observing a particular area at a given time, and consequently, to decide which locations to visit and the best times to go there. To validate the approach, a mobile robot was deployed continuously over 5 consecutive business days in a busy office environment. The results indicate that the robot’s ability to spot environmental changes improved as it refined its knowledge of the world dynamics.

A sharp adaptive confidence ball for self-similar functions

In the nonparametric Gaussian sequence space model an ℓ2-confidence ball Cn is constructed that adapts to unknown smoothness and Sobolev-norm of the infinite-dimensional parameter to be estimated. The confidence ball has exact and honest asymptotic coverage over appropriately defined ‘self-similar’ parameter spaces. It is shown by information-theoretic methods that this ‘self-similarity’ condition is weakest possible.

Aggregate entropy scoring for quantifying activity across endpoints with irregular correlation structure.

Robust computational approaches are needed to characterize systems-level responses to chemical perturbations in environmental and clinical toxicology applications. Appropriate characterization of response presents a methodological challenge when dealing with diverse phenotypic endpoints measured using in vivo systems. In this article, we propose an information-theoretic method named Aggregate Entropy (AggE) and apply it to scoring multiplexed, phenotypic endpoints measured in developing zebrafish (Danio rerio) across a broad concentration-response profile for a diverse set of 1060 chemicals. AggE accurately identified chemicals with significant morphological effects, including single-endpoint effects and multi-endpoint responses that would have been missed by univariate methods, while avoiding putative false-positives that confound traditional methods due to irregular correlation structure. By testing AggE in a variety of high-dimensional real and simulated datasets, we have characterized its performance and suggested implementation parameters that can guide its application across a wide range of experimental scenarios.

Hellinger volume and number-on-the-forehead communication complexity

Information-theoretic methods are a powerful tool in communication complexity, providing, in particular, elegant and tight lower bounds on disjointness in the number-in-the-hand (NIH) model. In this paper, we study the applicability of information theoretic methods to the multi-party number-on-the-forehead model (NOF). The lower bound on disjointness in the NIH model has two parts: a direct sum theorem and a lower bound on the one-bit AND function using a beautiful connection between Hellinger distance and protocols [4]. Inspired by this connection, we introduce the notion of Hellinger volume which lower bounds the information cost of multi-party NOF protocols. We provide tools for manipulating and proving lower bounds on Hellinger volume, and use these tools to obtain a lower bound on the information complexity of the ANDk function in the NOF setting. Finally, we discuss the difficulties of proving a direct sum theorem for information cost in the NOF model.

Information theoretic methods in small domain estimation

ABSTRACTSmall area estimation techniques are becoming increasingly used in survey applications to provide estimates for local areas of interest. The objective of this article is to develop and apply Information Theoretic (IT)-based formulations to estimate small area business and trade statistics. More specifically, we propose a Generalized Maximum Entropy (GME) approach to the problem of small area estimation that exploits auxiliary information relating to other known variables on the population and adjusts for consistency and additivity. The GME formulations, combining information from the sample together with out-of-sample aggregates of the population of interest, can be particularly useful in the context of small area estimation, for both direct and model-based estimators, since they do not require strong distributional assumptions on the disturbances. The performance of the proposed IT formulations is illustrated through real and simulated datasets.

Information Theoretical Methods as Discerning Quantifiers of the Equations of State of Neutron Stars

In this work we use the statistical measures of information entropy, disequilibrium and complexity to discriminate different approaches and parametrizations for different equations of state for quark stars. We confirm the usefulness of such quantities to quantify the role of interactions in such stars. We find that within this approach, a quark matter equation of state such as SU(2) NJL with vectorial coupling and phase transition is slightly favoured and deserves deeper studies.

Local Information Theoretic Methods for smooth Coefficients Dynamic Panel Data Models

This paper considers estimation and inference in semiparametric smooth coefficients dynamic panel data models. It proposes a class of local estimators that can be given an interesting information‐theoretic interpretation and a number of test statistics that can be used to test for the (local) correct specification of the model and for the constancy of the smooth coefficients. The results of the paper are rather general as they allow for the three cases of ‘large N, small T’, ‘small N, large T’ and ‘large N, large T’, for the possibility that some of the regressors might be correlated with the unobservable errors and for the possibility that some of the variables used in the estimation might not be directly observable. Simulations show that the proposed method have competitive finite sample properties.

An Informational Study of the Evolution of Codes and of Emerging Concepts in Populations of Agents.

We consider the problem of the evolution of a code within a structured population of agents. The agents try to maximize their information about their environment by acquiring information from the outputs of other agents in the population. A naive use of information-theoretic methods would assume that every agent knows how to interpret the information offered by other agents. However, this assumes that it knows which other agents it observes, and thus which code they use. In our model, however, we wish to preclude that: It is not clear which other agents an agent is observing, and the resulting usable information is therefore influenced by the universality of the code used and by which agents an agent is listening to. We further investigate whether an agent that does not directly perceive the environment can distinguish states by observing other agents' outputs. For this purpose, we consider a population of different types of agents talking about different concepts, and try to extract new ones by considering their outputs only.

Nonparametric Problem-Space Clustering: Learning Efficient Codes for Cognitive Control Tasks

We present an information-theoretic method permitting one to find structure in a problem space (here, in a spatial navigation domain) and cluster it in ways that are convenient to solve different classes of control problems, which include planning a path to a goal from a known or an unknown location, achieving multiple goals and exploring a novel environment. Our generative nonparametric approach, called the generative embedded Chinese restaurant process (geCRP), extends the family of Chinese restaurant process (CRP) models by introducing a parameterizable notion of distance (or kernel) between the states to be clustered together. By using different kernels, such as the the conditional probability or joint probability of two states, the same geCRP method clusters the environment in ways that are more sensitive to different control-related information, such as goal, sub-goal and path information. We perform a series of simulations in three scenarios—an open space, a grid world with four rooms and a maze having the same structure as the Hanoi Tower—in order to illustrate the characteristics of the different clusters (obtained using different kernels) and their relative benefits for solving planning and control problems.

Effects of Lead Exposure, Flock Behavior, and Management Actions on the Survival of California Condors (Gymnogyps californianus).

Translocation is an increasingly important tool for managing endangered species, but factors influencing the survival of translocated individuals are not well understood. Here we examine intrinsic and extrinsic drivers of survival for critically endangered California condors (Gymnogyps californianus) whose wild population recovery is reliant upon releases of captively bred stock. We used known fate models and information-theoretic methods to compare the ability of hypothesized covariates, most of which serve as proxies for lead exposure risk, to predict survival rates of condors in California. Our best supported model included the following predictors of survival: age of the recovery program, precipitation, proportion of days observed feeding on proffered carcasses, maximum blood lead concentration over the preceding 18months, and time since release. We found that as flocks have increased in size and age, condors are increasingly likely to range more widely and less likely to be observed feeding on proffered food, and these "wilder" behaviors were associated with lower survival. After accounting for these behaviors, we found a positive survival trend, which we attribute to ongoing improvements in management. Our findings illustrate that the survival of translocated animals, such as highly social California condors, is influenced by behaviors that change through time.

Information-Theoretic Active SOM for Improving Generalization Performance

In this paper, we introduce a new type of information-theoretic method called “information-theoretic active SOM”, based on the self-organizing maps (SOM) for training multi-layered neural networks. The SOM is one of the most important techniques in unsupervised learning. However, SOM knowledge is sometimes ambiguous and cannot be easily interpreted. Thus, we introduce the information-theoretic method to produce clearer and interpretable representations. The present method extends this information-theoretic approach into supervised learning. The main contribution can be summarized by three points. First, it is shown that clear representations by the information-theoretic method can be effective in training supervised learning. Second, the method is sufficiently simple where there are two separated components, namely, information maximization and error minimization component. Usually, two components are mixed in one framework, and it is difficult to compromise between them. In addition, the knowledge obtained by this information-theoretic SOM can be used to solve the shortage of unlabeled data, because the information maximization component is unsupervised and can process all input data with and without labels. The method was applied to the well-known image segmentation datasets. Experimental results showed that clear weights were produced and generalization performance was improved by using the information-theoretic SOM. In addition, the final results were stable, almost independent of the parameter values.

Soil phosphorus predicts feral pig (Sus scrofa) occupancy, detection probability and feeding activity in a temperate montane rainforest

Context Feral pigs (Sus scrofa) have a wide global distribution that includes large parts of Australia and New Zealand. There is concern about the impacts of feral pigs on above- and below-ground flora and fauna, but little is known about their habitat use and feeding activity in temperate rainforests. Aims We evaluated the importance of abiotic and biotic variables hypothesised to influence seasonal and annual feral pig habitat use and feeding activity in a montane conifer–angiosperm rainforest in Te Urewera, North Island, New Zealand. Methods We used a grid of 25 remote-camera locations to collect feral pig images in a 100-ha area during the winters and summers of 2010 and 2011. Plant composition, solar radiation and soil fertility variables were determined for each camera-image area. Multiseason, multistate occupancy models and information-theoretic methods were used to evaluate how these variables related to feral pig occupancy and feeding. Key results Feral pigs occupied more camera locations in summer than in winter, and detection probabilities increased if piglets were present and with increasing soil phosphorus (P). Piglets were detected only in summer, and their detection probability increased with increasing soil P. The probability of detecting feral pigs feeding also increased with soil P and was higher in 2010 than 2011. Conclusions Feral pigs selected locations with high soil P, probably because those sites had more food than did locations with low soil P. Mast fruiting of tawa (Beilschmiedia tawa) has been hypothesised to increase feral pig recruitment, and the higher detection probability of piglets in summer 2010 followed a heavier tawa fruit fall. Implications Our study highlighted the usefulness of camera traps and occupancy models for understanding seasonal and annual dynamics of cryptic ungulate species in remote, rugged forests, and suggests that the impacts of feral pigs will be greatest in areas of high soil P following widespread tawa masting.

A Transfer Entropy Method to Quantify Causality in Stochastic Nonlinear Systems

In modern chemical processes, identification of the process variable connectivity and their topology is vital for maintaining the operational safety. As a general information theoretic method, transfer entropy can analyze the causality between two variables based on estimation of conditional probability density functions. Transfer entropy estimation is typically a data driven method, however, the associated high computational complexity and poor accuracy are not acceptable in real applications. Using a nonlinear stochastic state-space model in conjunction with particle filters, a novel transfer entropy estimation method is proposed. The proposed approach requires less data, is fast and accurate.

Understanding Policy Diffusion in the U.S.: An Information-Theoretical Approach to Unveil Connectivity Structures in Slowly Evolving Complex Systems.

Detecting and explaining the relationships among interacting components has long been a focal point of dynamical systems research. In this paper, we extend these types of data-driven analyses to the realm of public policy, whereby individual legislative entities interact to produce changes in their legal and political environments. We focus on the U.S. public health policy landscape, whose complexity determines our capacity as a society to effectively tackle pressing health issues. It has long been thought that some U.S. states innovate and enact new policies, while others mimic successful or competing states. However, the extent to which states learn from others, and the state characteristics that lead two states to influence one another, are not fully understood. Here, we propose a model-free, information-theoretical method to measure the existence and direction of influence of one state's policy or legal activity on others. Specifically, we tailor a popular notion of causality to handle the slow time-scale of policy adoption dynamics and unravel relationships among states from their recent law enactment histories. The method is validated using surrogate data generated from a new stochastic model of policy activity. Through the analysis of real data in alcohol, driving safety, and impaired driving policy, we provide evidence for the role of geography, political ideology, risk factors, and demographic and economic indicators on a state's tendency to learn from others when shaping its approach to public health regulation. Our method offers a new model-free approach to uncover interactions and establish cause-and-effect in slowly-evolving complex dynamical systems.

Simplified Information Maximization for Improving Generalization Performance in Multilayered Neural Networks

A new type of information-theoretic method is proposed to improve prediction performance in supervised learning. The method has two main technical features. First, the complicated procedures used to increase information content are replaced by the direct use of hidden neuron outputs. Information is controlled by directly changing the outputs of the hidden neurons. In addition, to simultaneously increase information content and decrease errors between targets and outputs, the information acquisition and use phases are separated. In the information acquisition phase, the autoencoder tries to acquire as much information content on input patterns as possible. In the information use phase, information obtained in the acquisition phase is used to train supervised learning. The method is a simplified version of actual information maximization and directly deals with the outputs from neurons. The method was applied to the three data sets, namely, Iris, bankruptcy, and rebel participation data sets. Experimental results showed that the proposed simplified information acquisition method was effective in increasing the real information content. In addition, by using the information content, generalization performance was greatly improved.

The coupling analysis between stock market indices based on permutation measures

Many information-theoretic methods have been proposed for analyzing the coupling dependence between time series. And it is significant to quantify the correlation relationship between financial sequences since the financial market is a complex evolved dynamic system. Recently, we developed a new permutation-based entropy, called cross-permutation entropy (CPE), to detect the coupling structures between two synchronous time series. In this paper, we extend the CPE method to weighted cross-permutation entropy (WCPE), to address some of CPE’s limitations, mainly its inability to differentiate between distinct patterns of a certain motif and the sensitivity of patterns close to the noise floor. It shows more stable and reliable results than CPE does when applied it to spiky data and AR(1) processes. Besides, we adapt the CPE method to infer the complexity of short-length time series by freely changing the time delay, and test it with Gaussian random series and random walks. The modified method shows the advantages in reducing deviations of entropy estimation compared with the conventional one. Finally, the weighted cross-permutation entropy of eight important stock indices from the world financial markets is investigated, and some useful and interesting empirical results are obtained.