Information-theoretic Methods Research Articles

Detecting Undeclared-Leader-Follower Structure in Pedestrian Evacuation Using Transfer Entropy

In evacuation models, individual interactions are typically assumed to follow the certain rules which are decided using prior knowledge. Our proposed improvement in evacuation modelling is inspired by the real-life experience that during crowd movements, individuals usually place their visual focus on the movements of their neighbours, even in the absence of social affiliations. That is, undeclared leader–follower (ULF) structures tend to emerge in crowd motion. In this study, to clarify the mechanism underlying individual interactions in crowd motion, a force-based model integrated with the ULF structure (LFM) was developed. An information-theoretic and model-free method, transfer entropy (TE), was applied to measure the ULF structure in evacuation crowds. Movement information (e.g., velocity and acceleration) was used as the time series for computing TE. The results showed that the LFM provides more realistic trajectories than does the classical social force model. In addition, the leader–follower structures were found to change over time, and an individual could act as a leader and a follower simultaneously during evacuation. ULF behaviour is strong at the early stage of evacuation and becomes weak when evacuees’ differences in movement states diminish. Moreover, the density map of leaders’ distributions presented a ‘V-like’ formation. Recognising the leader–follower relationship is central to understanding the mechanisms underlying the emergence of collective behaviour. The proposed approach is expected to aid in simulating realistic local interactions among evacuees and in providing a data-driven perspective for understanding crowd evacuation.

Fuzzy information-theoretic feature selection via relevance, redundancy, and complementarity criteria

The concepts of feature relevance, redundancy, and complementarity are very important for identifying optimal feature subsets and designing effective feature evaluation criteria when developing information-theoretic feature selection methods. However, the aforementioned concepts are generally defined based on classical information-theoretic measures that work with discrete features. Therefore, it is very difficult to directly apply them to handle continuous features. In this paper, we investigate the concepts of feature relevance, redundancy, and complementarity based on fuzzy information-theoretic measures to address continuous features. More specifically, we examine some fuzzy information-theoretic measures for any finite number of fuzzy T-equivalence relations. From a conceptual point of view, we present theoretical definitions of feature relevance, redundancy, and complementarity by using these fuzzy information-theoretic measures. According to these theoretical definitions, we introduce a computationally effective feature evaluation criterion that employs a weighting scheme to combine feature relevance, redundancy, and complementarity. We propose a feature selection algorithm by combining the feature evaluation criterion with the sequential forward search strategy. To verify the effectiveness of our method, we compare it with some state-of-the-art feature selection methods through extensive experiments. The experimental results show that our method achieves better performance.

Information Theoretic Methods for Variable Selection-A Review.

We review the principal information theoretic tools and their use for feature selection, with the main emphasis on classification problems with discrete features. Since it is known that empirical versions of conditional mutual information perform poorly for high-dimensional problems, we focus on various ways of constructing its counterparts and the properties and limitations of such methods. We present a unified way of constructing such measures based on truncation, or truncation and weighing, for the Möbius expansion of conditional mutual information. We also discuss the main approaches to feature selection which apply the introduced measures of conditional dependence, together with the ways of assessing the quality of the obtained vector of predictors. This involves discussion of recent results on asymptotic distributions of empirical counterparts of criteria, as well as advances in resampling.

Cost-forced and repeated selective information minimization and maximization for multi-layered neural networks1

The present paper aims to propose a new information-theoretic method to minimize and maximize selective information repeatedly. In particular, we try to solve the incomplete information control problem, where information cannot be fully controlled due to the existence of many contradictory factors inside. For this problem, the cost in terms of the sum of absolute connection weights is introduced for neural networks to increase and decrease information against contradictory forces in learning, such as error minimization. Thus, this method is called a “cost-forced” approach to control information. The method is contrary to the conventional regularization approach, where the cost has been used passively or negatively. The present method tries to use the cost positively, meaning that the cost can be augmented if necessary. The method was applied to an artificial and symmetric data set. In the symmetric data set, we tried to show that the symmetric property of the data set could be obtained by appropriately controlling information. In the second data set, that of residents in a nursing home, obtained by the complicated procedures of natural language processing, the experimental results confirmed that the present method could control selective information to extract non-linear relations as well as linear ones in increasing interpretation and generalization performance.

The effect of combat exposure on sexually transmitted diseases

Traumatic exposures can affect beliefs and behaviors related to the spread of sexually transmitted diseases (STDs), a persistent public health problem. I leverage a natural experiment created by variation in US military deployment location assignments to estimate how combat exposure changes a surviving deployed male veteran’s probability of acquiring a sexually transmitted disease. I analyze longitudinal data from 1994 to 2008 on 485 deployed veterans with information theoretic methods to reduce the sensitivity of estimates to small samples, an infrequently observed outcome, and highly correlated covariates. For veterans assigned to a combat zone, I estimate combat exposure results in a 5.4 percentage point increase in the probability of acquiring an STD. Additional estimations provide evidence suggesting risky behaviors involving substance use or multiple sexual partners may serve as pathways from combat exposure to STDs. My results are relevant to discussions regarding STD screening and care needs for trauma exposed individuals.

Objective crystallographic symmetry classifications of a noisy crystal pattern with strong Fedorov-type pseudosymmetries and its optimal image-quality enhancement.

Statistically sound crystallographic symmetry classifications are obtained with information-theory-based methods in the presence of approximately Gaussian distributed noise. A set of three synthetic patterns with strong Fedorov-type pseudosymmetries and varying amounts of noise serve as examples. Contrary to traditional crystallographic symmetry classifications with an image processing program such as CRISP, the classification process does not need to be supervised by a human being and is free of any subjectively set thresholds in the geometric model selection process. This enables crystallographic symmetry classification of digital images that are more or less periodic in two dimensions (2D), also known as crystal patterns, as recorded with sufficient structural resolution from a wide range of crystalline samples with different types of scanning probe and transmission electron microscopes. Correct symmetry classifications enable the optimal crystallographic processing of such images. That processing consists of the averaging over all asymmetric units in all unit cells in the selected image area and significantly enhances both the signal-to-noise ratio and the structural resolution of a microscopic study of a crystal. For sufficiently complex crystal patterns, the information-theoretic symmetry classification methods are more accurate than both visual classifications by human experts and the recommendations of one of the popular crystallographic image processing programs of electron crystallography.

Response of grassland soil respiration to experimental warming: The long-term effects may be greater than we thought

Soil respiration (Rs) profoundly affects the global carbon cycle, and its response to climate warming in grasslands shows significant heterogeneity for poorly understood reasons. A meta-analysis was undertaken to evaluate the effects of warming (including the magnitude, methods, and duration of warming) and environmental factors (including climate, soils, and plants) on the response of Rs to warming in grasslands, using a global dataset of 168 independent measurements. Multi-model inference, an information-theoretical method for synthesizing results of multiple alternative models, was used to quantify the relative importance of the environmental factors. We found that the response of Rs to warming followed a three-phase pattern: Rs increased initially, then remained unchanged, and then increased again as the duration of warming increased. The long-term response was greater than the short-term response. The Rs response was also affected by the method of warming, with open-top chambers promoting Rs more than infrared heaters. The response of Rs to warming was also positively related to the magnitude of warming. Rs was greatly increased with warming in temperate grasslands compared with that in cold grasslands but did not change significantly in arid grasslands. According to multi-model inference, this can be attributable to differences in soil nutrients and the mean annual precipitation. Our findings suggest a greater long-term stimulation of Rs in a warming world than we previously thought, so short-term studies cannot provide a reliable scientific basis for carbon projections under long-term climate warming. Our results also demonstrate that the carbon released in cold grasslands could be overestimated if the water deficiency was not considered.

Decomposed POMDP Optimization-Based Sensor Management for Multi-Target Tracking in Passive Multi-Sensor Systems

This paper presents an efficient information-theoretic sensor management method to maximize the performance of the passive multi-sensor system for multi-target tracking. We model the multi-target state as a generalized labeled multi-Bernoulli (GLMB) random finite set and formulate the dynamic sensor selection process as a partially observable Markov decision process (POMDP). The optimization objective is to maximum the information gain obtained from the observed data, which is measured by the Cauchy-Schwarz divergence. This is accomplished with two main technical innovations. The first is a tractable decomposed POMDP based sensor selection solution, in which the informative sensors are selected sequentially from the candidates based on the Cauchy-Schwarz divergence. The second is a novel dual-stage multi-sensor fusion strategy based on the iterated-corrector GLMB filter. Since the uncertainty of the passive multi-sensor system is generally large, the performance of the iterated-corrector scheme can be greatly influenced by the order of sensor updates. To fix this problem, the selected sensors are ranked in order of the Cauchy-Schwarz divergence obtained in sensor selection, followed by the iterated-corrector update. For the multi-sensor fusion, the effect of poor performance sensors is weakened and that of better performance sensors is enhanced. Simulation studies demonstrate the effectiveness and efficiency of the proposed method in challenging passive multi-target tracking scenarios.

Cost-forced collective potentiality maximization by complementary potentiality minimization for interpreting multi-layered neural networks

The present paper aims to propose a new type of information-theoretic method to augment the ratio of collective potentiality to its cost for the interpretation. Collective potentiality represents how many components, such as connection weights or neurons, are collectively used to deal with relations between inputs and outputs, while individual potentiality shows how an individual component potentially contributes to the inputs or outputs. From our viewpoint, the conventional interpretation methods have focused on the individual potentiality of components, ignoring these collective behaviors. Thus, the present paper stresses that neural networks should try to deal with the collective properties of components and to examine the final results from multiple points of view. For implementing this concept of potentiality maximization, we introduce the complementary potentiality minimization, which aims to reduce the strength of larger weights as much as possible, and at the same time, it can be used for increasing the potentiality. The method was applied to two intuitively interpretable data sets, namely, the absenteeism and online shoppers data sets. With both data sets, experimental results confirmed that the method could increase collective potentiality and considerably reduce the corresponding cost. The new method could extract many groups of weights with the same small strength, all of which tried to respond to the coming inputs, while the conventional methods, including the regularization ones, tried to produce a smaller number of stronger connection weights very selectively. In particular, the present method could produce stable compressed weights similar to the original correlation coefficients of the data sets, meaning that simple, independent, and linear relations could be detected, contrary to the presupposed non-linear ones. Thus, the collective interpretation, applied to two business data sets, revealed a possibility that, behind seemingly complicated non-linear relations between inputs and outputs, neural networks with collective potentiality augmentation with a smaller cost can extract the very simple relations for easy interpretation. All complicated relations can possibly be generated, based on those simple ones.

Simulation of spontaneous leader–follower behaviour in crowd evacuation

The leader–follower (L-F) relationship is believed to be the fundamental mechanism explaining the collective behaviour of a crowd. However, previous models are inherently limited in that they simulate a pre-defined L-F behaviour using simple heuristics. In this paper, we propose a new approach to measure the emergence of spontaneous L-F behaviour in an evacuating crowd using an information-theoretic and model-free method, i.e., transfer entropy (TE). The dynamic characteristics of evacuees (e.g., their trajectories, movement patterns and density plot) in a room (or corridor) are clearly visualised by the proposed TE-incorporating force-based model. The results show that the overall evacuation time decreases as the TE threshold increases. The trajectories occur in congested clusters when the TE threshold is low, but become more dispersed and uniformly distributed in scenarios with higher TE thresholds. In terms of group size, a spontaneous L-F group normally consists of one leader and one follower. L-F groups of larger-size (i.e., more than three followers of a leader) tend to disappear with increasing TE threshold. Lastly, we examine the location of spontaneous leadership emergence. Irrespective of the geometric layout or the number of evacuees in a room or corridor, the threshold at which distinct L-F patterns emerge is always approximately I¯TE = 0.5. By innovatively integrating the TE approach into a force-based model, this study suggests a novel method for leadership identification during crowd evacuation given no prior knowledge.

Application of information theoretic feature selection and machine learning methods for the development of genetic risk prediction models

In view of the growth of clinical risk prediction models using genetic data, there is an increasing need for studies that use appropriate methods to select the optimum number of features from a large number of genetic variants with a high degree of redundancy between features due to linkage disequilibrium (LD). Filter feature selection methods based on information theoretic criteria, are well suited to this challenge and will identify a subset of the original variables that should result in more accurate prediction. However, data collected from cohort studies are often high-dimensional genetic data with potential confounders presenting challenges to feature selection and risk prediction machine learning models. Patients with psoriasis are at high risk of developing a chronic arthritis known as psoriatic arthritis (PsA). The prevalence of PsA in this patient group can be up to 30% and the identification of high risk patients represents an important clinical research which would allow early intervention and a reduction of disability. This also provides us with an ideal scenario for the development of clinical risk prediction models and an opportunity to explore the application of information theoretic criteria methods. In this study, we developed the feature selection and psoriatic arthritis (PsA) risk prediction models that were applied to a cross-sectional genetic dataset of 1462 PsA cases and 1132 cutaneous-only psoriasis (PsC) cases using 2-digit HLA alleles imputed using the SNP2HLA algorithm. We also developed stratification method to mitigate the impact of potential confounder features and illustrate that confounding features impact the feature selection. The mitigated dataset was used in training of seven supervised algorithms. 80% of data was randomly used for training of seven supervised machine learning methods using stratified nested cross validation and 20% was selected randomly as a holdout set for internal validation. The risk prediction models were then further validated in UK Biobank dataset containing data on 1187 participants and a set of features overlapping with the training dataset.Performance of these methods has been evaluated using the area under the curve (AUC), accuracy, precision, recall, F1 score and decision curve analysis(net benefit). The best model is selected based on three criteria: the ‘lowest number of feature subset’ with the ‘maximal average AUC over the nested cross validation’ and good generalisability to the UK Biobank dataset. In the original dataset, with over 100 different bootstraps and seven feature selection (FS) methods, HLA_C_*06 was selected as the most informative genetic variant. When the dataset is mitigated the single most important genetic features based on rank was identified as HLA_B_*27 by the seven different feature selection methods, consistent with previous analyses of this data using regression based methods. However, the predictive accuracy of these single features in post mitigation was found to be moderate (AUC= 0.54 (internal cross validation), AUC=0.53 (internal hold out set), AUC=0.55(external data set)). Sequentially adding additional HLA features based on rank improved the performance of the Random Forest classification model where 20 2-digit features selected by Interaction Capping (ICAP) demonstrated (AUC= 0.61 (internal cross validation), AUC=0.57 (internal hold out set), AUC=0.58 (external dataset)). The stratification method for mitigation of confounding features and filter information theoretic feature selection can be applied to a high dimensional dataset with the potential confounders.

Device Authentication Codes based on RF Fingerprinting using Deep Learning

In this paper, we propose Device Authentication Code (DAC), a novel method for authenticating IoT devices with wireless interface, by exploiting their radio frequency (RF) signatures. The proposed DAC is based on RF fingerprinting, an information-theoretic method, feature learning, and the discrimi

Biomarker Identification From Gene Expression Based on Symmetrical Uncertainty

In this paper, the authors present an effective information theoretic feature selection method, symmetrical uncertainty, to classify gene expression microarray data and detect biomarkers from it. Here, information gain and symmetrical uncertainty contribute for ranking the features. Based on computed values of symmetrical uncertainty, features were sorted from most informative to least informative ones. Then, the top features from the sorted list are passed to random forest, logistic regression, and other well-known classifiers with leave-one-out cross validation to construct the best classification model(s) and accordingly select the most important genes from microarray datasets. Obtained results in terms of classification accuracy, running time, root mean square error, and other parameters computed on leukemia and colon cancer datasets demonstrate the effectiveness of the proposed approach. The proposed method is relatively much faster than many other wrapper or ensemble methods.

Invariance Properties of Controlled Stochastic Nonlinear Systems Under Information Constraints

Given a stochastic nonlinear system controlled over a possibly noisy communication channel, the article studies the largest class of channels for which there exist coding and control policies so that the closed-loop system is stochastically stable. The stability criterion considered is asymptotic mean stationarity (AMS). In this article, we develop a general method based on ergodic theory and probability to derive fundamental bounds on information transmission requirements leading to stabilization. Through this method, we develop a new notion of entropy which is tailored to derive lower bounds for AMS for both noise-free and noisy channels. The bounds obtained through probabilistic and ergodic-theoretic analysis are more refined in comparison with the bounds obtained earlier via information-theoretic methods. Moreover, our approach is more versatile in view of the models considered and allows for finer lower bounds when the AMS measure is known to admit further properties, such as moment bounds.

Unifying information theory and machine learning in a model of electrode discrimination in cochlear implants.

Despite the development and success of cochlear implants over several decades, wide inter-subject variability in speech perception is reported. This suggests that cochlear implant user-dependent factors limit speech perception at the individual level. Clinical studies have demonstrated the importance of the number, placement, and insertion depths of electrodes on speech recognition abilities. However, these do not account for all inter-subject variability and to what extent these factors affect speech recognition abilities has not been studied. In this paper, an information theoretic method and machine learning technique are unified in a model to investigate the extent to which key factors limit cochlear implant electrode discrimination. The framework uses a neural network classifier to predict which electrode is stimulated for a given simulated activation pattern of the auditory nerve, and mutual information is then estimated between the actual stimulated electrode and predicted ones. We also investigate how and to what extent the choices of parameters affect the performance of the model. The advantages of this framework include i) electrode discrimination ability is quantified using information theory, ii) it provides a flexible framework that may be used to investigate the key factors that limit the performance of cochlear implant users, and iii) it provides insights for future modeling studies of other types of neural prostheses.

Optimal Allocation for Electric Vehicle Charging Stations

Currently, more than 30% of the fine dust generated in the Seoul metropolitan area is a pollutant emitted from automobiles such as diesel vehicles, and air pollution caused by this is becoming increasingly serious. In addition, the importance of electric vehicle distribution is increasing due to the strengthening of international environmental regulations on automobile exhaust gas and increasing the possibility of depletion of petroleum resources. This manuscript proposes a method for selecting an optimal electric vehicle charging station location in expanding charging facilities to activate electric vehicle distribution. For the sake of illustration, directions will be provided on how to select the best location for electric vehicle charging stations using data from Seoul, which has the best access. As the features, the number of living population and work force people and the number of guest facilities, which are determined to affect demand for quick charging, are considered. The missing values of the observed data are imputed based on the kriging technique from spatial correlation, and by segmenting the data through clustering, a representative technique of unsupervised learning, the characteristics of each cluster are examined and the characteristics of the clusters are identified. In addition, machine learning techniques such as the elastic net, random forest, support vector machine, and extreme gradient boosting are applied to examine the influence of the features used in predicting classes of data. In clustering analysis, the optimal number of clusters was determined to be 3 based on the heuristic and information-theoretic methods, and all the machine learning techniques considered showed that the number of work force population is the most important feature in predicting classes of data. All things considered from our results, it is reasonable to install quick electric vehicle charging stations in the places with the highest concentration of work force population and guest facility.

Information Bottleneck as Optimisation Method for SSVEP-Based BCI.

In this study, the information bottleneck method is proposed as an optimisation method for steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI). The information bottleneck is an information-theoretic optimisation method for solving problems with a trade-off between preserving meaningful information and compression. Its main practical application in machine learning is in representation learning or feature extraction. In this study, we use the information bottleneck to find optimal classification rule for a BCI. This is a novel application for the information bottleneck. This approach is particularly suitable for BCIs since the information bottleneck optimises the amount of information transferred by the BCI. Steady-state visual evoked potential-based BCIs often classify targets using very simple rules like choosing the class corresponding to the largest feature value. We call this classifier the arg max classifier. It is unlikely that this approach is optimal, and in this study, we propose a classification method specifically designed to optimise the performance measure of BCIs. This approach gives an advantage over standard machine learning methods, which aim to optimise different measures. The performance of the proposed algorithm is tested on two publicly available datasets in offline experiments. We use the standard power spectral density analysis (PSDA) and canonical correlation analysis (CCA) feature extraction methods on one dataset and show that the current approach outperforms most of the related studies on this dataset. On the second dataset, we use the task-related component analysis (TRCA) method and demonstrate that the proposed method outperforms the standard argmax classification rule in terms of information transfer rate when using a small number of classes. To our knowledge, this is the first time the information bottleneck is used in the context of SSVEP-based BCIs. The approach is unique in the sense that optimisation is done over the space of classification functions. It potentially improves the performance of BCIs and makes it easier to calibrate the system for different subjects.

Cross-Validation, Information Theory, or Maximum Likelihood? A Comparison of Tuning Methods for Penalized Splines

Functional data analysis techniques, such as penalized splines, have become common tools used in a variety of applied research settings. Penalized spline estimators are frequently used in applied research to estimate unknown functions from noisy data. The success of these estimators depends on choosing a tuning parameter that provides the correct balance between fitting and smoothing the data. Several different smoothing parameter selection methods have been proposed for choosing a reasonable tuning parameter. The proposed methods generally fall into one of three categories: cross-validation methods, information theoretic methods, or maximum likelihood methods. Despite the well-known importance of selecting an ideal smoothing parameter, there is little agreement in the literature regarding which method(s) should be considered when analyzing real data. In this paper, we address this issue by exploring the practical performance of six popular tuning methods under a variety of simulated and real data situations. Our results reveal that maximum likelihood methods outperform the popular cross-validation methods in most situations—especially in the presence of correlated errors. Furthermore, our results reveal that the maximum likelihood methods perform well even when the errors are non-Gaussian and/or heteroscedastic. For real data applications, we recommend comparing results using cross-validation and maximum likelihood tuning methods, given that these methods tend to perform similarly (differently) when the model is correctly (incorrectly) specified.

Endogenous Technology Space

We use probabilistic machine learning to span a new endogenous technology space from patent texts. We then rely on information-theoretic methods to construct measures of technological firm distances -- both fixed and time-varying. Using the latter, we present three sets of findings. First, we observe that industries are becoming more technologically specialised and segregated over time. Second, we identify the emergence of internet companies in the mid-1990s as a distinct group of firms with roots in traditional information and communication technologies. Third, we determine the unique set of time-varying rivals surrounding a focal firm in the endogenous technology space. We demonstrate the validity of this approach by means of a case study of the software company Oracle.

Recovering Vote Choice from Partial Incomplete Data

In voting rights cases, judges often infer unobservable individ ual vote choices from election data aggregated at the precinct level. That is, one must solve an ill-posed inverse problem to obtain the critical information used in these cases. The ill-posed nature of the problem means that tradi tional frequentist and Bayesian approaches cannot be employed without first imposing a range of assumptions. In order to mitigate the problems result ing from incorporating potentially inaccurate information in these cases, we propose the use of information theoretic methods as a basis for recovering an estimate of the unobservable individual vote choices. We illustrate the empirical non-parametric likelihood methods with some election data.