Information-theoretic Measures Research Articles

Renyi entropy analysis of a deep convolutional representation for texture recognition

Despite the recent success of convolutional neural networks in computer vision in general, texture images still pose an important challenge to those models, especially when dealing with textures “in the wild”. In this context, deep learning models can benefit from the introduction of features long known to be useful for texture modeling. And that is the case of entropy, a measure of texture regularity that has played an important role in classical computer vision. Based on this observation, here we propose an alternative analysis over deep convolutional neural features based on entropy for texture representation and, particularly, texture classification. More precisely, we couple a module to the convolutional backbone that locally computes the Renyi entropy of the latent representation at multiple levels. The rationale for using Renyi entropy is essentially two-fold: (1) It connects the concept of entropy with multifractal theory, another well explored measure especially in domains of application where we seek some physical interpretation of the descriptors, e.g., in medicine, biology, and others; (2) It has an extra degree of freedom (α parameter) that can be fine-tuned. The main contribution of this study is the development of a strategy that can improve the performance of convolutional neural networks in texture recognition tasks, adding low computational cost. The effectiveness of our method is verified in texture classification of benchmark datasets, as well as in a practical task of plant species identification. Our method achieves a competitive accuracy of 84.5% in the classification of KTH-TIPS-2b and 80.9% in FMD, which are two challenging benchmark databases. Besides, an accuracy of 91.6% is obtained in the plant identification application, to our knowledge outperforming all the results previously published on this task. In both scenarios, the proposed descriptors outperform several approaches from the state-of-the-art, confirming the method potential as a rich and robust solution for texture analysis in general. The results also suggest that information-theoretical measures like entropy can be a reliable source of information to compose a precise and robust latent representation of texture images.

Phi fluctuates with surprisal: An empirical pre-study for the synthesis of the free energy principle and integrated information theory.

The Free Energy Principle (FEP) and Integrated Information Theory (IIT) are two ambitious theoretical approaches. The first aims to make a formal framework for describing self-organizing and life-like systems in general, and the second attempts a mathematical theory of conscious experience based on the intrinsic properties of a system. They are each concerned with complementary aspects of the properties of systems, one with life and behavior, the other with meaning and experience, so combining them has potential for scientific value. In this paper, we take a first step towards such a synthesis by expanding on the results of an earlier published evolutionary simulation study, which show a relationship between IIT-measures and fitness in differing complexities of tasks. We relate a basic information theoretic measure from the FEP, surprisal, to this result, finding that the surprisal of simulated agents' observations is inversely related to the general increase in fitness and integration over evolutionary time. Moreover, surprisal fluctuates together with IIT-based consciousness measures in within-trial time. This suggests that the consciousness measures used in IIT indirectly depend on the relation between the agent and the external world, and that it should therefore be possible to relate them to the theoretical concepts used in the FEP. Lastly, we suggest a future approach for investigating this relationship empirically.

Impact of observation thresholds in the assessment of field-scale soil saturated hydraulic conductivity

The soil saturated hydraulic conductivity (Ks) plays a key role in the partitioning of rainfall into runoff and infiltration. Field-scale variability of Ks is often represented as a lognormal random field, and its parameters are frequently estimated by calibrating areal-average infiltration models with infiltration observations obtained under natural/artificial rainfall conditions. Such observations are obtained in terms of censored moments as infiltration is controlled by only a fraction of the Ks random field during a rainfall event. In this study, the impact of such observations on the maximum likelihood estimates (MLE) of the Ks distribution parameters is evaluated. Based on the data from several rainfall-runoff events observed under natural conditions over an experimental field, the results demonstrate the role of temporal variation of rainfall in resolving the Ks field for a rainfall event. Validation results show that the performance of the MLEs is impacted by both the fraction of the Ks field resolved by a rainfall event and its capacity to capture the variability of Ks. Finally, recommendations are provided as to what constitutes a good calibration event. The impact of incorporating reliability of the calibration events in consolidating information was tested using an information-theoretic measure.

Global and local information-theoretic measures of the inversely quadratic Hellmann–Kratzer potential

The information-theoretic measures of the inversely quadratic Hellmann-Kratzer (IQHK) potential are reported in this paper. The Dirac equation is solved via the generalized form of the Nikiforov-Uvarov (NU) method, and a non-relativistic mapping is used to derive the non-relativistic energy and normalized wave function of the IQHK potential. The Shannon entropy, Onicescu information energy, Fisher information, and uncertainty relations for the IQHK potential are presented for two low-lying states (n = 0, 1). The results of the study show that the probability density for the n = 0 state is more localized than the probability density for the n = 1 state. The Beckner-Bialynicki-Birula-Mycielsky (BBM) inequality for the total Shannon entropy, the Stam-Cramer-Rao inequality for the Fisher information product, and the Heisenberg uncertainty relation are satisfied by the system. Finally, some of the states experience squeezing phenomena in position space.

Information-Theoretic Approaches to Differential Privacy

This tutorial studies relations between differential privacy and various information-theoretic measures by using several selective articles. In particular, we present how these connections can provide new interpretations for the privacy guarantee in systems that deploy differential privacy in an information-theoretic framework. Accordingly, the tutorial delivers an extensive summary on the existing literature that makes use of information-theoretic measures and tools such as mutual information, min-entropy, Kullback-Leibler divergence, and rate-distortion function for quantification and characterization of differential privacy in various settings.

Quantifying embodiment towards building better robots based on muscle-driven models

In recent years, researchers have investigated different methods to quantify embodiment for a variety of robotic systems including robotic arms, grippers and legged robots. This paper will discuss some of these methods, focusing on their potential contribution to designing robotic systems based on muscle-driven models. We start with the definition of embodiment based on the relational dynamics between the system and its environments by drawing upon the idea of mutual perturbation and structural coupling between the two. We will discuss how such an understanding can provide potential approaches to quantify embodiment. These includes two information-theoretic measures which are particularly suitable for muscle-driven models. The two methods are based on (i) comparing the controller and behaviour complexity and (ii) Conditional Mutual Information, which compares the difference in distribution of the action conditional on the actuated state and purely on its morphological properties. These methods were used on muscle-driven, biologically realistic hopping models to quantify embodiment at different stages of the hopping gait. The results clearly demonstrate the contribution of morphology of the muscle fibers at different points in the hopping cycle. Furthermore, these methods have been used in latter studies to measure the contribution of embodiment across different levels in a hierarchical control system of a neuro-musculoskeletal model and also to quantify the effects of information cost during various actions in a muscle-driven robotic system. We discuss the practical implications as well as limitations and the future work in the application of these quantification methods.

Mathematical Characterization of Private and Public Immune Receptor Sequences

Diverse T and B cell repertoires play an important role in mounting effective immune responses against a wide range of pathogens and malignant cells. The number of unique T and B cell clones is characterized by T and B cell receptors (TCRs and BCRs), respectively. Although receptor sequences are generated probabilistically by recombination processes, clinical studies found a high degree of sharing of TCRs and BCRs among different individuals. In this work, we use a general probabilistic model for T/B cell receptor clone abundances to define “publicness” or “privateness” and information-theoretic measures for comparing the frequency of sampled sequences observed across different individuals. We derive mathematical formulae to quantify the mean and the variances of clone richness and overlap. Our results can be used to evaluate the effect of different sampling protocols on abundances of clones within an individual as well as the commonality of clones across individuals. Using synthetic and empirical TCR amino acid sequence data, we perform simulations to study expected clonal commonalities across multiple individuals. Based on our formulae, we compare these simulated results with the analytically predicted mean and variances of the repertoire overlap. Complementing the results on simulated repertoires, we derive explicit expressions for the richness and its uncertainty for specific, single-parameter truncated power-law probability distributions. Finally, the information loss associated with grouping together certain receptor sequences, as is done in spectratyping, is also evaluated. Our approach can be, in principle, applied under more general and mechanistically realistic clone generation models.

Information theory and thermodynamic properties of diatomic molecules using molecular potential.

Owing to the devise applications of molecules in industries, the bound state solution of the non-relativistic wave equation with a molecular potential function has been obtained in a closed-form using the Nikiforov-Uvarov method. The solutions of the bound state are then applied to study the information-theoretic measures such as the one-dimensional Shannon and Renyi entropic densities. The expectation values for the position and momentum spaces were obtained to verify the Heisenberg's uncertainty principle. Utilizing the energy spectrum equation, the thermodynamic vibrational partition function is obtained via the Poisson summation. Other thermodynamic function variations with absolute temperature have been obtained numerically for four diatomic molecules (H2, N2, O2, and HF) using Maple 18 software. The Shannon global entropic sum inequality has also been verified. The Renyi sum for constrained index parameters satisfies the global entropic inequality. The thermodynamic properties of the four molecules are similar and conform to works reported in the existing literature. The obtained vibrational energies are in fair agreement with the ones obtained using other forms of potential energy. The result further indicates that the lowest bounds for the Shannon, Renyi, and Heisenberg inequalities are ground states phenomena.

Capturing information in extreme events

This study integrates information theory and extreme value theory to enhance the prediction of extreme events. Information-theoretic measures provide a foundation for model comparison in tails. The theoretical findings suggest that (1) the entropy of block maxima converges to the entropy of the generalized extreme value distribution, (2) the rate of convergence is controlled by its shape parameter, and (3) the entropy of block maxima is a monotonically decreasing function of the block size. Empirical analysis of E-mini S&P, 500 futures data evaluates the financial risk, capturing information content of extreme events using entropy and Kullback–Leibler divergence.

View recommendation for multi-camera demonstration-based training

While humans can effortlessly pick a view from multiple streams, automatically choosing the best view is a challenge. Choosing the best view from multi-camera streams poses a problem regarding which objective metrics should be considered. Existing works on view selection lack consensus about which metrics should be considered to select the best view. The literature on view selection describes diverse possible metrics. And strategies such as information-theoretic, instructional design, or aesthetics-motivated fail to incorporate all approaches. In this work, we postulate a strategy incorporating information-theoretic and instructional design-based objective metrics to select the best view from a set of views. Traditionally, information-theoretic measures have been used to find the goodness of a view, such as in 3D rendering. We adapted a similar measure known as the viewpoint entropy for real-world 2D images. Additionally, we incorporated similarity penalization to get a more accurate measure of the entropy of a view, which is one of the metrics for the best view selection. Since the choice of the best view is domain-dependent, we chose demonstration-based training scenarios as our use case. The limitation of our chosen scenarios is that they do not include collaborative training and solely feature a single trainer. To incorporate instructional design considerations, we included the trainer’s body pose, face, face when instructing, and hands visibility as metrics. To incorporate domain knowledge we included predetermined regions’ visibility as another metric. All of those metrics are taken into account to produce a parameterized view recommendation approach for demonstration-based training. An online study using recorded multi-camera video streams from a simulation environment was used to validate those metrics. Furthermore, the responses from the online study were used to optimize the view recommendation performance with a normalized discounted cumulative gain (NDCG) value of 0.912, which shows good performance with respect to matching user choices.

Regularized Estimation of Information via Canonical Correlation Analysis on a Finite-Dimensional Feature Space

This paper aims to estimate the information between two random phenomena by using consolidated second-order statistics tools. The squared-loss mutual information, a surrogate of the Shannon mutual information, is chosen due to its property of being expressed as a second-order moment. We first review the rationale for i.i.d. discrete sources, which involves mapping the data onto the simplex space, and we highlight the links with other well-known related concepts in the literature based on local approximations of information-theoretic measures. Then, the problem is translated to analog sources by mapping the data onto the characteristic space, focusing on the adaptability between the discrete and the analog case and its limitations. The proposed approach gains interpretability and scalability for its use on large data sets, providing a unified rationale for the free regularization parameters. Moreover, the structure of the proposed mapping allows resorting to Szegö’s theorem to reduce the complexity for high dimensional mappings, exhibiting a strong duality with spectral analysis. The performance of the developed estimators is analyzed using Gaussian mixtures.

Decomposing the Mutual Information Rate of Heart Period and Respiration Variability Series to Assess Cardiorespiratory Interactions.

Heart rate variability results from the coupled activity of the cardiovascular and cardiorespiratory systems, which have their own internal regulation mechanisms but also interact with each other and with the autonomic nervous system to maintain homeostasis. In this work, the assessment of these physiological mechanisms is carried out decomposing the Mutual Information Rate (MIR), an information-theoretic measure of the interdependence between coupled processes, into terms of entropy rate or conditional mutual information related respectively to complexity and causality measures. These measures are computed using a non-parametric approach based on nearest-neighbors. The proposed framework is first tested on simulated autoregressive processes and then applied to experimental data consisting of heart period and respiratory time series measured in healthy subjects monitored at rest and during head-up tilt. Our results evidence that MIR decomposition is able to highlight the interdependence of short-term physiological mechanisms of cardiorespiratory interactions during postural stress.

Exploring the Short-Term Memory of Heart Rate Variability through Model-Free Information Measures.

In this work, we perform a comparative analysis of discrete- and continuous-time estimators of information-theoretic measures quantifying the concept of memory utilization in short-term heart rate variability (HRV). Specifically, considering heartbeat intervals in discrete time we compute the measure of information storage (IS) and decompose it into immediate memory utilization (IMU) and longer memory utilization (MU) terms; considering the timings of heartbeats in continuous time we compute the measure of MU rate (MUR). All measures are computed through model-free approaches based on nearest neighbor entropy estimators applied to the HRV series of a group of 15 healthy subjects measured at rest and during postural stress. We find, moving from rest to stress, statistically significant increases of the IS and the IMU, as well as of the MUR. Our results suggest that both discrete-time and continuous-time approaches can detect the higher predictive capacity of HRV occurring with postural stress, and that such increased memory utilization is due to fast mechanisms likely related to sympathetic activation.

Stability of a two-electron system under pressure confinement: structural and quantum information theoretical analysis

The stability of a two-electron Zee system trapped inside an impenetrable spherical cavity is analyzed using an explicitly correlated multi-exponent Hylleraas-type basis set in the framework of the Ritz variational method. The wave function is considered to be consistent with Dirichlet’s boundary condition. Four different values are considered, where denotes the critical nuclear charge beyond which the system is embedded in a discrete positive energy continuum due to the impenetrable nature of the cavity. The energy contribution due to the total correlation (in the presence of both radial and angular correlation) effect, the radial correlation limit, and the angular correlation limit are also studied in detail. The thermodynamic pressure felt by the two-electron Zee system inside the cavity is estimated, and a formula replicating the behavior between the pressure and volume of the cavity is deduced by a fitting procedure. Different geometrical properties, e.g. radial moments (, , , , ; p = ), angular moments (, ), and related important physical quantities, are determined. The variation of Kirkwood and Buckingham polarizabilities w.r.t. the pressure felt by the two-electron Zee system is analyzed. A one-electron radial density is estimated for each pair of (), which has been employed to generate the electrostatic potential as well as different information theoretical measures like Shannon entropy, Fisher entropy, Rényi entropy, Tsallis entropy, and Onicescu informational energy. The chosen information theoretical measures have been found to be sensitive tools for describing changes in the electronic structure due to spatial confinement. An interesting interplay between the electronic and nuclear contribution to the classical electrostatic potential is observed, leading to a shift in the position of its characteristic minimum due to compression. Wherever possible, a comparison is made in order to ascertain the high accuracy of our numerical results. The procedure of analytic evaluation of the integrals needed to estimate the atomic properties under consideration is discussed in detail.

Kendall transfer entropy: a novel measure for estimating information transfer in complex systems

Objective. Transfer entropy (TE) has been widely used to infer causal relationships among dynamical systems, especially in neuroscience. Kendall transformation provides a novel quantization method for estimating information-theoretic measures and shows potential advantages for small-sample neural signals. But it has yet to be introduced into the framework of TE estimation, which commonly suffers from the limitation of small sample sizes. This paper aims to introduce the idea of Kendall correlation into TE estimation and verify its effect. Approach. We proposed the Kendall TE (KTE) which combines the improved Kendall transformation and the TE estimation. To confirm its effectiveness, we compared KTE with two common TE estimation techniques: the adaptive partitioning algorithm (D-V partitioning) and the symbolic TE. Their performances were estimated by simulation experiments which included linear, nonlinear, linear + nonlinear models and neural mass models. Moreover, the KTE was also applied to real electroencephalography (EEG) recordings to quantify the directional connectivity between frontal and parietal regions with propofol-induced general anesthesia. Main results. The simulation results showed that the KTE outperformed the other two methods by many measures: (1) identifying the coupling direction under a small sample size; (2) the sensitivity to coupling strength; (3) noise resistance; and (4) the sensitivity to time-dependent coupling changes. For real EEG recordings, the KTE clearly detected the disrupted frontal-to-parietal connectivity in propofol-induced unconsciousness, which is in agreement with previous findings. Significance. We reveal that the proposed KTE method is a robust and powerful tool for estimating TE, and is particularly suitable for small sample sizes. The KTE also provides an innovative form of quantizing continuous time series for information-theoretic measures.

Automated identification of patient subgroups: A case-study on mortality of COVID-19 patients admitted to the ICU

Background– Subgroup discovery (SGD) is the automated splitting of the data into complex subgroups. Various SGD methods have been applied to the medical domain, but none have been extensively evaluated. We assess the numerical and clinical quality of SGD methods. Method– We applied the improved Subgroup Set Discovery (SSD++), Patient Rule Induction Method (PRIM) and APRIORI – Subgroup Discovery (APRIORI-SD) algorithms to obtain patient subgroups on observational data of 14,548 COVID-19 patients admitted to 73 Dutch intensive care units. Hospital mortality was the clinical outcome. Numerical significance of the subgroups was assessed with information-theoretic measures. Clinical significance of the subgroups was assessed by comparing variable importance on population and subgroup levels and by expert evaluation. Results– The tested algorithms varied widely in the total number of discovered subgroups (5-62), the number of selected variables, and the predictive value of the subgroups. Qualitative assessment showed that the found subgroups make clinical sense. SSD++ found most subgroups (n = 62), which added predictive value and generally showed high potential for clinical use. APRIORI-SD and PRIM found fewer subgroups (n = 5 and 6), which did not add predictive value and were clinically less relevant. Conclusion– Automated SGD methods find clinical subgroups that are relevant when assessed quantitatively (yield added predictive value) and qualitatively (intensivists consider the subgroups significant). Different methods yield different subgroups with varying degrees of predictive performance and clinical quality. External validation is needed to generalize the results to other populations and future research should explore which algorithm performs best in other settings.

Segmentation-based linear discriminant analysis with information theoretic feature selection for hyperspectral image classification

ABSTRACT The use of hyperspectral imaging sensors has greatly improved the classification of remotely sensed data because of the abundant spectral information they offer. However, the numerous contiguous, tiny wavelength bands captured in hyperspectral images (HSIs) often hinder the classification process. To overcome the aforementioned problem, various feature reduction techniques, including feature extraction (FE) and feature selection (FS), are commonly employed to improve classification performance. Linear discriminant analysis (LDA) is a well-established approach that has been utilized for the FE of HSI. LDA’s consideration of global characteristics and the variance accumulator for FS can lead to the poor reduction of HIS’s intrinsic characteristics. Furthermore, LDA’s limited ability to select a very low number of features, i.e. the number of classes minus one, restricts its effectiveness in HSI classification. Therefore, we introduce an FS method based on a non-linear information-theoretic measure, normalized mutual information (nMI) combined with minimum redundancy maximum relevance (mRMR). This approach is used to identify inherent features from the transformed space of our proposed correlation-based segmented-LDA (SLDA) and spectral region-based segmented-LDA (SSLDA) FE methods. We thoroughly compare the performance of the SLDA-mRMR and SSLDA-mRMR methods with the existing linear and non-linear state-of-the-art techniques, including unsupervised principal component analysis (PCA), PCA-based methods, supervised LDA, and LDA-based methods. Additionally, we explore the performance of nMI-based mRMR selection with all FE methods by incorporating a cumulative variance-based top features pick-up strategy. Based on the experimental results, we observe that SSLDA-mRMR and SLDA-mRMR achieve the highest classification accuracy, such as 91.91%, and 91.57% for agricultural Indian Pines, respectively, 97.66%, and 97.54%, respectively, for Kennedy Space Center, and 96.69%, and 96.57%, respectively, for Pavia University. In contrast, the classification accuracies using all original features of the HSIs are 71.39%, 70.01%, and 83.52%, respectively. Moreover, our proposed SSLDA-mRMR and SLDA-mRMR combinations outperformed all other FE and FS combinations that were examined using actual HSI datasets.

Spatial Structure of Places in Singapore from a Complexity Perspective of the Public Transport Network

As cities become increasingly developed, inclusive and equal provision of public services and affordable housing is essential in achieving sustainable growth and development. In Singapore, with much of the population living in public housing and taking public transport, there is a great need to understand their landscape across the country. In this study, using tools from Complexity Science, we elucidated the spatial organization of places in Singapore through public bus transport connectivity, and investigated the association between such pattern to the distribution of the population age groups and property price. From the spatial network of the public transport services, we probed the places’ scale using the continuum percolation approach to uncover their spatial pattern. After that, we employed an information-theoretic measure of complexity to quantify the spatial organization of the system. The results show that there is a critical distance embedded within the public transport system at which the structure of the places in Singapore emerges. In many areas, the connectivity between places map well, and are greatest within the official planning boundary, demonstrating the influence of upstream spatial planning in determining bus networks. Yet in mature towns (particularly those in the central region), bus connectivity spans beyond planning boundaries, suggesting a more organic pattern on the ground in places with a longer history of development. The complexity of the bus network maps remarkably well to residential areas, and was associated with the distribution of resident age groups, and to property prices. In particular, the well-connected areas in and around the central region of Singapore are associated with higher property price and an older population. In contrast, the peripheral less-connected areas tend to see a lower property price and a younger population. The findings can contribute to understanding the existing pattern of physical and social form within an urban system, highlighting the role of complexity science methods in uncovering ‘emergent’ patterns in bus transport networks, beyond initial spatial plans, as areas mature over time. It can also guide future planning by highlighting the areas where more attention is needed to enhance island wide public bus connectivity, and by doing so, enable the public transport system to contribute to reducing potential socio-economic polarization within the system.

An Information-Theoretic Branch-and-Prune Algorithm for Discrete Phase Optimization of RIS in Massive MIMO

In this paper, we consider passive RIS-assisted multi-user communication between wireless nodes to improve the blocked line-of-sight (LOS) link performance. The wireless nodes are assumed to be equipped with Massive Multiple-Input Multiple-Output antennas, hybrid precoder, combiner, and low-resolution analog-to-digital converters (ADCs). We first derive the expression for the Cramer-Rao lower bound (CRLB) of the Mean Squared Error (MSE) of the received and combined signal at the intended receiver under interference. By appropriate design of the hybrid precoder, combiner, and RIS phase settings, it can be shown that the MSE achieves the CRLB. We further show that minimizing the MSE w.r.t. the phase settings of the RIS is equivalent to maximizing the throughput and energy efficiency of the system. We then propose a novel Information-Directed Branch-and-Prune (IDBP) algorithm to derive the phase settings of the RIS. We, for the first time in the literature, use an information-theoretic measure to decide on the pruning rules in a tree-search algorithm to arrive at the RIS phase-setting solution, which is vastly different compared to the traditional branch-and-bound algorithm that uses bounds of the cost function to define the pruning rules. In addition, we provide the theoretical guarantees of the near-optimality of the RIS phase-setting solution thus obtained using the Asymptotic Equipartition property. This also ensures near-optimal throughput and MSE performance.

Information theoretic measures of causal influences during transient neural events.

Introduction: Transient phenomena play a key role in coordinating brain activity at multiple scales, however their underlying mechanisms remain largely unknown. A key challenge for neural data science is thus to characterize the network interactions at play during these events. Methods: Using the formalism of Structural Causal Models and their graphical representation, we investigate the theoretical and empirical properties of Information Theory based causal strength measures in the context of recurring spontaneous transient events. Results: After showing the limitations of Transfer Entropy and Dynamic Causal Strength in this setting, we introduce a novel measure, relative Dynamic Causal Strength, and provide theoretical and empirical support for its benefits. Discussion: These methods are applied to simulated and experimentally recorded neural time series and provide results in agreement with our current understanding of the underlying brain circuits.