Information-theoretic Framework Research Articles

Single-trajectory map equation

Community detection, the process of identifying module structures in complex systems represented on networks, is an effective tool in various fields of science. The map equation, which is an information-theoretic framework based on the random walk on a network, is a particularly popular community detection method. Despite its outstanding performance in many applications, the inner workings of the map equation have not been thoroughly studied. Herein, we revisit the original formulation of the map equation and address the existence of its “raw form,” which we refer to as the single-trajectory map equation. This raw form sheds light on many details behind the principle of the map equation that are hidden in the steady-state limit of the random walk. Most importantly, the single-trajectory map equation provides a more balanced community structure, naturally reducing the tendency of the overfitting phenomenon in the map equation.

A Discriminant Information Theoretic Learning Framework for Multi-modal Feature Representation

As sensory and computing technology advances, multi-modal features have been playing a central role in ubiquitously representing patterns and phenomena for effective information analysis and recognition. As a result, multi-modal feature representation is becoming a progressively significant direction of academic research and real applications. Nevertheless, numerous challenges remain ahead, especially in the joint utilization of discriminatory representations and complementary representations from multi-modal features. In this article, a discriminant information theoretic learning (DITL) framework is proposed to address these challenges. By employing this proposed framework, the discrimination and complementation within the given multi-modal features are exploited jointly, resulting in a high-quality feature representation. According to characteristics of the DITL framework, the newly generated feature representation is further optimized, leading to lower computational complexity and improved system performance. To demonstrate the effectiveness and generality of DITL, we conducted experiments on several recognition examples, including both static cases, such as handwritten digit recognition, face recognition, and object recognition, and dynamic cases, such as video-based human emotion recognition and action recognition. The results show that the proposed framework outperforms state-of-the-art algorithms.

A special issue on Bayesian inference: challenges, perspectives and prospects.

A special issue on Bayesian inference: challenges, perspectives and prospects.

Quick Estimate of Information Decomposition for Text Style Transfer

A growing number of papers on style transfer for texts rely on information decomposition. The performance of the resulting systems is usually assessed empirically in terms of the output quality or requires laborious experiments. This paper suggests a straightforward information theoretical framework to assess the quality of information decomposition for latent representations in the context of style transfer. Experimenting with several state-of-the-art models, we demonstrate that such estimates could be used as a fast and straightforward health check for the models instead of more laborious empirical experiments.

A hybrid model of spatial autoregressive-multivariate adaptive generalized Poisson regression spline

Several Multivariate Adaptive Regression Spline (MARS) approaches are available to model categorical and numerical (especially continuous) data. Currently, there are other numerical data types—discrete or count data—that call for specific consideration in modeling. Additionally, spatially correlated count data is frequently observed. This has been seen in the case of health data, for example, the number of newborn fatalities, tuberculosis patients, hospital visitors, etc. However, currently no structurally consistent nonparametric regression and MARS model for count data incorporating spatial lag autocorrelation. The SAR-MAGPRS estimator (Spatial Autoregressive - Multivariate Adaptive Generalized Poisson Regression Spline) is developed to fill this gap. Although it can be applied to different count distributions, the estimator was developed in this study under the assumption of a Generalized Poisson distribution. This paper provides an information-theoretic framework for incorporating knowledge of the spatial structure and non-parametric regression models, especially MARS for the count data types. Moreover, the proposed method can assist in modeling the number of diseases while health policies are being developed. The framework presents an application of the Penalized Least Square (PLS) method to estimate the SAR – MAGPRS model.

On the Coexistence of eMBB and URLLC in Multi-Cell Massive MIMO

The non-orthogonal coexistence between the enhanced mobile broadband (eMBB) and the ultra-reliable low-latency communication (URLLC) in the downlink of a multi-cell massive MIMO system is rigorously analyzed in this work. We provide a unified information-theoretic framework blending an infinite-blocklength analysis of the eMBB spectral efficiency (SE) in the ergodic regime with a finite-blocklength analysis of the URLLC error probability relying on the use of mismatched decoding, and of the so-called saddlepoint approximation. Puncturing (PUNC) and superposition coding (SPC) are considered as alternative downlink coexistence strategies to deal with the inter-service interference, under the assumption of only statistical channel state information (CSI) knowledge at the users. eMBB and URLLC performances are then evaluated over different precoding techniques and power control schemes, by accounting for imperfect CSI knowledge at the base stations, pilot-based estimation overhead, pilot contamination, spatially correlated channels, the structure of the radio frame, and the characteristics of the URLLC activation pattern. Simulation results reveal that SPC is, in many operating regimes, superior to PUNC in providing higher SE for the eMBB yet achieving the target reliability for the URLLC with high probability. Moreover, PUNC might cause eMBB service outage in presence of high URLLC traffic loads. However, PUNC turns to be necessary to preserve the URLLC performance in scenarios where the multi-user interference cannot be satisfactorily alleviated.

A unified low-order information-theoretic feature selection framework for multi-label learning

The approximation of low-order information-theoretic terms for feature selection approaches has achieved success in addressing high-dimensional multi-label data. However, three critical issues exist in such type of approaches: (1) existing approaches are devised based on single heuristic variable correlation assumption, which biases towards some specific scene; (2) high-order variable correlations are ignored by cumulative summation low-order information-theoretic terms; (3) abundant approaches confuse researchers to devise and utilize appropriate approaches. To address these issues, two types of probability distribution assumption in terms of candidate features and labels are derived based on low-order variable correlations. Afterwords, clearing up all information-theoretic terms, we propose a unified feature selection framework including three low-order information-theoretic terms for multi-label learning named Selected Terms of Feature Selection (STFS). STFS contains high-order variable correlations in the form of low-order information-theoretic terms. Furthermore, many previous multi-label feature selection approaches can be reduced to special forms of STFS. Finally, extensive experiments conducted on twelve benchmark data sets in comparison to seven state-of-the-art approaches demonstrate the classification superiority of STFS.

Integration of aerial surveys and resource selection analysis indicates human land use supports boreal deer expansion.

Landscape change is a driver of global biodiversity loss. In the western Nearctic, petroleum exploration and extraction is a major contributor to landscape change, with concomitant effects on large mammal populations. One of those effects is the continued expansion of invasive white-tailed deer populations into the boreal forest, with ramifications for the whole ecosystem. We explored deer resource selection within the oil sands region of the boreal forest using a novel application of aerial ungulate survey (AUS) data. Deer locations from AUS were "used" points and together with randomly allocated "available" points informed deer resource selection in relation to landscape variables in the boreal forest. We created a candidate set of generalized linear models representing competing hypotheses about the role of natural landscape features, forest harvesting, cultivation, roads, and petroleum features. We ranked these in an information-theoretic framework. A combination of natural and anthropogenic landscape features best explained deer resource selection. Deer strongly selected seismic lines and other linear features associated with petroleum exploration and extraction, likely as movement corridors and resource subsidies. Forest harvesting and cultivation, important contributors to expansion in other parts of the white-tailed deer range, were not as important here. Stemming deer expansion to conserve native ungulates and maintain key predator-prey processes will likely require landscape management to restore the widespread linear features crossing the vast oil sands region.

Weighted Error Entropy-Based Information Theoretic Learning for Robust Subspace Representation.

In most of the existing representation learning frameworks, the noise contaminating the data points is often assumed to be independent and identically distributed (i.i.d.), where the Gaussian distribution is often imposed. This assumption, though greatly simplifies the resulting representation problems, may not hold in many practical scenarios. For example, the noise in face representation is usually attributable to local variation, random occlusion, and unconstrained illumination, which is essentially structural, and hence, does not satisfy the i.i.d. property or the Gaussianity. In this article, we devise a generic noise model, referred to as independent and piecewise identically distributed (i.p.i.d.) model for robust presentation learning, where the statistical behavior of the underlying noise is characterized using a union of distributions. We demonstrate that our proposed i.p.i.d. model can better describe the complex noise encountered in practical scenarios and accommodate the traditional i.i.d. one as a special case. Assisted by the proposed noise model, we then develop a new information-theoretic learning framework for robust subspace representation through a novel minimum weighted error entropy criterion. Thanks to the superior modeling capability of the i.p.i.d. model, our proposed learning method achieves superior robustness against various types of noise. When applying our scheme to the subspace clustering and image recognition problems, we observe significant performance gains over the existing approaches.

Information-Theoretic Odometry Learning

In this paper, we propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation, a crucial component of many robotics and vision tasks such as navigation and virtual reality where relative camera poses are required in real time. We formulate this problem as optimizing a variational information bottleneck objective function, which eliminates pose-irrelevant information from the latent representation. The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language. Specifically, we bound the generalization errors of the deep information bottleneck framework and the predictability of the latent representation. These provide not only a performance guarantee but also practical guidance for model design, sample collection, and sensor selection. Furthermore, the stochastic latent representation provides a natural uncertainty measure without the needs for extra structures or computations. Experiments on two well-known odometry datasets demonstrate the effectiveness of our method.

Does functional redundancy determine the ecological severity of a mass extinction event?

Many authors have noted the apparent 'decoupling' of the taxonomic and ecological severity of mass extinction events, with no widely accepted mechanistic explanation for this pattern having been offered. Here, we test between two key factors that potentially influence ecological severity: biosphere entropy (a measure of functional redundancy), and the degree of functional selectivity (in terms of deviation from a pattern of random extinction with respect to functional entities). While theoretical simulations suggest that the Shannon entropy of a given community prior to an extinction event determines the expected outcome following a perturbation of a given magnitude, actual variation in Shannon entropy between major extinction intervals is insufficient to explain the observed variation in ecological severity. Within this information-theoretic framework, we show that it is the degree of functional selectivity that is expected to primarily determine the ecological impact of a given perturbation when levels of functional redundancy are not substantially different.

Localization of signaling receptors maximizes cellular information acquisition in spatially structured natural environments.

Cells in natural environments, such as tissue or soil, sense and respond to extracellular ligands with intricately structured and non-monotonic spatial distributions, sculpted by processes such as fluid flow and substrate adhesion. In this work, we show that spatial sensing and navigation can be optimized by adapting the spatial organization of signaling pathways to the spatial structure of the environment. We develop an information-theoretic framework for computing the optimal spatial organization of a sensing system for a given signaling environment. We find that receptor localization previously observed in cells maximizes information acquisition in simulated natural contexts, including tissue and soil. Specifically, information acquisition is maximized when receptors form localized patches at regions of maximal ligand concentration. Receptor localization extends naturally to produce a dynamic protocol for continuously redistributing signaling receptors, which when implemented using simple feedback, boosts cell navigation efficiency by 30-fold.

A Generalized Information-Theoretic Framework for the Emergence of Hierarchical Abstractions in Resource-Limited Systems.

In this paper, a generalized information-theoretic framework for the emergence of multi-resolution hierarchical tree abstractions is developed. By leveraging ideas from information-theoretic signal encoding with side information, this paper develops a tree search problem which considers the generation of multi-resolution tree abstractions when there are multiple sources of relevant and irrelevant, or possibly confidential, information. We rigorously formulate an information-theoretic driven tree abstraction problem and discuss its connections with information-theoretic privacy and resource-limited systems. The problem structure is investigated and a novel algorithm, called G-tree search, is proposed. The proposed algorithm is analyzed and a number of theoretical results are established, including the optimally of the G-tree search algorithm. To demonstrate the utility of the proposed framework, we apply our method to a real-world example and provide a discussion of the results from the viewpoint of designing hierarchical abstractions for autonomous systems.

Private Facial Prediagnosis as an Edge Service for Parkinson's DBS Treatment Valuation.

Facial phenotyping for medical prediagnosis has recently been successfully exploited as a novel way for the preclinical assessment of a range of rare genetic diseases, where facial biometrics is revealed to have rich links to underlying genetic or medical causes. In this paper, we aim to extend this facial prediagnosis technology for a more general disease, Parkinson's Diseases (PD), and proposed an Artificial-Intelligence-of-Things (AIoT) edge-oriented privacy-preserving facial prediagnosis framework to analyze the treatment of Deep Brain Stimulation (DBS) on PD patients. In the proposed framework, a novel edge-based privacy-preserving framework is proposed to implement private deep facial diagnosis as a service over an AIoT-oriented information theoretically secure multi-party communication scheme, while data privacy has been a primary concern toward a wider exploitation of Electronic Health and Medical Records (EHR/EMR) over cloud-based medical services. In our experiments with a collected facial dataset from PD patients, for the first time, we proved that facial patterns could be used to evaluate the facial difference of PD patients undergoing DBS treatment. We further implemented a privacy-preserving information theoretical secure deep facial prediagnosis framework that can achieve the same accuracy as the non-encrypted one, showing the potential of our facial prediagnosis as a trustworthy edge service for grading the severity of PD in patients.

Constrained iterative Hirshfeld charges: A variational approach

We develop a variational procedure for the iterative Hirshfeld (HI) partitioning scheme. The main practical advantage of having a variational framework is that it provides a formal and straightforward approach for imposing constraints (e.g., fixed charges on certain atoms or molecular fragments) when computing HI atoms and their properties. Unlike many other variants of the Hirshfeld partitioning scheme, HI charges do not arise naturally from the information-theoretic framework, but only as a reverse-engineered construction of the objective function. However, the procedure we use is quite general and could be applied to other problems as well. We also prove that there is always at least one solution to the HI equations, but we could not prove that its self-consistent equations would always converge for any given initial pro-atom charges. Our numerical assessment of the constrained iterative Hirshfeld method shows that it satisfies many desirable traits of atoms in molecules and has the potential to surpass existing approaches for adding constraints when computing atomic properties.

MAVE-NN: learning genotype-phenotype maps from multiplex assays of variant effect

Multiplex assays of variant effect (MAVEs) are a family of methods that includes deep mutational scanning experiments on proteins and massively parallel reporter assays on gene regulatory sequences. Despite their increasing popularity, a general strategy for inferring quantitative models of genotype-phenotype maps from MAVE data is lacking. Here we introduce MAVE-NN, a neural-network-based Python package that implements a broadly applicable information-theoretic framework for learning genotype-phenotype maps—including biophysically interpretable models—from MAVE datasets. We demonstrate MAVE-NN in multiple biological contexts, and highlight the ability of our approach to deconvolve mutational effects from otherwise confounding experimental nonlinearities and noise.

A model-free method for identifying time-delay characteristics of HVAC system based on multivariate transfer entropy

The time delays in the HVAC system are common, which seriously affect time series prediction and pre-control command implementation, so identifying them is promising research. The existing methods of identifying time-delays mainly focus on establishing physical or dynamical models first and then conducting control variable experiments, which don't facilitate engineering. Therefore, we develop a model-free identification method rooted in an information-theoretic framework by introducing transfer entropy (TE) into the HVAC field. The suggested multivariate TE method can pick out each time-delay characteristic by mining monitoring data. Compared with correlation coefficients, it can filter redundant information between variables and discern the nonlinearity. Among them, to estimate multivariate TE well, a kernel estimator is improved. It performs the precise detection ability, low computational burden and strong robustness, compared with the original and k-nearest-neighbor (KNN) estimator. Besides, for an unacquainted HVAC system, a hierarchical optimization algorithm combining the Nash-optimization algorithm with a second-order oscillatory particle swarm optimization (SOPSO) algorithm is proposed to identify its time delays, where the accuracy and time cost are improved. Lastly, the above-mentioned methods are validated with simulated and real time series. This work is enlightening and has a further reference to identifying time delays in HVAC systems.

On the origin of red spirals: does assembly bias play a role?

The formation of the red spirals is a puzzling issue in the standard picture of galaxy formation and evolution. Most studies attribute the colour of the red spirals to different environmental effects. We analyze a volume limited sample from the SDSS to study the roles of small-scale and large-scale environments on the colour of spiral galaxies. We compare the star formation rate, stellar age and stellar mass distributions of the red and blue spirals and find statistically significant differences between them at 99.9% confidence level. The red spirals inhabit significantly denser regions than the blue spirals, explaining some of the observed differences in their physical properties. However, the differences persist in all types of environments, indicating that the local density alone is not sufficient to explain the origin of the red spirals. Using an information theoretic framework, we find a small but non-zero mutual information between the colour of spiral galaxies and their large-scale environment that are statistically significant (99.9% confidence level) throughout the entire length scale probed. Such correlations between the colour and the large-scale environment of spiral galaxies may result from the assembly bias. Thus both the local environment and the assembly bias may play essential roles in forming the red spirals. The spiral galaxies may have different assembly history across all types of environments. We propose a picture where the differences in the assembly history may produce spiral galaxies with different cold gas content. Such a difference would make some spirals more susceptible to quenching. In all environments, the spirals with high cold gas content could delay the quenching and maintain a blue colour, whereas the spirals with low cold gas fractions would be easily quenched and become red.

Entropy, free energy, symmetry and dynamics in the brain

Neuroscience is home to concepts and theories with roots in a variety of domains including information theory, dynamical systems theory, and cognitive psychology. Not all of those can be coherently linked, some concepts are incommensurable, and domain-specific language poses an obstacle to integration. Still, conceptual integration is a form of understanding that provides intuition and consolidation, without which progress remains unguided. This paper is concerned with the integration of deterministic and stochastic processes within an information theoretic framework, linking information entropy and free energy to mechanisms of emergent dynamics and self-organization in brain networks. We identify basic properties of neuronal populations leading to an equivariant matrix in a network, in which complex behaviors can naturally be represented through structured flows on manifolds establishing the internal model relevant to theories of brain function. We propose a neural mechanism for the generation of internal models from symmetry breaking in the connectivity of brain networks. The emergent perspective illustrates how free energy can be linked to internal models and how they arise from the neural substrate.

Towards an information-theoretic framework of intrusion detection for composed systems and robustness analyses

Network-based Intrusion Detection Systems (NIDSs) are an important mechanism to identify malicious behaviour or policy violations within a network. Such detection systems typically face several challenges, among which are the base-rate fallacy and the resilience against adaptive adversaries. These challenges are often countered in modern NIDSs by combining multiple detection systems to diversify the used feature levels or utilize the advantages of multiple detection methods. However, currently there exists no suitable framework for a detailed analysis of such composed systems. Therefore, the contribution of this work is an evaluation framework for composed systems, which builds on previous information-theoretic approaches and highlights the utility of information-theoretic redundancies for robustness evaluations. This framework enables an attribution of the overall system performance to its individual components, to fine-tune parameters and to study the dynamics between classifiers. The versatility of the framework is demonstrated by designing and evaluating a composed NIDS example based on systems described in the literature and using an open data set. Studying the impact of an evasion attempt with adversarial examples on this system highlighted the importance of robustness against false-alarms as well as detection evasion. Moreover, the framework enables general insights on how to improve the design of composed NIDSs: based on the dynamics between classifiers, it can be shown that optimizing the operation point of each component individually does not necessarily maximize the overall system performance from an information-theoretic perspective. Additionally, it can be shown that existing classification redundancies might not be fully utilized during an attack on the NIDS components, due to a static system design.