Information-theoretic Perspective Research Articles

Shunting at Arbitrary Feature Levels via Spatial Disentanglement: Toward Selective Image Translation.

The past few years have witnessed considerable efforts devoted to translating images from one domain to another, mainly aiming at editing global style. Here, we focus on a more general case, selective image translation (SLIT), under an unsupervised setting. SLIT essentially operates through a shunt mechanism that involves learning gates to manipulate only the contents of interest (CoIs), which can be either local or global, while leaving the irrelevant parts unchanged. Existing methods typically rely on a flawed implicit assumption that CoIs are separable at arbitrary levels, ignoring the entangled nature of DNN representations. This leads to unwanted changes and learning inefficiency. In this work, we revisit SLIT from an information-theoretical perspective and introduce a novel framework, which equips two opposite forces to disentangle the visual features. One force encourages independence between spatial locations on the features, while the other force unites multiple locations to form a "block" that jointly characterizes an instance or attribute that a single location may not independently characterize. Importantly, this disentanglement paradigm can be applied to visual features of any layer, enabling shunting at arbitrary feature levels, which is a significant advantage not explored in existing works. Our approach has undergone extensive evaluation and analysis, confirming its effectiveness in significantly outperforming the state-of-the-art baselines.

A Novel Hypersonic Target Trajectory Estimation Method Based on Long Short-Term Memory and a Multi-Head Attention Mechanism.

To address the complex maneuvering characteristics of hypersonic targets in adjacent space, this paper proposes an LSTM trajectory estimation method combined with the attention mechanism and optimizes the model from the information-theoretic perspective. The method captures the target dynamics by using the temporal processing capability of LSTM, and at the same time improves the efficiency of information utilization through the attention mechanism to achieve accurate prediction. First, a target dynamics model is constructed to clarify the motion behavior parameters. Subsequently, an LSTM model incorporating the attention mechanism is designed, which enables the model to automatically focus on key information fragments in the historical trajectory. In model training, information redundancy is reduced, and information validity is improved through feature selection and data preprocessing. Eventually, the model achieves accurate prediction of hypersonic target trajectories with limited computational resources. The experimental results show that the method performs well in complex dynamic environments with improved prediction accuracy and robustness, reflecting the potential of information theory principles in optimizing the trajectory prediction model.

Redundancy Attributes of a Complex System: Application to Bioinformatics

In general, a complex system consists of a large number of interacting units, which when viewed in an information-theoretic perspective, could be seen to possess gross redundant features. Further, a complex system is inherently stochastical in its extensive spatiotemporal universe and hence, some of its statistical features could manifest as patterns occurring more frequently than others; and, likewise some other patterns could form rare elements of the same set. In information-theoretic perspectives (in Shannon's sense) such more (or less) probabilistic occurrences of features specify the redundant (or nonredundant) aspects of the complex system. This attribute of information redundancy in a complex system could be adopted to devise a useful complexity metric as described in this paper. The use of such a complexity metric is demonstrated in differentiating codon-noncodon domains in human and bacterial genomes (which constitute a complex system when seen at the scale of observing a DNA sequence made of vast domains of protein molecules).

IDEA: Invariant defense for graph adversarial robustness

Despite the success of graph neural networks (GNNs), their vulnerability to adversarial attacks poses tremendous challenges for practical applications. Existing defense methods suffer from severe performance decline under unseen attacks, due to either limited observed adversarial examples or pre-defined heuristics. To address these limitations, we analyze the causalities in graph adversarial attacks and conclude that causal features are key to achieve graph adversarial robustness, owing to their determinedness for labels and invariance across attacks. To learn these causal features, we innovatively propose an Invariant causal DEfense method against adversarial Attacks (IDEA). We derive node-based and structure-based invariance objectives from an information-theoretic perspective. IDEA ensures strong predictability for labels and invariant predictability across attacks, which is provably a causally invariant defense across various attacks. Extensive experiments demonstrate that IDEA attains state-of-the-art defense performance under all five attacks on all five datasets. The implementation of IDEA is available at https://github.com/TaoShuchang/IDEA_repo.

Mutual information for detecting multi-class biomarkers when integrating multiple bulk or single-cell transcriptomic studies.

Biomarker detection plays a pivotal role in biomedical research. Integrating omics studies from multiple cohorts can enhance statistical power, accuracy and robustness of the detection results. However, existing methods for horizontally combining omics studies are mostly designed for two-class scenarios (e.g., cases versus controls) and are not directly applicable for studies with multi-class design (e.g., samples from multiple disease subtypes, treatments, tissues, or cell types). We propose a statistical framework, namely Mutual Information Concordance Analysis (MICA), to detect biomarkers with concordant multi-class expression pattern across multiple omics studies from an information theoretic perspective. Our approach first detects biomarkers with concordant multi-class patterns across partial or all of the omics studies using a global test by mutual information. A post hoc analysis is then performed for each detected biomarkers and identify studies with concordant pattern. Extensive simulations demonstrate improved accuracy and successful false discovery rate control of MICA compared to an existing MCC method. The method is then applied to two practical scenarios: four tissues of mouse metabolism-related transcriptomic studies, and three sources of estrogen treatment expression profiles. Detected biomarkers by MICA show intriguing biological insights and functional annotations. Additionally, we implemented MICA for single-cell RNA-Seq data for tumor progression biomarkers, highlighting critical roles of ribosomal function in the tumor microenvironment of triple-negative breast cancer and underscoring the potential of MICA for detecting novel therapeutic targets. https://github.com/jianzou75/MICA.

Secure authentication in MIMO systems: exploring physical limits

Multiple-input multiple-output (MIMO) technology is employed to improve the reliability and capacity of wireless communication systems. However, the wireless communication environment creates vulnerabilities to spoofing attacks. Furthermore, the authentication challenges posed by the heterogeneous characteristics of wireless applications increase as diverse technologies facilitate the growing number of Internet of Things (IoT) devices. To address these challenges, adaptive physical-layer authentication (PLA) leveraging the inherent antenna diversity in MIMO systems is examined, and an information-theoretic perspective on PLA in MIMO systems is given. The real and imaginary components of the received reference signals are used as attributes with a single-class classification support vector machine (SCC-SVM). It is shown that the authentication performance improves with the number of antennas, and the proposed scheme provides robust authentication.

The Inverse of Exact Renormalization Group Flows as Statistical Inference.

We build on the view of the Exact Renormalization Group (ERG) as an instantiation of Optimal Transport described by a functional convection-diffusion equation. We provide a new information-theoretic perspective for understanding the ERG through the intermediary of Bayesian Statistical Inference. This connection is facilitated by the Dynamical Bayesian Inference scheme, which encodes Bayesian inference in the form of a one-parameter family of probability distributions solving an integro-differential equation derived from Bayes' law. In this note, we demonstrate how the Dynamical Bayesian Inference equation is, itself, equivalent to a diffusion equation, which we dub Bayesian Diffusion. By identifying the features that define Bayesian Diffusion and mapping them onto the features that define the ERG, we obtain a dictionary outlining how renormalization can be understood as the inverse of statistical inference.

CiteNet: Cross-modal incongruity perception network for multimodal sentiment prediction

Multimodal sentiment prediction poses a formidable challenge that necessitates a profound understanding of both visual and linguistic cues, as well as the intricate interactions between them. The current achievements of modern systems in this domain can plausibly be attributed to the development of sophisticated cross-modal fusion techniques. Nevertheless, such solutions often handle each modality equally, neglecting the discordant predictions arising from sentiment incongruity in unimodal sources, which may result in performance degradation in conventional extraction-fusion scenarios. In this work, we take a different route–introducing an extraction-estimation-fusion paradigm aimed at exploring more reliable multimodal representations under the supervision of unimodal sentiment prediction. To this end, we propose a Cross-modal IncongruiTy pErception NETwork, named CiteNet, for multimodal sentiment detection. In CiteNet, we initially develop a cross-modal alignment module tailored to synchronize modality-specific representations through contrastive learning. Subsequently, with a refined cross-modal integration module, CiteNet can achieve a synergistic and comprehensive multimodal representation. In addition, we explore a cross-modal incongruity learning module from an information-theoretic perspective, capable of estimating inherent sentiment disparities by analyzing modal distributions. This incongruity score is then employed as a crucial factor in the adaptive fusion of unimodal and multimodal representations, culminating in enhanced accuracy in sentiment prediction. Experimental results on two datasets demonstrate that CiteNet outperforms prior methods by a significant margin of approximately 1%–11% in accuracy.

A Rényi-type quasimetric with random interference detection

This paper introduces a new dissimilarity measure between two discrete and finite probability distributions. The followed approach is grounded jointly on mixtures of probability distributions and an optimization procedure. We discuss the clear interpretation of the constitutive elements of the measure under an information-theoretical perspective by also highlighting its connections with the Rényi divergence of infinite order. Moreover, we show how the measure describes the inefficiency in assuming that a given probability distribution coincides with a benchmark one by giving formal writing of the random interference between the considered probability distributions. We explore the properties of the considered tool, which are in line with those defining the concept of quasimetric—i.e. a divergence for which the triangular inequality is satisfied. As a possible usage of the introduced device, an application to rare events is illustrated. This application shows that our measure may be suitable in cases where the accuracy of the small probabilities is a relevant matter.

Revisiting the Information Capacity of Neural Network Watermarks: Upper Bound Estimation and Beyond

To trace the copyright of deep neural networks, an owner can embed its identity information into its model as a watermark. The capacity of the watermark quantify the maximal volume of information that can be verified from the watermarked model. Current studies on capacity focus on the ownership verification accuracy under ordinary removal attacks and fail to capture the relationship between robustness and fidelity. This paper studies the capacity of deep neural network watermarks from an information theoretical perspective. We propose a new definition of deep neural network watermark capacity analogous to channel capacity, analyze its properties, and design an algorithm that yields a tight estimation of its upper bound under adversarial overwriting. We also propose a universal non-invasive method to secure the transmission of the identity message beyond capacity by multiple rounds of ownership verification. Our observations provide evidence for neural network owners and defenders that are curious about the tradeoff between the integrity of their ownership and the performance degradation of their products.

Towards Explainable Joint Models via Information Theory for Multiple Intent Detection and Slot Filling

Recent joint models for multi-intent detection and slot filling have obtained promising results through modeling the unidirectional or bidirectional guidance between intent and slot. However, existing works design joint models heuristically and lack some theoretical exploration, including (1) theoretical measurement of the joint-interaction quality; (2) explainability of design and optimization methods of joint models, which may limit the performance and efficiency of designs. In this paper, we mathematically define the cross-task information gain (CIG) to measure the quality of joint processes from an information-theoretic perspective and discover an implicit optimization of CIG in previous models. Based on this, we propose a novel multi-stage iterative framework with theoretical effectiveness, explainability, and convergence, which can explicitly optimize information for cross-task interactions. Further, we devise an information-based joint model (InfoJoint) that conforms to this theoretical framework to gradually reduce the cross-task propagation of erroneous semantics through CIG iterative maximization. Extensive experiment results on two public datasets show that InfoJoint outperforms the state-of-the-art models by a large margin.

H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer

Multi-source transfer learning is an effective solution to data scarcity by utilizing multiple source tasks for the learning of the target task. However, access to source data and model details is limited in the era of commercial models, giving rise to the setting of multi-source-free (MSF) transfer learning that aims to leverage source domain knowledge without such access. As a newly defined problem paradigm, MSF transfer learning remains largely underexplored and not clearly formulated. In this work, we adopt an information theoretic perspective on it and propose a framework named H-ensemble, which dynamically learns the optimal linear combination, or ensemble, of source models for the target task, using a generalization of maximal correlation regression. The ensemble weights are optimized by maximizing an information theoretic metric for transferability. Compared to previous works, H-ensemble is characterized by: 1) its adaptability to a novel and realistic MSF setting for few-shot target tasks, 2) theoretical reliability, 3) a lightweight structure easy to interpret and adapt. Our method is empirically validated by ablation studies, along with extensive comparative analysis with other task ensemble and transfer learning methods. We show that the H-ensemble can successfully learn the optimal task ensemble, as well as outperform prior arts.

Contrastive regularization guided label refurbishment for fault diagnosis under label noise

Conventional intelligent fault diagnosis studies require precise labels by default. However, label noise is inevitable in real industrial environments due to human errors, measurement biases, data transmission and storage errors. The performance of existing data-driven methods may be severely affected by mislabels. In this paper, we investigate the fault diagnosis issue of rotating machinery under noisy labels. A generalized framework contrastive regularization guided label refurbishment (CRLR) is proposed, which utilizes contrastive regularization function extracting class prototype to guide label refurbishment. The learned contrastive representations can suppress the misleading effect of noisy labels. The theoretical analysis from an information theoretic perspective also ensures the effectiveness of CRLR. Validation experiments have been conducted on public and private collected datasets using the ResNet-50 as the backbone network. The experiments demonstrate that our proposed framework outperforms the current state-of-the-art methods at different noise rates.

To Compress or Not to Compress-Self-Supervised Learning and Information Theory: A Review.

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.

Maximum Geometric Quantum Entropy.

Any given density matrix can be represented as an infinite number of ensembles of pure states. This leads to the natural question of how to uniquely select one out of the many, apparently equally-suitable, possibilities. Following Jaynes' information-theoretic perspective, this can be framed as an inference problem. We propose the Maximum Geometric Quantum Entropy Principle to exploit the notions of Quantum Information Dimension and Geometric Quantum Entropy. These allow us to quantify the entropy of fully arbitrary ensembles and select the one that maximizes it. After formulating the principle mathematically, we give the analytical solution to the maximization problem in a number of cases and discuss the physical mechanism behind the emergence of such maximum entropy ensembles.

Sources of richness and ineffability for phenomenally conscious states.

Conscious states-state that there is something it is like to be in-seem both rich or full of detail and ineffable or hard to fully describe or recall. The problem of ineffability, in particular, is a longstanding issue in philosophy that partly motivates the explanatory gap: the belief that consciousness cannot be reduced to underlying physical processes. Here, we provide an information theoretic dynamical systems perspective on the richness and ineffability of consciousness. In our framework, the richness of conscious experience corresponds to the amount of information in a conscious state and ineffability corresponds to the amount of information lost at different stages of processing. We describe how attractor dynamics in working memory would induce impoverished recollections of our original experiences, how the discrete symbolic nature of language is insufficient for describing the rich and high-dimensional structure of experiences, and how similarity in the cognitive function of two individuals relates to improved communicability of their experiences to each other. While our model may not settle all questions relating to the explanatory gap, it makes progress toward a fully physicalist explanation of the richness and ineffability of conscious experience-two important aspects that seem to be part of what makes qualitative character so puzzling.

Adding causality to the information-theoretic perspective on individuality

I extend work from Krakauer et al. (2020), who propose a conception of individuality as the capacity to propagate information through time. From this conception, they develop information-theoretic measures. I identify several shortcomings with these measures—in particular, that they are associative rather than causal. I rectify this shortcoming by deriving a causal information-theoretic measure of individuality. I then illustrate how this measure can be implemented and extended in the context of evolutionary transitions in individuality.

Slack Is Needed to Solve the Shortage of Nurses.

Healthcare systems are facing a shortage of nurses. This article identifies some of the major causes of this and the issues that need to be solved. We take a perspective derived from queuing theory: the patient-nurse relationship is characterized by a scarcity of time and resources, requiring comprehensive coordination at all levels. For coordination, we take an information-theoretic perspective. Using both perspectives, we analyze the nature of healthcare services and show that ensuring slack, meaning a less than exhaustive use of human resources, is a sine qua non to having a good, functioning healthcare system. We analyze what coordination efforts are needed to manage relatively simple office hours, wards, and home care. Next, we address the level of care where providers cannot themselves prevent the complexity of organization that possibly damages care tasks and job quality. A lack of job quality may result in nurses leaving the profession. Job quality, in this context, depends on the ability of nurses to coordinate their activities. This requires slack resources. The availability of slack that is efficient depends on a stable inflow and retention rate of nurses. The healthcare system as a whole should ensure that the required nurse workforce will be able to coordinate and execute their tasks. Above that, workforce policies need more stability.

Information-theoretic perspective on performance assessment and fundamental limit of quantum imaging [Invited

Information-theoretic perspective on performance assessment and fundamental limit of quantum imaging [Invited

Causalized convergent cross-mapping and its approximate equivalence with directed information in causality analysis.

Convergent cross-mapping (CCM) has attracted increased attention recently due to its capability to detect causality in nonseparable systems under deterministic settings, which may not be covered by the traditional Granger causality. From an information-theoretic perspective, causality is often characterized as the directed information (DI) flowing from one side to the other. As information is essentially nondeterministic, a natural question is: does CCM measure DI flow? Here, we first causalize CCM so that it aligns with the presumption in causality analysis-the future values of one process cannot influence the past of the other, and then establish and validate the approximate equivalence of causalized CCM (cCCM) and DI under Gaussian variables through both theoretical derivations and fMRI-based brain network causality analysis. Our simulation result indicates that, in general, cCCM tends to be more robust than DI in causality detection. The underlying argument is that DI relies heavily on probability estimation, which is sensitive to data size as well as digitization procedures; cCCM, on the other hand, gets around this problem through geometric cross-mapping between the manifolds involved. Overall, our analysis demonstrates that cross-mapping provides an alternative way to evaluate DI and is potentially an effective technique for identifying both linear and nonlinear causal coupling in brain neural networks and other settings, either random or deterministic, or both.