Network Information Theory Research Articles

CapMax: A Framework for Dynamic Network Representation Learning From the View of Multiuser Communication.

In this article, a modified mutual information maximization (InfoMax) framework, named channel capacity maximization (CapMax), is proposed and applied to learn informative representations for dynamic networks with time-varying topology and/or time-evolving node attributes. The CapMax is based on the network information theory for multiuser communication, where the representation model is treated as a multiaccess communication channel with memory and feedback. Without requirements of the backbone structure, the learning objective of our CapMax is maximizing the channel capacity, which is measured by directed information (DI) rather than mutual information. For efficient implementation, we design an estimator of the channel capacity through the combination of graph neural networks (GNNs) and recurrent neural networks (RNNs). Under some mild conditions, we theoretically prove that DI is a better measure than mutual information in capturing useful information. The experiments are conducted on multiple real-world dynamic network datasets, and the outperformance of our CapMax on different backbone models on link detection and prediction validates the effectiveness of modeling the representation model as a communication channel.

Information Constrained Optimal Transport: From Talagrand, to Marton, to Cover

The optimal transport problem studies how to transport one measure to another in the most cost-effective way and has wide range of applications from economics to machine learning. In this paper, we introduce and study an information constrained variation of this problem. Our study yields a strengthening and generalization of Talagrand’s celebrated transportation cost inequality. Following Marton’s approach, we show that the new transportation cost inequality can be used to recover old and new concentration of measure results. Finally, we provide an application of this new inequality to network information theory. We show that it can be used to recover almost immediately a recent solution to a long-standing open problem posed by Cover regarding the capacity of the relay channel.

Composable end-to-end security of Gaussian quantum networks with untrusted relays

Gaussian networks are fundamental objects in network information theory. Here many senders and receivers are connected by physically motivated Gaussian channels, while auxiliary Gaussian components, such as Gaussian relays, are entailed. Whilst the theoretical backbone of classical Gaussian networks is well established, the quantum analog is yet immature. Here, we theoretically tackle composable security of arbitrary Gaussian quantum networks, with generally untrusted nodes, in the finite-size regime. We put forward a general methodology for parameter estimation, which is only based on the data shared by the remote end-users. Taking a chain of identical quantum links as an example, we further demonstrate our study. Additionally, we find that the key rate of a quantum amplifier-assisted chain can ideally beat the fundamental repeaterless limit with practical block sizes. However, this objective is practically questioned leading the way to future network/chain designs.

Information evolution in complex networks

Many biological phenomena or social events critically depend on how information evolves in complex networks. However, a general theory to characterize information evolution is yet absent. Consequently, numerous unknowns remain about the mechanisms underlying information evolution. Among these unknowns, a fundamental problem, being a seeming paradox, lies in the coexistence of local randomness, manifested as the stochastic distortion of information content during individual-individual diffusion, and global regularity, illustrated by specific non-random patterns of information content on the network scale. Here, we attempt to formalize information evolution and explain the coexistence of randomness and regularity in complex networks. Applying network dynamics and information theory, we discover that a certain amount of information, determined by the selectivity of networks to the input information, frequently survives from random distortion. Other information will inevitably experience distortion or dissipation, whose speeds are shaped by the diversity of information selectivity in networks. The discovered laws exist irrespective of noise, but noise accounts for disturbing them. We further demonstrate the ubiquity of our discovered laws by analyzing the emergence of neural tuning properties in the primary visual and medial temporal cortices of animal brains and the emergence of extreme opinions in social networks.

Self-organization of in vitro neuronal assemblies drives to complex network topology

Activity-dependent self-organization plays an important role in the formation of specific and stereotyped connectivity patterns in neural circuits. By combining neuronal cultures, and tools with approaches from network neuroscience and information theory, we can study how complex network topology emerges from local neuronal interactions. We constructed effective connectivity networks using a transfer entropy analysis of spike trains recorded from rat embryo dissociated hippocampal neuron cultures between 6 and 35 days in vitro to investigate how the topology evolves during maturation. The methodology for constructing the networks considered the synapse delay and addressed the influence of firing rate and population bursts as well as spurious effects on the inference of connections. We found that the number of links in the networks grew over the course of development, shifting from a segregated to a more integrated architecture. As part of this progression, three significant aspects of complex network topology emerged. In agreement with previous in silico and in vitro studies, a small-world architecture was detected, largely due to strong clustering among neurons. Additionally, the networks developed in a modular topology, with most modules comprising nearby neurons. Finally, highly active neurons acquired topological characteristics that made them important nodes to the network and integrators of modules. These findings leverage new insights into how neuronal effective network topology relates to neuronal assembly self-organization mechanisms.

Network Analysis of Time Series: Novel Approaches to Network Neuroscience.

In the last two decades, there has been an explosion of interest in modeling the brain as a network, where nodes correspond variously to brain regions or neurons, and edges correspond to structural or statistical dependencies between them. This kind of network construction, which preserves spatial, or structural, information while collapsing across time, has become broadly known as “network neuroscience.” In this work, we provide an alternative application of network science to neural data: network-based analysis of non-linear time series and review applications of these methods to neural data. Instead of preserving spatial information and collapsing across time, network analysis of time series does the reverse: it collapses spatial information, instead preserving temporally extended dynamics, typically corresponding to evolution through some kind of phase/state-space. This allows researchers to infer a, possibly low-dimensional, “intrinsic manifold” from empirical brain data. We will discuss three methods of constructing networks from nonlinear time series, and how to interpret them in the context of neural data: recurrence networks, visibility networks, and ordinal partition networks. By capturing typically continuous, non-linear dynamics in the form of discrete networks, we show how techniques from network science, non-linear dynamics, and information theory can extract meaningful information distinct from what is normally accessible in standard network neuroscience approaches.

Toward an information theoretical description of communication in brain networks.

Modeling communication dynamics in the brain is a key challenge in network neuroscience. We present here a framework that combines two measurements for any system where different communication processes are taking place on top of a fixed structural topology: path processing score (PPS) estimates how much the brain signal has changed or has been transformed between any two brain regions (source and target); path broadcasting strength (PBS) estimates the propagation of the signal through edges adjacent to the path being assessed. We use PPS and PBS to explore communication dynamics in large-scale brain networks. We show that brain communication dynamics can be divided into three main “communication regimes” of information transfer: absent communication (no communication happening); relay communication (information is being transferred almost intact); and transducted communication (the information is being transformed). We use PBS to categorize brain regions based on the way they broadcast information. Subcortical regions are mainly direct broadcasters to multiple receivers; Temporal and frontal nodes mainly operate as broadcast relay brain stations; visual and somatomotor cortices act as multichannel transducted broadcasters. This work paves the way toward the field of brain network information theory by providing a principled methodology to explore communication dynamics in large-scale brain networks.

Complexities of disturbance response in a marine food web

ABSTRACTA primary aspect of applied marine ecology assesses how food webs change in response to ecosystem disturbances. In 2010, the drilling rig Deepwater Horizon (DWH) discharged ~3.19 million barrels of crude oil into the northern Gulf of Mexico. The spill, followed by widespread dispersant application to enhance oil degradation, represented a significant anthropogenic disturbance in the region. We created network models of four multi‐year periods, to represent the acute and chronic food web responses to the DWH spill. Using ecological network analysis (ENA) and information theory, we compared multiple food web metrics among these periods in the context of food web resilience theory. This analysis was conducted at three levels of hierarchical organization: whole ecosystem, nekton community, and individual nekton taxa. We analyzed how individual taxa contribute to resilience of the food web with a novel informational index: Redundancy/Ascendency. Apparent responses to the disturbance differed across hierarchical levels. Some metrics dependent on biomass change and flow distribution temporarily increased during the years immediately following the discharge and subsequently returned to pre‐DWH levels. Metrics of energy flow linked to primary production rose in the last two periods, perhaps reflecting eutrophication. Other metrics changed little or had no obviously explainable patterns. Overall, our results indicate the nektonic food web in this region is flexible to disturbance and likely has redundant energy pathways explaining the reported ecosystem resilience to the DWH spill. We show that an ENA, when applied to multiple levels of ecosystem hierarchy, may aid understanding of marine food web resilience.

A Unified View of Causal and Non-causal Feature Selection

In this article, we aim to develop a unified view of causal and non-causal feature selection methods. The unified view will fill in the gap in the research of the relation between the two types of methods. Based on the Bayesian network framework and information theory, we first show that causal and non-causal feature selection methods share the same objective. That is to find the Markov blanket of a class attribute, the theoretically optimal feature set for classification. We then examine the assumptions made by causal and non-causal feature selection methods when searching for the optimal feature set, and unify the assumptions by mapping them to the restrictions on the structure of the Bayesian network model of the studied problem. We further analyze in detail how the structural assumptions lead to the different levels of approximations employed by the methods in their search, which then result in the approximations in the feature sets found by the methods with respect to the optimal feature set. With the unified view, we can interpret the output of non-causal methods from a causal perspective and derive the error bounds of both types of methods. Finally, we present practical understanding of the relation between causal and non-causal methods using extensive experiments with synthetic data and various types of real-world data.

Unions, intersections and a one-shot quantum joint typicality lemma

A fundamental tool to prove inner bounds in classical network information theory is the so-called ‘conditional joint typicality lemma’. In addition to the lemma, one often uses unions and intersections of typical sets in the inner bound arguments without so much as giving them a second thought. These arguments do not work in the quantum setting. This bottleneck shows up in the fact that so-called ‘simultaneous decoders’, as opposed to ‘successive cancellation decoders’, are known for very few channels in quantum network information theory. Another manifestation of this bottleneck is the lack of so-called ‘simultaneous smoothing’ theorems for quantum states. In this paper, we overcome the bottleneck by proving for the first time a one-shot quantum joint typicality lemma with robust union and intersection properties. To do so we develop two novel tools in quantum information theory, which may be of independent interest. The first tool is a simple geometric idea called tilting, which increases the angles between a family of subspaces in orthogonal directions. The second tool, called smoothing and augmentation, is a way of perturbing a multipartite quantum state such that the partial trace over any subset of registers does not increase the operator norm much. Our joint typicality lemma allows us to construct simultaneous quantum decoders for many multiterminal quantum channels. It provides a powerful tool to extend many results in classical network information theory to the one-shot quantum setting.

A Unified Framework for One-Shot Achievability via the Poisson Matching Lemma

We introduce a fundamental lemma called the Poisson matching lemma, and apply it to prove one-shot achievability results for various settings, namely channels with state information at the encoder, lossy source coding with side information at the decoder, joint source-channel coding, broadcast channels, distributed lossy source coding, multiple access channels, channel resolvability and wiretap channels. Our one-shot bounds improve upon the best known one-shot bounds in most of the aforementioned settings (except multiple access channels, channel resolvability and wiretap channels, where we recover bounds comparable to the best known bounds), with shorter proofs in some settings even when compared to the conventional asymptotic approach using typicality. The Poisson matching lemma replaces both the packing and covering lemmas, greatly simplifying the error analysis. This paper extends the work of Li and El Gamal on Poisson functional representation, which mainly considered variable-length source coding settings, whereas this paper studies fixed-length settings, and is not limited to source coding, showing that the Poisson functional representation is a viable alternative to typicality for most problems in network information theory.

Strong Converse Bounds in Quantum Network Information Theory

In this paper, we develop the first method for finding strong converse bounds in quantum network information theory. The general scheme relies on a recently obtained result in the field of non-commutative functional inequalities, namely the tensorization property of quantum reverse hypercontractivity for the quantum depolarizing semigroup. We develop a novel technique to employ this result to find both finite blocklength and exponential strong converse bounds for the tasks of quantum source coding with compressed classical side information, and distributed quantum hypothesis testing with communication constraints for a classical-quantum state. In the classical setting, these two problems can be reformulated in a unified framework in terms of the so-called image-size characterization problem, which we extend to the classical-quantum setting. We also use this technique to establish analogous strong converse bounds in broadcast communication scenarios. In particular, we consider the transmission of classical information through a degraded broadcast channel, whose outputs are two quantum systems, with the state of one being a degraded version of the other. In establishing this last result, we prove a second-order Fano-type inequality, which is of independent interest. Our method to study strong converses has potential applications in other important tasks of quantum network information theory.

Inner bounds via simultaneous decoding in quantum network information theory

We prove new inner bounds for several multiterminal channels with classical inputs and quantum outputs. Our inner bounds are all proved in the one-shot setting and are natural analogues of the best classical inner bounds for the respective channels. For some of these channels, similar quantum inner bounds were unknown even in the asymptotic independent and identically distributed setting. We prove our inner bounds by appealing to a new classical–quantum joint typicality lemma established in a companion paper. This lemma allows us to lift to the quantum setting many inner bound proofs for classical multiterminal channels that use intersections and unions of typical sets.

Combining network topology and information theory to construct representative brain networks.

Network neuroscience employs graph theory to investigate the human brain as a complex network, and derive generalizable insights about the brain’s network properties. However, graph-theoretical results obtained from network construction pipelines that produce idiosyncratic networks may not generalize when alternative pipelines are employed. This issue is especially pressing because a wide variety of network construction pipelines have been employed in the human network neuroscience literature, making comparisons between studies problematic. Here, we investigate how to produce networks that are maximally representative of the broader set of brain networks obtained from the same neuroimaging data. We do so by minimizing an information-theoretic measure of divergence between network topologies, known as the portrait divergence. Based on functional and diffusion MRI data from the Human Connectome Project, we consider anatomical, functional, and multimodal parcellations at three different scales, and 48 distinct ways of defining network edges. We show that the highest representativeness can be obtained by using parcellations in the order of 200 regions and filtering functional networks based on efficiency-cost optimization—though suitable alternatives are also highlighted. Overall, we identify specific node definition and thresholding procedures that neuroscientists can follow in order to derive representative networks from their human neuroimaging data.

Information theory, synchronization and topological order in complete dynamical networks of discontinuous maps

This paper is dedicated to the study of information measures, synchronization and a topological order in complete dynamical networks of discontinuous piecewise linear maps with different slopes. It stands out that the networks topologies are characterized by circulant matrices and the conditional Lyapunov exponents are explicitly determined. Some properties of the mutual information rate and the Kolmogorov–Sinai entropy, depending on the synchronization interval, are discussed. A topological order between the complete dynamical networks is presented, which is characterized by the monotony of the network topological entropy. It is proved that if the network topological entropy increases, then the mutual information rate and the Kolmogorov–Sinai entropy increase or decrease, according to the variation of the coupling parameter. Furthermore, various types of computer simulations show the experimental applications of these results and techniques.

Superior Iterative Detection for Co-Channel Interference in Multibeam Satellite Systems

Aggressive frequency reuse alleviates the frequency-spectrum shortage in multibeam satellite systems targeting multi-Terabits-per-second throughput but creates harsh co-channel interference (CCI) environment. For forward-link, state-of-the-art receivers at user terminals assume memoryless CCI to avoid exponential increase in complexity. However, memory effects of CCI are inevitable, arising when co-channel signals combine asynchronously. First, we provide analytical characterization of CCI accounting for memory, a formulation that captures synchronization impairments. Then, low-complexity receivers, capable of realizing full frequency reuse, are developed to compensate for spatial and temporal CCI profiles, with computational complexity increasing linearly in the memory span. This is achieved by our novel soft-in soft-out, iterative divide-and-conquer (IDAC) paradigm, decomposing interference into smaller sets depending on intensity. A set is created from weak interferers which are not decoded but considered as thermal noise. The interferers which are decoded are further split into a set that subtracts strong ones and another set addressing intense interferers in optimal-Bayesian fashion. Extensive numerical studies reveal that IDAC architecture offers lossless compensation of dominant CCI with memory for beam-edge terminals, when rate coordination is guided by network information theory. Further, its performance is superior compared with approaches using iterative subtractive cancellation, commonly employed for return-link, random-access satellite applications.

Compute-Forward for DMCs: Simultaneous Decoding of Multiple Combinations

Algebraic network information theory is an emerging facet of network information theory, studying the achievable rates of random code ensembles that have algebraic structure, such as random linear codes. A distinguishing feature is that linear combinations of codewords can sometimes be decoded more efficiently than codewords themselves. The present work further develops this framework by studying the simultaneous decoding of multiple messages. Specifically, consider a receiver in a multi-user network that wishes to decode several messages. Simultaneous joint typicality decoding is one of the most powerful techniques for determining the fundamental limits at which reliable decoding is possible. This technique has historically been used in conjunction with random i.i.d. codebooks to establish achievable rate regions for networks. Recently, it has been shown that, in certain scenarios, nested linear codebooks in conjunction with “single-user” or sequential decoding can yield better achievable rates. For instance, the compute–forward problem examines the scenario of recovering $L \le K$ linear combinations of transmitted codewords over a $K$ -user multiple-access channel (MAC), and it is well established that linear codebooks can yield higher rates. This paper develops bounds for simultaneous joint typicality decoding used in conjunction with nested linear codebooks, and applies them to obtain a larger achievable region for compute–forward over a $K$ -user discrete memoryless MAC. The key technical challenge is that competing codeword tuples that are linearly dependent on the true codeword tuple introduce statistical dependencies, which requires careful partitioning of the associated error events.

Capacity Upper Bounds for the Relay Channel via Reverse Hypercontractivity

We revisit the primitive relay channel, introduced by Cover in 1987. Recent work derived upper bounds on the capacity of this channel that are tighter than the classical cutset bound using the concentration of measure. In this paper, we recover, generalize, and improve upon some of these upper bounds with simpler proofs using reverse hypercontractivity. To our knowledge, this is the first application of reverse hypercontractivity in proving first-order converses in network information theory.

Shannon entropy in time-varying semantic networks of titles of scientific paper

Recent work has employed information theory in social and complex networks. Studies often discuss entropy in the degree distributions of a network. However, no specific work on entropy exists in clique networks. This work is an extension of a previous study that discussed this topic. We propose a method for calculating the entropy of a clique network and its minimum and maximum values in temporal semantic networks based on titles of scientific papers. In addition, the critical network of moments was extracted. We use the titles of scientific papers published in Nature and Science over ten-year period. The results show the diversity of vocabulary over time, based on the entropy values of vertices and edges. In each critical network, we discover the paths that connect important words and an interesting modular structure.

Classical information theory of networks

Existing information-theoretic frameworks based on maximum entropy network ensembles are not able to explain the emergence of heterogeneity in complex networks. Here, we fill this gap of knowledge by developing a classical framework for networks based on finding an optimal trade-off between the information content of a compressed representation of the ensemble and the information content of the actual network ensemble. We introduce a novel classical network ensemble satisfying a set of soft constraints and we find the optimal distribution of the constraints for this ensemble. We show that for the classical network ensemble in which the only constraints are the expected degrees a power-law degree distribution is optimal. Also, we study spatially embedded networks finding that the interactions between nodes naturally lead to non-uniform spread of nodes in the embedding space, leading in some cases to a fractal distribution of nodes. This result is consistent with the so called `blessing of non-uniformity' of data, i.e. the fact that real world data typically do not obey uniform distributions. The pertinent features of real-world air transportation networks are well described by the proposed framework.