Information-theoretic Performance Research Articles

Bottlenecks CLUB: Unifying Information-Theoretic Trade-Offs Among Complexity, Leakage, and Utility

Bottleneck problems are an important class of optimization problems that have recently gained increasing attention in the domain of machine learning and information theory. They are widely used in generative models, fair machine learning algorithms, design of privacy-assuring mechanisms, and appear as information-theoretic performance bounds in various multi-user communication problems. In this work, we propose a general family of optimization problems, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity-leakage-utility bottleneck (CLUB)</i> model, which (i) provides a unified theoretical framework that generalizes most of the state-of-the-art literature for the information-theoretic privacy models, (ii) establishes a new interpretation of the popular generative and discriminative models, (iii) constructs new insights for the generative compression models, and (iv) can be used to obtain fair generative models. We first formulate the CLUB model as a complexity-constrained privacy-utility optimization problem. We then connect it with the closely related bottleneck problems, namely information bottleneck (IB), privacy funnel (PF), deterministic IB (DIB), conditional entropy bottleneck (CEB), and conditional PF (CPF). We show that the CLUB model generalizes all these problems as well as most other information-theoretic privacy models. Then, we construct the deep variational CLUB (DVCLUB) models by employing neural networks to parameterize variational approximations of the associated information quantities. Building upon these information quantities, we present unified objectives of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">supervised</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised</i> DVCLUB models. Leveraging the DVCLUB model in an unsupervised setup, we then connect it with state-of-the-art generative models, such as variational auto-encoders (VAEs), generative adversarial networks (GANs), as well as the Wasserstein GAN (WGAN), Wasserstein auto-encoder (WAE), and adversarial auto-encoder (AAE) models through the optimal transport (OT) problem. We then show that the DVCLUB model can also be used in fair representation learning problems, where the goal is to mitigate the undesired bias during the training phase of a machine learning model. We conduct extensive quantitative experiments on colored-MNIST and CelebA datasets.

Information theoretic performance evaluation of 3D integral imaging.

Integral imaging (InIm) has proved useful for three-dimensional (3D) object sensing, visualization, and classification of partially occluded objects. This paper presents an information-theoretic approach for simulating and evaluating the integral imaging capture and reconstruction process. We utilize mutual information (MI) as a metric for evaluating the fidelity of the reconstructed 3D scene. Also we consider passive depth estimation using mutual information. We apply this formulation for optimal pitch estimation of integral-imaging capture and reconstruction to maximize the longitudinal resolution. The effect of partial occlusion in integral imaging 3D reconstruction using mutual information is evaluated. Computer simulation tests and experiments are presented.

Discriminant Analysis under f-Divergence Measures.

In statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical models). As the data dimension grows, computing the statistics involved in decision-making and the attendant performance limits (divergence measures) face complexity and stability challenges. Dimensionality reduction addresses these challenges at the expense of compromising the performance (the divergence reduces by the data-processing inequality). This paper considers linear dimensionality reduction such that the divergence between the models is maximally preserved. Specifically, this paper focuses on Gaussian models where we investigate discriminant analysis under five f-divergence measures (Kullback–Leibler, symmetrized Kullback–Leibler, Hellinger, total variation, and ). We characterize the optimal design of the linear transformation of the data onto a lower-dimensional subspace for zero-mean Gaussian models and employ numerical algorithms to find the design for general Gaussian models with non-zero means. There are two key observations for zero-mean Gaussian models. First, projections are not necessarily along the largest modes of the covariance matrix of the data, and, in some situations, they can even be along the smallest modes. Secondly, under specific regimes, the optimal design of subspace projection is identical under all the f-divergence measures considered, rendering a degree of universality to the design, independent of the inference problem of interest.

Information Theoretic Analysis and Performance Gains of 3-Color Shift Keying

In this letter, we investigate the information theoretic performance of 3-Color Shift Keying (3-CSK), a promising modulation scheme for visible light communications that is based on the use of sequences of symbols. More specifically, considering its advantages, which include improved error performance and lower power consumption compared to 4-CSK and On-Off Keying, we evaluate its performance in terms of the maximum achievable rate, which is expressed as a function of the number of symbols per sequence, while a closed-form lower-bound and a tractable approximation are also obtained. Monte-Carlo simulations of the maximum achievable rate verify the analysis and illustrate that 3-CSK outperforms 4-CSK, when channel state information is not available at the receiver, with 3-CSK -in contrast to 4-CSK- reaching its maximum capacity in this scenario. Finally, to get further insights on the inferred 3-CSK gains, the bit-error-rate is also analyzed under different channel realizations, assuming no channel state information is available.

Advances in Inference and Representation for Simultaneous Localization and Mapping

Simultaneous localization and mapping (SLAM) is the process of constructing a global model of an environment from local observations of it; this is a foundational capability for mobile robots, supporting such core functions as planning, navigation, and control. This article reviews recent progress in SLAM, focusing on advances in the expressive capacity of the environmental models used in SLAM systems (representation) and the performance of the algorithms used to estimate these models from data (inference). A prominent theme of recent SLAM research is the pursuit of environmental representations (including learned representations) that go beyond the classical attributes of geometry and appearance to model properties such as hierarchical organization, affordance, dynamics, and semantics; these advances equip autonomous agents with a more comprehensive understanding of the world, enabling more versatile and intelligent operation. A second major theme is a revitalized interest in the mathematical properties of the SLAM estimation problem itself (including its computational and information-theoretic performance limits); this work has led to the development of novel classes of certifiable and robust inference methods that dramatically improve the reliability of SLAM systems in real-world operation. We survey these advances with an emphasis on their ramifications for achieving robust, long-duration autonomy, and conclude with a discussion of open challenges and a perspective on future research directions.

Communicating Correlated Sources Over MAC and Interference Channels I: Separation-Based Schemes

We consider the two scenarios of communicating a pair S <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , S <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> of distributed correlated sources over 2-user multiple access (MAC) and interference channels (IC), respectively. While in the MAC problem, the receiver needs to reconstruct both sources, in the IC problem, receiver j needs to reconstruct S <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3.</sub> We undertake a Shannon theoretic study and focus on characterizing sufficient conditions. Building on Dueck's findings (1981), we propose a coding scheme based on fixed block-length (B-L) codes. We characterize its information theoretic performance and characterize a new set of sufficient conditions. We identify examples of the MAC and IC problems for which the latter conditions are proven to be strictly weaker than the current known tightest.

Mutual Information Based Sensor Tasking with Applications to Space Situational Awareness

This paper focuses on the problem of managing or tasking a network of sensors to accurately track a number of objects while using information theoretic sensor performance metrics. The mathematical formulation of optimally tasking a group of sensors using the mutual information as a sensor utility measure is discussed along with the relative merits of maximizing mutual information. The resulting sensor-tasking optimization problem is shown to be combinatorial in nature, for which the computational complexity increases with an increase in the number of objects as well as the number of sensors. Depending upon the number of objects and available sensors, appropriate suboptimal approximations are presented to alleviate the computational complexity of the tasking problem. The submodular property of the mutual information measure is used to provide guarantees on the optimality of different approximations. Numerical simulations involving tracking ground objects with moving unmanned aerial vehicles and tracking resident space objects with ground-based sensors are considered to show the efficacy of the developed methods.

Dynamic TDD Systems for 5G and Beyond: A Survey of Cross-Link Interference Mitigation

Dynamic time division duplex (D-TDD) dynamically allocates the transmission directions for traffic adaptation in each cell. D-TDD systems are receiving a lot of attention because they can reduce latency and increase spectrum utilization via flexible and dynamic duplex operation in 5G New Radio (NR). However, the advantages of the D-TDD system are difficult to fully utilize due to the cross-link interference (CLI) arising from the use of different transmission directions between adjacent cells. This paper is a survey of the research from academia and the standardization efforts being undertaken to solve this CLI problem and make the D-TDD system a reality. Specifically, we categorize and present the approaches to mitigating CLI according to operational principles. Furthermore, we present the signaling necessary to apply the CLI mitigation schemes. We also present information-theoretic performance analysis of D-TDD systems in various environments. As topics for future works, we discuss the research challenges and opportunities associated with the CLI mitigation schemes and signaling design in a variety of environments. This survey is recommended for those who are in the initial stage of studying D-TDD systems and those who wish to develop a more feasible D-TDD system as a baseline for reviewing the research flow and standardization trends surrounding D-TDD systems and to identify areas of focus for future works.

Band selection of hyperspectral image by sparse manifold clustering

Band selection of hyperspectral images is an optimal feature selection method, which aims at reducing the computational burden associated with processing the whole data. The significant and informative bands identified by the band selection process lead to efficient, compact representation of the image data and produce a satisfactory performance in the succeeding applications viz. classification, unmixing, target detection and so on. In this study, the authors present an unsupervised manifold clustering approach for band selection, which accounts for different types of scenarios. Unlike other band selection approaches, the authors’ proposed manifold clustering framework identifies the informative bands by utilising the interrelation between the bands and accounts for the multi-manifold structure prevalent in some real images. The proposed band selection framework identifies the optimal number of clusters by cluster validity index, clusters the bands by manifold clustering and select representative bands from each cluster according to graph weight. Their proposed manifold clustering approach is a generic clustering approach, which produces a satisfactory result even when the data contains non-linearity. The information theoretic performance measures, classification and unmixing performance on real image experiments demonstrate the proficiency of their proposed band selection algorithm.

Fundamental Limits of Cloud and Cache-Aided Interference Management With Multi-Antenna Edge Nodes

In fog-aided cellular systems, content delivery latency can be minimized by jointly optimizing edge caching and transmission strategies. In order to account for the cache capacity limitations at the Edge Nodes (ENs), transmission generally involves both fronthaul transfer from a cloud processor with access to the content library to the ENs, as well as wireless delivery from the ENs to the users. In this paper, the resulting problem is studied from an information-theoretic viewpoint by making the following practically relevant assumptions: 1) the ENs have multiple antennas; 2) only uncoded fractional caching is allowed; 3) the fronthaul links are used to send fractions of contents; and 4) the ENs are constrained to use one-shot linear precoding on the wireless channel. Assuming offline proactive caching and focusing on a high signal-to-noise ratio (SNR) latency metric, the optimal information-theoretic performance is investigated under both serial and pipelined fronthaul-edge transmission modes. The analysis characterizes the minimum high-SNR latency in terms of Normalized Delivery Time (NDT) for worst-case users' demands. The characterization is exact for a subset of system parameters, and is generally optimal within a multiplicative factor of 3/2 for the serial case and of 2 for the pipelined case. The results bring insights into the optimal interplay between edge and cloud processing in fog-aided wireless networks as a function of system resources, including the number of antennas at the ENs, the ENs' cache capacity and the fronthaul capacity.

Performance Limits With Additive Error Metrics in Noisy Multimeasurement Vector Problems

Real-world applications such as magnetic resonance imaging with multiple coils, multi-user communication, and diffuse optical tomography often assume a linear model where several sparse signals sharing common sparse supports are acquired by several measurement matrices and then contaminated by noise. Multi-measurement vector (MMV) problems consider the estimation or reconstruction of such signals. In different applications, the estimation error that we want to minimize could be the mean squared error or other metrics such as the mean absolute error and the support set error. Seeing that minimizing different error metrics is useful in MMV problems, we study information-theoretic performance limits for MMV signal estimation with arbitrary additive error metrics. We also propose a message passing algorithmic framework that achieves the optimal performance, and rigorously prove the optimality of our algorithm for a special case. We further conjecture the optimality of our algorithm for some general cases, and back it up through numerical examples. As an application of our MMV algorithm, we propose a novel setup for active user detection in multi-user communication and demonstrate the promise of our proposed setup.

Information-Theoretic Performance Analysis of Sensor Networks via Markov Modeling of Time Series Data.

This paper presents information-theoretic performance analysis of passive sensor networks for detection of moving targets. The proposed method falls largely under the category of data-level information fusion in sensor networks. To this end, a measure of information contribution for sensors is formulated in a symbolic dynamics framework. The network information state is approximately represented as the largest principal component of the time series collected across the network. To quantify each sensor's contribution for generation of the information content, Markov machine models as well as x-Markov (pronounced as cross-Markov) machine models, conditioned on the network information state, are constructed; the difference between the conditional entropies of these machines is then treated as an approximate measure of information contribution by the respective sensors. The x-Markov models represent the conditional temporal statistics given the network information state. The proposed method has been validated on experimental data collected from a local area network of passive sensors for target detection, where the statistical characteristics of environmental disturbances are similar to those of the target signal in the sense of time scale and texture. A distinctive feature of the proposed algorithm is that the network decisions are independent of the behavior and identity of the individual sensors, which is desirable from computational perspectives. Results are presented to demonstrate the proposed method's efficacy to correctly identify the presence of a target with very low false-alarm rates. The performance of the underlying algorithm is compared with that of a recent data-driven, feature-level information fusion algorithm. It is shown that the proposed algorithm outperforms the other algorithm.

GF(q)-based precoding: information theoretical analysis and performance evaluation

<i>G</i><i>F</i>(<i>q</i>)-based precoding: information theoretical analysis and performance evaluation

Information theoretical performance analysis and optimisation of cooperative underwater acoustic communication systems

In this study, the authors investigate the information theoretical limits on the performance of single-carrier cooperative underwater acoustic communication systems in the presence of intersymbol interference. The authors assume decode-and-forward relaying and consider orthogonal half-duplex cooperation. Under the assumptions of sparse and frequency-selective Rician fading channel and non-white correlated Gaussian ambient noise, the authors derive expressions for bounds on the individual and achievable rates for both cases where channel state information is available at the transmitter or not. Using these expressions, the authors optimise input signalling and relay location to maximise average achievable rates.

Models, statistics, and rates of binary correlated sources

This paper discusses and analyzes various models of binary correlated sources, which may be relevant in several distributed communication scenarios. These models are statistically characterized in terms of joint Probability Mass Function (PMF) and covariance. Closed-form expressions for the joint entropy of the sources are also overviewed. The asymptotic entropy rate for very large number of sources is shown to converge to a common limit for all the considered models. This fact generalizes recent results on the information-theoretic performance limit of communication schemes which exploit the correlation among sources at the receiver.

Physical Layer Security for Multiple-Antenna Systems: A Unified Approach

Secrecy capacity is a fundamental information-theoretic performance metric to predict the maximum data rate of reliable communication, while the intended message is not revealed to the eavesdropper. Motivated by this consideration in this paper, a unified communication-theoretic framework for the analysis of the probability of nonzero secrecy capacity, the secrecy outage probability, and the secrecy capacity of multiple-antenna systems over fading channels is proposed. Specifically, a powerful frequency-domain approach is first developed in which the integrals involved in the evaluation of the probability of nonzero secrecy capacity and secrecy outage probability are transformed into the frequency domain, by employing Parseval’s theorem. A generic approach for the evaluation of the asymptotic secrecy outage probability at high signal-to-noise ratio (SNR) region is also introduced, thus providing useful insight as to the parameters affecting the secrecy performance. Finally, a unified numerical approach for computing the average secrecy capacity of multiple-antenna systems under arbitrary fading environments is developed. The proposed framework is general enough to accommodate any well-known multiantenna transmission technique and fading model. Finally, the secrecy performance of several multiple-antenna system setups is assessed, in the presence of generalized fading conditions and arbitrary antenna correlation, while various numerical and computer simulation results are shown and compared to substantiate the proposed mathematical analysis.

Performance Analysis of Parallel Relaying in Free-Space Optical Systems

In this paper, we investigate the information theoretic performance of parallel relaying in free-space optical (FSO) communications over Gamma-Gamma fading channels. We consider an intensity modulation/direct-detection (IM/DD) FSO system with a single relay and a line-of-sight link between the source and the destination. Considering both amplify-and-forward and decode-and-forward relaying, we develop performance characterizations of each relaying mode in terms of outage probability, diversity gain, coding gain, and diversity-multiplexing tradeoff (DMT). Our results demonstrate that parallel relaying improves the performance of FSO systems by bringing diversity advantages. In addition, both relaying modes achieve the same DMT and diversity gain while their coding gains can be different depending on underlying channels' characteristics.

Energy Harvesting Wireless Communications: A Review of Recent Advances

This paper summarizes recent contributions in the broad area of energy harvesting wireless communications. In particular, we provide the current state of the art for wireless networks composed of energy harvesting nodes, starting from the information-theoretic performance limits to transmission scheduling policies and resource allocation, medium access, and networking issues. The emerging related area of energy transfer for self-sustaining energy harvesting wireless networks is considered in detail covering both energy cooperation aspects and simultaneous energy and information transfer. Various potential models with energy harvesting nodes at different network scales are reviewed, as well as models for energy consumption at the nodes.

In-Band Full-Duplex Relaying: A Survey, Research Issues and Challenges

Recent advances in self-interference cancellation techniques enable in-band full-duplex wireless systems, which transmit and receive simultaneously in the same frequency band with high spectrum efficiency. As a typical application of in-band full-duplex wireless, in-band full-duplex relaying (FDR) is a promising technology to integrate the merits of in-band full-duplex wireless and relaying technology. However, several significant research challenges remain to be addressed before its widespread deployment, including small-size full-duplex device design, channel modeling and estimation, cross-layer/joint resource management, interference management, security, etc. In this paper, we provide a brief survey on some of the works that have already been done for in-band FDR, and discuss the related research issues and challenges. We identify several important aspects of in-band FDR: basics, enabling technologies, information-theoretical performance analysis, key design issues and challenges. Finally, we also explore some broader perspectives for in-band FDR.

Information theoretical performance limits of single‐carrier underwater acoustic systems

In this study, the authors investigate the information theoretical limits on the performance of point-to-point single-carrier acoustic systems over frequency-selective underwater channels with intersymbol interference. Under the assumptions of sparse and frequency-selective Rician fading channel and non-white correlated Gaussian ambient noise, the authors derive an expression for channel capacity and demonstrate the dependency on channel parameters such as the number, location and power delay profile of significant taps, as well as environmental parameters such as distance, temperature, salinity, pressure and depth. Then, the authors use this expression to determine the optimal carrier frequency, input signalling and bandwidth for capacity maximisation.