Input-output Mutual Information Research Articles

Capacity-Driven Autoencoders for Communications

The autoencoder concept has fostered the reinterpretation and the design of modern communication systems. It consists of an encoder, a channel and a decoder block that modify their internal neural structure in an end-to-end learning fashion. However, the current approach to train an autoencoder relies on the use of the cross-entropy loss function. This approach can be prone to overfitting issues and often fails to learn an optimal system and signal representation (code). In addition, less is known about the autoencoder ability to design channel capacity-approaching codes, i.e., codes that maximize the input-output mutual information under a certain power constraint. The task being even more formidable for an unknown channel for which the capacity is unknown and therefore it has to be learnt. In this paper, we address the challenge of designing capacity-approaching codes by incorporating the presence of the communication channel into a novel loss function for the autoencoder training. In particular, we exploit the mutual information between the transmitted and received signals as a regularization term in the cross-entropy loss function, with the aim of controlling the amount of information stored. By jointly maximizing the mutual information and minimizing the cross-entropy, we propose a theoretical approach that a) computes an estimate of the channel capacity and b) constructs an optimal coded signal approaching it. Theoretical considerations are made on the choice of the cost function and the ability of the proposed architecture to mitigate the overfitting problem. Simulation results offer an initial evidence of the potentiality of the proposed method.

Characterization of Event-Based Sampling Encoders for Industrial Internet of Things Using Input–Output Mutual Information

The emergence of Industry 4.0 has resulted in a rapid increase in the demand for bandwidth due to the proliferation of the industrial Internet of Things (IIoT). Increased utilization of the network also gives rise to undesirable consequences such as high latency and increased likelihood of data loss through packet drops. Encoding inputs using event-based sampling is a potential solution for decreasing network traffic generated when continuous input variables are sampled. The problem addressed by this article is the fact that currently there is no effective method for comparing different encoders. The resulting contribution is the use of mutual information to compare the input and encoded output in terms of accuracy for the currently most efficient encoders that use either memory based event triggering (MBET) or deadband error modulation (DEM). This allows a practitioner to select a suitable encoder for a given input specification.

Uplink multi‐carrier multiple access scheme LDS‐IOTA

In this study, a multiple access scheme called low density signature isotropic orthogonal transfer algorithm (LDS-IOTA) is proposed for overloading transmissions, and an analytical model for the performance limitations of this system is developed. LDS-IOTA utilises a low density signature structure for spreading the symbols in frequency domain, and applies such technique to IOTA-based filter bank multicarrier system. The input-output mutual information of LDS-IOTA is analysed for the case of finite size modulation, and LDS-IOTA is compared with OFDM as well as LDS-OFDM over multipath fadding channels.

Optimal Signaling Schemes and Capacity of Non-Coherent Rician Fading Channels With Low-Resolution Output Quantization

Low-resolution analog-to-digital converter (ADC) has been considered as a promising solution to save power and cost in communication systems using high bandwidth and/or multiple RF chains. The goal of this work is to address the design of optimal signaling schemes and establish the capacity limit of Rician fading channels with low-resolution output quantization. This fading channel can be used to accurately model a wide range of wireless channels with the line-of-sight (LOS) components, including emerging mm-wave communications. The focus is on non-coherent fast fading channels where neither the transmitter nor the receiver knows the channel state information (CSI). By examining the continuity of the input-output mutual information, the existence of the optimal input signal is first validated. Then, considering the case of 1-bit ADC, we show that the optimal input is $\pi /2$ circularly symmetric. A necessary and sufficient condition for an input signal to be optimal, which is referred to as the Kuhn-Tucker condition (KTC), and Lagrangian optimization problem are then established. By exploiting the novel log-quadratic bounds on the Gaussian $Q$ -function, it is then demonstrated that for a given mass point’s amplitude, the corresponding rotated mass points through the phase of LOS component must form a square grid centered at zero. Furthermore, the amplitude of the mass points in the optimal distribution can take on only one value. As a result, the capacity-achieving input with 1-bit ADC is a rotated quadrature phase-shift keying (QPSK) constellation, and the rotation angle depends on the Rician factor. The characterization of the optimal input has also been extended to the case of multi-bit ADCs. Specifically, it is shown that for a $K$ -bit ADC, the optimal input is discrete having atmost $2^{2K}$ mass points. In both the cases of 1-bit and $K$ -bit ADCs, the channel capacities are established in closed-form.

Gradients of the fundamental information measures: Theory and applications

Gradient of the information measures can be considered as the building block of the fundamental relations between information and estimation theories established in the literature. This paper introduces the gradients of two fundamental information measures, i.e., differential entropy and mutual information in some general cases. Our approach is based on the utilizing a general functional model to describe the stochastic systems. This approach enables us to give a simple new proof for De Bruijn’s identity. We also derive a general expression for the gradient of the mutual information in systems with both additive and multiplicative noises. Systems with additive noise, as the most common model for the communication channels, is considered as well, where in the Gaussian case some of the available results in the literature are recovered. Moreover, the aforementioned general functional model allows us to study the sensitivity of the input–output mutual information to the system parameters in some other practical scenarios. Specifically, Orthogonal Frequency Division Multiplexing (OFDM) systems with High Power Amplifier (HPA) are considered and an information theoretic analysis for the behavior of these systems is given.

Massive MIMO With Antenna Selection: Fundamental Limits and Applications

Antenna selection is an effective means to address the cost and complexity issues in massive MIMO systems. This paper studies the performance limits of massive MIMO systems under practical antenna selection algorithms. A massive MIMO system is considered in which the transmitter employs only a fixed-size subset of the available antennas with the strongest channel gains. For this setup, the input-output mutual information of the system is shown to be well-approximated by a normal random variable when the number of transmit antennas is large. The mean of this random variable grows proportional to the number of antennas and its variance vanishes in the large-system limit. This behavior of the mutual information generalizes the well-known channel hardening property in massive MIMO systems to the cases with antenna selection. Our investigations show that 90% of the ergodic rate achieved by full antenna selection can be achieved by selecting less than 30% of the transmit antennas. Using large-system analysis, we drive an analytical expression for the number of selected antennas that maximizes the energy efficiency. This number is also derived for the case where a certain fraction of the totally achievable rate is aimed to be achieved. Our numerical investigations demonstrate a close match between the analytical and simulation results even for scenarios with not-so-large dimensions.

Probabilistic MIMO Symbol Detection With Expectation Consistency Approximate Inference

In this paper we explore low-complexity probabilistic algorithms for soft symbol detection in high-dimensional multiple-input multiple-output (MIMO) systems. We present a novel algorithm based on the Expectation Consistency (EC) framework, which describes the approximate inference problem as an optimization over a non-convex function. EC generalizes algorithms such as Belief Propagation and Expectation Propagation. For the MIMO symbol detection problem, we discuss feasible methods to find stationary points of the EC function and explore their tradeoffs between accuracy and speed of convergence. The accuracy is studied, first in terms of input-output mutual information and show that the proposed EC MIMO detector greatly improves state-of-the-art methods, with a complexity order cubic in the number of transmitting antennas. Second, these gains are corroborated by combining the probabilistic output of the EC detector with a low-density parity-check (LDPC) channel code.

IT Formulae for Gamma Target: Mutual Information and Relative Entropy

In this paper, we introduce new Stein identities for gamma target distribution as well as a new non-linear channel specifically designed for gamma inputs. From these two ingredients, we derive an explicit and simple formula for the derivative of the input-output mutual information of this non-linear channel with respect to the channel quality parameter. This relation is reminiscent of the well-known link between the derivative of the input-output mutual information of additive Gaussian noise channel with respect to the signal-to-noise ratio and the minimum mean-square error. The proof relies on a rescaled version of De Bruijn identity for gamma target distribution together with a stochastic representation for the gamma specific Fisher information. Finally, we are able to derive precise bounds and asymptotics for the input-output mutual information of the non-linear channel with gamma inputs.

Linear Precoder Design for SWIPT in MIMO Broadcasting Systems With Discrete Input Signals: Manifold Optimization Approach

In this paper, we investigate the design of linear precoders for simultaneously wireless information and power transfer (SWIPT) in a multi-input multi-output (MIMO) broadcasting system with discrete input signals. The considered system model consists of one base station (BS), one information receiver (IR), and one energy receiver (ER). The design objective is to maximize the input–output mutual information of the IR subject to the power constraint and the harvested energy requirement for the ER. We derive the structure of the optimal linear precoder by using manifold optimization, and propose an algorithm to find the optimal precoder. Simulation results show that the proposed algorithm can achieve better performance than the time sharing scheme and the Gaussian optimal precoder when Gaussian inputs are replaced by discrete input signals.

Reservoir Computing with Both Neuronal Intrinsic Plasticity and Multi-Clustered Structure

In the echo state networks, both reservoir states and network structure are essential for the performance of reservoir computing. In neuroscience, it has been confirmed that a single neuron can adaptively change its intrinsic excitability to fit various synaptic inputs. This mechanism is called intrinsic plasticity (IP) mechanism in the literature. This adaptive adjustment of neuronal response to external inputs is believed to maximize input-output mutual information. Meanwhile, the existence of multi-clustered structure with small-world-like property in the brain has been strongly supported by many neurophysiological experiments. Thus, it is advisable to consider both the intrinsic plasticity and multi-clustered structure of a reservoir network, rather than a random network with a non-adaptive reservoir response. In this paper, reservoir models with neuronal intrinsic plasticity and multi-clustered structure are investigated. The effects of two types of IP rules on the performance of several computational tasks have been investigated in detail by combining neuronal IP with multi-clustered reservoir structures. The first type is the Triesch’s IP rule, which drives the output activities of neurons to approximate exponential distributions; another is the Li’s IP rule, which generates a Gaussian distribution of neuronal firing. Results show that both the multi-clustered structures and IP rules can improve the computational accuracy of reservoir computing. However, before the application of the IP rules, the enhancement of computational performance for multi-clustered reservoirs is minor. Both IP rules contribute to improvement of the computational performance, where the Li’s IP rule is more advantageous than the Triesch’s IP. The results indicate that the combination of multi-clustered reservoir structures and IP learning can increase the dynamic diversity of reservoir states, especially for the IP’s learning. The adaptive tuning of reservoir states based on IP improves the dynamic complexity of neuronal activity, which helps train output weights. This biologically inspired reservoir model may give insights for the optimization of reservoir computing.

Free Deterministic Equivalents for the Analysis of MIMO Multiple Access Channel

In this paper, a free deterministic equivalent is proposed for the capacity analysis of the multi-input multi-output (MIMO) multiple access channel (MAC) with a more general channel model compared to previous works. In particular, a MIMO MAC with one base station (BS) equipped with several distributed antenna sets is considered. Each link between a user and a BS antenna set forms a jointly correlated Rician fading channel. The analysis is based on operator-valued free probability theory, which broadens the range of applicability of free probability techniques tremendously. By replacing independent Gaussian random matrices with operator-valued random variables satisfying certain operator-valued freeness relations, the free deterministic equivalent of the considered channel Gram matrix is obtained. The Shannon transform of the free deterministic equivalent is derived, which provides an approximate expression for the ergodic input-output mutual information of the channel. The sum-rate capacity achieving input covariance matrices are also derived based on the approximate ergodic input-output mutual information. The free deterministic equivalent results are easy to compute, and simulation results show that these approximations are numerically accurate and computationally efficient.

Linear Precoding Gain for Large MIMO Configurations with QAM and Reduced Complexity

In this paper, the problem of designing a linear precoder for multiple-input multiple-output (MIMO) systems in conjunction with quadrature amplitude modulation (QAM) is addressed. First, a novel and efficient methodology to evaluate the input–output mutual information for a general MIMO system as well as its corresponding gradients is presented, based on the Gauss–Hermite quadrature rule. Then, the method is exploited in a block coordinate gradient ascent optimization process to determine the globally optimal linear precoder with respect to the MIMO input–output mutual information for QAM systems with relatively moderate MIMO channel sizes. The proposed methodology is next applied in conjunction with the complexity-reducing per-group processing technique to both perfect channel state information (CSI) at the transmitter as well as statistical CSI (Statistical CSI) scenarios, with large transmitting and receiving antenna sizes, and for constellation size up to $M=64$ . We show by the numerical results that the precoders developed offer significantly better performance than the configuration with no precoder as well as the maximum diversity precoder for QAM with constellation sizes $M=16$ , 32, and 64 and for MIMO channel size up to $100\times 100$ .

Lower Bounds and Approximations for the Information Rate of the ISI Channel

We consider the discrete-time intersymbol interference (ISI) channel model, with additive Gaussian noise and fixed independent identically distributed inputs. In this setting, we investigate the expression put forth by Shamai and Laroia as a conjectured lower bound for the input–output mutual information after application of a minimum mean-square error decision-feedback equalizer receiver. A low-signal to noise ratio (SNR) expansion is used to prove that the conjectured bound does not hold under general conditions, and to characterize inputs for which it is particularly ill-suited. One such input is used to construct a counterexample, indicating that the Shamai–Laroia expression does not always bound even the achievable rate of the channel, thus excluding a natural relaxation of the original conjectured bound. However, this relaxed bound is then shown to hold for any finite entropy input and ISI channel, when the SNR is sufficiently high. We derive two conditions under which the relaxed bound holds, involving compound channel capacity and quasiconvexity arguments. Finally, new simple bounds for the achievable rate are proven, and compared with other known bounds. Information-estimation relations and estimation–theoretic bounds play a key role in establishing our results.

Performance Analysis of Cognitive Radio Systems With Imperfect Channel Sensing and Estimation

In cognitive radio systems, employing sensing-based spectrum access strategies, secondary users are required to perform channel sensing to detect the activities of licensed primary users in a channel, and in realistic scenarios, channel sensing occurs with possible errors due to miss-detections and false alarms. As another challenge, time-varying fading conditions in the channel between the secondary transmitter and the secondary receiver have to be learned via channel estimation. In this paper, performance of causal channel estimation methods in correlated cognitive radio channels under imperfect channel sensing results is analyzed, and achievable rates for reliable communication under both channel and sensing uncertainty are investigated by considering the input-output mutual information. Initially, cognitive radio channel model with channel sensing error and channel estimation is described. Then, using pilot symbols, minimum mean square error (MMSE) and linear-MMSE (L-MMSE) estimation methods are employed at the secondary receiver to learn the channel fading coefficients. Expressions for the channel estimates and mean-squared errors (MSE) are determined, and their dependencies on channel sensing results, and pilot symbol period and energy are investigated. Since sensing uncertainty leads to uncertainty in the variance of the additive disturbance, channel estimation strategies and performance are interestingly shown to depend on the sensing reliability. It is further shown that the L-MMSE estimation method, which is in general suboptimal, performs very close to MMSE estimation. Furthermore, assuming the channel estimation errors and the interference introduced by the primary users as zero-mean and Gaussian distributed, achievable rate expressions of linear modulation schemes and Gaussian signaling are determined. Subsequently, the training period, and data and pilot symbol energy allocations are jointly optimized to maximize the achievable rates for both signaling schemes.

Steepest-Descent Optimization of CDMA Signatures for Multiple Correlated Sources With Applications to Joint Source-Channel Coding

An approach to transmitting multiple correlated sources over a Gaussian multiple-access channel (GMAC) using code-division multiple-access (CDMA) is investigated. A block coordinate descent (BCD) algorithm is presented which can be used to optimize CDMA signatures for a given source covariance matrix under individual transmitter power constraints. This algorithm can be used to design non-orthogonal CDMA signatures for GMACs with arbitrary input alphabets, fading, and colored noise. While the main focus has been the design of signatures which minimize the mean square error (MSE) of linear decoding, the algorithm can also be used to design signatures which maximize the input-output mutual information of the channel. This paper investigates the uncoded transmission of Gaussian sources and the coded transmission of binary sources over a GMAC using joint source-channel (JSC) optimized CDMA signatures under constraints on transmitter power and channel bandwidth, scenarios relevant to wireless sensor networks. By using simple coding schemes which only rely on linear multi-user detection and independent channel decoding of a large number of sources, it is demonstrated that CDMA-based JSC coding can exploit inter-source correlation to achieve lower distortion (Gaussian sources) or higher rates (binary sources) compared to conventional separate source-channel (SSC) coding over the same channel bandwidth.

Information-Theoretic Modeling of Trichromacy Coding of Light Spectrum

Trichromacy is the representation of a light spectrum by three scalar coordinates. Such representation is universally implemented by the human visual system and by RGB (Red Green Blue) cameras. We propose here an informational model for trichromacy. Based on a statistical analysis of the dynamics of individual photons, the model demonstrates a possibility for describing trichromacy as an information channel, for which the input–output mutual information can be computed to serve as a measure of performance. The capabilities and significance of the informational model are illustrated and motivated in various situations. The model especially enables an assessment of the influence of the spectral sensitivities of the three types of photodetectors realizing the trichromatic representation. It provides a criterion to optimize possibly adjustable parameters of the spectral sensitivities such as their center wavelength, spectral width or magnitude. The model shows, for instance, the usefulness of some overlap with smooth graded spectral sensitivities, as observed for instance in the human retina. The approach also, starting from hyperspectral images with high spectral resolution measured in the laboratory, can be used to devise low-cost trichromatic imaging systems optimized for observation of specific spectral signatures. This is illustrated with an example from plant science, and demonstrates a potential of application especially to life sciences. The approach particularizes connections between physics, biophysics and information theory.

Spurious correlation as an approximation of the mutual information between redundant outputs and an unknown input

Stochastic resonance (SR) is a counterintuitive phenomenon, observed in a wide variety of nonlinear systems, for which the addition of noise of opportune magnitude can improve signal detection. Tuning the noise for maximizing the SR effect is important both for artificial and biological systems. In the case of artificial systems, full exploitation of the SR effect opens the possibility of measuring otherwise unmeasurable signals. In biology, identification of possible SR maximization mechanisms is of great interest for explaining the low-energy high-sensitivity perception capabilities often observed in animals. SR maximization approaches presented in literature use knowledge on the input signal (or stimulus, in the case of living beings), and maximize the mutual information between the input and the output signal. The input signal, however, is unknown in many practical settings. To cope with this problem, this paper introduces an approximation of the input–output mutual information based on the spurious correlation among a set of redundant units. A proof of the approximation, as well as numerical examples of its application are given.

Likelihood Ratios and Inference for Poisson Channels

In recent years, infinite-dimensional methods have been introduced for Gaussian channel estimation. The aim of this paper is to study the application of similar methods to Poisson channels. In particular, we compute the noncausal conditional mean estimator of a Poisson channel using the likelihood ratio and the discrete Malliavin gradient. This algorithm is suitable for numerical implementation via the Monte-Carlo scheme. As an application, we provide a new proof of a very deep and remarkable formula in Information Theory obtained recently in the literature and relating the derivatives of the input-output mutual information of a general Poisson channel and the conditional mean estimator of the input regardless the distribution of the latter. The use of the aforementioned stochastic analysis techniques allows us to extend these results to more general channels such as mixed Gaussian-Poisson channels.

Support Recovery With Sparsely Sampled Free Random Matrices

Consider a Bernoulli-Gaussian complex <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> -vector whose components are <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> = <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> , with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> ~ <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C N</i> (0, <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</sub> ) and binary <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> mutually independent and iid across <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> . This random <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i> -sparse vector is multiplied by a square random matrix <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</b> , and a randomly chosen subset, of average size <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n p</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> ∈ [0,1], of the resulting vector components is then observed in additive Gaussian noise. We extend the scope of conventional noisy compressive sampling models where <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</b> is typically a matrix with iid components, to allow <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</b> satisfying a certain freeness condition. This class of matrices encompasses Haar matrices and other unitarily invariant matrices. We use the replica method and the decoupling principle of Guo and Verdú, as well as a number of information-theoretic bounds, to study the input-output mutual information and the support recovery error rate in the limit of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> → ∞. We also extend the scope of the large deviation approach of Rangan and characterize the performance of a class of estimators encompassing thresholded linear MMSE and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> relaxation.

Sequential Linear MIMO Precoder Optimization for Hybrid ARQ Retransmission of QAM Signals

Our investigation centers on the issue of linear precoding for hybrid-ARQ retransmissions in linear multi-input-multi-output (MIMO) channels driven by non-Gaussian inputs. Leveraging recent works on single transmission MIMO precoding , we derive the gradient of input-output mutual information with respect to precoder matrix for optimization. We further establish that the right eigenvectors of optimum precoder are determined by the associated minimum mean-square error (MMSE) matrix. We then propose a gradient algorithm for maximizing the mutual information. Compared with precoders designed for Gaussian input, our new precoders can significantly improve the input-output mutual information, especially at low signal-to-noise ratio (SNR) and for lower dimensional constellations.