Memoryless Symmetric Channels Research Articles

Linear Programming-Based Converses for Finite Blocklength Lossy Joint Source-Channel Coding

A linear programming (LP)-based framework is presented for obtaining converses for finite blocklength lossy joint source-channel coding problems. The framework applies for any loss criterion, generalizes certain previously known converses, and also extends to multi-terminal settings. The finite blocklength problem is posed equivalently as a nonconvex optimization problem and using a lift-and-project-like method, a close but tractable LP relaxation of this problem is derived. Lower bounds on the original problem are obtained by the construction of feasible points for the dual of the LP relaxation. A particular application of this approach leads to new converses, which recover and improve on the converses of Kostina and Verdu for finite blocklength lossy joint source-channel coding and lossy source coding. For finite blocklength channel coding, the LP relaxation recovers the converse of Polyanskiy, Poor and Verdu and leads to a new improvement on the converse of Wolfowitz, showing thereby that our LP relaxation is asymptotically tight with increasing blocklengths for channel coding, lossless source coding, and joint source-channel coding with the excess distortion probability as the loss criterion. Using a duality-based argument, a new converse is derived for finite blocklength joint source-channel coding for a class of source-channel pairs. Employing this converse, the LP relaxation is also shown to be tight for all blocklengths for the minimization of the expected average symbolwise Hamming distortion of a $q$ -ary uniform source over a $q$ -ary symmetric memoryless channel for any $q\in {\mathbb {N}}$ . The optimization formulation and the lift-and-project method are extended to networked settings and demonstrated by obtaining an improvement on a converse of Zhou et al. for the successive refinement problem for successively refinable source-distortion measure triplets.

Improving on the Cut-Set Bound via Geometric Analysis of Typical Sets

We consider the discrete memoryless symmetric primitive relay channel, where, a source $X$ wants to send information to a destination $Y$ with the help of a relay $Z$ and the relay can communicate to the destination via an error-free digital link of rate $R_{0}$ , while $Y$ and $Z$ are conditionally independent and identically distributed given $X$ . We develop two new upper bounds on the capacity of this channel that are tighter than existing bounds, including the celebrated cut-set bound. Our approach significantly deviates from the standard information-theoretic approach for proving upper bounds on the capacity of multi-user channels. We build on the blowing-up lemma to analyze the probabilistic geometric relations between the typical sets of the $n$ -letter random variables associated with a reliable code for communicating over this channel. These relations translate to new entropy inequalities between the $n$ -letter random variables involved. As an application of our bounds, we study an open question posed by (Cover, 1987), namely, what is the minimum rate $R_{0}^{*}$ needed for the $Z$ – $Y$ link in order for the capacity of the relay channel to be equal to that of the broadcast cut. We consider the special case when the $X$ – $Y$ and $X$ – $Z$ links are both binary symmetric channels. Our tighter bounds on the capacity of the relay channel immediately translate to tighter lower bounds for $R_{0}^{*}$ . More interestingly, we show that when $p\to 1/2$ , $R_{0}^{*}\geq 0.1803$ ; even though the broadcast channel becomes completely noisy as $p\to 1/2$ and its capacity, and therefore the capacity of the relay channel, goes to zero, a strictly positive rate $R_{0}$ is required for the relay channel capacity to be equal to the broadcast bound. Existing upper bounds on the capacity of the relay channel, and the cut-set bound in particular, would rather imply $R_{0}^{*}\to 0$ , while achievability schemes require $R_{0}^{*}\to 1$ . We conjecture that $R_{0}^{*}\to 1$ as $p\to 1/2$ .

Polar Code Design for Irregular Multidimensional Constellations

Polar codes, ever since their introduction, have been shown to be very effective for various wireless communication channels. This, together with their relatively low implementation complexity, has made them an attractive coding scheme for wireless communications. Polar codes have been extensively studied for use with binary-input symmetric memoryless channels but little is known about their effectiveness in other channels. In this paper, a novel methodology for designing multilevel polar codes that works effectively with arbitrary multidimensional constellations is presented. In order for this multilevel design to function, a novel set merging algorithm, able to label such constellations, is proposed. We then compare the error rate performance of our design with that of existing schemes and show that we were able to obtain unprecedented results in many cases over the previously known best techniques at relatively low decoding complexity.

Unified Scaling of Polar Codes: Error Exponent, Scaling Exponent, Moderate Deviations, and Error Floors

Consider the transmission of a polar code of block length $N$ and rate $R$ over a binary memoryless symmetric channel $W$ and let $P_{\mathrm{ e}}$ be the block error probability under successive cancellation decoding. In this paper, we develop new bounds that characterize the relationship of the parameters $R$ , $N$ , $P_{\mathrm{ e}}$ , and the quality of the channel $W$ quantified by its capacity $I(W)$ and its Bhattacharyya parameter $Z(W)$ . In previous work, two main regimes were studied. In the error exponent regime, the channel $W$ and the rate $R are fixed, and it was proved that the error probability $P_{\mathrm{ e}}$ scales roughly as $2^{-\sqrt {N}}$ . In the scaling exponent approach, the channel $W$ and the error probability $P_{\mathrm{ e}}$ are fixed and it was proved that the gap to capacity $I(W)-R$ scales as $N^{-1/\mu }$ . Here, $\mu $ is called scaling exponent and this scaling exponent depends on the channel $W$ . A heuristic computation for the binary erasure channel (BEC) gives $\mu =3.627$ and it was shown that, for any channel $W$ , $3.579 \le \mu \le 5.702$ . Our contributions are as follows. First, we provide the tighter upper bound $\mu \le 4.714$ valid for any $W$ . With the same technique, we obtain the upper bound $\mu \le 3.639$ for the case of the BEC; this upper bound approaches very closely the heuristically derived value for the scaling exponent of the erasure channel. Second, we develop a trade-off between the gap to capacity $I(W)-R$ and the error probability $P_{\mathrm{ e}}$ as the functions of the block length $N$ . In other words, we neither fix the gap to capacity (error exponent regime) nor the error probability (scaling exponent regime), but we do consider a moderate deviations regime in which we study how fast both quantities, as the functions of the block length $N$ , simultaneously go to 0. Third, we prove that polar codes are not affected by error floors . To do so, we fix a polar code of block length $N$ and rate $R$ . Then, we vary the channel $W$ and study the impact of this variation on the error probability. We show that the error probability $P_{\mathrm{ e}}$ scales as the Bhattacharyya parameter $Z(W)$ raised to a power that scales roughly like ${\sqrt {N}}$ . This agrees with the scaling in the error exponent regime.

On the Scaling Exponent of Polar Codes for Binary-Input Energy-Harvesting Channels

This paper investigates the scaling exponent of polar codes for binary-input energy-harvesting (EH) channels with infinite-capacity batteries. The EH process is characterized by a sequence of i.i.d. random variables with finite variances. The scaling exponent $\mu$ of polar codes for a binary-input memoryless channel (BMC) characterizes the closest gap between the capacity and non-asymptotic rates achieved by polar codes with error probabilities no larger than some non-vanishing $\varepsilon\in(0,1)$. It has been shown that for any $\varepsilon\in(0,1)$, the scaling exponent $\mu$ for any binary-input memoryless symmetric channel (BMSC) with $I(q_{Y|X})\in(0,1)$ lies between 3.579 and 4.714 , where the upper bound $4.714$ was shown by an explicit construction of polar codes. Our main result shows that $4.714$ remains to be a valid upper bound on the scaling exponent for any binary-input EH channel, i.e., a BMC subject to additional EH constraints. Our result thus implies that the EH constraints do not worsen the rate of convergence to capacity if polar codes are employed. The main result is proved by leveraging the following three existing results: scaling exponent analyses for BMSCs, construction of polar codes designed for binary-input memoryless asymmetric channels, and the save-and-transmit strategy for EH channels.

From Polar to Reed-Muller Codes

The year 2016, in which I am writing these words, marks the centenary of Claude Shannon, the father of information theory. In his landmark 1948 paper Mathematical Theory of Communication, Shannon established the largest rate at which reliable communication is possible, and he referred to it as the channel capacity. Since then, researchers have focused on the design of practical coding schemes that could approach such a limit. The road to channel capacity has been almost 70 years long and, after many ideas, occasional detours, and some rediscoveries, it has culminated in the description of low-complexity and provably capacity-achieving coding schemes, namely, polar codes and iterative codes based on sparse graphs. However, next-generation communication systems require an unprecedented performance improvement and the number of transmission settings relevant in applications is rapidly increasing. Hence, although Shannon's limit seems finally close at hand, new challenges are just around the corner. In this thesis, we trace a road that goes from polar to Reed-Muller codes and, by doing so, we investigate three main topics: unified scaling, non-standard channels, and capacity via symmetry. First, we consider unified scaling. A coding scheme is capacity-achieving when, for any rate smaller than capacity, the error probability tends to 0 as the block length becomes increasingly larger. However, the practitioner is often interested in more specific questions such as, How much do we need to increase the block length in order to halve the gap between rate and capacity?. We focus our analysis on polar codes and develop a unified framework to rigorously analyze the scaling of the main parameters, i.e., block length, rate, error probability, and channel quality. Furthermore, in light of the recent success of a list decoding algorithm for polar codes, we provide scaling results on the performance of list decoders. Next, we deal with non-standard channels. When we say that a coding scheme achieves capacity, we typically consider binary memoryless symmetric channels. However, practical transmission scenarios often involve more complicated settings. For example, the downlink of a cellular system is modeled as a broadcast channel, and the communication on fiber links is inherently asymmetric. We propose provably optimal low-complexity solutions for these settings. In particular, we present a polar coding scheme that achieves the best known rate region for the broadcast channel, and we describe three paradigms to achieve the capacity of asymmetric channels. To do so, we develop general coding primitives, such as the chaining construction that has already proved to be useful in a variety of communication problems. Finally, we show how to achieve capacity via symmetry. In the early days of coding theory, a popular paradigm consisted in exploiting the structure of algebraic codes to devise practical decoding algorithms. However, proving the optimality of such coding schemes remained an elusive goal. In particular, the conjecture that Reed-Muller codes achieve capacity dates back to the 1960s. We solve this open problem by showing that Reed-Muller codes and, in general, codes with sufficient symmetry are capacity-achieving over erasure channels under optimal MAP decoding. As the proof does not rely on the precise structure of the codes, we are able to show that symmetry alone guarantees optimal performance.

Polar Codes: Speed of Polarization and Polynomial Gap to Capacity

We prove that, for all binary-input symmetric memoryless channels, polar codes enable reliable communication at rates within an additive gap e > 0 to the Shannon capacity with a block length, construction complexity, and decoding complexity, all bounded by a polynomial in 1/e. Polar coding gives the first known explicit construction with rigorous proofs of all these properties; previous constructions were not known to achieve capacity with less than exp(1/e) decoding complexity except for erasure channels. We establish the capacity-achieving property of polar codes via a direct analysis of the underlying martingale of conditional entropies, without relying on the martingale convergence theorem. This step gives rough polarization (noise levels ≈ e for the good channels), which can then be adequately amplified by tracking the decay of the channel Bhattacharyya parameters. Our effective bounds imply that polar codes can have block length (and encoding/decoding complexity) bounded by a polynomial in 1/e. The generator matrix of such polar codes can be constructed in polynomial time by algorithmically computing an adequate approximation of the polarization process.

Threshold Saturation for Spatially Coupled LDPC and LDGM Codes on BMS Channels

Spatially-coupled low-density parity-check (LDPC) codes, which were first introduced as LDPC convolutional codes, have been shown to exhibit excellent performance under low-complexity belief-propagation decoding. This phenomenon is now termed threshold saturation via spatial coupling. Spatially-coupled codes have been successfully applied in numerous areas. In particular, it was proven that spatially-coupled regular LDPC codes universally achieve capacity over the class of binary memoryless symmetric (BMS) channels under belief-propagation decoding. Recently, potential functions have been used to simplify threshold saturation proofs for scalar and vector recursions. In this paper, potential functions are used to prove threshold saturation for irregular LDPC and low-density generator-matrix (LDGM) codes on BMS channels, extending the simplified proof technique to BMS channels. The corresponding potential functions are closely related to the average Bethe free entropy of the ensembles in the large-system limit. These functions also appear in statistical physics when the replica method is used to analyze optimal decoding.

A Simple Proof of Maxwell Saturation for Coupled Scalar Recursions

Low-density parity-check (LDPC) convolutional codes (or spatially coupled codes) were recently shown to approach capacity on the binary erasure channel (BEC) and binary-input memoryless symmetric channels. The mechanism behind this spectacular performance is now called threshold saturation via spatial coupling. This new phenomenon is characterized by the belief-propagation threshold of the spatially coupled ensemble increasing to an intrinsic noise threshold defined by the uncoupled system. In this paper, we present a simple proof of threshold saturation that applies to a wide class of coupled scalar recursions. Our approach is based on constructing potential functions for both the coupled and uncoupled recursions. Our results actually show that the fixed point of the coupled recursion is essentially determined by the minimum of the uncoupled potential function and we refer to this phenomenon as Maxwell saturation. A variety of examples are considered including the density-evolution equations for: irregular LDPC codes on the BEC, irregular low-density generator-matrix codes on the BEC, a class of generalized LDPC codes with BCH component codes, the joint iterative decoding of LDPC codes on intersymbol-interference channels with erasure noise, and the compressed sensing of random vectors with independent identically distributed components.

Refinement of the DS2 Bound and its Extensions for Discrete Memoryless Symmetric Channels

Strengthened versions of the DS2 bound and its extensions for the transmission of linear codes over discrete memoryless output symmetric channels are presented. Probability likelihood ratios under the list decoding rule are approximated by sigmoid functions. The effect of the erroneous decoding region in the calculation of list error decoding probability is considerably reduced. Numerical results illustrate the new bounds.

Tree-Structured Expectation Propagation for LDPC Decoding over BMS Channels

In this paper, we put forward the tree-structured expectation propagation (TEP) algorithm for decoding block and convolutional low-density parity-check codes over any binary channel. We have already shown that TEP improves belief propagation (BP) over the binary erasure channel (BEC) by imposing marginal constraints over a set of pairs of variables that form a tree or a forest. The TEP decoder is a message-passing algorithm that sequentially builds a tree/forest of erased variables to capture additional information disregarded by the standard BP decoder, which leads to a noticeable reduction of the error rate for finite-length codes. In this paper, we show how the TEP can be extended to any channel, specifically to binary memoryless symmetric (BMS) channels. We particularly focus on how the TEP algorithm can be adapted for any channel model and, more importantly, how to choose the tree/forest to keep the gains observed for block and convolutional LDPC codes over the BEC.

Compute-and-Forward Strategies for Cooperative Distributed Antenna Systems

We study a distributed antenna system where L antenna terminals (ATs) are connected to a central processor (CP) via digital error-free links of finite capacity R0, and serve K user terminals (UTs). This model has been widely investigated both for the uplink (UTs to CP) and for the downlink (CP to UTs), which are instances of the general multiple-access relay and broadcast relay networks. We contribute to the subject in the following ways: 1) For the uplink, we consider the recently proposed “compute and forward” (CoF) approach and examine the corresponding system optimization at finite SNR. 2) For the downlink, we propose a novel precoding scheme nicknamed “reverse compute and forward” (RCoF). 3) In both cases, we present low-complexity versions of CoF and RCoF based on standard scalar quantization at the receivers, that lead to discrete-input discrete-output symmetric memoryless channel models for which near-optimal performance can be achieved by standard single-user linear coding. 4) We provide extensive numerical results and finite SNR comparison with other “state of the art” information theoretic techniques, in scenarios including fading and shadowing. The proposed uplink and downlink system optimization focuses specifically on the ATs and UTs selection problem. In both cases, for a given set of transmitters, the goal consists of selecting a subset of the receivers such that the corresponding system matrix has full rank and the sum rate is maximized. We present low-complexity ATs and UTs selection schemes and demonstrate through Monte Carlo simulation that the proposed schemes essentially eliminate the problem of rank deficiency of the system matrix and greatly mitigate the noninteger penalty affecting CoF/RCoF at high SNR. Comparison with other state-of-the art information theoretic schemes, show competitive performance of the proposed approaches with significantly lower complexity.

Bounds on Maximum Likelihood Decoding Performance for Linear Codes at Low Rates

For a given linear code (ensemble), upper bounds on the error probability under maximum likelihood decoding are investigated at low rates. A class of symmetric memoryless channels suitable for Bhattacharyya-type bounds, which are particularly of importance at low rates, is introduced. Over a symmetric channel, a lower bound on the error exponent is derived for a given linear code. A sufficient condition for achieving the expurgated exponent, which is the best among known error exponents at low rates, within a fixed discrepancy is given. Over a general discrete memoryless channel, the same analysis provides a lower bound on the average error exponent for an ensemble of coset codes generated by a given linear code. The bounding technique is extended to the case of generalized maximum likelihood decoding with erasure and list- decoding options.

Code Design for the Noisy Slepian-Wolf Problem

We consider a noisy Slepian-Wolf problem where two correlated sources are separately encoded (using codes of fixed rate) and transmitted over two independent binary memoryless symmetric channels. The capacity of each channel is characterized by a single parameter that is not known at the transmitter. System performance is evaluated by computing the set of channel parameters for which the system can successfully decode. This set is called the achievable channel parameter region (ACPR). The goal is to design systems whose ACPRs are as large as possible. The main result is the design of irregular low-density parity-check (LDPC) ensembles whose ACPRs are significantly larger than previous designs. Some previous attempts to achieve large ACPRs with LDPC codes failed because systematic codes were used. In this work, we start with systematic encoders but puncture all the systematic bits before transmission. We also show that additional gains are possible using a staggered structure which enables codes optimized for single-user channels to perform well under symmetric channel conditions. The main analysis tool is a generic density-evolution framework for the analysis of joint iterative decoding for this problem.

On Achieving an Asymptotically Error-Free Fixed-Point of Iterative Decoding for Perfect A Priori Information

In this paper we provide necessary and sufficient conditions for constituent codes in (multiple) concatenated and graph-based coding schemes to achieve an asymptotically error-free iterative decoding fixed-point if the maximum possible a priori information is available. At least one constituent code in an iterative decoding scheme must satisfy these conditions in order to ensure an asymptotically vanishing bit error probability at the convergence point of the decoder. Our results are proved for arbitrary binary-input symmetric memoryless channels (BISMCs) and thus can be universally applied to many transmission scenarios. Specifically, using a factor graph framework, it is shown that non-inner codes in a serial concatenation or check nodes in generalized LDPC codes achieve perfect extrinsic information if and only if the minimum Hamming distance between codewords is two or greater. For inner codes in a serial concatenation, constituent codes in a parallel concatenation, or variable nodes in doubly-generalized LDPC codes the corresponding encoder condition for acquiring perfect extrinsic information is an infinite codeword weight for a weight-one input sequence. For this case we provide a general proof which holds for all linear encoders and BISMCs. We also show that these results can improve the performance of concatenated coding schemes.

Extremes of Error Exponents

This paper determines the range of feasible values of standard error exponents for binary-input memoryless symmetric channels of fixed capacity C and shows that extremes are attained by the binary symmetric and the binary erasure channel. The proof technique also provides analogous extremes for other quantities related to Gallager's E0 function, such as the cutoff rate, the Bhattacharyya parameter, and the channel dispersion.

Bilayer LDPC Convolutional Codes for Decode-and-Forward Relaying

In this paper we present bilayer LDPC convolutional codes for half-duplex relay channels. Two types of codes, bilayer expurgated LDPC convolutional codes and bilayer lengthened LDPC convolutional codes, are proposed for decode-and-forward (DF) relaying. In the case of the binary erasure relay channel, we prove analytically that both code constructions achieve the capacities of the source-relay link and the source-destination link simultaneously, provided that the channel conditions are known when designing the codes. Meanwhile, both codes enable the highest transmission rate possible with DF relaying for a wide range of channel parameters. In addition, the regular degree distributions can easily be computed from the channel parameters, which significantly simplifies the code optimization. The code construction and performance analysis are extended to the general binary memoryless symmetric channel, where a capacity-achieving performance is conjectured. Numerical results are provided for both types of codes with finite node degrees over binary erasure channels and binary-input additive white Gaussian noise channels, which verify the aforementioned theoretical analysis.

Mathematical Programming Decoding of Binary Linear Codes: Theory and Algorithms

Mathematical programming is a branch of applied mathematics and has recently been used to derive new decoding approaches, challenging established but often heuristic algorithms based on iterative message passing. Concepts from mathematical programming used in the context of decoding include linear, integer, and nonlinear programming, network flows, notions of duality as well as matroid and polyhedral theory. This survey article reviews and categorizes decoding methods based on mathematical programming approaches for binary linear codes over binary-input memoryless symmetric channels.

Rate-Compatible LDPC Convolutional Codes Achieving the Capacity of the BEC

In this paper, we propose a new family of rate-compatible regular low-density parity-check (LDPC) convolutional codes. The construction is based on graph extension, i.e., the codes of lower rates are generated by successively extending the graph of the base code with the highest rate. Theoretically, the proposed rate-compatible family can cover all the rational rates from 0 to 1. In addition, the regularity of degree distributions simplifies the code optimization. We prove analytically that all the LDPC convolutional codes of different rates in the family are capable of achieving the capacity of the binary erasure channel (BEC). The analysis is extended to the general binary memoryless symmetric channel, for which a capacity-approaching performance can be achieved. Analytical thresholds and simulation results for finite check and variable node degrees are provided for both BECs and binary-input additive white Gaussian noise channels. The results confirm that the decoding thresholds of the rate-compatible codes approach the corresponding Shannon limits over both channels.

Analysis of Error Floors for Non-binary LDPC Codes over General Linear Group through q-Ary Memoryless Symmetric Channels

Analysis of Error Floors for Non-binary LDPC Codes over General Linear Group through <i>q</i>-Ary Memoryless Symmetric Channels