Ensemble Of Random Codes Research Articles

A Universal Random Coding Ensemble for Sample-Wise Lossy Compression.

We propose a universal ensemble for the random selection of rate-distortion codes which is asymptotically optimal in a sample-wise sense. According to this ensemble, each reproduction vector, x^, is selected independently at random under the probability distribution that is proportional to 2-LZ(x^), where LZ(x^) is the code length of x^ pertaining to the 1978 version of the Lempel-Ziv (LZ) algorithm. We show that, with high probability, the resulting codebook gives rise to an asymptotically optimal variable-rate lossy compression scheme under an arbitrary distortion measure, in the sense that a matching converse theorem also holds. According to the converse theorem, even if the decoder knew the ℓ-th order type of source vector in advance (ℓ being a large but fixed positive integer), the performance of the above-mentioned code could not have been improved essentially for the vast majority of codewords pertaining to source vectors in the same type. Finally, we present a discussion of our results, which includes among other things, a clear indication that our coding scheme outperforms the one that selects the reproduction vector with the shortest LZ code length among all vectors that are within the allowed distortion from the source vector.

Nonbinary LDPC Coded QAM Signals With Optimized Mapping: Bounds and Simulation Results

This paper studies specific properties of nonbinary low-density parity-check (NB LDPC) codes when used in coded modulation systems. The paper is focused on the practically important NB LDPC codes over the Galois extension fields GF(2m) with m ≤ 6 used with QAM signaling. Performance of NB LDPC coded transmission strongly depends on the mapping of nonbinary symbols to signal constellation points. We obtain a random coding bound on the maximum-likelihood decoding error probability for an ensemble of random irregular NB LDPC codes used with QAM signaling for specific symbol-to-signal point mappings. This bound is based on the ensemble average squared Euclidean distance spectra derived for these mappings. The simulation results for the belief-propagation decoding in the coded modulation schemes with the NB quasi-cyclic (QC)-LDPC codes under different mappings are given. Comparisons with the optimized binary QC-LDPC codes in the WiFi and 5G standards, as well as with the new bound, are performed.

Design of Polar Codes and PAC Codes for SCL Decoding

The performance of a code under the maximum-likelihood (ML) decoder highly depends on the weight enumerating function (WEF). However, how to compute efficiently the WEF of a polar code or a polarization-adjusted convolutional (PAC) code is still an open problem. For the design of stand-alone polar codes, we consider enumerating the number of minimum-weight non-zero codewords of the polar code. The block error rate (BLER) under the ML decoder can be approximated as a function of the minimum weight of non-zero codewords and its multiplicity. On the other hand, for the design of PAC codes, we consider enumerating the WEF averaged over the ensemble of random PAC codes. The ML-BLER upper bound can be represented as a function of the WEF. The bit-channel selection algorithms for polar codes and PAC codes are proposed, which take both utilization of the polarization effect and the ML decoding performance as selection criteria. Simulation results show that the proposed stand-alone polar codes are competitive when the block length gets larger. Also, simulation results show that the proposed PAC codes yield excellent performance for a wide range of code rates and block lengths and outperform the 5G polar codes.

Multi-Class Cost-Constrained Random Coding for Correlated Sources over the Multiple-Access Channel.

This paper studies a generalized version of multi-class cost-constrained random-coding ensemble with multiple auxiliary costs for the transmission of N correlated sources over an N-user multiple-access channel. For each user, the set of messages is partitioned into classes and codebooks are generated according to a distribution depending on the class index of the source message and under the constraint that the codewords satisfy a set of cost functions. Proper choices of the cost functions recover different coding schemes including message-dependent and message-independent versions of independent and identically distributed, independent conditionally distributed, constant-composition and conditional constant composition ensembles. The transmissibility region of the scheme is related to the Cover-El Gamal-Salehi region. A related family of correlated-source Gallager source exponent functions is also studied. The achievable exponents are compared for correlated and independent sources, both numerically and analytically.

On Approximation, Bounding & Exact Calculation of Average Block Error Probability for Random Code Ensembles

This paper presents a method to calculate the exact average block error probability of some random code ensembles under maximum-likelihood decoding. The proposed method is applicable to various channels and ensembles. The focus is on both spherical and Gaussian random codes on the additive white Gaussian noise channel as well as binary random codes on both the binary symmetric channel and the binary erasure channel. While for the uniform spherical ensemble Shannon, in 1959, argued with solid angles in N-dimensional space, the presented approach projects the problem into two dimensions and applies standard trigonometry. This simplifies the derivation and also allows for the analysis of the independent identically distributed (i.i.d.) Gaussian ensemble which turns out to perform better for short blocklengths and high rates. Moreover, a new lower bound on the average block error probability of the uniform spherical ensemble is found. For codes with more than three codewords, it is tighter than the sphere packing bound, but requires exactly the same computing effort. Furthermore, tight approximations are proposed to simplify the computation of both the exact average error probability and the two bounds. For the binary symmetric channel and the binary erasure channel, bounds on the average block error probability for i.i.d. random coding are derived and compared to the exact calculations.

Trade-offs between Error Exponents and Excess-Rate Exponents of Typical Slepian-Wolf Codes.

Typical random codes (TRCs) in a communication scenario of source coding with side information in the decoder is the main subject of this work. We study the semi-deterministic code ensemble, which is a certain variant of the ordinary random binning code ensemble. In this code ensemble, the relatively small type classes of the source are deterministically partitioned into the available bins in a one-to-one manner. As a consequence, the error probability decreases dramatically. The random binning error exponent and the error exponent of the TRCs are derived and proved to be equal to one another in a few important special cases. We show that the performance under optimal decoding can be attained also by certain universal decoders, e.g., the stochastic likelihood decoder with an empirical entropy metric. Moreover, we discuss the trade-offs between the error exponent and the excess-rate exponent for the typical random semi-deterministic code and characterize its optimal rate function. We show that for any pair of correlated information sources, both error and excess-rate probabilities exponential vanish when the blocklength tends to infinity.

Compute-Forward for DMCs: Simultaneous Decoding of Multiple Combinations

Algebraic network information theory is an emerging facet of network information theory, studying the achievable rates of random code ensembles that have algebraic structure, such as random linear codes. A distinguishing feature is that linear combinations of codewords can sometimes be decoded more efficiently than codewords themselves. The present work further develops this framework by studying the simultaneous decoding of multiple messages. Specifically, consider a receiver in a multi-user network that wishes to decode several messages. Simultaneous joint typicality decoding is one of the most powerful techniques for determining the fundamental limits at which reliable decoding is possible. This technique has historically been used in conjunction with random i.i.d. codebooks to establish achievable rate regions for networks. Recently, it has been shown that, in certain scenarios, nested linear codebooks in conjunction with “single-user” or sequential decoding can yield better achievable rates. For instance, the compute–forward problem examines the scenario of recovering $L \le K$ linear combinations of transmitted codewords over a $K$ -user multiple-access channel (MAC), and it is well established that linear codebooks can yield higher rates. This paper develops bounds for simultaneous joint typicality decoding used in conjunction with nested linear codebooks, and applies them to obtain a larger achievable region for compute–forward over a $K$ -user discrete memoryless MAC. The key technical challenge is that competing codeword tuples that are linearly dependent on the true codeword tuple introduce statistical dependencies, which requires careful partitioning of the associated error events.

Error Correction Based on Partial Information

We consider the decoding of linear and array codes from errors when we are only allowed to download a part of the codeword. More specifically, suppose that we have encoded k data symbols using an (n, k) code with code length n and dimension k. During storage, some of the codeword coordinates might be corrupted by errors. We aim to recover the original data by reading the corrupted codeword with a limit on the transmission bandwidth, namely, we can only download an α proportion of the corrupted codeword. For a given α, our objective is to design a code and a decoding scheme such that we can recover the original data from the largest possible number of errors. A naive scheme is to read αn coordinates of the codeword. This method used in conjunction with MDS codes guarantees recovery from any ⌊(αn - k)/2⌋ errors. In this paper we show that we can instead download an α proportion from each of the codeword's coordinates. For a well-designed MDS code, this method can guarantee recovery from ⌊(n - k/α)/2⌋ errors, which is 1/α times more than the naive method, and is also the maximum number of errors that an (n, k) code can correct by downloading only an α proportion of the codeword. We present two families of such optimal constructions and decoding schemes of which one is based on Interleaved Reed-Solomon codes and the other on Folded Reed-Solomon codes. We further show that both code constructions attain asymptotically optimal list decoding radius when downloading only a part of the corrupted codeword. We also construct an ensemble of random codes that with high probability approaches the upper bound on the number of correctable errors when the decoder downloads an α proportion of the corrupted codeword.

The Bee-Identification Problem: Bounds on the Error Exponent

Consider the problem of identifying a massive number of bees, uniquely labeled with barcodes, using noisy measurements. We formally introduce this “bee-identification problem”, define its error exponent, and derive efficiently computable upper and lower bounds for this exponent. We show that joint decoding of barcodes provides a significantly better exponent compared to separate decoding followed by permutation inference. For low rates, we prove that the lower bound on the bee-identification exponent obtained using typical random codes (TRC) is strictly better than the corresponding bound obtained using a random code ensemble (RCE). Further, as the rate approaches zero, we prove that the upper bound on the bee-identification exponent meets the lower bound obtained using TRC with joint barcode decoding.

Arimoto–Rényi Conditional Entropy and Bayesian $M$ -Ary Hypothesis Testing

This paper gives upper and lower bounds on the minimum error probability of Bayesian $M$ -ary hypothesis testing in terms of the Arimoto–Renyi conditional entropy of an arbitrary order $\alpha $ . The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy ( $\alpha =1$ ) is demonstrated. In particular, in the case where $M$ is finite, we show how to generalize Fano’s inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano’s inequality, allowing $M$ to be infinite, a lower bound on the Arimoto–Renyi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto–Renyi conditional entropy for both positive and negative $\alpha $ . Furthermore, we give upper bounds on the minimum error probability as functions of the Renyi divergence. In the setup of discrete memoryless channels, we analyze the exponentially vanishing decay of the Arimoto–Renyi conditional entropy of the transmitted codeword given the channel output when averaged over a random-coding ensemble.

Exact Random Coding Secrecy Exponents for the Wiretap Channel

We analyze the exact exponential decay rate of the expected amount of information leaked to the wiretapper in Wyner's wiretap channel setting using wiretap channel codes constructed from both i.i.d. and constant-composition random codes. Our analysis for those sampled from i.i.d. random coding ensemble shows that the previously-known achievable secrecy exponent using this ensemble is indeed the exact exponent for an average code in the ensemble. Furthermore, our analysis on wiretap channel codes constructed from the ensemble of constant-composition random codes leads to an exponent which, in addition to being the exact exponent for an average code, is larger than the achievable secrecy exponent that has been established so far in the literature for this ensemble (which in turn was known to be smaller than that achievable by wiretap channel codes sampled from i.i.d. random coding ensemble). We show examples where the exact secrecy exponent for the wiretap channel codes constructed from random constant-composition codes is larger than that of those constructed from i.i.d. random codes and examples where the exact secrecy exponent for the wiretap channel codes constructed from i.i.d. random codes is larger than that of those constructed from constant-composition random codes. We, hence, conclude that, unlike the error correction problem, there is no general ordering between the two random coding ensembles in terms of their secrecy exponent.

Linear Time-Invariant Anytime Codes for Control Over Noisy Channels

The problem of stabilizing an unstable plant over a noisy communication link is an increasingly important one that arises in problems of distributed control and networked control systems. Although the work of Schulman, and Sahai and Mitter over the past two decades, and their development of the notions of “tree codes” and “anytime capacity” respectively, provides the theoretical framework for studying such problems, there has been scant practical progress in this area because explicit constructions of tree codes with efficient encoding and decoding did not exist. To stabilize an unstable plant driven by bounded noise over a noisy channel one often needs real-time encoding and real-time decoding and a reliability which increases exponentially with delay, which is what tree codes guarantee. We propose an ensemble of random causal linear codes with a time invariant structure and show that they are tree codes with probability one. For erasure channels, we show that the average complexity of maximum likelihood decoding is bounded by a constant for all time if the code rate is smaller than the computational cutoff rate. For rates larger than the computational cutoff rate, we present an alternate way to perform maximum likelihood decoding with a complexity that grows linearly with time. We give novel sufficient conditions on the rate and reliability required of the tree codes to stabilize vector plants and argue that they are asymptotically tight.

On the Rényi Divergence, Joint Range of Relative Entropies, and a Channel Coding Theorem

This paper starts by considering the minimization of the Rényi divergence subject to a constraint on the total variation distance. Based on the solution of this optimization problem, the exact locus of the points (D(QIIP <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> ), D(QIIP <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> )) is determined when P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> , and Q are arbitrary probability measures which are mutually absolutely continuous, and the total variation distance between P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> is not below a given value. It is further shown that all the points of this convex region are attained by probability measures which are defined on a binary alphabet. This characterization yields a geometric interpretation of the minimal Chernoff information subject to a constraint on the variational distance. This paper also derives an exponential upper bound on the performance of binary linear block codes (or code ensembles) under maximum-likelihood decoding. Its derivation relies on the Gallager bounding technique, and it reproduces the Shulman-Feder bound as a special case. The bound is expressed in terms of the Rényi divergence from the normalized distance spectrum of the code (or the average distance spectrum of the ensemble) to the binomially distributed distance spectrum of the capacity-achieving ensemble of random block codes. This exponential bound provides a quantitative measure of the degradation in performance of binary linear block codes (or code ensembles) as a function of the deviation of their distance spectra from the binomial distribution. An efficient use of this bound is considered.

The Random Coding Bound Is Tight for the Average Linear Code or Lattice

In 1973, Gallager proved that the random-coding bound is exponentially tight for the random code ensemble at all rates, even below expurgation. This result explained that the random-coding exponent does not achieve the expurgation exponent due to the properties of the random ensemble, irrespective of the utilized bounding technique. It has been conjectured that this same behavior holds true for a random ensemble of linear codes. This conjecture is proved in this paper. In addition, it is shown that this property extends to Poltyrev’s random-coding exponent for a random ensemble of lattices.

Optimal Achievable Rates for Interference Networks With Random Codes

The optimal rate region for interference networks is characterized when encoding is restricted to random code ensembles with superposition coding and time sharing. A simple simultaneous nonunique decoding rule, under which each receiver decodes for the intended message as well as the interfering messages, is shown to achieve this optimal rate region regardless of the relative strengths of signal, interference, and noise. This result implies that the Han-Kobayashi bound, the best known inner bound on the capacity region of the two-user-pair interference channel, cannot be improved merely by using the optimal maximum likelihood decoder.

Expurgated Random-Coding Ensembles: Exponents, Refinements, and Connections

This paper studies expurgated random-coding bounds and exponents for channel coding with a given (possibly suboptimal) decoding rule. Variations of Gallager's analysis are presented, yielding several asymptotic and non-asymptotic bounds on the error probability for an arbitrary codeword distribution. A simple non-asymptotic bound is shown to attain an exponent of Csisz\'ar and K\"orner under constant-composition coding. Using Lagrange duality, this exponent is expressed in several forms, one of which is shown to permit a direct derivation via cost-constrained coding which extends to infinite and continuous alphabets. The method of type class enumeration is studied, and it is shown that this approach can yield improved exponents and better tightness guarantees for some codeword distributions. A generalization of this approach is shown to provide a multi-letter exponent which extends immediately to channels with memory. Finally, a refined analysis expurgated i.i.d. random coding is shown to yield a O\big(\frac{1}{\sqrt{n}}\big) prefactor, thus improving on the standard O(1) prefactor. Moreover, the implied constant is explicitly characterized.

Mismatched Decoding: Error Exponents, Second-Order Rates and Saddlepoint Approximations

This paper considers the problem of channel coding with a given (possibly suboptimal) maximum-metric decoding rule. A cost-constrained random-coding ensemble with multiple auxiliary costs is introduced, and is shown to achieve error exponents and second-order coding rates matching those of constant-composition random coding, while being directly applicable to channels with infinite or continuous alphabets. The number of auxiliary costs required to match the error exponents and second-order rates of constant-composition coding is studied, and is shown to be at most two. For independent identically distributed random coding, asymptotic estimates of two well-known non-asymptotic bounds are given using saddlepoint approximations. Each expression is shown to characterize the asymptotic behavior of the corresponding random-coding bound at both fixed and varying rates, thus unifying the regimes characterized by error exponents, second-order rates, and moderate deviations. For fixed rates, novel exact asymptotics expressions are obtained to within a multiplicative 1+o(1) term. Using numerical examples, it is shown that the saddlepoint approximations are highly accurate even at short block lengths.

Statistical mechanical analysis of a hierarchical random code ensemble in signal processing

We study a random code ensemble with a hierarchical structure, which is closely related to the generalized random energy model with discrete energy values. Based on this correspondence, we analyze the hierarchical random code ensemble by using the replica method in two situations: lossy data compression and channel coding. For both the situations, the exponents of large deviation analysis characterizing the performance of the ensemble, the distortion rate of lossy data compression and the error exponent of channel coding in Gallager's formalism, are accessible by a generating function of the generalized random energy model. We discuss that the transitions of those exponents observed in the preceding work can be interpreted as phase transitions with respect to the replica number. We also show that the replica symmetry breaking plays an essential role in these transitions.

Asymptotic estimation of the fraction of errors correctable by q-ary LDPC codes

We consider an ensemble of random q-ary LDPC codes. As constituent codes, we use q-ary single-parity-check codes with d = 2 and Reed-Solomon codes with d = 3. We propose a hard-decision iterative decoding algorithm with the number of iterations of the order of the logarithm of the code length. We show that under this decoding algorithm there are codes in the ensemble with the number of correctable errors linearly growing with the code length. We weaken a condition on the vertex expansion of the Tanner graph corresponding to the code.

Linear Precoder Design Through Cut-Off Rate Maximization in MIMO-OFDM Coded Systems With Imperfect CSIT

This paper proposes a linear transmitter design that aims at minimizing the packet error rate (PER) using partial channel state information at the transmitter (CSIT). The design is based on the maximization of the channel cut-off rate, which provides an upper bound on the codeword error probability over the ensemble of random codes. The advantage of this optimization criterion is that it allows to get a low PER while keeping the design independent of the specific channel code used in the system. The algorithm is derived for a multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) system, and adopts a Bayesian approach to design an algorithm robust to CSI errors. The proposed algorithm converges to the open-loop transmission scheme for the case of no channel knowledge and it converges to the closed-loop design with perfect CSI for the case of no uncertainty. Simulation results evidence the performance gain in terms of PER and throughput of the proposed algorithm when compared to more commonly used optimization criteria.