Correlated Side Information Research Articles

Sharp Second-Order Pointwise Asymptotics for Lossless Compression with Side Information.

The problem of determining the best achievable performance of arbitrary lossless compression algorithms is examined, when correlated side information is available at both the encoder and decoder. For arbitrary source-side information pairs, the conditional information density is shown to provide a sharp asymptotic lower bound for the description lengths achieved by an arbitrary sequence of compressors. This implies that for ergodic source-side information pairs, the conditional entropy rate is the best achievable asymptotic lower bound to the rate, not just in expectation but with probability one. Under appropriate mixing conditions, a central limit theorem and a law of the iterated logarithm are proved, describing the inevitable fluctuations of the second-order asymptotically best possible rate. An idealised version of Lempel-Ziv coding with side information is shown to be universally first- and second-order asymptotically optimal, under the same conditions. These results are in part based on a new almost-sure invariance principle for the conditional information density, which may be of independent interest.

Source Separation With Side Information Based on Gaussian Mixture Models With Application in Art Investigation

In this paper, we propose an algorithm for source separation with side information where one observes the linear superposition of two source signals plus two additional signals that are correlated with the mixed ones. Our algorithm is based on two ingredients: first, we learn a Gaussian mixture model (GMM) for the joint distribution of a source signal and the corresponding correlated side information signal; second, we separate the signals using standard computationally efficient conditional mean estimators. The paper also puts forth new recovery guarantees for this source separation algorithm. In particular, under the assumption that the signals can be perfectly described by a GMM model, we characterize necessary and sufficient conditions for reliable source separation in the asymptotic regime of low-noise as a function of the geometry of the underlying signals and their interaction. It is shown that if the subspaces spanned by the innovation components of the source signals with respect to the side information signals have zero intersection, provided that we observe a certain number of linear measurements from the mixture, then we can reliably separate the sources; otherwise we cannot. Our proposed framework – which provides a new way to incorporate side information to aid the solution of source separation problems where the decoder has access to linear projections of superimposed sources and side information – is also employed in a real-world art investigation application involving the separation of mixtures of X-ray images. The simulation results showcase the superiority of our algorithm against other state-of-the-art algorithms.

On the Rate-Distortion Function for Binary Source Coding With Side Information

We present an in-depth analysis of the problem of lossy compression of binary sources in the presence of correlated side information, where the correlation is given by a generic binary asymmetric channel and the Hamming distance is the distortion metric. Our analysis is motivated by systematic rate-distortion gains observed when applying asymmetric correlation models in Wyner-Ziv video coding. First, we derive for the first time the rate-distortion function for conventional predictive coding in the binary-asymmetric-correlation-channel scenario. Second, we propose a new bound for the case where the side information is only available at the decoder—Wyner-Ziv coding. We conjecture this bound to be tight. We show that the maximum rate needed to encode as well as the maximum rate-loss of Wyner-Ziv coding relative to predictive coding corresponds to uniform sources and symmetric correlations. Importantly, we show that the upper bound on the rate-loss established by Zamir is not tight and that the maximum value is actually significantly lower. Moreover, we prove that the only binary correlation channel that incurs no rate-loss for Wyner-Ziv coding compared with predictive coding is the Z-channel. Finally, we complement our analysis with new compression performance results obtained with our state-of-the-art Wyner-Ziv video coding system.

Distortion Exponent in MIMO Fading Channels With Time-Varying Source Side Information

Transmission of a Gaussian source over a time-varying multiple-input multiple-output (MIMO) channel is studied under strict delay constraints. Availability of a correlated side information at the receiver is assumed, whose quality, i.e., its correlation with the source signal, also varies over time. A block-fading model is considered for the states of the time-varying channel and side information; perfect state information at the receiver is assumed, while the transmitter knows only the statistics. The high signal to noise ratio performance, characterized by the distortion exponent, is studied for this joint source-channel coding problem. An upper bound is derived and compared with several lower bounds based on list decoding (LD), hybrid digital–analog transmission, as well as multi-layer schemes, which transmit successive refinements of the source, relying on progressive or superposition transmission with LD. The optimal distortion exponent is characterized for the single-input multiple-output and multiple-input single-output scenarios by showing that the distortion exponent achieved by multi-layer superposition encoding with joint decoding meets the proposed upper bound. In the MIMO scenario, the optimal distortion exponent is characterized in the low bandwidth ratio regime, and it is shown that the multi-layer superposition encoding performs very close to the upper bound in the high bandwidth ratio regime.

Joint Source-Channel Coding With Time-Varying Channel and Side-Information

Transmission of a Gaussian source over a time-varying Gaussian channel is studied in the presence of time-varying correlated side information at the receiver. A block fading model is considered for both the channel and the side information, whose states are assumed to be known only at the receiver. The optimality of separate source and channel coding in terms of average end-to-end distortion is shown when the channel is static, while the side information state follows a discrete or a continuous and quasiconcave distribution. When both the channel and side information states are time-varying, separate source and channel coding is suboptimal in general. A partially informed encoder lower bound is studied by providing the channel state information to the encoder. Several achievable transmission schemes are proposed based on uncoded transmission, separate source and channel coding, joint decoding, and hybrid digital–analog transmission. Uncoded transmission is shown to be optimal for a class of continuous and quasiconcave side information state distributions, while the channel gain may have an arbitrary distribution. To the best of our knowledge, this is the first example, in which the uncoded transmission achieves the optimal performance thanks to the time-varying nature of the states, while it is suboptimal in the static version of the same problem. Then, the optimal distortion exponent, which quantifies the exponential decay rate of the expected distortion in the high SNR regime, is characterized for Nakagami distributed channel and side information states, and it is shown to be achieved by hybrid digital–analog and joint decoding schemes in certain cases, illustrating the suboptimality of pure digital or analog transmission in general.

Slepian-Wolf Coding for Broadcasting With Cooperative Base-Stations

We propose a base-station (BS) cooperation model for broadcasting a discrete memoryless source in a cellular or heterogeneous network. The model allows the receivers to use helper BSs to improve network performance, and it permits the receivers to have prior side information about the source. We establish the model's information-theoretic limits in two operational modes: In Mode 1, the helper BSs are given information about the channel codeword transmitted by the main BS, and in Mode 2 they are provided correlated side information about the source. Optimal codes for Mode 1 use hash-and-forward coding at the helper BSs; while, in Mode 2, optimal codes use source codes from Wyner's helper source-coding problem at the helper BSs. We prove the optimality of both approaches by way of a new list-decoding generalisation used in Theorem 6 of Tuncel (2006), and in doing so, show an operational duality between Modes 1 and 2.

Source Coding Problems With Conditionally Less Noisy Side Information

A computable expression for Heegard and Berger's rate-distortion function has eluded information theory for nearly three decades. Heegard and Berger's single-letter achievability bound is well known to be optimal for physically degraded side information; however, it is not known whether the bound is optimal for arbitrarily correlated side information (general discrete memoryless sources). In this paper, we consider a new setup where the side information at one receiver is conditionally less noisy than that at the other. The new setup includes degraded side information as a special case, and it is motivated by the literature on degraded and less noisy broadcast channels. Our key contribution is a converse proving the optimality of Heegard and Berger's achievability bound in a new setting, where the side information is conditionally less noisy and one distortion function is deterministic. The less noisy setup is also generalized to two different successive-refinement problems.

Pilot Symbol Based Fixed Bipartite Graph Rate Adaptable Codes for Distributed Video Coding

The Distributed Video Coding with correlated side information for Quarter Common Interface Format (QCIF) was considered for the implementation presented in this paper. The size of the QCIF video was used to develop the Short length progressive Edge Growth codes. A novel pilot symbol based rate compatible code with same Bipartite graph for all rates is proposed. The pilot symbol algorithm makes the computation of the punctured nodes deterministic and one time process. The insertion of pilot symbols converts the parallel virtual channel to a uniform channel, thereby improves the decoding performance. The proposed solution reduces the complexity of the encoder by 25 % and decoder by a factor of 20---40 %. The performance analysis is also done for non-uniform sources as most of the practical data falls into non-uniform distribution.

Identification and Lossy Reconstruction in Noisy Databases

A high-dimensional database system is studied where the noisy versions of the underlying feature vectors are observed in both the enrollment and query phases. The noisy observations are compressed before being stored in the database, and the user wishes to both identify the correct entry corresponding to the noisy query vector and reconstruct the original feature vector within a desired distortion level. A fundamental capacity-storage-distortion tradeoff is identified for this system in the form of single-letter information theoretic expressions. The relation of this problem to the classical Wyner-Ziv rate-distortion problem is shown, where the noisy query vector acts as the correlated side information available only in the lossy reconstruction of the feature vector.

Secure Transmission of Sources Over Noisy Channels With Side Information at the Receivers

This paper investigates the problem of source-channel coding for secure transmission with arbitrarily correlated side informations at both receivers. This scenario consists of an encoder (referred to as Alice) that wishes to compress a source and send it through a noisy channel to a legitimate receiver (referred to as Bob). In this context, Alice must simultaneously satisfy the desired requirements on the distortion level at Bob and the equivocation rate at the eavesdropper (referred to as Eve). This setting can be seen as a generalization of the problems of secure source coding with (uncoded) side information at the decoders and the wiretap channel. A general outer bound on the rate-distortion-equivocation region, as well as an inner bound based on a pure digital scheme, is derived for arbitrary channels and side informations. In some special cases of interest, it is proved that this digital scheme is optimal and that separation holds. However, it is also shown through a simple counterexample with a binary source that a pure analog scheme can outperform the digital one while being optimal. According to these observations and assuming matched bandwidth, a novel hybrid digital/analog scheme that aims to gather the advantages of both digital and analog ones is then presented. In the quadratic Gaussian setup when side information is only present at the eavesdropper, this strategy is proved to be optimal. Furthermore, it outperforms both digital and analog schemes and cannot be achieved via time-sharing. Through an appropriate coding, the presence of any statistical difference among the side informations, the channel noises, and the distortion at Bob can be fully exploited in terms of secrecy.

Rate-adaptive BCH codes for distributed source coding

This paper considers Bose-Chaudhuri-Hocquenghem (BCH) codes for distributed source coding. A feedback channel is employed to adapt the rate of the code during the decoding process. The focus is on codes with short block lengths for independently coding a binary source X and decoding it given its correlated side information Y. The proposed codes have been analyzed in a high-correlation scenario, where the marginal probability of each symbol, X i in X, given Y is highly skewed (unbalanced). Rate-adaptive BCH codes are presented and applied to distributed source coding. Adaptive and fixed checking strategies for improving the reliability of the decoded result are analyzed, and methods for estimating the performance are proposed. In the analysis, noiseless feedback and noiseless communication are assumed. Simulation results show that rate-adaptive BCH codes achieve better performance than low-density parity-check accumulate (LDPCA) codes in the cases studied.

Discriminatory Lossy Source Coding: Side Information Privacy

A lossy source coding problem is studied in which a source encoder communicates with two decoders, one with and one without correlated side information with an additional constraint on the privacy of the side information at the uninformed decoder. Two cases of this problem arise depending on the availability of the side information at the encoder. The set of all feasible rate-distortion-equivocation tuples is characterized for each case. The difference between the informed and uninformed cases and the advantages of encoder side information for enhancing privacy are highlighted for a binary symmetric source with erasure side information and Hamming distortion.

Reliable Joint Source-Channel Cooperative Transmission Over Relay Networks

Reliable transmission of a discrete memoryless source to multiple destinations over a relay network is considered. Motivated by sensor network applications, it is assumed that the relays and the destinations all have access to side information correlated with the underlying source signal. Joint source-channel cooperative transmission is studied in which the terminals in the network help the transmission of the source signal to the destinations by using their overheard signals, as in the classical channel cooperation scenario, as well as the available correlated side information. Decode-and-forward-based cooperative transmission is studied in a network of multiple relay terminals and two different achievability schemes are proposed: 1) a regular encoding and sliding-window decoding scheme without explicit source binning at the encoder; and 2) a semiregular encoding and backward decoding scheme with binning based on the side information statistics. It is shown that both of these schemes lead to the same source-channel code rate, which is shown to be the source-channel capacity in the case of 1) a physically degraded relay network with a single destination in which the side information signals are degraded in the same order as the channel; and 2) a relay network with multiple destinations, in which all the terminals want to reconstruct the source reliably, while at most one of them can act as a relay.

Source and Channel Coding for Cooperative Relaying

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> User cooperation is a powerful tool to combat fading and increase robustness for communication over wireless channels. Although it is doubtless a promising technique for enhancing channel reliability, its performance in terms of average source distortion is not clear since source–channel separation theorem fails under the most common nonergodic slow-fading channel assumption, when channel state information (CSI) is only available at the receiving terminals. This work sheds some light on the end-to-end performance of joint source–channel coding for cooperative relay systems in the high signal-to-noise ratio (SNR) regime. Considering distortion exponent as a figure of merit, we propose various strategies for cooperative source and channel coding that significantly improve the performance compared to the conventional scheme of source coding followed by cooperative channel coding. We characterize the optimal distortion exponent of a full-duplex relay channel for all bandwidth ratios. For the half-duplex relay channel, we provide an upper bound which is tight for small and large bandwidth ratios. We consider the effect of correlated side information on the distortion exponent as well. </para>

Guessing Under Source Uncertainty

This paper considers the problem of guessing the realization of a finite alphabet source when some side information is provided. The only knowledge the guesser has about the source and the correlated side information is that the joint source is one among a family. A notion of redundancy is first defined and a new divergence quantity that measures this redundancy is identified. This divergence quantity shares the Pythagorean property with the Kullback-Leibler divergence. Good guessing strategies that minimize the supremum redundancy (over the family) are then identified. The min-sup value measures the richness of the uncertainty set. The min-sup redundancies for two examples - the families of discrete memoryless sources and finite-state arbitrarily varying sources - are then determined

On Successive Refinement for the Wyner–Ziv Problem

Achievable rates are characterized for successive refinement (SR) in the Wyner-Ziv scenario, namely, in the presence of correlated side information (SI) at the receivers. In this setting, the encoder is assumed to operate in two stages, where the first corresponds to relatively low rate and high distortion, and the second, comprising a refinement code on top of the first code, is aimed at reproduction at reduced distortion. Both decoders (for low rate/high distortion and for high rate/low distortion) are equipped with SI streams, correlated to the source, but unavailable to the encoder. Furthermore, it is assumed that the decoder that receives the higher rate bitstream, i.e., the additional refinement bits, accesses also SI of better quality than that of the lower resolution decoder. By better quality, we mean that the source and the SI are modeled together as a stochastically degraded joint source, where the encoded symbols, the SI at the refinement stage, and the SI at the initial (coarse) stage, form a Markov chain in this order. For a memoryless joint process (that includes the source to be encoded and its instantaneously correlated SI streams), necessary and sufficient conditions are furnished, in terms of single-letter formulas, for the achievability of a pair of rates, corresponding to two given distortion levels. Special attention is devoted to the degenerate, but important, case where the two SI streams, at the two decoders, are identical. For this case, conditions are provided for successive refinability in the sense of the existence of codes that asymptotically achieve the Wyner-Ziv rate-distortion function, simultaneously at both distortion levels. In this context, the doubly symmetric binary source (with the Hamming distortion measure) and the jointly Gaussian source (with the squared error distortion measure) are successively refinable in the Wyner-Ziv setting. It is demonstrated that a source that is not successively refinable in the ordinary sense (i.e., without SI) may become successively refinable in the presence of SI at the decoders.

Rate distortion when side information may be absent

The problem is considered of encoding a discrete memoryless source when correlated side information may or may not be available to the decoder. It is assumed that the side information is not available to the encoder. The rate-distortion function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R (D_{l}, D_{2})</tex> is determined where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D_{1}</tex> is the distortion achieved with side information and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D_{2}</tex> is the distortion achieved without it. A generalization is made to the case of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</tex> decoders, each of which is privy to its own side information. An appropriately defined <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</tex> -admissible rate for this general case is shown to equal <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R(D)</tex> when the side information sources satisfy a specified degradedness condition. Explicit results are obtained in the quadratic Gaussian case and in the binary Hamming case.