Successive Refinement Source Research Articles

Stochastic chaining and strengthened information-theoretic generalization bounds

We propose a new approach to apply the chaining technique in conjunction with information-theoretic measures to bound the generalization error of machine learning algorithms. Different from the deterministic chaining approach based on hierarchical partitions of a metric space, previously proposed by Asadi et al., we propose a stochastic chaining approach, which replaces the hierarchical partitions with an abstracted Markovian model borrowed from successive refinement source coding. This approach has three benefits over deterministic chaining: (1) the metric space is not necessarily bounded, (2) facilitation of subsequent analysis to yield more explicit bound, and (3) further opportunity to optimize the bound by removing the geometric rigidity of the partitions. The proposed approach includes the traditional chaining as a special case, and can therefore also utilize any deterministic chaining construction. We illustrate these benefits using the problem of estimating Gaussian mean and that of phase retrieval. For the former, we derive a bound that provides an order-wise improvement over previous results, and for the latter we provide a stochastic chain that allows optimization over the chaining parameter.

On Asymptotic Analysis of Energy-Distortion Tradeoff for Low-Delay Transmission Over Gaussian Channels

Asymptotic energy-distortion performance of zero- and low-delay transmission of Gaussian sources over energy-limited Gaussian channels is studied. A lower bound for the leading term in the negative logarithm of the distortion, termed the energy-distortion exponent, is derived through an achievable scheme based on high-resolution quantization coupled with orthogonal signaling. The higher-order term in the negative logarithm of the distortion, termed the energy-distortion dispersion, is optimized while keeping the leading term, the energy-distortion exponent, at its optimal (respectively, the best known) value for the zero-delay (respectively, low-delay) regime. In contrast with the decaying dispersion previously reported in the literature, the proposed coding scheme achieves a constant dispersion. When the scheme is optimized, this constant can be improved with respect to its naïve value, i.e., that achieved by optimizing purely the source coding performance instead of the end-to-end distortion. Lastly, a tradeoff of achievable energy-distortion exponents is derived for broadcast scenarios by extending the point-to-point scheme to include a successive refinement source coder coupled with two rounds of orthogonal signaling. A simple parametric computation algorithm is derived for the tradeoff.

Minimizing the average achievable distortion using multi-layer coding approach in two-hop networks

Minimizing the average achievable distortion (AAD) of a Gaussian source at the destination of a two-hop block fading relay channel is studied in this paper. The communication is carried out through the use of a Decode and Forward (DF) relay with no source-destination direct links. The associated receivers of both hops are assumed to be aware of the corresponding channel state information (CSI), while the transmitters are unaware of their corresponding CSI. The current paper explores the effectiveness of incorporating the successive refinement source coding together with multi-layer channel coding in minimizing the AAD. In this regard, the closed form and optimal power allocation policy across code layers of the second hop is derived, and using a proper curve fitting approach, a close-to-optimal power allocation policy associated with the first hop is devised. It is numerically shown that the DF strategy closely follows the Amplify and Forward (AF) relaying, while there is a sizable gap between the AAD of multi-layer coding and single-layer source coding.

End-to-End Distortion Analysis of Nonorthogonal Layered Coding Over Relay-Assisted Networks

This article investigates cross-layer design of nonorthogonal layered (NOL) coding schemes over relay-assisted downlink of cellular networks that incorporates with successive refinement (SR) source code for transmission of adaptive multiresolution multimedia content. Two novel-layered relaying strategies, namely NOL compress-forward (NOL-CF) and NOL decode-forward (NOL-DF) coding, concatenated to the SR source coding scheme are outlined. The network parameters are jointly optimized, to derive optimal end-to-end system performance. Taking the expected distortion (ED) as the end-to-end system performance metric, the explicit analytical solution achieving optimal ED under the proposed NOL-CF scheme with two layers is derived. For the NOL-DF coding, the numerical analysis demonstrates the merits of the proposed scheme, which dynamically adapts the transmission to the instantaneous channel realization in the absence of channel knowledge at the source node. The superiority of the proposed layered coding over several available nonorthogonal multiple access relaying schemes is demonstrated for Gaussian channels. Also, simulation results confirm the effectiveness of the proposed relay-assisted NOL codes to improve the ED over Rayleigh fading channel.

Second-Order and Moderate Deviation Asymptotics for Successive Refinement

We derive the optimal second-order coding region and moderate deviations constant for successive refinement source coding with a joint excess-distortion probability constraint. We consider two scenarios: (i) a discrete memoryless source (DMS) and arbitrary distortion measures at the decoders and (ii) a Gaussian memoryless source (GMS) and quadratic distortion measures at the decoders. For a DMS with arbitrary distortion measures, we prove an achievable second-order coding region, using type covering lemmas by Kanlis and Narayan and by No, Ingber and Weissman. We prove the converse using the perturbation approach by Gu and Effros. When the DMS is successively refinable, the expressions for the second-order coding region and the moderate deviations constant are simplified and easily computable. For this case, we also obtain new insights on the second-order behavior compared to the scenario where separate excess-distortion proabilities are considered. For example, we describe a DMS, for which the optimal second-order region transitions from being characterizable by a bivariate Gaussian to a univariate Gaussian, as the distortion levels are varied. We then consider a GMS with quadratic distortion measures. To prove the direct part, we make use of the sphere covering theorem by Verger-Gaugry, together with appropriately-defined Gaussian type classes. To prove the converse, we generalize Kostina and Verd\'u's one-shot converse bound for point-to-point lossy source coding. We remark that this proof is applicable to general successively refinable sources. In the proofs of the moderate deviations results for both scenarios, we follow a strategy similar to that for the second-order asymptotics and use the moderate deviations principle.

Cooperative Relaying of Superposition Coding with Simple Feedback for Layered Source Transmission

We consider a relay network that delivers a Gaussian source by employing successive refinement source coding and superposition coding of layers at the source node and successive decoding at the relay and destination nodes. For the network, making use of the decoding results at the relay and destination nodes of the first transmission, an efficient relaying strategy of layers is proposed to minimize the expected distortion (ED) when only the average channel state information is available at the source node. Three types of the proposed scheme, defined as Prop-DF, using decode-and-forward signals, Prop-AF, using amplify-and-forward signals, and Prop-MF, using mixed-forward signals, are addressed and analyzed in terms of the outage probability and distortion exponent. Unlike other studies, we have also taken the relay location into account in deriving the distortion exponent showing the high SNR behavior of the ED. The results show that the proposed scheme increases the distortion exponent up to twice that of the conventional relaying schemes when the relay is close to the source node, and that Prop-MF provides the best performance for most relay locations.

Lossy Multicasting Over Binary Symmetric Broadcast Channels

Lossy multicasting of a set of independent, discrete-time, continuous-amplitude source components under the mean square error distortion measure over binary symmetric broadcast channels is investigated. The practically appealing concatenation of successive refinement source coding with broadcast coding and, specifically, time-sharing of linear binary codes, is considered. Three different system optimization criteria are formulated for the lossy multicasting problem. The resulting system optimization is fairly general and applies to a variety of combinations of successive refinement source codes and channel codes. The system optimization is investigated in depth for a class of channel optimized quantization with successive refinement, obtained by using standard embedded scalar quantizers and linear mapping of the (redundant) quantizer bitplanes onto channel codewords by using a systematic Raptor encoder. This scheme is referred to as quantization with linear index coding (QLIC). Unlike existing literature on progressive transmission with unequal error protection or channel optimized quantization, the focus here is on the regime of moderate-to-large code block length and the power of modern sparse-graph codes with iterative belief propagation decoding is leveraged. In this regime, the system optimization takes on the form of simple convex programming that reduces to linear programming for QLIC. The performance of QLIC compares favorably with respect to the state of the art channel optimized quantization in the conventional setting of a single Gaussian source over a binary symmetric channel. For the multicast scenario, the performance gap incurred by the practical QLIC design with respect to ideal source and channel codes is quantified.

Joint Source-Channel Optimization over Wireless Relay Networks

Cooperative communications have received considerable attention for wireless multimedia applications as a technique to improve reliability and service coverage. To apply such cooperation over slow fading channels, we consider exploiting the spatial diversity in multiple relay networks. In this paper, we focus on amplify-and-forward relaying schemes, namely, orthogonal amplify-and-forward, selective relaying, and distributed beamforming, for the cooperative wireless multimedia transmission. Furthermore, the successive refinement source coding is exploited such that the multimedia signal is encoded into multiple layers and the quality of its reconstruction at the receiver is improved when more layers are received correctly. We study two strategies for the layered source transmission, namely, progressive transmission and superposition coding. With our developed framework, we propose suboptimal resource allocation algorithms to efficiently assign rate, power, and channel uses to different layers so as to maximize the quality of the multimedia signal reconstructed at the receiver. The proposed optimization methodology is simple and only requires the knowledge of the channel statistics compared to the existing algorithms. As a result, the receiver only requires to feedback the determined rate, power, and channel uses to the transmitter whenever the channel statistics or the layered source transmission strategy changes.

On Successive Refinement With Causal Side Information at the Decoders

Consider a process, {Xi Yi,Zi}infin i=1 producing independent copies of a triplet of jointly distributed random variables (RVs). The part of the process - the source - is observed at the encoder, and is supposed to be reproduced at two decoders, decoder 1 and decoder 2, where the {Zi} and the {Yi} parts of the process are observed, respectively, in a causal manner. The communication between the encoder and the decoders is carried out across two memoryless channels in two successive communication stages. In the first stage, the compressed transmission is available to both decoders, but only decoder 1 reconstructs the source (according to the received data stream and its causal side information (SI){Zi}). In the second stage, the second decoder reconstructs the source according to {Yi} and the transmissions of the encoder at both stages. It is desired to find necessary and sufficient conditions such that the distortions incurred (in each stage) will not exceed given thresholds. First, a single-letter characterization of achievable rates is derived for a pure source-coding problem with successive refinement and causal SI at the decoders. Then, for a joint source-channel coding setting, a separation theorem is proved, asserting that in the limit of long blocks, no optimality is lost by first applying lossy successive-refinement source coding, regardless of the channels, and then applying good channel codes to each one of the resulting bit streams, regardless of the source. Next, conditions for a source to be successively refinable in two different senses are established, and finally, it is shown that the binary-symmetric source is successively refinable.

On hierarchical joint source-channel coding with degraded side information

We extend the setting of two-stage lossy source coding with successive refinement structures into a joint source-channel coding setting. In particular, we consider a problem where two descriptions of a memoryless source are to be transmitted across two independent memoryless channels and where the output of the channel corresponding to the first (coarse) description is also available to the decoder of the second (refinement) decoder. Side information (SI), correlated to the source, may also be available to the decoders. In such a case, we confine attention to degraded SI, in the sense that the source, the SI available at the refinement decoder, and the SI available at the coarse decoder form a Markov chain in this order. Our first result is a separation theorem asserting that in the limit of long blocks, no optimality is lost by first applying lossy successive-refinement source coding, regardless of the channels, and then applying good channel codes to each one of the resulting bitstreams, regardless of the source and the SI. It is also shown that (even noiseless) feedback from the output of the first channel to the input of the second encoder cannot improve performance, but may sometimes significantly facilitate the implementation of optimum codes. We provide two examples where single-letter codes (of unit block length) achieve optimum performance, if feedback from the channel output of the first stage is provided to the encoder of the refinement stage. In one of these examples, it is evident that if feedback is not provided, optimality cannot be achieved with unit length code. Motivated by these examples, we then investigate single-letter codes for this system. Necessary and sufficient conditions are furnished for the optimality of single-letter codes with and without feedback. A corollary of these conditions is that for the quadratic distortion measure, feedback is necessary to achieve optimality in single-letter codes, regardless of the source distribution and the channel statistics

Sequential coding of correlated sources

We study a generalization of the successive refinement coding problem called the sequential coding of correlated sources. In successive refinement source coding one first describes the given source using a few bits of information, and then subsequently improves the description of the same source when more information is supplied. Sequential coding differs from successive refinement in that the second-stage encoding involves describing a correlated source as opposed to improving the description of the same source. We introduce the notion of a coupled fidelity criterion to quantify perceived distortion in certain applications of sequential coding. We characterize the achievable rate region for this source coding problem and show that the rate region reduces to the successive refinement rate region when the two sources are the same. Then we consider the specific case of a pair of correlated Gaussian sources as an example. We give an explicit characterization that reveals an interesting generalization of a property of successive refinement of a single Gaussian source.

Distortion-rate bounds for fixed- and variable-rate multiresolution source codes

The source coding theorem for stationary sources describes the optimal performance theoretically achievable by fixed- and variable-rate block quantizers. The source coding theorem may be generalized by considering the problem of multiresolution or successive refinement source coding, which is the topic of this work. Given a distortion vector (D/sub 1/,...,D/sub L/), this work describes the family of achievable rate vectors (R/sub 1/,...,R/sub L/) for describing a stationary source at L resolutions, where the description at the first resolution is given at rate R/sub 1/ and achieves an expected distortion no greater than D/sub 1/, the description at the second resolution includes both the first description and a refining description of rate R/sub 2/ and achieves expected distortion no greater than D/sub 2/, and so on. The work includes performance bounds for both fixed- and variable-rate source codes on discrete-time stationary ergodic sources and discrete-time stationary nonergodic sources for any integer number of resolutions L/spl ges/1. For L=1, the source coding theorems for stationary sources result. For L>1, the results extend previous theorems for discrete-alphabet memoryless sources.