Mean Squared Error Distortion Research Articles

Video coding using closed-loop texture analysis and synthesis

The compression of video content containing high-frequency textures using the Mean Squared Error (MSE) criterion or other similar distortion measures typically leads to very large bit rate assignments to these areas. However, we assume that the exact reproduction (in an MSE sense) is subjectively not required and that other measures and coding methods can be used leading to substantially lower bit rate assignments. One such measure is a global similarity measure and we demonstrate its suitability for assessing the distortion of such textures in the context of video coding. The above-mentioned concept is adapted to a new, generic, and fully automated video coding scheme that provides a texture analyzer at the encoder and a texture synthesizer at the decoder. The texture analyzer thereby identifies the target textures, where it is assumed that the viewer perceives the semantic meaning of the displayed texture rather than the specific details therein. The texture synthesizer regenerates an approximate version of the original texture based on corresponding meta data transmitted by the encoder. The texture itself is not coded in the sense of minimizing MSE or other similar distortion measures. The new approacch is integrated into an H.264/AVC video codec and yields bit rate reductions of up to 41% when compared at the same visual quality to a reference H.264/AVC codec.

The Role of Fidelity in Goal-Oriented Semantic Communication: A Rate Distortion Approach

We study a variant of a robust description source coding framework, which is a relevant model for goal-oriented semantic information transmission, via its corresponding characterization. Considering two individual single-letter separable distortion constraints and input and output data acting as the intrinsic and extrinsic message, respectively, we first derive a lower bound on the optimal rates of the problem, as well as necessary and sufficient conditions for this bound to be tight. Subsequently, we prove a general result that provides in parametric form the optimal solution of the characterization of this problem. Capitalizing on these results, we examine the structure of the solution for one case study of general binary alphabets under Hamming distortions and solve in closed form a special case. We also solve another general binary alphabet case where a Hamming and an erasure distortion coexist, as a means to highlight the importance of selecting the type of the distortion constraint in goal-oriented semantic communication. Furthermore, we develop a goal-oriented Blahut-Arimoto (BA) algorithm, which can be used for the computation of any finite alphabet intrinsic or extrinsic message under individual distortion criteria. Finally, we revisit the problem for multidimensional independent and identically distributed (i.i.d.) jointly Gaussian processes with individual mean-square error (MSE) distortion constraints, providing new insights that have previously been overlooked. This work reveals the cardinal role of context-dependent fidelity criteria in goal-oriented semantic communication.

Optimal added noise for minimizing distortion in quantizer-array linear estimation

For linearly estimating the input signal, the optimization of suprathreshold stochastic resonance in an array of N parallel M-ary quantizers is theoretically investigated. We first prove that the mean square error (MSE) of the designed quantizer-array is a convex functional with respect to the cumulative probability function (CDF) of the added noise. Then, for an arbitrary input signal and the quantization interval being N times the decoding step size, we theoretically demonstrate that minimum MSE distortion can be obtained for optimal added uniform noise. Furthermore, for a uniform input signal, the optimality condition also holds if the system parameters and the boundary of the signal satisfy an inequality constraint condition. Moreover, under this condition, the optimal parameters of the quantizer-array can be also determined exactly. By applying both the optimal added noise and optimal parameters to the quantizer-array, the MSE can be further improved for larger array size N or quantization order M. Finally, the optimal added noise is also discussed for minimizing distortions of the quantizer-array linear estimation in the presence of the background noise.

Learning Progressive Distributed Compression Strategies From Local Channel State Information

This paper proposes a deep learning framework to design distributed compression strategies in which distributed agents need to compress high-dimensional observations of a source, then send the compressed bits via bandwidth limited links to a fusion center for source reconstruction. Further, we require the compression strategy to be progressive so that it can adapt to the varying link bandwidths between the agents and the fusion center. Moreover, to ensure scalability, we investigate strategies that depend only on the local channel state information (CSI) at each agent. Toward this end, we use a data-driven approach in which the progressive linear combination and uniform quantization strategy at each agent are trained as a function of its local CSI. To deal with the challenges of modeling the quantization operations (which always produce zero gradients in the training of neural networks), we propose a novel approach of exploiting the statistics of the batch training data to set the dynamic ranges of the uniform quantizers. Numerically, we show that the proposed distributed estimation strategy designed with only local CSI can significantly reduce the signaling overhead and can achieve a lower mean-squared error distortion for source reconstruction than state-of-the-art designs that require global CSI at comparable overall communication cost.

Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

Quantization and compression of neural network parameters using the uniform scalar quantization is carried out in this paper. The attractiveness of the uniform scalar quantizer is reflected in a low complexity and relatively good performance, making it the most popular quantization model. We present a design approach for the memoryless Laplacian source with zero-mean and unit variance, which is based on iterative rule and uses the minimal mean-squared error distortion as a performance criterion. In addition, we derive closed-form expressions for SQNR (Signal to Quantization Noise Ratio) in a wide dynamic range of variance of input data. To show effectiveness on real data, the proposed quantizer is used to compress the weights of neural networks using bit rates from 9 to 16 bps (bits/sample) instead of standardly used 32 bps full precision bit rate. The impact of weights compression on the NN (neural network) performance is analyzed, indicating good matching with the theoretical results and showing negligible decreasing of the prediction accuracy of the NN even in the case of high variance-mismatch between the variance of NN weights and the variance used for the design of quantizer, if the value of the bit-rate is properly chosen according to the rule proposed in the paper.

New Formulation and Computation of NRDF for Time-Varying Multivariate Gaussian Processes With Correlated Noise

We derive a new formulation of nonanticipative rate distortion function (NRDF) for time-varying multivariate Gauss-Markov processes driven by correlated noise described by a first order autoregressive moving average (ARMA(1,1)) process with mean-squared error (MSE) distortion constraint. To arrive to this formulation, we first show that the Gauss-Markov process with correlated noise can be compactly written as a linear functional of the lagged by one sample of the sufficient statistic of the correlated noise and its orthogonal innovations process. Then, we use this structural result to a general low delay quantization problem where we choose to design the encoder and the decoder policies of a multi-input multi-output (MIMO) system with given the past sample of the sufficient statistic of the correlated noise process. For jointly Gaussian processes, we find the minimization problem that needs to be solved, obtain its optimal realization and solve it by showing that is semidefinite representable. Interestingly, the optimal realization of this problem reveals that it suffices only the decoder to have access to the given past sufficient statistic of the correlated noise process but not necessarily the encoder. For scalar-valued processes, we also derive a new analytical expression. The generality of our results (both for vector and scalar processes) is shown by recovering various special cases and known results obtained for independent noise processes.

On the energy self-sustainability of IoT via distributed compressed sensing

This paper advocates the use of the distributed compressed sensing (DCS) paradigm to deploy energy harvesting (EH) Internet of Thing (IoT) devices for energy self-sustainability. We consider networks with signal/energy models that capture the fact that both the collected signals and the harvested energy of different devices can exhibit correlation. We provide theoretical analysis on the performance of both the classical compressive sensing (CS) approach and the proposed distributed CS (DCS)-based approach to data acquisition for EH IoT. Moreover, we perform an in-depth comparison of the proposed DCS- based approach against the distributed source coding (DSC) system. These performance characterizations and comparisons embody the effect of various system phenomena and parameters including signal correlation, EH correlation, network size, and energy availability level. Our results unveil that, the proposed approach offers significant increase in data gathering capability with respect to the CS-based approach, and offers a substantial reduction of the mean-squared error distortion with respect to the DSC system.

Design and Analysis of Binary Scalar Quantizer of Laplacian Source with Applications

A compression method based on non-uniform binary scalar quantization, designed for the memoryless Laplacian source with zero-mean and unit variance, is analyzed in this paper. Two quantizer design approaches are presented that investigate the effect of clipping with the aim of reducing the quantization noise, where the minimal mean-squared error distortion is used to determine the optimal clipping factor. A detailed comparison of both models is provided, and the performance evaluation in a wide dynamic range of input data variances is also performed. The observed binary scalar quantization models are applied in standard signal processing tasks, such as speech and image quantization, but also to quantization of neural network parameters. The motivation behind the binary quantization of neural network weights is the model compression by a factor of 32, which is crucial for implementation in mobile or embedded devices with limited memory and processing power. The experimental results follow well the theoretical models, confirming their applicability in real-world applications.

Real Time Coding and Estimation of Linear Discrete Time System Over Networks

This study deals with the real time coding and estimation of linear discrete time scalar system over communication networks. With the mean-squared error (MSE) distortion criterion, the information rate distortion function describing the performance limit of the coding-estimation system is analyzed and discussed. To achieve near instantaneous encoding and decoding, an asymptotic design scheme has been presented as a realization of real time coding-estimation system. The outputs of standard Kalman filter relying on unquantized observations are encoded using the Lloyd-Max quantization rules and transmitted over the noiseless channel. The decoder side runs the corresponding decoding and reconstruction algorithms to produce the optimal real time state estimate of system. To synchronize the encoder and decoder, a double-predictor regime on the updating rules of the encoder-decoder pair is proposed, and it does not require any feedback information. The rate distortion function of the proposed scheme is derived and when comparing with the information theoretical lower bound, there is only a factor discrepancy related to the quantization rules. The rate distortion performance results of various design schemes are compared and demonstrated with numerical simulations.

Joint State Estimation and Communication Over a State-Dependent Gaussian Multiple Access Channel

A hybrid communication network with a common analog source signal and independent digital data streams at the transmitters of a multiple access network is considered. The receiver has to estimate the analog signal samples with a given fidelity, and decode the digital streams with a low error probability. The main goal of this paper is to characterize the optimal tradeoff between the mean-squared error distortion in source estimation and the data rates available to each user. To this end, we consider a Gaussian multiple access channel (GMAC) setup with additive state, where the state is nothing but a scaled version of the source process itself. The state process is assumed to be non-causally available to all the transmitting nodes. The problem now becomes that of the joint state estimation and message communication in a GMAC with state. We provide a complete characterization of the optimal distortion-rate tradeoff for an $N$ —sender GMAC. Our results show that, similar to the single-user results, it is optimal to amplify the state using uncoded transmissions, whereas the digital streams are superposed using appropriate Gaussian codebooks in conjunction with dirty paper coding (DPC). Since the variance of the additive state is controlled by a scaling factor in our model, we also recover the results for communicating a common source and independent messages over a GMAC without state as a special case.

The Time-Invariant Multidimensional Gaussian Sequential Rate-Distortion Problem Revisited

We revisit the sequential rate-distortion (SRD) tradeoff problem for vector-valued Gauss–Markov sources with mean-squared error distortion constraints. Our study is partly motivated by the question recently raised in the paper “Rate-cost tradeoffs in control” (in Proc. 54th Annu. Allerton Conf. Commun., Control, Comput. , 2016, pp. 1157–1164) regarding the correctness of its solution algorithm known in the literature. We show via a counterexample that the dynamic reverse water-filling algorithm suggested by (15) of the paper “Stochastic linear control over a communication channel” ( IEEE Trans. Autom. Control , vol. 49, pp. 1549–1561, 2004) is not applicable to this problem, and consequently, the closed-form expression of the asymptotic SRD function derived in (17) of the paper “Stochastic linear control over a communication channel” ( IEEE Trans. Autom. Control , vol. 49, pp. 1549–1561, 2004) is not correct in general. Nevertheless, we show that the multidimensional Gaussian SRD function is semidefinite representable, and thus, it is readily computable.

Monotonicity of Step Sizes of MSE-Optimal Symmetric Uniform Scalar Quantizers

For generalized gamma probability densities, this paper studies the monotonicity of step sizes of optimal symmetric uniform scalar quantizers with respect to mean squared-error distortion. The principal results are that for the special cases of Gaussian, Laplacian, two-sided Rayleigh, and gamma densities, optimal step size monotonically decreases when the number of levels $N$ increases by two, and that for any generalized gamma density and all sufficiently large $N$ , optimal step size again decreases when $N$ increases by two. Also, it is shown that for a Laplacian density and sufficiently large $N$ , optimal step size decreases when $N$ increases by just one.

Asymptotic Rate-Distortion Analysis of Symmetric Remote Gaussian Source Coding: Centralized Encoding vs. Distributed Encoding.

Consider a symmetric multivariate Gaussian source with ℓ components, which are corrupted by independent and identically distributed Gaussian noises; these noisy components are compressed at a certain rate, and the compressed version is leveraged to reconstruct the source subject to a mean squared error distortion constraint. The rate-distortion analysis is performed for two scenarios: centralized encoding (where the noisy source components are jointly compressed) and distributed encoding (where the noisy source components are separately compressed). It is shown, among other things, that the gap between the rate-distortion functions associated with these two scenarios admits a simple characterization in the large ℓ limit.

Zero-Delay Rate Distortion via Filtering for Vector-Valued Gaussian Sources

We deal with zero-delay source coding of a vector-valued Gauss-Markov source subject to a mean-squared error (MSE) fidelity criterion characterized by the operational zero-delay vector-valued Gaussian rate distortion function (RDF). We address this problem by considering the nonanticipative RDF (NRDF) which is a lower bound to the causal optimal performance theoretically attainable (OPTA) function and operational zero-delay RDF. We recall the realization that corresponds to the optimal "test-channel" of the Gaussian NRDF, when considering a vector Gauss-Markov source subject to a MSE distortion in the finite time horizon. Then, we introduce sufficient conditions to show existence of solution for this problem in the infinite time horizon. For the asymptotic regime, we use the asymptotic characterization of the Gaussian NRDF to provide a new equivalent realization scheme with feedback which is characterized by a resource allocation (reverse-waterfilling) problem across the dimension of the vector source. We leverage the new realization to derive a predictive coding scheme via lattice quantization with subtractive dither and joint memoryless entropy coding. This coding scheme offers an upper bound to the operational zero-delay vector-valued Gaussian RDF. When we use scalar quantization, then for "r" active dimensions of the vector Gauss-Markov source the gap between the obtained lower and theoretical upper bounds is less than or equal to 0.254r + 1 bits/vector. We further show that it is possible when we use vector quantization, and assume infinite dimensional Gauss-Markov sources to make the previous gap to be negligible, i.e., Gaussian NRDF approximates the operational zero-delay Gaussian RDF. We also extend our results to vector-valued Gaussian sources of any finite memory under mild conditions. Our theoretical framework is demonstrated with illustrative numerical experiments.

Noise effect on signal quantization in an array of binary quantizers

Abstract The effect of noise is examined in both the uniform quantizer and a stochastic quantizer. The stochastic quantizer is an array of identical binary quantizers combined with a linear decoder. In uniform quantization scenario, we find that noise cannot help decrease mean square error (MSE) distortion. However, in the array of binary quantizers with identical thresholds, noise may play a positive role, and stochastic resonance (SR) can be observed. First, based on MSE distortion, the optimal noise in the array is derived by Gateaux differential, which is shown to be a uniform noise. And then, some other noises, including uniform noise with zero mean, Gaussian noise, Laplacian noise, and discrete noise, are considered for comparison. In the case that the granular region is fixed, the quantization performances induced by those noises are shown, and the SR effects are discussed. Furthermore, when the granular region is adjustable, quantization performance may be better. Especially, the MSE distortion, in the optimal noise case, will approximate zero as the rate becomes high. At last, some further studies are discussed, which may extend our results.

On the Convexity of the MSE Distortion of Symmetric Uniform Scalar Quantization

This paper investigates the convexity of the mean squared-error distortion of symmetric uniform scalar quantization with respect to step size. The principal results include proofs for odd numbers of levels that distortion is not convex for any symmetric density and that it is convex for even numbers of levels for densities, such as Gaussian, Laplacian, and gamma, but is not, in general for two-sided Rayleigh. For the latter case, an interval is derived that includes the optimal step size and over which the distortion is convex. The proofs of convexity use the Euler–Maclaurin formula applied to the second derivative of distortion, with upper bounds on the remainder term. These results imply that a zero of the derivative of the distortion for these densities, which has been previously conjectured optimal, is indeed the optimal step size, because the distortion is convex either globally or locally over a sufficiently wide interval to ensure a global minimizer.

A Lower Bound on the Differential Entropy of Log-Concave Random Vectors with Applications.

We derive a lower bound on the differential entropy of a log-concave random variable X in terms of the p-th absolute moment of X. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. Specifically, we study the rate-distortion function for log-concave sources and distortion measure , with , and we establish that the difference between the rate-distortion function and the Shannon lower bound is at most bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most bit, regardless of d. We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most bit. Our results generalize to the case of a random vector X with possibly dependent coordinates. Our proof technique leverages tools from convex geometry.

Zero-Delay Source-Channel Coding With a Low-Resolution ADC Front End

Motivated by the practical constraints arising in emerging sensor network and Internet-of-Things (IoT) applications, the zero-delay transmission of a Gaussian measurement over a real single-input multiple-output (SIMO) additive white Gaussian noise (AWGN) channel is studied with a low-resolution analog-to-digital converter (ADC) front end. Joint optimization of the encoder and the decoder mapping is tackled under both the mean squared error (MSE) distortion and the distortion outage probability (DOP) criteria, with an average power constraint on the channel input. Optimal encoder and decoder mappings are identified for a one-bit ADC front end under both criteria. For the MSE distortion, the optimal encoder mapping is shown to be non-linear in general, while it tends to a linear encoder in the low signal-to-noise ratio (SNR) regime, and to an antipodal digital encoder in the high SNR regime. This is in contrast to the optimality of linear encoding at all SNR values in the presence of a full-precision front end. For the DOP criterion, it is shown that the optimal encoder mapping is piecewise constant and can take only two opposite values when it is non-zero. For both the MSE distortion and the DOP criteria, necessary optimality conditions are then derived for $K$ -level ADC front ends as well as front ends with multiple one-bit ADCs. These conditions are used to obtain numerically optimized solutions. Extensive numerical results are also provided in order to gain insights into the structure of the optimal encoding and decoding mappings.

Finite-Time Nonanticipative Rate Distortion Function for Time-Varying Scalar-Valued Gauss-Markov Sources

We derive the finite-time horizon nonanticipative rate distortion function (NRDF) of time-varying scalar Gauss–Markov sources under an average mean squared-error (MSE) distortion fidelity. Further, we show that a conditionally Gaussian reproduction process realizes the optimal reproduction distribution, and this is determined from the solution of a dynamic reverse-waterfilling optimization problem. We provide an iterative algorithm that approximates the solution of the dynamic reverse-waterfilling problem. From the above results, we also obtain, as a special case, the NRDF under a per-letter or pointwise MSE distortion fidelity, and we draw connections to the classical RDF of Gaussian processes. Our results are corroborated with illustrative examples.

The Golden Quantizer: The Complex Gaussian Random Variable Case

The problem of quantizing a circularly symmetric complex Gaussian random variable is considered. For this purpose, we design two non-uniform quantizers, a high-rate- and a Lloyd-Max-quantizer, that are both based on the (golden angle) spiral-phyllotaxis packing principle. We find that the proposed schemes have lower mean-square error distortion compared to (non)-uniform polar/rectangular-quantizers, and near-identical to the best performing trained vector quantizers. The proposed quantizer scheme offers a structured design, a simple natural index ordering, and allows for any number of centroids.