Mean Square Quantization Research Articles

Application analysis of clipping and digital resolution enhancer in high-speed direct-detection PAM4 transmission.

In this paper, the application of a clipping and digital resolution enhancer (DRE) is numerically investigated in 56-GBaud direct-detection four-ary pulse amplitude modulation (PAM4) transmission systems. The influence of the tap length and the digital-to-analog converter (DAC) samples per symbol on DRE gain is first studied. After optimizing clipping probability, sampling jitter is introduced. The combination of clipping and DRE can increase jitter tolerance by ∼1.7% sampling time for 4-bit DAC. Then, the required optical signal-to-noise ratio (OSNR) at the hard-decision forward error correction (HD-FEC) threshold of 3.8×10-3 is investigated in back-to-back case and 80-km fiber transmission. It is shown that the combination of clipping and DRE can reduce the required OSNR significantly for 3-bit and 4-bit DAC. Finally, the performance gain introduced by clipping and DRE is analyzed from a perspective of the transmitted average signal power and the received mean-squared quantization error.

Fixed-Point Implementation of Fast QR-Decomposition Recursive Least-Squares Algorithms (FQRD-RLS): Stability Conditions and Quantization Errors Analysis

The fast QR-decomposition based recursive least-squares (FQRD-RLS) algorithms offer RLS-like convergence and misadjustment at a lower computational cost, and therefore are desirable for implementation on a fixed-point digital signal processor (DSP). Furthermore, the FQRD-RLS algorithms are derived from QR-decomposition based RLS algorithms that are well known for their numerical stability in finite precision. Hence, these algorithms are often assumed numerically stable, although there is no rigorous analysis addressing the stability of such algorithms in finite precision. In this paper, we derive the conditions that guarantee stability of the FQRD-RLS algorithms, and also derive mathematical expressions for the mean-squared quantization error (MSQE) of internal variables of the FQRD-RLS algorithms at steady state. The objective is to quantify the propagation error due to quantization effects. The derived MSQE expressions have been verified by comparisons with fixed-point computer simulations.

Lattice Quantization of Phases for Equal Gain Transmission

Equal gain transmission (EGT) requires only the feedback of phase information and is known to enjoy several implementational advantages compared to maximum ratio transmission (MRT). This paper investigates the use of lattice quantization for quantizing phases in EGT systems. In practice, the feedback rate is finite and the lattices need to be truncated to have finite code rate. The truncation of lattices results in boundary effect; hence, earlier results developed for untruncated lattices, e.g., fast algorithm and mean-square error (MSE) analysis, cannot be applied directly. Exploiting the fact that the quantized signals are phases, we show how to quantize using truncated lattices so that there is no boundary effect. Moreover, we propose efficient methods to convert between lattice codewords and binary digits so that the lattice quantization can be used in practical systems. Furthermore, the mean-square quantization error is analyzed for several useful truncated lattices. We also show how to analyze the signal-to-noise ratio (SNR) loss when the proposed quantization is applied to an EGT system.

An Automated Fixed-Point Optimization Tool in MATLAB XSG/SynDSP Environment

This paper presents an automated tool for floating-point to fixed-point conversion. The tool is based on previous work that was built in MATLAB/Simulink environment and Xilinx System Generator support. The tool is now extended to include Synplify DSP blocksets in a seamless way from the users' view point. In addition to FPGA area estimation, the tool now also includes ASIC area estimation for end-users who choose the ASIC flow. The tool minimizes hardware cost subject to mean-squared quantization error (MSE) constraints. To obtain more accurate ASIC area estimations with synthesized results, 3 performance levels are available to choose from, suitable for high-performance, typical, or low-power applications. The use of the tool is first illustrated on an FIR filter to achieve over 50% area savings for MSE specification of 10−6 as compared to all 16-bit realization. More complex optimization results for chip-level designs are also demonstrated.

Canonical coordinates for transform coding of noisy sources

Based on the additive white quantization noise model, linear transform coders are derived for Gaussian sources corrupted by noise. There are two alternative design objectives: minimizing the trace of the error correlation matrix and thus minimizing the mean-squared error, or minimizing the determinant of the error correlation matrix and thus maximizing information rate. It is shown that a solution to both problems is to first transform the noisy observations into canonical coordinates, quantize and apply a Wiener filter in this coordinate system, and then transform the result back to the original coordinates. Canonical coordinates are uncorrelated, and quantization and Wiener filtering are applied to each component independently. The type of canonical coordinate system depends on the design objective: Quantization in half-canonical coordinates minimizes the mean-squared error and quantization in full-canonical coordinates maximizes information rate. Finally, it is also demonstrated in this paper that majorization is the fundamental principle underlying proofs of optimal transform coding.

Robust low-delay audio coding using multiple descriptions

This paper proposes an encoding method for high-quality, low-delay audio communication that is robust to losses in packetized transmission. Robustness is provided by a multiple description vector quantization (MDVQ) technique that is designed to minimize the mean-squared error (MSE). The key to applying this technique effectively is the use of psycho-acoustically controlled preand post-filters that make the mean-squared quantization error perceptually relevant. Experiments show that the MDVQ-based encoder yields better results-in both MSE and subjective audio quality-than simple alternative coders with the same low delay.

Minimum mean-squared error transform coding and subband coding

Knowledge of the power spectrum of a stationary random sequence can be used for quantizing the signal efficiently and with minimum mean-squared error. A multichannel filter is used to transform the random sequence into an intermediate set of variables that are quantized using independent scalar quantizers, and then inverse-filtered, producing a quantized version of the original sequence. Equal word-length and optimal word-length quantization at high bit rates is considered. An analytical solution for the filter that minimizes the mean-squared quantization error is obtained in terms of its singular value decomposition. The performance is characterized by a set of invariants termed second-order modes, which are derived from the eigenvalue decomposition of the matrix-valued power spectrum. A more general rank-reduced model is used for decreasing distortion by introducing bias. The results are specialized to the case when the vector-valued time series is obtained from a scalar random sequence, which gives rise to a filter bank model for quantization. The asymptotic performance of such a subband coder is derived and shown to coincide with the asymptotic bound for transform coding. Quantization employing a single scalar pre- and postfilter, traditional transform coding using a square linear transformation, and subband coding in filter banks, arise as special cases of the structure analyzed here.

Pre/post-filter for performance improvement of transform coding

Abstract A technique for generating independent transform coefficients from any p -dependent signal has been developed. Since these coefficients are independent, the Lloyd-Max quantization efficiency is improved. In addition, these coefficients are shown to be Gaussian distributed. Therefore, the probability density function estimate is bypassed during the quantizer design. An all-pass (encryption) pre-filter is required by the proposed technique, and the filtered signal is obtained efficiently by an algorithm developed in this paper. An added benefit of this technique is the compatibility and higher security with respect to the conventional transform coding (TC), and the method is called the transform encryption coding (TEC). In addition, a post-filter is necessary for signal reconstruction. Due to the all-pass nature of the pre/post-filtering process, the mean-square quantization error for the complete process is equal to that for the intermediate independent Gaussian transform coefficients. Simulation results show TEC achieves about 1 dB coding gain compared with TC. Coded images without blocking effects are obtained at 0.5 bit/pixel (bpp) using TEC. The image quality is similar to that obtained by TC at 1 bpp. Even at 0.35 bpp, TEC performs in the same way as the recent ‘DCT/DST’ technique of Rose et al. (1990) and shows no blocking effects. Most significantly, TEC encoded image is insensitive and robust to the channel noise.

A comparison of the Hartley, Cas-Cas, Fourier, and discrete cosine transforms for image coding

The coding method used for the comparison utilizes a marginal bit-allocation scheme and Lloyd-Max quantizer. Gamma, Laplacian, and Gaussian models are compared for the distribution of the transform coefficients. The discrete cosine transform outperforms the other three transforms based on the mean-square quantization error. >

Oversampled Sigma-Delta Modulation

Oversampled sigma-delta modulation has been proposed as a practical implementation for high rate analog-to-digital conversion because of its simplicity and its robustness against circuit imperfections. To date, mathematical developments of the basic properties of such systems have been based either on simplified continuous-time approximate models or on linearized discrete-time models where the quantizer is replaced by an additive white uniform noise source. In this paper, we rigorously derive several basic properties of a simple discrete-time single integrator loop sigma-delta modulator with an accumulate-and-dump demodulator. The derivation does not require any assumptions on the correlation or distribution of the quantizer error, and hence involves no linearization of the nonlinear system, but it does show that when the input is constant, the state sequence of the integrator in the encoder loop can be modeled exactly as a linear system in an appropriate space. Two basic properties are developed: 1) the behavior of the sigma-delta quantizer when driven by a constant input and its relation to uniform quantization, and 2) the rate-distortion tradeoffs between the oversampling ratio and the average mean-squared quantization error.

DCT/DPCM Processing of NTSC Composite Video Signal

This paper describes two different schemes of DCT/ DPCM processing of digital NTSC composite video signal. The first scheme (line processing) is based on applying the discrete cosine transform (DCT) along each horizontal line and then implementing the differential pulse code modulation (DPCM) in the vertical direction on the semitransformed image. The second scheme (block processing) involves DCT of (4 × 4) subblocks followed by adaptive DPCM to reduce intersubblock correlation. Based on the statistics of the prediction error, variable bit quantizers optimized for minimizing the mean-square quantization error are developed. These techniques lead to reduced bit rates for transmitting the color video at broadcast standards. Both original and processed color images in composite form are displayed for subjective evaluation. The effects of channel error on block processing are also investigated.

Design of absolutely optimal quantizers for a wide class of distortion measures

In this paper designs of two types of quantizers are presented. The first type is designed to minimize a distortion measure under the constraint that the number of levels is fixed or the entropy of the output signal is bounded below a given value. The distortion measure is defined as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E[f(x,\eta)]</tex> , the expected value of an error weighting function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f(x, \eta)</tex> , where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</tex> is the quantizer input and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\eta</tex> is the corresponding quantization error. This paper departs from the quautization work reported in the literature heretofore in allowing <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</tex> to be a function of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</tex> as well as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\eta</tex> . Algorithms to minimize such a distortion measure ander the constraints mentioned above are presented. They use a combination of dynamic programming and Fibonacci search. It is shown that if <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f(x,\eta)</tex> is semiconvex in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\eta</tex> for all fixed values of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</tex> , Fibonacci search can be used in one of the steps of the minimization algorithm. This reduces the number of multiplications by a factor of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M/(5 \log M)</tex> when the range of input values is divided into <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</tex> parts. Some examples are considered. The first deals with an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f(x,\eta)</tex> which is zero if <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\eta</tex> is below a certain threshold <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T(x)</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\eta_{2}-T_{2}(x)</tex> otherwise. It arises in coding video signals by differential pulse-code modulation (PCM). The second deals with the minimum mean-square quantization of a truncated Laplacian input density. The step sizes of the near-optimal uniform quantizers are obtained under varions entropy constraints. The third example shows that the optimal quantizer can be asymmetric, even when the probability density and the error weighting function are symmetric. The second type of quantizer is designed to minimize the number of levels or the output entropy, when the quantization error is constrained not to exceed a threshold function. Methods to design them are presented that involve, respectively, a geometric construction and a dynamic programming algorithm in which the domain of search is modified according to the constraint mentioned above.

On quantizers for DPCM coding of picture signals

A measure of picture quality for simple element, differentially coded pictures is developed based on certain subjective tests. The measure weights the quantization noise according to its visibility. It is shown that the measure correlates well with the picture quality determined on a standard impairment scale. Optimization of DPCM quantizers is done for this and for the mean-square measure of picture quality. Performance of the following types of quantizers is evaluated in terms of entropy of the quantized output and the picture quality: a) minimum mean-square error quantizers with a fixed number of levels, b) minimum mean-square error quantizers with fixed entropy, c) minimum mean-square subjective distortion quantizers with a fixed number of levels, d) minimum mean-square subjective distortion quantizers with fixed entropy, and e) uniform quantizers. It is concluded that for a fixed number of levels and a fixed word-length coding of the quantizer outputs, the quantizers in c) outperform those in a); and with variable length coding, the quantizers in d) perform better than all of the other quantizers having the same entropy. The sensitivity of the approach to variation of picture content is also investigated.

Design of Quantizers for Real-Time Hadamard-Transform Coding of Pictures

A methodology is developed to obtain subjectively optimum quantizers for Hadamard-transformed still pictures. To exploit the perceptual redundancies that depend upon the local properties of the picture, a small block (2 × 2 × 2, horizontal-vertical-temporal) is used. A series of subjective tests was carried out to determine the visibility of impairment in the reconstructed picture when noise, which simulated the quantization noise, was added to the Hadamard coefficients in the transform domain (H-noise). A design procedure for quantizers was developed using these visibility functions. These quantizers minimize the “mean-square subjective distortion” (mmssd) due to quantization noise. The resulting picture Quality and entropy were compared with that of Max-type quantizers which minimize the “mean-square error” (mmse). This comparison indicates that the mmssd quantizers based on subjective visibility of the quantization noise are less companded than the MMSE quantizers. Also for the same number of quantization levels, pictures coded with mmssd quantizers have better quality and less entropy than the pictures coded with minimum mean-square quantizers.

On optimum quantization

The problem of minimizing mean-square quantization error is considered and simple closed form approximations based on the work of Max and Roe are derived for the quantization error and entropy of signals quantized by the optimum fixed- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> quantizer. These approximations are then used to show that, when <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> is moderately large, it is better to use equl-interval quantizing than the optimum fixed- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> quantizer if the signal is to be subsetiuently buffered and transmitted at a fixed bit rate. Finally, the problem of optimum quantizing in the presence of buffering is examined, and the numerical results presented for Gaussian signals indicate that equllevel quantizing yields nearly optimum results.