Laplacian Source Research Articles

Learned Progressive Image Compression With Dead-Zone Quantizers

Progressive coding is essential to the practical deployment of learned image compression over heterogeneous networks and clients. Existing methods for learned progressive image compression require complex and empirical design to achieve near-optimal rate-distortion performance over a wide range of bit-rates. However, these methods are limited by the implicit learned mechanism based on neural networks and introduction of uniform quantizers. In this paper, we propose generalized learned progressive image compression with analytic rate-distortion optimization using dead-zone quantizers on the latent representation. Specifically, we reveal that dead-zone quantizers, as a general case of uniform quantizers, are equivalent to uniform quantizers in fixed-rate nonlinear transform coding and can prevent extra redundancy in embedded quantization for progressive coding. Consequently, we propose rate-distortion optimized learned progressive coding by approximating the optimal quantizer in the source spaces using dead-zone quantizers in an analytic manner on the Laplacian source. To our best knowledge, this paper is the first to achieve general learned progressive coding from the perspective of optimal quantizers. The proposed method achieves theoretically sound and practically efficient embedded quantization and learned progressive coding of latent representations with improved rate-distortion performance. It can also enable embedded quantization with diverse assignments of truncation points and support flexible configuration of quality layers of varying numbers and at varying target bit-rates. Furthermore, we successfully incorporate the proposed method into existing pre-trained fixed-rate models to realize progressive learned image compression without re-training. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performance in learned progressive image compression compared with traditional codecs and recent learned methods.

Lossy P-LDPC Codes for Compressing General Sources Using Neural Networks

It is challenging to design an efficient lossy compression scheme for complicated sources based on block codes, especially to approach the theoretical distortion-rate limit. In this paper, a lossy compression scheme is proposed for Gaussian and Laplacian sources. In this scheme, a new route using "transformation-quantization" was designed to replace the conventional "quantization-compression". The proposed scheme utilizes neural networks for transformation and lossy protograph low-density parity-check codes for quantization. To ensure the system's feasibility, some problems existing in the neural networks were resolved, including parameter updating and the propagation optimization. Simulation results demonstrated good distortion-rate performance.

Optimization of the 24-Bit Fixed-Point Format for the Laplacian Source

The 32-bit floating-point (FP32) binary format, commonly used for data representation in computers, introduces high complexity, requiring powerful and expensive hardware for data processing and high energy consumption, hence being unsuitable for implementation on sensor nodes, edge devices, and other devices with limited hardware resources. Therefore, it is often necessary to use binary formats of lower complexity than FP32. This paper proposes the usage of the 24-bit fixed-point format that will reduce the complexity in two ways, by decreasing the number of bits and by the fact that the fixed-point format has significantly less complexity than the floating-point format. The paper optimizes the 24-bit fixed-point format and examines its performance for data with the Laplacian distribution, exploiting the analogy between fixed-point binary representation and uniform quantization. Firstly, the optimization of the 24-bit uniform quantizer is performed by deriving two new closed-form formulas for a very accurate calculation of its maximal amplitude. Then, the 24-bit fixed-point format is optimized by optimization of its key parameter and by proposing two adaptation procedures, with the aim to obtain the same performance as of the optimal uniform quantizer in a wide range of variance of input data. It is shown that the proposed 24-bit fixed-point format achieves for 18.425 dB higher performance than the floating-point format with the same number of bits while being less complex.

Performance Analysis of a 2-bit Dual-Mode Uniform Scalar Quantizer for Laplacian Source

The main issue when dealing with the non-adaptive scalar quantizers is their sensitivity to variance-mismatch, the effect that occurs when the data variance differs from the one used for the quantizer design. In this paper, we consider the influence of that effect in low-rate (2-bit) uniform scalar quantization (USQ) of Laplacian source and also we propose adequate measure to suppress it. Particularly, the approach we propose represents the upgraded version of the previous approaches used to improve performance of the single quantizer. It is based on dual-mode quantization that combines two 2-bit USQs (with adequately chosen parameters) to process input data, selected by applying the special rule. Analysis conducted in theoretical domain has shown that the proposed approach is less sensitive to variance-mismatch, making the dual-mode USQ more efficient in terms of robustness than the single USQ. Also, a gain is achieved compared to other 2-bit quantizer solutions. Experimental results are also provided for quantization of weights of the multi-layer perceptron (MLP) neural network, where good matching with the theoretical results is observed. Due to these achievements, we believe that the solution we propose can be a good choice for compression of non-stationary data modeled by Laplacian distribution, such as neural network parameters.

Two Novel Non-Uniform Quantizers with Application in Post-Training Quantization

With increased network downsizing and cost minimization in deployment of neural network (NN) models, the utilization of edge computing takes a significant place in modern artificial intelligence today. To bridge the memory constraints of less-capable edge systems, a plethora of quantizer models and quantization techniques are proposed for NN compression with the goal of enabling the fitting of the quantized NN (QNN) on the edge device and guaranteeing a high extent of accuracy preservation. NN compression by means of post-training quantization has attracted a lot of research attention, where the efficiency of uniform quantizers (UQs) has been promoted and heavily exploited. In this paper, we propose two novel non-uniform quantizers (NUQs) that prudently utilize one of the two properties of the simplest UQ. Although having the same quantization rule for specifying the support region, both NUQs have a different starting setting in terms of cell width, compared to a standard UQ. The first quantizer, named the simplest power-of-two quantizer (SPTQ), defines the width of cells that are multiplied by the power of two. As it is the case in the simplest UQ design, the representation levels of SPTQ are midpoints of the quantization cells. The second quantizer, named the modified SPTQ (MSPTQ), is a more competitive quantizer model, representing an enhanced version of SPTQ in which the quantizer decision thresholds are centered between the nearest representation levels, similar to the UQ design. These properties make the novel NUQs relatively simple. Unlike UQ, the quantization cells of MSPTQ are not of equal widths and the representation levels are not midpoints of the quantization cells. In this paper, we describe the design procedure of SPTQ and MSPTQ and we perform their optimization for the assumed Laplacian source. Afterwards, we perform post-training quantization by implementing SPTQ and MSPTQ, study the viability of QNN accuracy and show the implementation benefits over the case where UQ of an equal number of quantization cells is utilized in QNN for the same classification task. We believe that both NUQs are particularly substantial for memory-constrained environments, where simple and acceptably accurate solutions are of crucial importance.

Performance of Post-Training Two-Bits Uniform and Layer-Wise Uniform Quantization for MNIST Dataset from the Perspective of Support Region Choice

This paper contributes to the goal of finding an efficient compression solution for post-training quantization from the perspective of support region choice under the framework of low-bit uniform quantization. The decision to give preference to uniform quantization comes from the fact that support region choice is the most sensitive in the uniform quantization of nonuniform sources (e.g., Laplacian sources). Therefore, in this paper, we analyse in detail how the choice of the support region influences the performance of two-bits uniform quantization, measured with signal to quantization noise ratio (SQNR), and the accuracy of the compressed neural network (NN) model. We provide experimental and theoretical results for a few significant cases of two-bits uniform quantizer design, where we assume that Laplacian source models the distribution of weights in our fully connected NN. We opt for Laplacian distribution since it models well weights of NNs. Specifically, we analyse whether it is possible to apply the simplest uniform quantization in trained NN model weight representation with a bit rate of R = 2 bit/sample while preserving the accuracy of the model to a great extent. Also, our goal is to determine whether the choice of the key parameter of two-bits uniform quantizer (support region threshold) equally reflects on both, SQNR and accuracy. Moreover, we extend our analysis to the application of layer-wise two-bits uniform quantization in order to examine whether it is possible to achieve an additional improvement of the accuracy of our NN model for the MNIST dataset. We believe that the detailed analysis of post-training quantization described and conducted in this paper is very useful for all further research studies of this very current topic, especially due to the fact that the problem regarding post-training quantization is addressed from a particularly important perspective of choosing the support region.

Iterative Algorithm for Parameterization of Two-Region Piecewise Uniform Quantizer for the Laplacian Source

Motivated by the fact that uniform quantization is not suitable for signals having non-uniform probability density functions (pdfs), as the Laplacian pdf is, in this paper we have divided the support region of the quantizer into two disjunctive regions and utilized the simplest uniform quantization with equal bit-rates within both regions. In particular, we assumed a narrow central granular region (CGR) covering the peak of the Laplacian pdf and a wider peripheral granular region (PGR) where the pdf is predominantly tailed. We performed optimization of the widths of CGR and PGR via distortion optimization per border–clipping threshold scaling ratio which resulted in an iterative formula enabling the parametrization of our piecewise uniform quantizer (PWUQ). For medium and high bit-rates, we demonstrated the convenience of our PWUQ over the uniform quantizer, paying special attention to the case where 99.99% of the signal amplitudes belong to the support region or clipping region. We believe that the resulting formulas for PWUQ design and performance assessment are greatly beneficial in neural networks where weights and activations are typically modelled by the Laplacian distribution, and where uniform quantization is commonly used to decrease memory footprint.

Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

Quantization and compression of neural network parameters using the uniform scalar quantization is carried out in this paper. The attractiveness of the uniform scalar quantizer is reflected in a low complexity and relatively good performance, making it the most popular quantization model. We present a design approach for the memoryless Laplacian source with zero-mean and unit variance, which is based on iterative rule and uses the minimal mean-squared error distortion as a performance criterion. In addition, we derive closed-form expressions for SQNR (Signal to Quantization Noise Ratio) in a wide dynamic range of variance of input data. To show effectiveness on real data, the proposed quantizer is used to compress the weights of neural networks using bit rates from 9 to 16 bps (bits/sample) instead of standardly used 32 bps full precision bit rate. The impact of weights compression on the NN (neural network) performance is analyzed, indicating good matching with the theoretical results and showing negligible decreasing of the prediction accuracy of the NN even in the case of high variance-mismatch between the variance of NN weights and the variance used for the design of quantizer, if the value of the bit-rate is properly chosen according to the rule proposed in the paper.

Binary Quantization Analysis of Neural Networks Weights on MNIST Dataset

This paper considers the design of a binary scalar quantizer of Laplacian source and its application in compressed neural networks. The quantizer performance is investigated in a wide dynamic range of data variances, and for that purpose, we derive novel closed-form expressions. Moreover, we propose two selection criteria for the variance range of interest. Binary quantizers are further implemented for compressing neural network weights and its performance is analysed for a simple classification task. Good matching between theory and experiment is observed and a great possibility for implementation is indicated.

Support region of μ-law logarithmic quantizers for Laplacian source applied in neural networks

The main aim of the paper is to provide effective and accurate solutions for the calculation of the support region of the μ-law logarithmic companding quantizers. A new solution for the starting point of iterative methods will be proposed, that provides very accurate value of the support region (being the main parameter needed for the design of the quantizer) only after one iteration of the iterative method. Based on this new starting point, an accurate closed-form approximate expression for the calculation of the support region will be derived, as one of the main contributions of the paper. To significantly simplify implementation of the μ-law companding quantizer, piecewise linearization is performed. A new linearization method is presented, based on the optimization of the last segments. Derivation of an accurate closed-form formula for the support region of the linearized quantizer is done, as an important contribution. The obtained linearized μ-law companding quantizer is very simple to design (due to closed-form formulas) and to implement (due to linearization), providing at the same time very high performance (due to optimization of the last segments). Due to these and other advantages (robustness, adjustability to the statistical distribution of the input signal), the proposed quantizer can be used in many topical applications, such as in receivers of 5G wireless systems or in neural networks for quantization of weights and activations. The paper provides an application of the designed quantizers for quantization of weights of a neural network, showing significant decreasing of the bit-rate compared to the standard full-precision representation (from 32 bits to just 5 bits), with the same prediction accuracy of the network.

Design of a 2-Bit Neural Network Quantizer for Laplacian Source.

Achieving real-time inference is one of the major issues in contemporary neural network applications, as complex algorithms are frequently being deployed to mobile devices that have constrained storage and computing power. Moving from a full-precision neural network model to a lower representation by applying quantization techniques is a popular approach to facilitate this issue. Here, we analyze in detail and design a 2-bit uniform quantization model for Laplacian source due to its significance in terms of implementation simplicity, which further leads to a shorter processing time and faster inference. The results show that it is possible to achieve high classification accuracy (more than 96% in the case of MLP and more than 98% in the case of CNN) by implementing the proposed model, which is competitive to the performance of the other quantization solutions with almost optimal precision.

An Anti-Clustering Model for Stability Enhancement of a 3D Moving Particle Semi-Implicit Method and Two-Phase Coupling between MPS and Euler Grids

As a Lagrangian gridless particle method, the MPS (Moving Particle Semi-implicit) method has a wide engineering application. However, for complex 3D flows, unphysical pressure oscillations often occur and result in the failure of simulations. This paper compares the stability enhancement methods proposed by different researchers to develop a 3D, stable MPS method. The results indicate that the proposed methods are incapable of eliminating the particle clustering that leads to instability as the main source in coarser particle spacing cases. An anti-clustering model, referring to the SPH (Smoothed Particle Hydrodynamics) artificial viscosity model, is proposed to further reduce instability. Combining various proposed methods and models, several typical examples are simulated comparatively. The results are compared with those of the VOF (Volume of Fluid) model using commercial software to validate the accuracy and stability of the combination of the proposed methods and models. It is concluded that (1) 3D cases that adopt a high-order Laplacian model and high-order source terms in PPE are more accurate than those adopting the low-order operators; (2) the proposed anti-clustering model can produce a tuned interparticle force to prevent particle clustering and introduce no additional viscosity effects in the flow of the normal state, which plays a very positive role for further stability enhancement of MPS; (3) particle resolution significantly maintains simulation accuracy given the stable algorithms by the combination of stability enhancement methods. The 3D MPS method is coupled with the Euler grid (FLUENT V17 software, ANSYS, Pittsburgh, PA, USA) in two phases. In particular, the 3D MPS algorithm is used to calculate the liquid-phase change from the continuous to the dispersed, and the finite volume method based on the Euler grid is adopted to measure the corresponding gas-phase motion. The atomization of the liquid jet under static air flow is calculated and compared with the results of the VOF method, which can capture the continuous interface.

Symmetric Quantile Quantizer Parameterization for the Laplacian Source: Qualification for Contemporary Quantization Solutions

In this paper, we consider the opportunities and constraints, which rest on quantization as a guiding principle for data representation and compression. In particular, we propose a novel model of Symmetric Quantile Quantizer (SQQ) and we describe in detail its parameterization. We suggest a simple method for offline precalculation of its parameters and we examine the inevitable loss of information introduced by SQQ, as an important part of bit optimization task at the traditional network level, which can be globally mapped out in many contemporary solutions. Our anticipation is that such precalculated values can be leveraged in deterministic quantization process. We highlight that this notice heavily relies on the fact that the values of interest are distributed according to the Laplacian distribution, which we consider in the paper. The basic difference of our SQQ and the previously established asymptotically optimal quantizer model, that is, Scalar Companding Quantizer (SCQ), is reflected in the fact that, in SCQ model, both decision thresholds and representation levels are determined in accordance with the specified compressor function, whereas in our SQQ model, a precedence of SCQ model for the straightforward decision thresholds calculation is used, while the representation levels are optimally determined for the specified decision thresholds and assumed Laplacian distribution. As a result, our SQQ outperforms SCQ in terms of signal-to-quantization noise ratio (SQNR). As stated in this paper, there are numerous indications to make us believe that appropriate quantizer parameterization will move us closer to an optimization in the amount of the transferred data in bits, which is strongly dependent on the amount of SQNR.

Design and Analysis of Binary Scalar Quantizer of Laplacian Source with Applications

A compression method based on non-uniform binary scalar quantization, designed for the memoryless Laplacian source with zero-mean and unit variance, is analyzed in this paper. Two quantizer design approaches are presented that investigate the effect of clipping with the aim of reducing the quantization noise, where the minimal mean-squared error distortion is used to determine the optimal clipping factor. A detailed comparison of both models is provided, and the performance evaluation in a wide dynamic range of input data variances is also performed. The observed binary scalar quantization models are applied in standard signal processing tasks, such as speech and image quantization, but also to quantization of neural network parameters. The motivation behind the binary quantization of neural network weights is the model compression by a factor of 32, which is crucial for implementation in mobile or embedded devices with limited memory and processing power. The experimental results follow well the theoretical models, confirming their applicability in real-world applications.

Support Region of Semilogarithmic Quantizer for Laplacian Source

The overall distortion of semilogarithmic A law quantizer is influenced by its support region, as well as parameter A. Current solutions require numerical determination of optimal support region and larger processing time, i.e. simple closed-form solutions for finding optimal values of these two parameters are not provided. In this paper, iterative and closed-form solutions for determination of the support region of semilogarithmic A law quantizer for Laplacian source, unite variance and a set of bit-rates 6 bits/sample−10 bits/sample are proposed. The performance of proposed solutions is compared to the optimal numerical results.DOI: http://dx.doi.org/10.5755/j01.eie.24.4.21481

Forward Adaptive Laplacian Source Coding Based on Restricted Quantization

A novel solution for Laplacian source coding based on three-level quantization is proposed in this paper. The restricted three-level quantizer is designed by assuming the restricted Laplacian distribution of the input signal. Quantizer and Huffman encoder are jointly designed. Forward adaptive scheme was employed, where the adaptation to the signal variance (power) was performed on frame-by frame basis. We employ switched model that consists of two restricted quantizers having unequal support regions. The simulation results (measured as SQNR) of the proposed scheme with a switched restricted three-level quantizer are compared to the cases when it involves three-level unrestricted quantizer and the Lloyd-Max quantizers having N=2 and N=4 levels. It is shown that the proposed solution offers performance comparable to the one of N=4 levels Lloyd-Max’s baseline with large savings in bit rate, while outperforming two other baselines.DOI: http://dx.doi.org/10.5755/j01.itc.47.2.16670

A Rigorous Revisit to the Partial Distortion Theorem in the Case of a Laplacian Source

This letter studies rigorously the partial distortion theorem in the case of a Laplacian probability density function and proves that, while essentially each cell of a minimum mean-square error nonuniform scalar quantizer contributes to distortion more or less the same as its neighbor, each outer cell contributes slightly more than its inner neighbor. Also it is proved that the outermost cell distortion cannot exceed asymptotically 1.914 times that of the innermost cell. Thus, this letter clarifies in the Laplacian case the meaning of approximate equidistortion stated in the partial distortion theorem.

Performance of Quasi-logarithmic Quantizer for Discrete Input Signal

In this paper, performance of quasi-logarithmic quantizer, designed for correlated discrete input signal is analyzed. Quantizer design is done for Laplacian source due to its both hardware and software significance, whereas experiments are done by processing test wideband speech signal sampled at 16 [kHz]. The quantizer is exploited as a second stage of two-stage quantization system, where the first step is used for continuous signal sampling, while the second stage provides additional data compression. The main goal is to provide improved design by discussing theoretical performance of two quantization models. As the traditional models for performance estimation provide estimation of average performance, we have decided to propose a novel model for performance estimation and to analyze performance in details for each random input signal variance. Finally, the experimental results have shown excellent matching with theoretical results. DOI: http://dx.doi.org/10.5755/j01.itc.46.3.16197

Projection to Mixture Families and Rate-Distortion Bounds with Power Distortion Measures

The explicit form of the rate-distortion function has rarely been obtained, except for few cases where the Shannon lower bound coincides with the rate-distortion function for the entire range of the positive rate. From an information geometrical point of view, the evaluation of the rate-distortion function is achieved by a projection to the mixture family defined by the distortion measure. In this paper, we consider the β -th power distortion measure, and prove that β -generalized Gaussian distribution is the only source that can make the Shannon lower bound tight at the minimum distortion level at zero rate. We demonstrate that the tightness of the Shannon lower bound for β = 1 (Laplacian source) and β = 2 (Gaussian source) yields upper bounds to the rate-distortion function of power distortion measures with a different power. These bounds evaluate from above the projection of the source distribution to the mixture family of the generalized Gaussian models. Applying similar arguments to ϵ -insensitive distortion measures, we consider the tightness of the Shannon lower bound and derive an upper bound to the distortion-rate function which is accurate at low rates.

Analysis of switched quantizer based on the quadratic spline functions

ABSTRACTIn this paper, an approximation of the optimal compressor function using the quadratic spline functions with 2L = 8 segments is described. Since the quadratic spline with 2L = 8 segments provides better approximation of the optimal compression function than quadratic spline with 2L = 4 segments, capitalizing on the benefits of the obtained spline approximation, quantizer designing process is firstly performed for the so assumed number of segments and the Laplacian source of a unit variance. Then, to enhance the usability of the proposed model, the switched quantization technique is applied and a beneficial analysis is derived, providing insight in the robustness of the proposed quantizer performances with respect to the mismatch in designed for and applied to variances. Reached quality has been compared to another model from the literature, and it has been shown that the proposed model outperforms the previous model by almost 1.3 dB.