Quantization Compression Research Articles

Intelligent Fault Diagnosis Method Based on Neural Network Compression for Rolling Bearings

Rolling bearings are often exposed to high speeds and pressures, leading to the symmetry in their rotating structure being disrupted, which can lead to serious failures. Intelligent rolling bearing fault diagnosis is a critical part of ensuring operation of machinery, and it has been facilitated by the growing popularity of convolutional neural networks (CNNs). The outstanding performance of fault diagnosis CNNs results from complex and redundant network structures and parameters, resulting in huge storage and computational requirements, which makes it challenging to implement these models in resource-limited industrial devices. This study aims to address this problem by proposing a comprehensive compression method for CNNs that is applied to intelligent fault diagnosis. It involves several different compression methods, including tensor train decomposition, parameter quantization, and knowledge distillation for deep network compression. This results in a significant decrease in redundancy and speeding up the training of CNN models. Firstly, tensor train decomposition is applied to reduce redundant connections in both convolutional and fully connected layers. The next step is to perform parameter quantization to minimize the bits needed for parameter representation and storage. Finally, knowledge distillation is used to restore accuracy to the compressed model. The effectiveness of the proposed approach is confirmed by an experiment and ablation study with different models on several datasets. The results show that it can significantly reduce redundant information and floating-point operations with little degradation in accuracy. Notably, on the CWRU dataset, with about 60% parameter reduction, there is no degradation in our model’s accuracy. The proposed approach is a new attempt at the intelligent fault diagnosis of rolling bearings in industrial equipment.

Embedding Secret Data in a Vector Quantization Codebook Using a Novel Thresholding Scheme

In recent decades, information security has become increasingly valued, including many aspects of privacy protection, copyright protection, and digital forensics. Therefore, many data hiding schemes have been proposed and applied to various carriers such as text, images, audio, and videos. Vector Quantization (VQ) compression is a well-known method for compressing images. In previous research, most methods related to VQ compressed images have focused on hiding information in index tables, while only a few of the latest studies have explored embedding data in codebooks. We propose a data hiding scheme for VQ codebooks. With our approach, a sender XORs most of the pixel values in a codebook and then applies a threshold to control data embedding. The auxiliary information generated during this process is embedded alongside secret data in the index reordering phase. Upon receiving the stego codebook and the reordered index table, the recipient can extract the data and reconstruct the VQ-compressed image using the reverse process. Experimental results demonstrate that our scheme significantly improves embedding capacity compared to the most recent codebook-based methods. Specifically, we observe an improvement rate of 223.66% in a small codebook of size 64 and an improvement rate of 85.19% in a codebook of size 1024.

Edge-Federated Self-Supervised Communication Optimization Framework Based on Sparsification and Quantization Compression

Edge-Federated Self-Supervised Communication Optimization Framework Based on Sparsification and Quantization Compression

Joint Optimization of Dimension Reduction and Mixed-Precision Quantization for Activation Compression of Neural Networks

Recently, deep convolutional neural networks (CNNs) have achieved eye-catching results in various applications. However, intensive memory access of activations introduces considerable energy consumption, resulting in a great challenge for deploying CNNs on resource-constrained edge devices. Existing research utilizes dimension reduction and mixed-precision quantization separately to reduce computational complexity without paying attention to their interaction. Such naïve concatenation of different compression strategies ends up with sub-optimal performance. To develop a comprehensive compression framework, we propose an optimization system by jointly considering dimension reduction and mixed-precision quantization, which is enabled by independent group-wise learnable mixed-precision schemes. Group partitioning is guided by a well-designed automatic group partition mechanism that can distinguish compression priorities among channels, and it can deal with the trade-off between model accuracy and compressibility. Moreover, to preserve model accuracy under low bit-width quantization, we propose a dynamic bit-width searching technique to enable continuous bit-width reduction. Our experimental results show that the proposed system reaches 69.03%/70.73% with average 2.16/2.61 bits per value on Resnet18/MobileNetV2, while introducing only approximately 1% accuracy loss of the uncompressed full-precision models. Compared with individual activation compression schemes, the proposed joint optimization system reduces 55%/9% (-2.62/-0.27 bits) memory access of dimension reduction and 55%/63% (-2.60/-4.52 bits) memory access of mixed-precision quantization, respectively, on Resnet18/ MobileNetV2 with comparable or even higher accuracy.

Efficient compression at the edge for real-time data acquisition in a billion-pixel X-ray camera

Recent advances in radiation detectors have significantly improved the resolution and frame rate of X-ray imaging systems at the cost of large data throughputs that can be challenging to transfer, store, and process. Such is the case of the billion-pixel X-ray camera, where the estimated throughput of 10 terabytes per second (TB/s) exceeds the capabilities of current data acquisition systems (DAQs) for real-time transfer and processing. To address this, we propose a lossy, multi-stage compression scheme to reduce data at the source. This work leverages machine learning (ML) to produce sparse representations of the camera’s images, followed by quantization and entropy coding for compression. The proposed scheme achieves a 100:1 compression ratio with high PSNR and SSIM scores on X-ray source images. Finally, we implement the sparse coding neural network (NN) on an FPGA to showcase the feasibility, low power, and low latency of edge data compression.

Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

In this paper, a new methodology is proposed that allows for the low-complexity development of neural network (NN) based equalizers for the mitigation of impairments in high-speed coherent optical transmission systems. In this work, we provide a comprehensive description and comparison of various deep model compression approaches that have been applied to feed-forward and recurrent NN designs. Additionally, we evaluate the influence these strategies have on the performance of each NN equalizer. Quantization, weight clustering, pruning, and other cutting-edge strategies for model compression are taken into consideration. In this work, we propose and evaluate a Bayesian optimization-assisted compression, in which the hyperparameters of the compression are chosen to simultaneously reduce complexity and improve performance. Next, this paper presents four distinct metrics (RMpS, BoP, NABS, and NLGs) that are discussed here that can be used to evaluate the amount of computing complexity required by various compression algorithms. These measurements can serve as a benchmark for evaluating the relative effectiveness of various NN equalizers when compression approaches are used. In conclusion, the trade-off between the complexity of each compression approach and its performance is evaluated by utilizing both simulated and experimental data in order to complete the analysis. By utilizing optimal compression approaches, we show that it is possible to design an NN-based equalizer that is simpler to implement and has better performance than the conventional digital back-propagation (DBP) equalizer with only one step per span. This is accomplished by reducing the number of multipliers used in the NN equalizer after applying the weighted clustering and pruning algorithms. Furthermore, we demonstrate that an equalizer based on NN can also achieve superior performance while still maintaining the same degree of complexity as the full electronic chromatic dispersion compensation block. We conclude our analysis by highlighting open questions and existing challenges, as well as possible future research directions.

FedMPT: Federated Learning for Multiple Personalized Tasks Over Mobile Computing

Federated learning (FL) is a privacy-preserving collaborative learning framework that can be used in mobile computing where multiple user devices jointly train a deep learning model without uploading their data to a centralized server. An essential issue of FL is to reduce the significant communication overhead during training. Existing FL schemes mostly address this issue regarding single task learning. However, each user generally has multiple related tasks on the mobile device such as multi-content recommendation, and traditional FL schemes need to train an individual model per task which consumes a substantial number of resources. In this work, we formulate an FL problem with multiple personalized tasks, which aims to minimize the communication cost in learning different personalized tasks on each device. To solve the formulated problem, we incorporate multi-task learning into FL which trains a model for multiple tasks concurrently and propose an FL framework named FedMPT. FedMPT modifies the efficient acceleration algorithm and quantization compression strategy delicately to achieve superior performance regarding the communication efficiency. We implement and evaluate FedMPT on two datasets, Multi-MNIST and CelebA, in the FL environment. Experimental results show that FedMPT significantly outperforms the traditional FL scheme considering communication cost and average accuracy.

Non-Decimated Wavelet Transform and Vector Quantization for Lossy Medical Images Compression

This study presents a new approach for lossy medical image compression using vector quantization. Recently, the digital image has been a reliable replacement for a hard copy of medical images, therefore, an effort has been made to ensure maintaining high-quality images to use for archiving, classification, or automated diagnostics support. Although the medical application contains all sorts of the images like microscopic, X-rays, tomography, and fiber optics imaging by angioplasty, all of this comes at the cost of using digital storage that needs to be regularly backed up and maintained and to help minimize the need for larger storage media, this study is focusing on applying Non-Decimated Wavelet Transform (NDWT) and combined lossy and lossless compression techniques that will allow the images to take much smaller storage space while maintaining the high level of quality for these images. This study is focusing on chest X-ray images compression using a combination of lossy compression techniques using two Vector Quantization (VQ) algorithms such as k-means clustering and Linde, Buzo, and Gray (LBG) algorithm, and three lossless compression techniques such as Arithmetic Coding (AC), Run Length Encoding (RLE) and Huffman Coding (HC) and choose the optimum combination of them. Then, the performance is measured using Compression Ratio (CR), processing time, or called run time, Peak Signal to Noise Ratio (PSNR), and Bit Rate.

The Improvement of Federated Learning in communication efficiency

In recent years, the need to protect user privacy and data security as well as multi-party data cooperation has become increasingly strong. Privacy computing is considered a fundamental solution that is growing rapidly and gaining unprecedented popularity. Currently, federated learning is the most popular product in privacy computing. With its lightweight technology route and deployment, it has become a mainstream solution and product of choice for many privacy computing applications, such as providing advanced medical image collection segmentation technology in the medical field to protect patient privacy, where it can play a unique role. But Federated learning faces a need to improve communication efficiency. This paper introduces FedAvg, the most basic algorithm for federated learning, including its basic process, advantages and disadvantages. We analyze algorithms that optimize federated learning by adding local computation, Communication Compression (including model Compression, Periodic Averaging and Quantization Compression, and gradient Compression), computation and communication overlap, and compare the strengths and weaknesses of representative algorithms for each approach. At the end of this paper, the summary and prospect are made.

A gradient optimization and manifold preserving based binary neural network for point cloud

With significant progress of deep learning on 3D point cloud, the demand for deployment of point cloud neural network on the edge devices is growing. Binary neural network, a type of quantization compression method, with extreme low bit and fast inference speed, attracts more attention. It is more challenging, but has greater potentiality. Most of the researches on binary networks focus on images rather than point cloud. Considering the particularity of point cloud neural network, this paper presents a novel binarization framework, which includes two main contributions. Firstly, a gradient optimization method is proposed to overcome the shortcomings of Straight Through Estimator (STE) commonly used in the back propagation of binary network training. Secondly, based on the analysis of manifold distortion caused by the binary convolution and pooling operations, we propose an optimized scaling recovery method to restore manifold for the convoluted feature, and also, a pooling correction method to improve the pooled feature's fidelity. Manifold distortion leads to the severe feature homogeneity problem, which brings trouble in generating features with sufficient discrimination for classification and segmentation. The manifold preserving optimizations are designed to introduce minimum extra parameters to balance the accuracy with the computation and storage consumption. Experiments show that the proposed method outperforms state-of-the-art in accuracy with ignored overhead, and also has good scalability.

Differential Weight Quantization for Multi-Model Compression

Differential Weight Quantization for Multi-Model Compression

A Study on the Application of TensorFlow Compression Techniques to Human Activity Recognition

In the human activity recognition (HAR) application domain, the use of deep learning (DL) algorithms for feature extractions and training purposes delivers significant performance improvements with respect to the use of traditional machine learning (ML) algorithms. However, this comes at the expense of more complex and demanding models, making harder their deployment on constrained devices traditionally involved in the HAR process. The efficiency of DL deployment is thus yet to be explored. We thoroughly investigated the application of TensorFlow Lite simple conversion, dynamic, and full integer quantization compression techniques. We applied those techniques not only to convolutional neural networks (CNNs), but also to long short-term memory (LSTM) networks, and a combined version of CNN and LSTM. We also considered two use case scenarios, namely cascading compression and stand-alone compression mode. This paper reports the feasibility of deploying deep networks onto an ESP32 device, and how TensorFlow compression techniques impact classification accuracy, energy consumption, and inference latency. Results show that in the cascading case, it is not possible to carry out the performance characterization. Whereas in the stand-alone case, dynamic quantization is recommended because yields a negligible loss of accuracy. In terms of power efficiency, both dynamic and full integer quantization provide high energy saving with respect to the uncompressed models: between 31% and 37% for CNN networks, and up to 45% for LSTM networks. In terms of inference latency, dynamic and full integer quantization provide comparable performance.

Construction of a Knowledge Map of Speech Emotion Features Based on Impulse-Coupled Neural Networks.

This paper constructs knowledge graphs of speech emotion feature to support the expression of rich semantic information due to their unique graphical structure and to provide new ideas for studying speech emotion recognition recommendations. Impulse-coupled neural networks, as a mathematical abstraction of the visual properties of the human eye, have been widely used in various fields of speech processing. In this paper, we take the classical impulse-coupled neural network model as the research object, aiming to explore, analyze, and study the method of constructing knowledge maps of speech emotion features based on the impulse-coupled neural network model, and propose several improved impulse-coupled neural network models, which are used in the fields of target detection, speech segmentation, and quantization compression. This paper further constructs various domain semantic concept maps and performs multidimensional semantic enhancement understanding of search text based on the concept maps and the constructed entity-relationship knowledge maps and proposes a knowledge-based interpretable recommendation method for cloud services and a generalized recommendation and sample enhancement method for cloud scenarios. The fusion algorithm based on multiscale analysis can decompose the source speech into subspeech at different scales and then fuse each subspeech separately, while typical objects in the real world are also composed of many components at different scales. Based on the factors of different degrees in the speech context, the imagery work ruler evaluation model is refined with the basic principles of risk perception and risk resolution in the context, and an exploratory design is carried out separately for the verbal description, graphics, text, color, animation, and sound signals in the audiovisual signal using the workshop format to obtain a large number of design samples and evaluate the important units in turn, finally integrating the context involved in the study. The perceptual design system is given as a specific contextual design method.

Adaptive sparse ternary gradient compression for distributed DNN training in edge computing

In edge computing, though distributed training of Deep Neural Networks (DNNs) is expected to exchange massive gradients between parameter servers and working nodes, the high communication cost constrains the training speed. To break this limitation, gradient compression algorithms expect the ultimate compression ratio at the expense of the accuracy of the trained model. Therefore, new gradient compression techniques are necessary to ensure both communication efficiency and model accuracy. This paper introduces a novel technique—an Adaptive Sparse Ternary Gradient Compression (ASTC) scheme, which relies on the number of gradients in model layers to compress gradients. ASTC establishes the model compression selection criterion by gradients’ amount, compresses the network layer that meets the model’s standard, evaluates the gradients’ importance based on entropy to adaptively perform sparse compression, and finally conducts ternary quantization compression and a lossless code scheme on sparse gradients. Using public datasets (MNIST, CIFAR-10, Tiny ImageNet) and deep learning models (CNN, LeNet5, ResNet18) for experimental evaluation, we exhibit excellent results that the training efficiency of ASTC is about 1.6 times, 1.37 times, and 1.1 times higher than that of Top-1, AdaComp, and SBC, respectively. Furthermore, ASTC can be improved by an average of about 1.9% in training accuracy compared with the above approaches.

UAV Anti-Jamming Video Transmissions With QoE Guarantee: A Reinforcement Learning-Based Approach

Unmanned aerial vehicles (UAVs) that are widely utilized for video capturing, processing and transmission have to address jamming attacks with dynamic topology and limited energy. In this paper, we propose a reinforcement learning (RL)-based UAV anti-jamming video transmission scheme to choose the video compression quantization parameter, the channel coding rate, the modulation and power control strategies against jamming attacks. More specifically, this scheme applies RL to choose the UAV video compression and transmission policy based on the observed video task priority, the UAV-controller channel state and the received jamming power. This scheme enables the UAV to guarantee the video quality-of-experience (QoE) and reduce the energy consumption without relying on the jamming model or the video service model. A safe RL-based approach is further proposed, which uses deep learning to accelerate the UAV learning process and reduce the video transmission outage probability. The computational complexity is provided and the optimal utility of the UAV is derived and verified via simulations. Simulation results show that the proposed schemes significantly improve the video quality and reduce the transmission latency and energy consumption of the UAV compared with existing schemes.

Rethinking Adaptive Computing: Building a Unified Model Complexity-Reduction Framework With Adversarial Robustness.

Adaptive computing (AC) is a technique to dynamically select the layers to pass in a prespecified deep neural network (DNN) according to the input samples. In previous literature, AC was deemed as a standalone complexity-reduction skill. This brief studies AC through a different lens: we investigate how this strategy interacts with mainstream compression techniques in a unified complexity-reduction framework and whether its "input sample related" feature helps with the improvement of model robustness. Following this direction, we first propose a defensive accelerating branch (DAB) based on the AC strategy that can reduce the average computational cost and inference time of DNNs with higher accuracy compared with its counterparts. Then, the proposed DAB is jointly applied with the mainstream parameterwise compression skills, pruning and quantization, to build a unified complexity-reduction framework. Extensive experiments are conducted, and the results reveal quasi-orthogonality between the input-related and parameterwise complexity-reduction skills, which means that the proposed AC can be integrated into an off-the-shelf compressed model without hurting its accuracy. Besides, the robustness of the proposed compression framework is explored, and the experimental results demonstrate that DAB can be used as both the detector and the defensive tool when the model is under adversarial attacks. All these findings shed light on the great potential of DAB in building a unified complexity-reduction framework with both a high compression ratio and great adversarial robustness.

Lossless Image Compression Using Wavelet and Vector Quantization.(Dept.E)

In this paper, we propose a new low bit rate scheme that combines the Discrete Wavelet transform (IWT) and Vector Quantization (VQ) for image compression with high Compression Ratio (CR) in order to achieve acceptable quality of output images, Image compression using the combination of DWT willy VQ has been considered in many recent works; here in our paper the DWT was performed on several gray scale standard images, each of (256X256) byte. We used 4 images as the training images of the train VO using LBG algorithm and got the final optimal codebook. Then we used another two images to compare our output image with JPEG 2000 standard winch based on Wavelet and Scalar quantization. Our proposed system used a VO codebook with Variable cod hook size and dimensions were used to compress wavelet Transform) coefficients resulting using Danny wavelet types. Our pro system is slowed so that the best Wavelet types that you can use for compression is COIF 2. Our proposed system gave better PSNR than G2000 standard at high (CR), above than (5L), and this achieved a VQ codebook size (1024 -512) with dimension equal to 256. In the paper a typical system for the application of image compression system and new scientific results are present.

Dynamic Logarithmic Vector Quantizer Design for Stochastic Continuous Time Linear Systems With State Feedback Control

This paper addresses logarithmic vector quantizers with dynamic sensitivity design for stochastic continuous time linear system with a feedback control law. Logarithmic vector quantization is adopted so that the coarsest density of quantization and data compression efficiency are both guaranteed. In addition, the problem of quantizer saturation is treated with the mechanism of dynamic sensitivity. The dynamic of the sensitivity during “zoom-in” / “zoom-out” stages is proposed. It is shown that with the proposed algorithm, a single input continuous time linear system can be stabilized in the sense of mean square by quantized feedback control via adopting sensitivity varying algorithm. Results from theoretical analysis are verified through simulation at the end of the paper.

Large system analysis of downlink C-RAN with phase noise and fronthaul compression

This paper studies the effect of phase noise and fronthaul compression on a downlink cloud radio access network (C-RAN), where several remote radio heads (RRHs) are coordinated to communicate with users by a baseband unit (BBU) on the cloud server. In the system, the baseband signals are precoded at BBU, and then compressed before being transmitted to RRHs through capacity-limited fronthaul links which results in the compressive quantization noise. We assume the regularized zero-forcing precoding is performed with an imperfect channel state information and a compression strategy is applied at BBU. The effect of phase noise arising from nonideal local oscillators both at RRHs and users is considered. We propose an approximate expression for the downlink ergodic sum-rate of considered C-RAN utilizing large dimensional random matrix theory in the large-system regime. From simulation results, the accuracy of the approximate expression is validated, and the effect of phase noise and fronthaul compression can be analyzed theoretically based on the approximate expression.

Image compression and encryption scheme with hyper-chaotic system and mean error control

ABSTRACTBased on a hyper-chaotic system, an image compression and encryption scheme is presented by adopting mean error control to improve the quantization compression performance of the discrete cosine transform (DCT) coefficients. The transform coefficients of DCT forming a structure similar to the sub-band can be classified and encoded. Then the image is scanned with Zigzag operation in different directions to achieve sufficiently scrambled effects. The cyclic shift matrix for image encryption is generated by the hyper-chaotic system. The four initial inputs for generating the cyclic shift matrix serve as the key. Simulation results demonstrate the security and the effectiveness of the proposed image compression and encryption scheme.