Model Compression Methods Research Articles

Convolutional neural network model compression method

In the field of deep learning, Convolutional Neural Networks (CNN) has become a focal point due to its multi-layered structure and wide application. The success of deep learning is due to the model has more layers and more parameters, which gives it a stronger nonlinear fitting ability. Traditionally, CNNs are primarily run on Central Processing Units (CPUs) and Graphics Processing Units (GPUs). However, CPUs have lower computational power, and GPUs consume a lot of energy. In contrast, Field-Programmable Gate Arrays (FPGAs) offer high parallelism, low power consumption, flexible programming, and rapid development cycles. These combined advantages make FPGAs more suitable for the forward inference processes of deep learning compared to other platforms. However, CNNs has the characteristics of parameter redundancy, the storage cost of deploying it on FPGA is too high. In order to apply CNNs to FPGA, we need to optimize CNNs for compression. Because after compression, This paper analyzes several specific cases of model compression for convolutional neural networks, and summarizes and compares the efficient methods of model compression. The results show that the model compression methods are mainly divided into the structure change of convolutional neural network and the quantization of parameters. The two compression methods can be cascaded at the same time to achieve a better optimization effect.

DCCD: Reducing Neural Network Redundancy via Distillation.

Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.

MCMC: Multi-Constrained Model Compression via One-Stage Envelope Reinforcement Learning.

Model compression methods are being developed to bridge the gap between the massive scale of neural networks and the limited hardware resources on edge devices. Since most real-world applications deployed on resource-limited hardware platforms typically have multiple hardware constraints simultaneously, most existing model compression approaches that only consider optimizing one single hardware objective are ineffective. In this article, we propose an automated pruning method called multi-constrained model compression (MCMC) that allows for the optimization of multiple hardware targets, such as latency, floating point operations (FLOPs), and memory usage, while minimizing the impact on accuracy. Specifically, we propose an improved multi-objective reinforcement learning (MORL) algorithm, the one-stage envelope deep deterministic policy gradient (DDPG) algorithm, to determine the pruning strategy for neural networks. Our improved one-stage envelope DDPG algorithm reduces exploration time and offers greater flexibility in adjusting target priorities, enhancing its suitability for pruning tasks. For instance, on the visual geometry group (VGG)-16 network, our method achieved an 80% reduction in FLOPs, a 2.31× reduction in memory usage, and a 1.92× acceleration, with an accuracy improvement of 0.09% compared with the baseline. For larger datasets, such as ImageNet, we reduced FLOPs by 50% for MobileNet-V1, resulting in a 4.7× faster speed and 1.48× memory compression, while maintaining the same accuracy. When applied to edge devices, such as JETSON XAVIER NX, our method resulted in a 71% reduction in FLOPs for MobileNet-V1, leading to a 1.63× faster speed, 1.64× memory compression, and an accuracy improvement.

Semi-Supervised Knowledge Distillation Via Teaching Assistant

As deep neural networks are widely used with computer vision, Model Compression Methods for knowledge distillation are being actively investigated in order to deploy them into smaller devices. However, when there are significant differences between models for students and teachers, a large amount of labeling wastes a great quantity of manpower while the performance of student learning decreases. In this paper, we propose an approach that uses semi-supervised teacher assistants to fuse knowledge distillation with teachers, effectively bridging the large teacher-student gap on a large number of unlabeled datasets and a small number of labeled datasets. We enable unsupervised teacher pre-training, fine-tuning, followed by teacher-assistant offline distillation, student-teacher-assistant oline distillation, and student-teacher offline distillation. We validate the effectiveness of the proposed method for the classification task using CIFAR10, CIFAR-100, and ImageNet.

RamanCMP: A Raman spectral classification acceleration method based on lightweight model and model compression techniques

In recent years, Raman spectroscopy combined with deep learning techniques has been widely used in various fields such as medical, chemical, and geological. However, there is still room for optimization of deep learning techniques and model compression algorithms for processing Raman spectral data. To further optimize deep learning models applied to Raman spectroscopy, in this study time, accuracy, sensitivity, specificity and floating point operations numbers(FLOPs) are used as evaluation metrics to optimize the model, which is named RamanCompact(RamanCMP). The experimental data used in this research are selected from the RRUFF public dataset, which consists of 723 Raman spectroscopy data samples from 10 different mineral categories. In this paper, 1D-EfficientNet adapted to the spectral data as well as 1D-DRSN are proposed to improve the model classification accuracy. To achieve better classification accuracy while optimizing the time parameters, three model compression methods are designed: knowledge distillation using 1D-EfficientNet model as a teacher model to train convolutional neural networks(CNN), proposing a channel conversion method to optimize 1D-DRSN model, and using 1D-DRSN model as a feature extractor in combination with linear discriminant analysis(LDA) model for classification. Compared with the traditional LDA and CNN models, the accuracy of 1D-EfficientNet and 1D-DRSN is improved by more than 20%. The time of the distilled model is reduced by 9680.9s compared with the teacher model 1D-EfficientNet under the condition of losing 2.07% accuracy. The accuracy of the distilled model is improved by 20% compared to the CNN student model while keeping inference efficiency constant. The 1D-DRSN optimized with channel conversion method saves 60% inference time of the original 1D-DRSN model. Feature extraction reduces the inference time of 1D-DRSN model by 93% with 94.48% accuracy. This study innovatively combines lightweight models and model compression algorithms to improve the classification speed of deep learning models in the field of Raman spectroscopy, forming a complete set of analysis methods and laying the foundation for future research.

A CNN Compression Method via Dynamic Channel Ranking Strategy

In recent years, the rapid development of mobile devices and embedded system raises a demand for intelligent models to address increasingly complicated problems. However, the complexity of the structure and extensive parameters press significantly on efficiency, storage space, and energy consumption. Additionally, the explosive growth of tasks with enormous model structures and parameters makes it impossible to compress models manually. Thus, a standardized and effective model compression solution achieving lightweight neural networks is established as an urgent demand by the industry. Accordingly, Dynamic Channel Ranking Strategy (DCRS) method is proposed to compress deep convolutional neural networks. DCRS selects channels with high contribution of each prunable layer according to compression ratio searched by reinforcement learning agent. Compared with current model compression methods, DCRS efficaciously applies various channel ranking strategies on prunable layers. Experiments indicate with a 50% compression ratio, compressed MobileNet achieved 70.62% top1 and 88.2% top5 accuracy on ImageNet, and compressed ResNet achieved 92.03% accuracy on CIFAR-10. DCRS reduces more FLOPS in these neural networks. The compressed model achieves the best Top-1 and Top-5 accuracy on ResNet50, the best Top-1 accuracy on MobilNetV1.

Deep Learning Model Compression Techniques: Advances, Opportunities, and Perspective

Recently, deep learning (DL) models have excelled in a wide range of fields. All of these successes are built on intricate DL models. The hundreds of millions or even billions of parameters and high-performance computing graphical processing units or tensor processing units are largely responsible for their achievement. DL model integration into real-time devices with tight latency limitations, limited memory, and power-constrained requirements is the key driving force behind investigation of DL model compression techniques. Also, there is an increase in data availability that encourages multimodal fusion in DL models to boost the models' predictive accuracy. In order to create compact DL models for deployment that is memory- and computationally efficient, the data included in the network parameters is compressed as much as possible, leaving only the bits necessary to carry out the task. A better trade-off between compression rate and accuracy loss should be established to take model acceleration and compression into consideration without severely reducing the model's performance. In this paper, we examine various DL model compression techniques used for both single- modality and multi-modal deep learning tasks. We explore over numerous DL model compression methods that have advanced in a number of applications. We then come up with the benefits and drawbacks of various compression and acceleration methods such as ineffectiveness in compressing more complicated networks with dimensionality-dependent complex structures, and, ultimately, the field's future prospects are given.

Network Collaborative Pruning Method for Hyperspectral Image Classification Based on Evolutionary Multi-Task Optimization

Neural network models for hyperspectral images classification are complex and therefore difficult to deploy directly onto mobile platforms. Neural network model compression methods can effectively optimize the storage space and inference time of the model while maintaining the accuracy. Although automated pruning methods can avoid designing pruning rules, they face the problem of search efficiency when optimizing complex networks. In this paper, a network collaborative pruning method is proposed for hyperspectral image classification based on evolutionary multi-task optimization. The proposed method allows classification networks to perform the model pruning task on multiple hyperspectral images simultaneously. Knowledge (the important local sparse structure of the network) is automatically searched and updated by using knowledge transfer between different tasks. The self-adaptive knowledge transfer strategy based on historical information and dormancy mechanism is designed to avoid possible negative transfer and unnecessary consumption of computing resources. The pruned networks can achieve high classification accuracy on hyperspectral data with limited labeled samples. Experiments on multiple hyperspectral images show that the proposed method can effectively realize the compression of the network model and the classification of hyperspectral images.

DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Vision Transformers.

Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. In this paper, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, that can generalized well on multiple dimensions in both CNNs and transformers (e.g. kernel size, embedding dimension, number of heads, etc.). Instead of adaptively selecting important weight elements in a sparse way, we pre-define dense weight slices with different importance level by nested residual learning. During inference, weights are progressively sliced beginning with the most important elements to less important ones to achieve different model capacity for inputs with diverse difficulty levels. Based on this conception, we present DS-CNN++ and DS-ViT++, by carefully designing the double headed dynamic gate and the overall network architecture. We further propose dynamic idle slicing to address the drastic reduction of embedding dimension in DS-ViT++. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme. In Stage I, in-place bootstrapping (IB) and multi-view consistency (MvCo) are proposed to stablize and improve the training of DS-CNN++ and DS-ViT++ supernet, respectively. In Stage II, sandwich gate sparsification (SGS) is proposed to assist the gate training. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our methods consistently outperform the state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, we achieves 2-4× computation reduction and up to 61.5% real-world acceleration on MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops on ImageNet. Code release: https://github.com/changlin31/DS-Net.

Energy-Aware AI-Driven Framework for Edge-Computing-Based IoT Applications

The significant growth in the number of Internet of Things (IoT) devices has given impetus to the idea of edge computing for several applications. In addition, energy harvestable or wireless-powered wearable devices are envisioned to empower the edge intelligence in IoT applications. However, the intermittent energy supply and network connectivity of such devices in scenarios including remote areas and hard-to-reach regions such as in-body applications can limit the performance of edge computing-based IoT applications. Hence, deploying state-of-the-art convolutional neural networks (CNNs) on such energy-constrained devices is not feasible due to their computational cost. Existing model compression methods, such as network pruning and quantization can reduce complexity, but these methods only work for fixed computational or energy requirements, which is not the case for edge devices with an intermittent energy source. In this work, we propose a pruning scheme based on deep reinforcement learning (DRL), which can compress the CNN model adaptively according to the energy dictated by the energy management policy and accuracy requirements for IoT applications. The proposed energy policy uses predictions of energy to be harvested and dictates the amount of energy that can be used by the edge device for deep learning inference. We compare the performance of our proposed approach with existing state-of-the-art CNNs and data sets using different filter-ranking criteria and pruning ratios. We observe that by using DRL-driven pruning, the convolutional layers that consume relatively higher energy are pruned more as compared to their counterparts. Thereby, our approach outperforms existing approaches by reducing energy consumption and maintaining accuracy.

Towards Automatic Model Compression via a Unified Two-Stage Framework

Deep Neural Networks have become ubiquitous in various domains. Meanwhile, the problems of massive storage and computation costs have hindered the deployment of these models to real-world applications. This paper proposes a novel and unified two-stage framework for automatic model compression. To determine the compression ratio of each layer, we improve the optimization from two aspects. First, to predict the performance of each compression policy, we propose Dynamic BN, which improves the correlation significantly with little computation overhead. Second, to search for the compression ratio allocation, we propose an efficient and hyperparameter-free solving algorithm based on the proposed Hessian matrix approximation and Knapsack problem reformulation. Moreover, comprehensive experiments and analyses are conducted on the CIFAR-100&ImageNet datasets and various network architectures to demonstrate its performance advantages over existing model compression methods under the quantization-only, pruning-only, and pruning-quantization settings.

Model Compression for Deep Neural Networks: A Survey

Currently, with the rapid development of deep learning, deep neural networks (DNNs) have been widely applied in various computer vision tasks. However, in the pursuit of performance, advanced DNN models have become more complex, which has led to a large memory footprint and high computation demands. As a result, the models are difficult to apply in real time. To address these issues, model compression has become a focus of research. Furthermore, model compression techniques play an important role in deploying models on edge devices. This study analyzed various model compression methods to assist researchers in reducing device storage space, speeding up model inference, reducing model complexity and training costs, and improving model deployment. Hence, this paper summarized the state-of-the-art techniques for model compression, including model pruning, parameter quantization, low-rank decomposition, knowledge distillation, and lightweight model design. In addition, this paper discusses research challenges and directions for future work.

Real‐time seed sorting system via 2D information entropy‐based CNN pruning and TensorRt acceleration

AbstractSeed sorting based on deep neural networks is one of the important applications of seed variety identification and quality purification. However, DNNs is difficult to deploy on embedded devices since the consumption of computational and storage resource. To address these problems, this paper proposes a pipeline‐style neural network framework for real‐time seed sorting. First, we propose a novel algorithm, 2D information entropy, pruning redundant filters to realize structured pruning. Then, the pruning rate of each convolution layer is determined by visualizing the results of 2D entropy. Meanwhile, the pruned network is fine‐tuned to recover the performance. Finally, TensorRT is utilized to optimize and accelerate the pruned model for deployment in Jeston Nano. Experiments on two large‐scale seed‐sorting datasets demonstrate the significant improvement of the proposed method over existing model compression methods. Experimental results on Jeston Nano show that the pruned model 2EFP‐E achieves a single image inference speed of 107 FPS, with the best accuracy of 95.94% on the red kidney bean dataset.

Automatic Compression of Neural Network with Deep Reinforcement Learning Based on Proximal Gradient Method

In recent years, the model compression technique is very effective for deep neural network compression. However, many existing model compression methods rely heavily on human experience to explore a compression strategy between network structure, speed, and accuracy, which is usually suboptimal and time-consuming. In this paper, we propose a framework for automatically compressing models through the actor–critic structured deep reinforcement learning (DRL) which interacts with each layer in the neural network, where the actor network determines the compression strategy and the critic network ensures the decision accuracy of the actor network through predicted values, thus improving the compression quality of the network. To enhance the prediction performance of the critic network, we impose the L1 norm regularizer on the weights of the critic network to obtain a distinct activation output feature on the representation, thus enhancing the prediction accuracy of the critic network. Moreover, to improve the decision performance of the actor network, we impose the L1 norm regularizer on the weights of the actor network to improve the decision accuracy of the actor network by removing the redundant weights in the actor network. Furthermore, to improve the training efficiency, we use the proximal gradient method to optimize the weights of the actor network and the critic network, which can obtain an effective weight solution and thus improve the compression performance. In the experiment, in MNIST datasets, the proposed method has only a 0.2% loss of accuracy when compressing more than 70% of neurons. Similarly, in CIFAR-10 datasets, the proposed method compresses more than 60% of neurons, with only 7.1% accuracy loss, which is superior to other existing methods. In terms of efficiency, the proposed method also cost the lowest time among the existing methods.

Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach

One of the main focuses in distributed learning is communication efficiency, since model aggregation at each round of training can consist of millions to billions of parameters. Several model compression methods, such as gradient quantization and sparsification, have been proposed to improve the communication efficiency of model aggregation. However, the information-theoretic minimum communication cost for a given distortion of gradient estimators is still unknown. In this paper, we study the fundamental limit of communication cost of model aggregation in distributed learning from a rate-distortion perspective. By formulating the model aggregation as a vector Gaussian CEO problem, we derive the rate region bound and sum-rate-distortion function for the model aggregation problem, which reveals the minimum communication rate at a particular gradient distortion upper bound. We also analyze the communication cost at each iteration and total communication cost based on the sum-rate-distortion function with the gradient statistics of real-world datasets. It is found that the communication gain by exploiting the correlation between worker nodes is significant for SignSGD, and a high distortion of gradient estimator can achieve low total communication cost in gradient compression.

A Survey on Deep-Learning-Based Real-Time SAR Ship Detection

Recently, deep learning has greatly promoted the development of SAR ship detection. But the detectors are usually heavy and computation intensive which hinder the usage on the edge. In order to solve this problem, a lot of lightweight networks and acceleration ideas are proposed. In this survey, we review the papers that about real-time SAR ship detection. We firstly introduce the model compression and acceleration methods. They are pruning, quantization, knowledge distillation, low-rank factorization, lightweight networks and model deployment. They are the source of innovation in real-time SAR ship detection. Then we summarize the real-time object detection methods. They are two-stage, single-stage, anchor free, trained from scratch, model compression and acceleration. Researchers in SAR ship detection usually learn from these ideas. We then spend a lot of content on the review of the 70 real-time SAR ship detection papers. The years, datasets, journals, deep learning frameworks, and hardwares are introduced firstly. After that, the 10 public datasets and the evaluation metrics are shown. Then, we survey the 70 papers according to anchor free, trained from scratch, YOLO series, CFAR+CNN, lightweight backbone, pruning, quantization, knowledge distillation and hardware deployment. The experimental results show that the algorithms have been greatly developed in speed and accuracy. In the end we pointed out the problems of the 70 papers and the directions to be studied in the future. Our work can enable researchers to quickly understand the research status in this field.

Differentiable Neural Network Pruning to Enable Smart Applications on Microcontrollers

Wearable, embedded, and IoT devices are a centrepiece of many ubiquitous computing applications, such as fitness tracking, health monitoring, home security and voice assistants. By gathering user data through a variety of sensors and leveraging machine learning (ML), applications can adapt their behaviour: in other words, devices become "smart". Such devices are typically powered by microcontroller units (MCUs). As MCUs continue to improve, smart devices become capable of performing a non-trivial amount of sensing and data processing, including machine learning inference, which results in a greater degree of user data privacy and autonomy, compared to offloading the execution of ML models to another device. Advanced predictive capabilities across many tasks make neural networks an attractive ML model for ubiquitous computing applications; however, on-device inference on MCUs remains extremely challenging. Orders of magnitude less storage, memory and computational ability, compared to what is typically required to execute neural networks, impose strict structural constraints on the network architecture and call for specialist model compression methodology. In this work, we present a differentiable structured pruning method for convolutional neural networks, which integrates a model's MCU-specific resource usage and parameter importance feedback to obtain highly compressed yet accurate models. Compared to related network pruning work, compressed models are more accurate due to better use of MCU resource budget, and compared to MCU specialist work, compressed models are produced faster. The user only needs to specify the amount of available computational resources and the pruning algorithm will automatically compress the network during training to satisfy them. We evaluate our methodology using benchmark image and audio classification tasks and find that it (a) improves key resource usage of neural networks up to 80x; (b) has little to no overhead or even improves model training time; (c) produces compressed models with matching or improved resource usage up to 1.4x in less time compared to prior MCU-specific model compression methods.

Cluster-Based Structural Redundancy Identification for Neural Network Compression

The increasingly large structure of neural networks makes it difficult to deploy on edge devices with limited computing resources. Network pruning has become one of the most successful model compression methods in recent years. Existing works typically compress models based on importance, removing unimportant filters. This paper reconsiders model pruning from the perspective of structural redundancy, claiming that identifying functionally similar filters plays a more important role, and proposes a model pruning framework for clustering-based redundancy identification. First, we perform cluster analysis on the filters of each layer to generate similar sets with different functions. We then propose a criterion for identifying redundant filters within similar sets. Finally, we propose a pruning scheme that automatically determines the pruning rate of each layer. Extensive experiments on various benchmark network architectures and datasets demonstrate the effectiveness of our proposed framework.

Feature distillation Siamese networks for object tracking

In recent years, Siamese-based trackers have shown remarkable improvement in visual tracking. The general trends are interested in making deeper and more complicated networks to pursue higher accuracy. However, these advances result in cumbersome trackers with respect to size and speed, which hinders the deployment of deep trackers on edge devices. Due to the tracking scenario complexity and temporal coherence, the backbone network of deep trackers emphasizes the target appearance information more. However, the appearance feature is sensitive to parameter variation, making the traditional deep model compression methods hard to compress a deep tracker. To bridge the gap between deep Siamese trackers and practical use, we propose a new feature distillation algorithm suitable for deep trackers in this paper. Firstly, motivated by the concept of divide-and-conquer, we formulate the feature distillation into a stepwise distillation problem and perform distillation on each minimum unit to relieve the hard-to-distill problem of appearance feature. Secondly, we reconstruct the student model into a combination of convolution kernels and a point-wise convolutional layer, which enables the student model to inherit all the parameters of the teacher model during initialization. Finally, we propose a 3-step warm-up training strategy to address the student model’s degradation and structural adaptation problems during training. Extensive experiments on eight benchmarks demonstrate that our proposed method compresses the fully convolutional Siamese Networks (SiamFC) and its variant Siamd and achieves leading tracking performance with only 2.1 MB model size while running at 225 fps.

Model Compression and Acceleration: Lip Recognition Based on Channel-Level Structured Pruning

In recent years, with the rapid development of deep learning, the requirements for the performance of the corresponding real-time recognition system are getting higher and higher. However, the rapid expansion of data volume means that time delay, power consumption, and cost have become problems that cannot be ignored. In this case, the traditional neural network is almost impossible to use to achieve productization. In order to improve the potential problems of a neural network facing a huge number of datasets without affecting the recognition effect, the model compression method has gradually entered people’s vision. However, the existing model compression methods still have some shortcomings in some aspects, such as low rank decomposition, transfer/compact convolution filter, knowledge distillation, etc. These problems enable the traditional model compression to cope with the huge amount of computation brought by large datasets to a certain extent, but also make the results unstable on some datasets, and the system performance has not been improved satisfactorily. To address this, we proposed a structured network compression and acceleration method for the convolutional neural network, which integrates the pruned convolutional neural network and the recurrent neural network, and applied it to the lip-recognition system in this paper.