Channel Pruning Research Articles

Taxonomy of Saliency Metrics for Channel Pruning

Pruning unimportant parameters can allow deep neural networks (DNNs) to reduce their heavy computation and memory requirements. A saliency metric estimates which parameters can be safely pruned with little impact on the classification performance of the DNN. Many saliency metrics have been proposed, each within the context of a wider pruning algorithm. The result is that it is difficult to separate the effectiveness of the saliency metric from the wider pruning algorithm that surrounds it. Similar-looking saliency metrics can yield very different results because of apparently minor design choices. We propose a taxonomy of saliency metrics based on four mostly-orthogonal principal components. We show that a broad range of metrics from the pruning literature can be grouped according to these components. Our taxonomy not only serves as a guide to prior work, but allows us to construct new saliency metrics by exploring novel combinations of our taxonomic components. We perform an in-depth experimental investigation of more than 300 saliency metrics. Our results provide decisive answers to open research questions, and demonstrate the importance of reduction and scaling when pruning groups of weights. We find that some of our constructed metrics can outperform the best existing state-of-the-art metrics for convolutional neural network channel pruning.

YOLOv4-5D: An Effective and Efficient Object Detector for Autonomous Driving

The use of object detection algorithms has become extremely important in autonomous vehicles. Object detection at high accuracy and a fast inference speed is essential for safe autonomous driving. Therefore, the balance between the effectiveness and efficiency of the object detector must be considered. This article proposes a one-stage object detection framework for improving the detection accuracy while supporting a true real-time operation based on the YOLOv4. The backbone network in the proposed framework is the CSPDarknet53_dcn(P). The last output layer in the CSPDarknet53 is replaced with deformable convolution to improve the detection accuracy. In order to perform feature fusion, a new feature fusion module PAN++ is designed and five scales detection layers are used to improve the detection accuracy of small objects. In addition, this article proposes an optimized network pruning algorithm to solve the problem that the real-time performance of the algorithm cannot be satisfied due to the limited computing resources of the vehicle-mounted computing platform. The method of sparse scaling factor is used to improve the existing channel pruning algorithm. Compared to the YOLOv4, the YOLOV4-5D improves the mean average precision by 4.23% on the BDD data sets and 1.68% on the KITTI data sets. Finally, by pruning the model, the inference speed of YOLOV4-5D is increased 31.3% and the memory is only 98.1 MB when the detection accuracy is almost unchanged. Nevertheless, the proposed algorithm is capable of real-time detection at faster than 66 frames/s (fps) and shows higher accuracy than the previous approaches with a similar fps.

Learning Low Resource Consumption CNN through Pruning and Quantization

Deep learning models have evolved into powerful tools that can be used for many artificial intelligence tasks. However, deploying deep neural networks into real-world applications is still challenging due to their high computational complexity and storage overhead. Fortunately, a densely connected neural network can be converted into a sparsely connected network with low resource demand by the neural network compression. Since deep neural networks are complicated, compression mechanism should find a tradeoff between compression ratio and model accuracy. In this article, by analyzing the statistics of channel connection, we propose an interactive neural network compression mechanism including out-in-channel pruning and neural network quantization. Many channel pruning works apply structured sparsity regularization on each layer separately. We consider correlations between successive layers to retain predictive power of the compact network. A global greedy pruning algorithm is designed to remove redundant out-in-channels in an iterative way. Moreover, in order to solve the shortcomings of the one-shot quantization, we propose the incremental quantization algorithm in the dimension of the output channel, which can smooth network fluctuations and recover accuracy better during retraining. Our mechanism is comprehensively evaluated with various Convolutional Neural Networks (CNN) architectures on popular datasets. Notably, on ImageNet-1K, the out-in-channel pruning reduce 54.0 percent FLOPS on AlexNet and 50.0 percent FLOPs on ResNet-50 with only 0.15 and 0.37 percent top-1 accuracy drop respectively. On classification and style transfer tasks, the superiority of incremental quantization increases with the decrease of the number of quantization bits.

Where to Prune: Using LSTM to Guide Data-Dependent Soft Pruning.

While convolutional neural network (CNN) has achieved overwhelming success in various vision tasks, its heavy computational cost and storage overhead limit the practical use on mobile or embedded devices. Recently, compressing CNN models has attracted considerable attention, where pruning CNN filters, also known as the channel pruning, has generated great research popularity due to its high compression rate. In this paper, a new channel pruning framework is proposed, which can significantly reduce the computational complexity while maintaining sufficient model accuracy. Unlike most existing approaches that seek to-be-pruned filters layer by layer, we argue that choosing appropriate layers for pruning is more crucial, which can result in more complexity reduction but less performance drop. To this end, we utilize a long short-term memory (LSTM) to learn the hierarchical characteristics of a network and generate a global network pruning scheme. On top of it, we propose a data-dependent soft pruning method, dubbed Squeeze-Excitation-Pruning (SEP), which does not physically prune any filters but selectively excludes some kernels involved in calculating forward and backward propagations depending on the pruning scheme. Compared with the hard pruning, our soft pruning can better retain the capacity and knowledge of the baseline model. Experimental results demonstrate that our approach still achieves comparable accuracy even when reducing 70.1% Floating-point operation per second (FLOPs) for VGG and 47.5% for Resnet-56.

A Slimmer Network with Polymorphic and Group Attention Modules for More Efficient Object Detection in Aerial Images

Object detection is one of the core technologies in aerial image processing and analysis. Although existing aerial image object detection methods based on deep learning have made some progress, there are still some problems remained: (1) Most existing methods fail to simultaneously consider multi-scale and multi-shape object characteristics in aerial images, which may lead to some missing or false detections; (2) high precision detection generally requires a large and complex network structure, which usually makes it difficult to achieve the high detection efficiency and deploy the network on resource-constrained devices for practical applications. To solve these problems, we propose a slimmer network for more efficient object detection in aerial images. Firstly, we design a polymorphic module (PM) for simultaneously learning the multi-scale and multi-shape object features, so as to better detect the hugely different objects in aerial images. Then, we design a group attention module (GAM) for better utilizing the diversiform concatenation features in the network. By designing multiple detection headers with adaptive anchors and the above-mentioned two modules, we propose a one-stage network called PG-YOLO for realizing the higher detection accuracy. Based on the proposed network, we further propose a more efficient channel pruning method, which can slim the network parameters from 63.7 million (M) to 3.3M that decreases the parameter size by 94.8%, so it can significantly improve the detection efficiency for real-time detection. Finally, we execute the comparative experiments on three public aerial datasets, and the experimental results show that the proposed method outperforms the state-of-the-art methods.

EDP: An Efficient Decomposition and Pruning Scheme for Convolutional Neural Network Compression.

Model compression methods have become popular in recent years, which aim to alleviate the heavy load of deep neural networks (DNNs) in real-world applications. However, most of the existing compression methods have two limitations: 1) they usually adopt a cumbersome process, including pretraining, training with a sparsity constraint, pruning/decomposition, and fine-tuning. Moreover, the last three stages are usually iterated multiple times. 2) The models are pretrained under explicit sparsity or low-rank assumptions, which are difficult to guarantee wide appropriateness. In this article, we propose an efficient decomposition and pruning (EDP) scheme via constructing a compressed-aware block that can automatically minimize the rank of the weight matrix and identify the redundant channels. Specifically, we embed the compressed-aware block by decomposing one network layer into two layers: a new weight matrix layer and a coefficient matrix layer. By imposing regularizers on the coefficient matrix, the new weight matrix learns to become a low-rank basis weight, and its corresponding channels become sparse. In this way, the proposed compressed-aware block simultaneously achieves low-rank decomposition and channel pruning by only one single data-driven training stage. Moreover, the network of architecture is further compressed and optimized by a novel Pruning & Merging (PM) module which prunes redundant channels and merges redundant decomposed layers. Experimental results (17 competitors) on different data sets and networks demonstrate that the proposed EDP achieves a high compression ratio with acceptable accuracy degradation and outperforms state-of-the-arts on compression rate, accuracy, inference time, and run-time memory.

One‐dimensional convolutional neural networks for high‐resolution range profile recognition via adaptively feature recalibrating and automatically channel pruning

One‐dimensional convolutional neural networks for high‐resolution range profile recognition via adaptively feature recalibrating and automatically channel pruning

Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments

Achieving the rapid and accurate detection of apple flowers in natural environments is essential for yield estimation and the development of an automatic flower thinner. A real-time apple flower detection method using the channel pruned YOLO v4 deep learning algorithm was proposed. First, the YOLO v4 model under the CSPDarknet53 framework was built, and then, to simplify the apple flower detection model and ensure the efficiency of the model, the channel pruning algorithm was used to prune the model. Finally, a total of 2230 manually labeled apple flower images (including three varieties of Fuji, Red Love, and Gala) were used to fine-tune the model to achieve the fast and accurate detection of apple flowers. The test results showed that the number of parameters of the apple flower detection model after pruning was reduced by 96.74%, the model size was reduced by 231.51 MB, the inference time was decreased by 39.47%, and the mAP was 97.31%, which was only 0.24% lower than the model before pruning. To verify the effectiveness of the proposed method, five different deep learning algorithms including the Faster R-CNN, Tiny-YOLO v2, YOLO v3, SSD 300 and EfficientDet-D0 were compared. The comparative results showed that the mAP of the apple flower detection using the proposed method was 97.31%; the detection speed was 72.33f/s; the model size was 12.46 MB; the mAP was 12.21%, 15.56%, 14.19%, 5.67% and 7.79% higher than the other five algorithms, respectively; and the detection speed could meet the real-time requirements. Furthermore, the detection performance of apple flowers under different species of apple trees and illumination conditions was discussed. The results indicated that the proposed method had strong robustness to the changes of fruit tree varieties and illumination directions. The results showed that it was feasible to apply the proposed method for the real-time and accurate detection of apple flowers. The research could provide technical references for orchard yield estimation and the development of apple flower thinning robots.

Lightweight Driver Behavior Identification Model with Sparse Learning on In-Vehicle CAN-BUS Sensor Data

This study focuses on driver-behavior identification and its application to finding embedded solutions in a connected car environment. We present a lightweight, end-to-end deep-learning framework for performing driver-behavior identification using in-vehicle controller area network (CAN-BUS) sensor data. The proposed method outperforms the state-of-the-art driver-behavior profiling models. Particularly, it exhibits significantly reduced computations (i.e., reduced numbers both of floating-point operations and parameters), more efficient memory usage (compact model size), and less inference time. The proposed architecture features depth-wise convolution, along with augmented recurrent neural networks (long short-term memory or gated recurrent unit), for time-series classification. The minimum time-step length (window size) required in the proposed method is significantly lower than that required by recent algorithms. We compared our results with compressed versions of existing models by applying efficient channel pruning on several layers of current models. Furthermore, our network can adapt to new classes using sparse-learning techniques, that is, by freezing relatively strong nodes at the fully connected layer for the existing classes and improving the weaker nodes by retraining them using data regarding the new classes. We successfully deploy the proposed method in a container environment using NVIDIA Docker in an embedded system (Xavier, TX2, and Nano) and comprehensively evaluate it with regard to numerous performance metrics.

Icing-EdgeNet: A Pruning Lightweight Edge Intelligent Method of Discriminative Driving Channel for Ice Thickness of Transmission Lines

The icing monitoring of transmission lines is of great significance to prevent freezing disasters of transmission lines and ensure the security and stability of power system. Measurement methods based on image recognition is gradually applied for icing monitoring but fails to distinguish the icing thickness under the extreme meteorological conditions. Therefore, this article built an ice monitoring system based on edge intelligence to achieve the ice thickness identification at the icing monitoring terminal and proposed a lightweight vision identification method based on discriminative-driven channel pruning for icing monitoring terminals with limited computing resources. In this method, the lightweight convolutional neural network (CNN) MobileNetV3 is utilized for feature extraction, and the multiscale target detection network SSD is used to extract the high-dimensional feature information for ice monitoring. Then, the channel pruning driven by discrimination is used to further compress model, so as to obtain a lightweight identification method suitable for ice monitoring terminals with limited resources. The effectiveness and superiority of the proposed method are verified in comparison with other CNN-based target detection methods and traditional ice monitoring methods. The experimental results showed that the proposed method has 74.5% recognition accuracy for the ice images collected under extreme meteorological conditions. Moreover, the model size is only 11 MB, which can be deployed in ice thickness monitoring terminals with limited computing resources.

Pruning multi-view stereo net for efficient 3D reconstruction

How can we perform an efficient 3D reconstruction with high accuracy and completeness, in the presence of non-Lambertian surface and low textured regions? This paper aims at fast quality 3D reconstruction, best near real time. While deep learning approaches perform very well in multi-view stereo (MVS), the high complexity of models makes them inapplicable in practical applications. Few works were explored to accelerate deep learning-based 3D reconstruction approaches. In this paper, we take an unprecedented attempt to compress and accelerate these models via pruning their redundant parameters. We introduce an efficient channel pruning method for 2D convolutional neural networks (CNNs) based on a mixed back propagation process, where a soft mask is learned to prune the channels using a fast iterative shrinkage-thresholding algorithm. While in 3D CNNs, we train a large multi-scale CNNs architecture and observe that only utilizing one module enough for the 3D reconstruction, which can still maintain the performance of the full-precision model. We achieve an efficient MVS reconstruction system up to 2 times faster, in contrast to the state-of-the-arts, while maintaining comparable model accuracy and even better completeness.

A Fusion Algorithm of Multi-model Pruning and Collaborative Distillation Learning

In order to improve the prediction performance, the complex depth network model needs a large number of calculation parameters, and the negative impact is the fact that it greatly increases the calculation time and energy consumption. In this paper, a fusion algorithm of model pruning and joint multi-model knowledge distillation learning is proposed. By constructing an adaptive joint learning loss function including distillation, multi-model training is carried out. This method can replace the training process of model fine-tuning after pruning. In this paper, firstly, multi classification tasks are carried out on different model structures and data sets, and a complex network model with good effect is trained as a teacher’s model. Then, the channel pruning process of random pruning degree is executed to generate multiple student models. Finally, the accuracy of multiple student models is improved by using this method. The experimental results show that this method effectively improves the accuracy of each model after pruning, and the experimental results achieve a suitable balance between model performance and accuracy.

AutoPruner: An end-to-end trainable filter pruning method for efficient deep model inference

Channel pruning is an important method to speed up CNN model’s inference. Previous filter pruning algorithms regard importance evaluation and model fine-tuning as two independent steps. This paper argues that combining them into a single end-to-end trainable system will lead to better results. We propose an efficient channel selection layer, namely AutoPruner, to find less important filters automatically in a joint training manner. Our AutoPruner takes previous activation responses as an input and generates a true binary index code for pruning. Hence, all the filters corresponding to zero index values can be removed safely after training. By gradually erasing several unimportant filters, we can prevent an excessive drop in model accuracy. Compared with previous state-of-the-art pruning algorithms (including training from scratch), AutoPruner achieves significantly better performance. Furthermore, ablation experiments show that the proposed novel mini-batch pooling and binarization operations are vital for the success of model pruning.

Model Compression Using Progressive Channel Pruning

In this work, we propose a simple but effective channel pruning framework called Progressive Channel Pruning (PCP) to accelerate Convolutional Neural Networks (CNNs). In contrast to the existing channel pruning methods that prune the channels only once per layer in a layer-by-layer fashion, our new progressive framework iteratively prunes a small number of channels from several selected layers, which consists of a three-step attempting-selecting-pruning pipeline in each iteration. In the attempting step, we attempt to prune a pre-defined number of channels from one layer by using any existing channel pruning methods and estimate the accuracy drop for this layer based on the labelled samples in the validation set. In the selecting step, based on the estimated accuracy drops for all layers, we propose a greedy strategy to automatically select a set of layers that will lead to less overall accuracy drop after pruning these layers. In the pruning step, we prune a small number of channels from these selected layers. We further extend our PCP framework to prune channels for the deep transfer learning methods like Domain Adversarial Neural Network (DANN), in which we effectively reduce the data distribution mismatch in the channel pruning process by using both labelled samples from the source domain and pseudo-labelled samples from the target domain. Our comprehensive experiments on two benchmark datasets demonstrate that our PCP framework outperforms the existing channel pruning approaches under both supervised learning and transfer learning settings.

Network pruning using sparse learning and genetic algorithm

In recent years, convolutional neural networks (CNNs) have achieved success in the field of computer vision. However, their large storage requirements and high computational consumption limit their scope of application, especially, in mobile devices. Network pruning has been proven to be an effective approach for relieving such burdens. However, the pruning ratio and accuracy of the existing methods need to be improved. In this paper, a scheme of network channel pruning, based on sparse learning and the genetic algorithm, is proposed to achieve a better balance between the pruning ratio and accuracy. A gradient-limited L1/2 regularization is introduced into the model training, to achieve channel-level sparsity. In addition, a dynamically adjustable evaluation factor is proposed to evaluate the importance of channels and the genetic algorithm is utilized to acquire a suitable subnetwork from the sparse model. Our method is evaluated against several state-of-the-art CNNs on three different classification datasets, and is found to achieve, on an average, 63.8%/50.8%/58.6% reduction in parameters and 49.1%/42.9%/58.7% reduction in computational complexity, with negligible loss of accuracy. Our method also achieves 42.84% reduction in parameters and 30.06% reduction in computational complexity of the detection network.

Simplification of neural networks for skin lesion image segmentation using color channel pruning

Automatic analysis of skin abnormality is an effective way for medical experts to facilitate diagnosis procedures and improve their capabilities. Efficient and accurate methods for analysis of the skin abnormalities such as convolutional neural networks (CNNs) are typically complex. Hence, the implementation of such complex structures in portable medical instruments is not feasible due to power and resource limitations. CNNs can extract features from the skin abnormality images automatically. To reduce the burden of the network for feature extraction, which can lead to the network simplicity, proper input color channels could be selected. In this paper, a pruning framework is proposed to simplify these complex structures through the selection of most informative color channels and simplification of the network. Moreover, hardware requirements of different network structures are identified to analyze the complexity of different networks. Experimental results are conducted for segmentation of images from two publicly available datasets of both dermoscopy and non-dermoscopy images. Simulation results show that using the proposed color channel selection method, simple and efficient neural network structures can be applied for segmentation of skin abnormalities.

Design of lightweight pedestrian detection network in railway scenes

The deployment of many advanced pedestrian detection applications is largely hindered by the high computational cost of deep convolutional neural networks (CNNs). In this paper, we propose a two-step pruning method to design a lightweight pedestrian detection network in railway scenes. The first step is feature pyramid network (FPN) pruning, which utilizes the characteristic of pedestrian in railway scenes and the FPN structure in YOLOv3. The second step is regular channel pruning, which utilizes network slimming knowledge and is an accelerator-friendly pruning strategy. Our two-step pruning method gives about 88% reduction in parameters and about 74% reduction in computing complexity with comparable detection accuracy.

Loss-Driven Channel Pruning of Convolutional Neural Networks

Loss-Driven Channel Pruning of Convolutional Neural Networks

Channel Pruning Guided by Classification Loss and Feature Importance

In this work, we propose a new layer-by-layer channel pruning method called Channel Pruning guided by classification Loss and feature Importance (CPLI). In contrast to the existing layer-by-layer channel pruning approaches that only consider how to reconstruct the features from the next layer, our approach additionally take the classification loss into account in the channel pruning process. We also observe that some reconstructed features will be removed at the next pruning stage. So it is unnecessary to reconstruct these features. To this end, we propose a new strategy to suppress the influence of unimportant features (i.e., the features will be removed at the next pruning stage). Our comprehensive experiments on three benchmark datasets, i.e., CIFAR-10, ImageNet, and UCF-101, demonstrate the effectiveness of our CPLI method.

Reborn Filters: Pruning Convolutional Neural Networks with Limited Data

Channel pruning is effective in compressing the pretrained CNNs for their deployment on low-end edge devices. Most existing methods independently prune some of the original channels and need the complete original dataset to fix the performance drop after pruning. However, due to commercial protection or data privacy, users may only have access to a tiny portion of training examples, which could be insufficient for the performance recovery. In this paper, for pruning with limited data, we propose to use all original filters to directly develop new compact filters, named reborn filters, so that all useful structure priors in the original filters can be well preserved into the pruned networks, alleviating the performance drop accordingly. During training, reborn filters can be easily implemented via 1×1 convolutional layers and then be fused in the inference stage for acceleration. Based on reborn filters, the proposed channel pruning algorithm shows its effectiveness and superiority on extensive experiments.