Unstructured Pruning Research Articles

Deep Learning Model Compression Techniques Performance on Edge Devices

Artificial intelligence at the edge can help solve complex tasks faced by various sectors such as automotive, healthcare and surveillance. However, challenged by the lack of computational power from the edge devices, artificial intelligence models are forced to adapt. Many have developed and quantified model compres-sion approaches over the years to tackle this problem. However, not many have considered the overhead of on-device model compression, even though model compression can take a considerable amount of time. With the added metric, we provide a more complete view on the efficiency of model compression on the edge. The objective of this research is identifying the benefit of compression methods and it’s tradeoff between size and latency reduction versus the accuracy loss as well as compression time in edge devices. In this work, quantitative method is used to analyze and rank three common ways of model compression: post-training quantization, unstructured pruning and knowledge distillation on the basis of accuracy, latency, model size and time to compress overhead. We concluded that knowledge distillation is the best, with potential of up to 11.4x model size reduction, and 78.67% latency speed up, with moderate loss of accura-cy and compression time.

Structured Pruning for Deep Convolutional Neural Networks: A Survey.

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at: https://github.com/he-y/Awesome-Pruning. A dedicated website offering a more interactive comparison of structured pruning methods can be found at: https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey.

Edge Federated Optimization for Heterogeneous Data

This study focuses on optimizing federated learning in heterogeneous data environments. We implement the FedProx and a baseline algorithm (i.e., the FedAvg) with advanced optimization strategies to tackle non-IID data issues in distributed learning. Model freezing and pruning techniques are explored to showcase the effective operations of deep learning models on resource-constrained edge devices. Experimental results show that at a pruning rate of 10%, the FedProx with structured pruning in the MIT-BIH and ST databases achieved the best F1 scores, reaching 96.01% and 77.81%, respectively, which achieves a good balance between system efficiency and model accuracy compared to those of the FedProx with the original configuration, reaching F1 scores of 66.12% and 89.90%, respectively. Similarly, with layer freezing technique, unstructured pruning method, and a pruning rate of 20%, the FedAvg algorithm effectively balances classification performance and degradation of pruned model accuracy, achieving F1 scores of 88.75% and 72.75%, respectively, compared to those of the FedAvg with the original configuration, reaching 56.82% and 85.80%, respectively. By adopting model optimization strategies, a practical solution is developed for deploying complex models in edge federated learning, vital for its efficient implementation.

Dynamic Unstructured Pruning Neural Network Image Super-resolution Reconstruction

Dynamic Unstructured Pruning Neural Network Image Super-resolution Reconstruction

Fluctuation-Based Adaptive Structured Pruning for Large Language Models

Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.

Compact Neural Network via Stacking Hybrid Units.

As an effective tool for network compression, pruning techniques have been widely used to reduce the large number of parameters in deep neural networks (NNs). Nevertheless, unstructured pruning has the limitation of dealing with the sparse and irregular weights. By contrast, structured pruning can help eliminate this drawback but it requires complex criteria to determine which components to be pruned. Therefore, this paper presents a new method termed BUnit-Net, which directly constructs compact NNs by stacking designed basic units, without requiring additional judgement criteria anymore. Given the basic units of various architectures, they are combined and stacked systematically to build up compact NNs which involve fewer weight parameters due to the independence among the units. In this way, BUnit-Net can achieve the same compression effect as unstructured pruning while the weight tensors can still remain regular and dense. We formulate BUnit-Net in diverse popular backbones in comparison with the state-of-the-art pruning methods on different benchmark datasets. Moreover, two new metrics are proposed to evaluate the trade-off of compression performance. Experiment results show that BUnit-Net can achieve comparable classification accuracy while saving around 80% FLOPs and 73% parameters. That is, stacking basic units provides a new promising way for network compression.

Unstructured Pruning and Low Rank Factorisation of Self-Supervised Pre-Trained Speech Models

Unstructured Pruning and Low Rank Factorisation of Self-Supervised Pre-Trained Speech Models

Heuristic Compression Method for CNN Model Applying Quantization to a Combination of Structured and Unstructured Pruning Techniques

Heuristic Compression Method for CNN Model Applying Quantization to a Combination of Structured and Unstructured Pruning Techniques

Dynamic Probabilistic Pruning: A General Framework for Hardware-Constrained Pruning at Different Granularities.

Unstructured neural network pruning algorithms have achieved impressive compression ratios. However, the resulting-typically irregular-sparse matrices hamper efficient hardware implementations, leading to additional memory usage and complex control logic that diminishes the benefits of unstructured pruning. This has spurred structured coarse-grained pruning solutions that prune entire feature maps or even layers, enabling efficient implementation at the expense of reduced flexibility. Here, we propose a flexible new pruning mechanism that facilitates pruning at different granularities (weights, kernels, and feature maps) while retaining efficient memory organization (e.g., pruning exactly k -out-of- n weights for every output neuron or pruning exactly k -out-of- n kernels for every feature map). We refer to this algorithm as dynamic probabilistic pruning (DPP). DPP leverages the Gumbel-softmax relaxation for differentiable k -out-of- n sampling, facilitating end-to-end optimization. We show that DPP achieves competitive compression ratios and classification accuracy when pruning common deep learning models trained on different benchmark datasets for image classification. Relevantly, the dynamic masking of DPP facilitates for joint optimization of pruning and weight quantization in order to even further compress the network, which we show as well. Finally, we propose novel information-theoretic metrics that show the confidence and pruning diversity of pruning masks within a layer.

A multi-granularity CNN pruning framework via deformable soft mask with joint training

Model pruning is a commonly used technique for compressing DNNs and reducing computation requirements to accelerate inference. However, the required granularity of pruning varies across different application scenarios, making it difficult and cumbersome to customize different pruning methods for each hardware or computing platform. Therefore, a unified framework is necessary to accommodate various levels of granularity of pruning. Furthermore, some available methods require additional fine-tuning or model retraining to restore accuracy, which can result in significant time costs. With this motivation, this paper proposes a Multi-Granularity Pruning Framework, namely MGPF, to obtain sparse models of different granularity without fine-tuning the remaining connections. Specifically, a deformable soft mask is introduced as the pruning initiator to achieve different levels of pruning granularity, such as weight pruning, channel pruning, and filter pruning, etc. The model parameters and soft masks are jointly trained, and we just apply L1 regularization on soft masks for sparsity to ensure that the model can be repaired during training without fine-tuning or retraining. After pruning, the soft masks are absorbed into the model parameters in the form of element product without changing the model structures. Experimental results on three image classification benchmarks CIFAR-10/100 and ImageNet-1K demonstrate the effectiveness of our method for various CNN architectures, datasets, and pruning rates. Particularly, for ResNet-50 on ImageNet-1K, we achieve a higher accuracy under the pruning rate of 98% for unstructured pruning which leads the advanced method by 12%.

Multiply-and-Fire: An Event-Driven Sparse Neural Network Accelerator

Deep neural network inference has become a vital workload for many systems from edge-based computing to data centers. To reduce the performance and power requirements for deep neural networks (DNNs) running on these systems, pruning is commonly used as a way to maintain most of the accuracy of the system while significantly reducing the workload requirements. Unfortunately, accelerators designed for unstructured pruning typically employ expensive methods to either determine non-zero activation-weight pairings or reorder computation. These methods require additional storage and memory accesses compared to the more regular data access patterns seen in structurally pruned models. However, even existing works that focus on the more regular access patterns seen in structured pruning continue to suffer from inefficient designs, which either ignore or expensively handle activation sparsity leading to low performance. To address these inefficiencies, we leverage structured pruning and propose the multiply-and-fire (MnF) technique, which aims to solve these problems in three ways: (a) the use of a novel event-driven dataflow that naturally exploits activation sparsity without complex, high-overhead logic; (b) an optimized dataflow takes an activation-centric approach, which aims to maximize the reuse of activation data in computation and ensures the data are only fetched once from off-chip global and on-chip local memory; and (c) based on the proposed event-driven dataflow, we develop an energy-efficient, high-performance sparsity-aware DNN accelerator. Our results show that our MnF accelerator achieves a significant improvement across a number of modern benchmarks and presents a new direction to enable highly efficient AI inference for both CNN and MLP workloads. Overall, this work achieves a geometric mean of 11.2× higher energy efficiency and 1.41× speedup compared to a state-of-the-art sparsity-aware accelerator.

DNNShifter: An efficient DNN pruning system for edge computing

Deep neural networks (DNNs) underpin many machine learning applications. Production quality DNN models achieve high inference accuracy by training millions of DNN parameters which has a significant resource footprint. This presents a challenge for resources operating at the extreme edge of the network, such as mobile and embedded devices that have limited computational and memory resources. To address this, models are pruned to create lightweight, more suitable variants for these devices. Existing pruning methods are unable to provide similar quality models compared to their unpruned counterparts without significant time costs and overheads or are limited to offline use cases. Our work rapidly derives suitable model variants while maintaining the accuracy of the original model. The model variants can be swapped quickly when system and network conditions change to match workload demand. This paper presents DNNShifter, an end-to-end DNN training, spatial pruning, and model switching system that addresses the challenges mentioned above. At the heart of DNNShifter is a novel methodology that prunes sparse models using structured pruning - combining the accuracy-preserving benefits of unstructured pruning with runtime performance improvements of structured pruning. The pruned model variants generated by DNNShifter are smaller in size and thus faster than dense and sparse model predecessors, making them suitable for inference at the edge while retaining near similar accuracy as of the original dense model. DNNShifter generates a portfolio of model variants that can be swiftly interchanged depending on operational conditions. DNNShifter produces pruned model variants up to 93x faster than conventional training methods. Compared to sparse models, the pruned model variants are up to 5.14x smaller and have a 1.67x inference latency speedup, with no compromise to sparse model accuracy. In addition, DNNShifter has up to 11.9x lower overhead for switching models and up to 3.8x lower memory utilisation than existing approaches. DNNShifter is available for public use from https://github.com/blessonvar/DNNShifter.

A Tiny Accelerator for Mixed-Bit Sparse CNN Based on Efficient Fetch Method of SIMO SPad

Convolution neural networks (CNNs) have been implemented with custom hardware on edge devices since its algorithms were successful in many artificial intelligence applications. Although lots of unstructured pruning and mix-bit quantization algorithms have been proposed to successfully compress CNNs, there are few hardware accelerators which can support both sparse and mix-bit CNNs. Besides, sparse matrix computation consumes lots of hardware resources such as registers or BRAM to fetch the needed input activations into processing element (PE). This paper presents a tiny accelerator for mixed-bit sparse CNNs featuring a novel scheme of single vector-based compressed sparse filter (CSF) method and single input multiple output scratch pad (SIMO SPad) to effectively compress weight and fetch the needed input activation. SIMO SPad is shared by multiple PEs, which saves 13.34% CLB LUTs, 46.24% CLB Registers. Furthermore, the accelerator supports mixed-bit sparse computation to obtain better accuracy and performance. When tested on VGG16, compared with 8-bit non-sparsity baselines, the performance of mixed-bit sparsity on Cifar10 and ImageNet improved by 4.85× and 3.33×, respectively, with small accuracy decrease degradation. Compared to state-of-the-art accelerators, the accelerator achieves 1.40× to 2.98× greater DSP efficiency, and offers 1.91× greater energy efficiency.

Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems

The rapid development of neural networks has come at the cost of increased computational complexity. Neural networks are both computationally intensive and memory intensive; as such, the minimal energy and computing power of satellites pose a challenge for automatic target recognition (ATR). Knowledge distillation (KD) can distill knowledge from a cumbersome teacher network to a lightweight student network, transferring the essential information learned by the teacher network. Thus, the concept of KD can be used to improve the accuracy of student networks. Even when learning from a teacher network, there is still redundancy in the student network. Traditional networks fix the structure before training, such that training does not improve the situation. This paper proposes a distillation sparsity training (DST) algorithm based on KD and network pruning to address the above limitations. We first improve the accuracy of the student network through KD, and then through network pruning, allowing the student network to learn which connections are essential. DST allows the teacher network to teach the pruned student network directly. The proposed algorithm was tested on the CIFAR-100, MSTAR, and FUSAR-Ship data sets, with a 50% sparsity setting. First, a new loss function for the teacher-pruned student was proposed, and the pruned student network showed a performance close to that of the teacher network. Second, a new sparsity model (uniformity half-pruning UHP) was designed to solve the problem that unstructured pruning does not facilitate the implementation of general-purpose hardware acceleration and storage. Compared with traditional unstructured pruning, UHP can double the speed of neural networks.

Progressive multi-level distillation learning for pruning network

Although the classification method based on the deep neural network has achieved excellent results in classification tasks, it is difficult to apply to real-time scenarios because of high memory footprints and prohibitive inference times. Compared to unstructured pruning, structured pruning techniques can reduce the computation cost of the model runtime more effectively, but inevitably reduces the precision of the model. Traditional methods use fine tuning to restore model damage performance. However, there is still a large gap between the pruned model and the original one. In this paper, we use progressive multi-level distillation learning to compensate for the loss caused by pruning. Pre-pruning and post-pruning networks serve as the teacher and student networks. The proposed approach utilizes the complementary properties of structured pruning and knowledge distillation, which allows the pruned network to learn the intermediate and output representations of the teacher network, thus reducing the influence of the model subject to pruning. Experiments demonstrate that our approach performs better on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets with different pruning rates. For instance, GoogLeNet can achieve near lossless pruning on the CIFAR-10 dataset with 60% pruning. Moreover, this paper also proves that using the proposed distillation learning method during the pruning process achieves more significant performance gains than after completing the pruning.

Research on lightweight Yolo coal gangue detection algorithm based on resnet18 backbone feature network

Nowadays coal gangue presorting is still mostly conducted manually, with high labor intensity, low sorting efficiency, and potential safety hazards. A coal gangue sorting robot that is an effective device to complete the coal gangue presorting instead of the worker, for which the coal gangue intelligent detection is one of its key technologies. The lightweight and real-time performance of the detection model has an important impact on the performance of coal gangue sorting robot. Lots of models for the coal gangue detection have been proposed based on machine learning, but there are still some problems such as slow detection speed, structural redundancy and large model size. This paper adopts the lightweight network of ResNet18 as the backbone feature extraction network of YOLOv3 and proposes a coal gangue detection algorithm of ResNet18-YOLO, studied the feature scale reduction and unstructured pruning of the model to further improve its lightweight and real-time performance, prepared the coal gangue data set referring to the actual gangue sorting situation and discussed the performance of the model. The results show that the improved ResNet18-YOLO algorithm can detect the coal gangue at a speed of 45.5 ms/piece and the model size is only approximately 65.34 MB when the mAP of the coal gangue detection is 96.27%. It has better real-time performance and smaller model size with the condition of equivalent mAP performance to the other algorithms, and has good gangue detection performance, which is conducive to reducing the technical requirements for the gangue sorting robot.

Sensitivity pruner: Filter-Level compression algorithm for deep neural networks

As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.

A 1.6-mW Sparse Deep Learning Accelerator for Speech Separation

Low-power deep learning accelerators (DLAs) on the speech processing enable real-time applications on edge devices. However, most of the existing accelerators suffer from high-power consumption and focus on image applications only. This article presents a low-power accelerator for speech separation through algorithm and hardware optimizations. At the algorithm level, the model is compressed with structured sensitivity as well as unstructured pruning, and further quantized to the shifted 8-bit floating-point format instead of the 32-bit floating-point format. The computations with the zero kernel and zero activation values are skipped by decomposition of the dilated and transposed convolutions. At the hardware level, the compressed model is then supported by an architecture with eight independent multipliers and accumulators (MACs) with a simple zero-skipping hardware to take advantage of the activation sparsity and low-power processing. The proposed approach reduces the model size by 95.44% and computation complexity by 93.88%. The final implementation with the TSMC 40-nm process can achieve real-time speech separation and consumes 1.6-mW power when operated at 150 MHz. The normalized energy efficiency and area efficiency are 2.344 TOPS/W and 14.42 GOPS/mm2, respectively.

Speedup deep learning models on GPU by taking advantage of efficient unstructured pruning and bit-width reduction

This work is focused on the UnSparse-Opt framework for the efficient unstructured pruning and quantisation of feedforward neural networks and the improvement of their efficiency on graphic processing units (GPU) by using a direct sparse algorithm. The Nvidia deep neural network (cuDnn) library is the most effective implementation of deep learning (DL) algorithms for GPUs. One of the most common techniques for improving the efficiency of Convolutional Neural Network (CNN) models is weight pruning and quantisation. There are two main types of pruning: structural and non-structural. The first enables much easier acceleration on many type of accelerators, but with structural it is difficult to achieve a sparsity level and accuracy as high as that obtained with the non-structural version. Non-structural pruning with retraining can generate the weight tensors up to ∼90% or more of sparsity in some deep CNN models. In this article, the pruning algorithm is presented which achieve high sparsity levels without drop in accuracy. In the next stage, the linear and non-linear quantisation is adapted for further reductions in time and memory footprint. Additionally, this work presents real CNN models pruned with high sparsities in which some subset of layers can have comparable or better efficiency than cuDnn by using a direct sparse method. Finally, it shows sparse CNN-based architectures with reduced precision which can be more efficient than CuDnn library.

Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation

The unstructured sparsity after pruning poses a challenge to the efficient implementation of deep learning models in existing regular architectures like systolic arrays. On the other hand, coarse-grained structured pruning is suitable for implementation in regular architectures but tends to have higher accuracy loss than unstructured pruning when the pruned models are of the same size. In this work, we propose a model compression method based on a novel weight permutation scheme to fully exploit the fine-grained weight sparsity in the hardware design. Through permutation, the optimal arrangement of the weight matrix is obtained, and the sparse weight matrix is further compressed to a small and dense format to make full use of the hardware resources. Two pruning granularities are explored. In addition to the unstructured weight pruning, we also propose a more fine-grained subword-level pruning to further improve the compression performance. Compared to the state-of-the-art works, the matrix compression rate is significantly improved from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.88\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$14.13\times $ </tex-math></inline-formula> . As a result, the throughput and energy efficiency are improved by 2.75 and 1.86 times, respectively.