Pruning Method Research Articles

Compact Neural Network via Stacking Hybrid Units.

As an effective tool for network compression, pruning techniques have been widely used to reduce the large number of parameters in deep neural networks (NNs). Nevertheless, unstructured pruning has the limitation of dealing with the sparse and irregular weights. By contrast, structured pruning can help eliminate this drawback but it requires complex criteria to determine which components to be pruned. Therefore, this paper presents a new method termed BUnit-Net, which directly constructs compact NNs by stacking designed basic units, without requiring additional judgement criteria anymore. Given the basic units of various architectures, they are combined and stacked systematically to build up compact NNs which involve fewer weight parameters due to the independence among the units. In this way, BUnit-Net can achieve the same compression effect as unstructured pruning while the weight tensors can still remain regular and dense. We formulate BUnit-Net in diverse popular backbones in comparison with the state-of-the-art pruning methods on different benchmark datasets. Moreover, two new metrics are proposed to evaluate the trade-off of compression performance. Experiment results show that BUnit-Net can achieve comparable classification accuracy while saving around 80% FLOPs and 73% parameters. That is, stacking basic units provides a new promising way for network compression.

SAAF: Self-Adaptive Attention Factor-Based Taylor-Pruning on Convolutional Neural Networks.

Nowadays, pruning techniques have drawn attention to convolutional neural networks (CNNs) for reducing the consumption of computation resources. In particular, the Taylor-based method simplifies the evaluation of importance for each filter as the product of the gradient and weight value of the output features, which outperforms other methods in reductions of parameters and floating point operations (FLOPs). However, the Taylor-based method sacrifices too much accuracy when the overall pruning rate is relatively large compared with other pruning algorithms. In this article, we propose a self-adaptive attention factor (SAAF) to improve the performance of the slimmed model when conventional Taylor-based pruning is utilized under higher pruning. Specifically, SAAF can be calculated by leveraging the remaining ratio of filters at the early pruning stage of the Taylor-based method, and then, some pruned filters can be recovered for improving the accuracy of the slimmed model in terms of SAAF. It means that SAAF can protect filters from being overslimmed to eliminate the degeneration of Taylor-based pruning when the pruning rate is large as well as can compress models apparently across various datasets. We test the efficiency of SAAF on VGG-16 and ResNet-50 with CIFAR-10, Tiny-ImageNet, ImageNet-1000, and remote sensing images. Our method outperforms the traditional Taylor-based method obviously in accuracy, and there are only tiny sacrifices in the reduction of parameters and FLOPs, which is better than other pruning methods.

Class-Separation Preserving Pruning for Deep Neural Networks

Neural network pruning has been deemed essential in the deployment of deep neural networks on resource-constrained edge devices, greatly reducing the number of network parameters without drastically compromising accuracy. A class of techniques proposed in the literature assigns an importance score to each parameter and prunes those of the least importance. However, most of these methods are based on generalised estimations of the importance of each parameter, ignoring the context of the specific task at hand. In this paper, we propose a task specific pruning approach, CSPrune, which is based on how efficiently a neuron or a convolutional filter is able to separate classes. Our axiomatic approach assigns an importance score based on how separable different classes are in the output activations or feature maps, preserving separation of classes which avoids reduction in classification accuracy. Additionally, most pruning algorithms prune individual connections or weights leading to a sparse network without taking into account whether the hardware the network is deployed on can take advantage of that sparsity or not. CSPrune prunes whole neurons or filters which results in a more structured pruned network whose sparsity can be more efficiently utilised by the hardware. We evaluate our pruning method against various benchmark datasets, both small and large, and network architectures and show that our approach outperforms comparable pruning techniques.

Features formation of blueberry crop in conditions of Transcarpathia with different pruning methods

Features formation of blueberry crop in conditions of Transcarpathia with different pruning methods

Dimensions of crowns and their level of productivity of apple trees of the Jonaweld variety depending on the method and term of crown pruning

Dimensions of crowns and their level of productivity of apple trees of the Jonaweld variety depending on the method and term of crown pruning

A poda pode aumentar o efeito alelopático? Um estudo com pingo-de-ouro

Abstract Golden-Dewdrop (Duranta erecta L.), known for its landscaping applications, undergoes successive pruning to maintain its ornamental structure, a practice known to induce numerous physiological and chemical responses in the plant. Therefore, this study aimed to evaluate the influence of pruning and different extraction methods on the allelopathic potential of Golden-Dewdrop on the germination and initial growth of lettuce. Leaf extracts were obtained by decoction (hot) and maceration (cold) at concentrations of 20%, 40%, 60%, 80%, and 100%, with distilled water as a negative control. Phytochemical screening of the extracts was performed for phenolic compounds, flavonoids, anthocyanidins, tannins, cardioactive heterosides, saponins, alkaloids, and terpenes. For the bioassay, germinative and morphological parameters were measured and characterized. The data were subjected to analysis of variance (ANOVA), and for quantitative data, regression models were adjusted, while for qualitative data, means were compared using the Scott-Knott test with 5% significance. Significant differences were observed between extraction methods and management practices, with the hot aqueous extract from pruned plants significantly reducing germination at higher concentrations. The initial growth of lettuce was severely affected, presenting numerous morphological abnormalities, with severe damage to the root system at all concentrations. This study suggests that pruning practices have a modulating effect on the allelopathic activity of Golden-Dewdrop, providing valuable information for landscaping projects and botanical studies considering its evaluation and production of phytotoxic phytochemicals.

Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks.

Developmental plasticity plays a prominent role in shaping the brain's structure during ongoing learning in response to dynamically changing environments. However, the existing network compression methods for deep artificial neural networks (ANNs) and spiking neural networks (SNNs) draw little inspiration from brain's developmental plasticity mechanisms, thus limiting their ability to learn efficiently, rapidly, and accurately. This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the "use it or lose it, gradually decay" principle. The proposed DPAP model considers multiple biologically realistic mechanisms (such as dendritic spine dynamic plasticity, activitydependent neural spiking trace, and local synaptic plasticity), with additional adaptive pruning strategy, so that the network structure can be dynamically optimized during learning without any pre-training and retraining. Extensive comparative experiments show consistent and remarkable performance and speed boost with the extremely compressed networks on a diverse set of benchmark tasks for deep ANNs and SNNs, especially the spatio-temporal joint pruning of SNNs in neuromorphic datasets. This work explores how developmental plasticity enables complex deep networks to gradually evolve into brain-like efficient and compact structures, eventually achieving state-of-the-art (SOTA) performance for biologically realistic SNNs.

Method of pruning apple tree crown size and increase in photosynthesis productivity

The most important condition for increasing the productivity of fruit trees in intensive orchards is the optimal and efficient use of solar radiation, as well as an increase in leaf surface area. For apple trees, the size of the leaf surface, the illumination of the tree crowns and the increase in the productivity of photosynthesis are often determined by the combination of variety-rootstock combinations, the method of pruning and forming trees, and the size of the tree crowns. It has been established that the most effective rejuvenating and normalizing pruning is the shortening of fruit-bearing branches by 8-16 fruit buds.

Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers

Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers

Spartus: A 9.4 TOp/s FPGA-Based LSTM Accelerator Exploiting Spatio-Temporal Sparsity.

Long short-term memory (LSTM) recurrent networks are frequently used for tasks involving time-sequential data, such as speech recognition. Unlike previous LSTM accelerators that either exploit spatial weight sparsity or temporal activation sparsity, this article proposes a new accelerator called "Spartus" that exploits spatio-temporal sparsity to achieve ultralow latency inference. Spatial sparsity is induced using a new column-balanced targeted dropout (CBTD) structured pruning method, producing structured sparse weight matrices for a balanced workload. The pruned networks running on Spartus hardware achieve weight sparsity levels of up to 96% and 94% with negligible accuracy loss on the TIMIT and the Librispeech datasets. To induce temporal sparsity in LSTM, we extend the previous DeltaGRU method to the DeltaLSTM method. Combining spatio-temporal sparsity with CBTD and DeltaLSTM saves on weight memory access and associated arithmetic operations. The Spartus architecture is scalable and supports real-time online speech recognition when implemented on small and large FPGAs. Spartus per-sample latency for a single DeltaLSTM layer of 1024 neurons averages 1 μ s. Exploiting spatio-temporal sparsity on our test LSTM network using the TIMIT dataset leads to 46 × speedup of Spartus over its theoretical hardware performance to achieve 9.4-TOp/s effective batch-1 throughput and 1.1-TOp/s/W power efficiency.

Poda de verão e dobramento de galhos de pessegueiros em região subtropical

Abstract The objective of this work was to quantify peach (Prunus persica) tree production in the Y-training and open-vase management systems, subjected to three summer pruning methods. The experiment was set up in a 2x3 factorial arrangement, with two management systems and three summer pruning methods (no summer pruning, summer pruning, and branch bending). The plants were arranged in two spacings: 1.5×5.0 m for the Y-training system, with 1,334 plants per hectare; and 4.0×5.0 m for open-vase system, with 500 plants per hectare. In the early summer of 2019, summer pruning was performed, being repeated in the three following years. In the branch bending treatment, new branches were twisted manually until reaching an angle of 110°. In the treatment without summer pruning, the branches were not removed. The Y-training system increases peach production. The methods of branch bending in summer and no prunning in summer provide an equivalent peach production, whereas the summer pruning treatment results in the lowest production.

Applications of Pruning Methods in Natural Language Processing

Applications of Pruning Methods in Natural Language Processing

Adding Before Pruning: Sparse Filter Fusion for Deep Convolutional Neural Networks via Auxiliary Attention.

Filter pruning is a significant feature selection technique to shrink the existing feature fusion schemes (especially on convolution calculation and model size), which helps to develop more efficient feature fusion models while maintaining state-of-the-art performance. In addition, it reduces the storage and computation requirements of deep neural networks (DNNs) and accelerates the inference process dramatically. Existing methods mainly rely on manual constraints such as normalization to select the filters. A typical pipeline comprises two stages: first pruning the original neural network and then fine-tuning the pruned model. However, choosing a manual criterion can be somehow tricky and stochastic. Moreover, directly regularizing and modifying filters in the pipeline suffer from being sensitive to the choice of hyperparameters, thus making the pruning procedure less robust. To address these challenges, we propose to handle the filter pruning issue through one stage: using an attention-based architecture that adaptively fuses the filter selection with filter learning in a unified network. Specifically, we present a pruning method named adding before pruning (ABP) to make the model focus on the filters of higher significance by training instead of man-made criteria such as norm, rank, etc. First, we add an auxiliary attention layer into the original model and set the significance scores in this layer to be binary. Furthermore, to propagate the gradients in the auxiliary attention layer, we design a specific gradient estimator and prove its effectiveness for convergence in the graph flow through mathematical derivation. In the end, to relieve the dependence on the complicated prior knowledge for designing the thresholding criterion, we simultaneously prune and train the filters to automatically eliminate network redundancy with recoverability. Extensive experimental results on the two typical image classification benchmarks, CIFAR-10 and ILSVRC-2012, illustrate that the proposed approach performs favorably against previous state-of-the-art filter pruning algorithms.

Three-Stage Global Channel Pruning for Resources-Limited Platform.

Deep neural networks (DNNs) have demonstrated remarkable performance in many fields, and deploying them on resource-limited devices has drawn more and more attention in industry and academia. Typically, there are great challenges for intelligent networked vehicles and drones to deploy object detection tasks due to the limited memory and computing power of embedded devices. To meet these challenges, hardware-friendly model compression approaches are required to reduce model parameters and computation. Three-stage global channel pruning, which involves sparsity training, channel pruning, and fine-tuning, is very popular in the field of model compression for its hardware-friendly structural pruning and ease of implementation. However, existing methods suffer from problems such as uneven sparsity, damage to the network structure, and reduced pruning ratio due to channel protection. To solve these issues, the present article makes the following significant contributions. First, we present an element-level heatmap-guided sparsity training method to achieve even sparsity, resulting in higher pruning ratio and improved performance. Second, we propose a global channel pruning method that fuses both global and local channel importance metrics to identify unimportant channels for pruning. Third, we present a channel replacement policy (CRP) to protect layers, ensuring that the pruning ratio can be guaranteed even under high pruning rate conditions. Evaluations show that our proposed method significantly outperforms the state-of-the-art (SOTA) methods in terms of pruning efficiency, making it more suitable for deployment on resource-limited devices.

A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations.

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and to accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than three thousand pruning papers have been published from 2020 to 2024. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pre-trained weights, etc.) and explore several emerging topics, including pruning for large language models, vision transformers, diffusion models, and large multimodal models, post-training pruning, and different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We maintain a repository on https://github.com/hrcheng1066/awesome-pruning that serves as a comprehensive resource for neural network pruning papers and corresponding open-source codes. We will keep updating this repository to include the latest advancements in the field.

Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving

Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving

MCMC: Multi-Constrained Model Compression via One-Stage Envelope Reinforcement Learning.

Model compression methods are being developed to bridge the gap between the massive scale of neural networks and the limited hardware resources on edge devices. Since most real-world applications deployed on resource-limited hardware platforms typically have multiple hardware constraints simultaneously, most existing model compression approaches that only consider optimizing one single hardware objective are ineffective. In this article, we propose an automated pruning method called multi-constrained model compression (MCMC) that allows for the optimization of multiple hardware targets, such as latency, floating point operations (FLOPs), and memory usage, while minimizing the impact on accuracy. Specifically, we propose an improved multi-objective reinforcement learning (MORL) algorithm, the one-stage envelope deep deterministic policy gradient (DDPG) algorithm, to determine the pruning strategy for neural networks. Our improved one-stage envelope DDPG algorithm reduces exploration time and offers greater flexibility in adjusting target priorities, enhancing its suitability for pruning tasks. For instance, on the visual geometry group (VGG)-16 network, our method achieved an 80% reduction in FLOPs, a 2.31× reduction in memory usage, and a 1.92× acceleration, with an accuracy improvement of 0.09% compared with the baseline. For larger datasets, such as ImageNet, we reduced FLOPs by 50% for MobileNet-V1, resulting in a 4.7× faster speed and 1.48× memory compression, while maintaining the same accuracy. When applied to edge devices, such as JETSON XAVIER NX, our method resulted in a 71% reduction in FLOPs for MobileNet-V1, leading to a 1.63× faster speed, 1.64× memory compression, and an accuracy improvement.

Research on rapid detection of cross-scale defects in surface based on deep learning

The complex and diverse forms of surface defects in metal cutting, as well as their large scale span, present new challenges for deep learning algorithms. In addition, the existing defect detection models are generally characterized by high computational amount and complex structures, which is contrary to the high real-time performance and limited computing resources required in industrial applications. Based on this, this paper proposes a rapid detection method for cross-scale defects in surfaces based on deep learning. Firstly, a dataset of defect surfaces is collected and constructed through cutting experiments. Then, the experimental analysis reveals the insufficiency of the You Only Look Once version-5 s (YOLOv5s) network model for the detection of cross-scale defects on the surface. As a result, a RepVGG-Coordinate Attention-YOLOv5s (Rep-CA-YOLOv5s) network model, suitable for cross-scale defect detection, is proposed. This model optimizes the YOLOv5s network model from three perspectives, enhancing its ability to extract and fuse features for cross-scale defects. Finally, this paper investigates methods to improve detection speed while ensuring model accuracy. Two optimal Rep-CA-YOLOv5s sparse models are obtained through sparse training based on the γ scaling factor of the Batch Normalization (BN) layer and the filter weight, respectively. The relationship between detection accuracy, parameter quantity, and inference speed of these two sparse models under different pruning rates is explored. Experimental results indicate that the filter pruning method significantly improves model inference speed. At a 50 % pruning rate, optimal detection results can be achieved. Compared with the unpruning model, the pruned model reduced the inference speed by 55.67 %, while the mean Average Precision (mAP) only decreased by 0.1 %.

A multi-granularity CNN pruning framework via deformable soft mask with joint training

Model pruning is a commonly used technique for compressing DNNs and reducing computation requirements to accelerate inference. However, the required granularity of pruning varies across different application scenarios, making it difficult and cumbersome to customize different pruning methods for each hardware or computing platform. Therefore, a unified framework is necessary to accommodate various levels of granularity of pruning. Furthermore, some available methods require additional fine-tuning or model retraining to restore accuracy, which can result in significant time costs. With this motivation, this paper proposes a Multi-Granularity Pruning Framework, namely MGPF, to obtain sparse models of different granularity without fine-tuning the remaining connections. Specifically, a deformable soft mask is introduced as the pruning initiator to achieve different levels of pruning granularity, such as weight pruning, channel pruning, and filter pruning, etc. The model parameters and soft masks are jointly trained, and we just apply L1 regularization on soft masks for sparsity to ensure that the model can be repaired during training without fine-tuning or retraining. After pruning, the soft masks are absorbed into the model parameters in the form of element product without changing the model structures. Experimental results on three image classification benchmarks CIFAR-10/100 and ImageNet-1K demonstrate the effectiveness of our method for various CNN architectures, datasets, and pruning rates. Particularly, for ResNet-50 on ImageNet-1K, we achieve a higher accuracy under the pruning rate of 98% for unstructured pruning which leads the advanced method by 12%.

INCORPORATING ATTENTION SCORE TO IMPROVE FORESIGHT PRUNING ON TRANSFORMER MODELS

With rapid development of technologies and growing number of application of neural networks, the problem of optimization arises. Among other methods to optimize training and inference time, neural network pruning has attracted attention in recent years. The main goal of pruning is to reduce the computational complexity of neural network models while retaining performance metrics on desired level. Among the various approaches to pruning, Single-shot Network Pruning (SNIP) methods was designed as a straightforward and effective approach to optimize number of parameters before training. However, as neural network architectures have evolved, particularly with the growing popularity of transformers, a need to reevaluate traditional pruning methods arises. This paper aims to revisit SNIP pruning method, evaluate its performance on transformer model, and introduce an enhanced version of SNIP, specifically designed for transformer architectures. The paper outlines the mathematical framework of SNIP algorithm, and proposes a modification, based on specifics of transformers models. Transformer models achieved impressive results because of their attention mechanisms for a multitude of tasks such as language modeling, translation, computer vision tasks and many others. The proposed modification takes into account this unique feature and combines this information with traditional loss gradients. Traditional method calculates importance score for weights of the network using only gradients from loss function, in the case of enhanced algorithm. In the enhanced version, the importance score is a composite metric that incorporates not only the gradient from the loss function but also from the attention activations. To evaluate the efficiency of proposed modifications, a series of experiments were conducted on image classification task, using Linformer variation of transformer architectures. The results of experiments demonstrate the efficiency of incorporating attention scores in pruning. Conducted experiments show that model pruned by modified algorithm outperforms model pruned by original SNIP by 34% in validation accuracy, confirming the validity of the improvements introduced.