Pruning Method Research Articles

YOLO-ResTinyECG: ECG-based lightweight embedded AI arrhythmia small object detector with pruning methods

YOLO-ResTinyECG: ECG-based lightweight embedded AI arrhythmia small object detector with pruning methods

Fast Knowledge Discovery from Big Data--Large-scale Data Analysis with Accuracy Guarantee via Efficient Pruning Methods

Fast Knowledge Discovery from Big Data--Large-scale Data Analysis with Accuracy Guarantee via Efficient Pruning Methods

Growth Activity of Apple Trees Depending on the Method and Timing of Crown Pruning

Growth Activity of Apple Trees Depending on the Method and Timing of Crown Pruning

Path Planning of Inspection Robot Based on Improved Ant Colony Algorithm

The conventional Ant Colony Optimization (ACO) algorithm, applied to logistics robot path planning in a two-dimensional grid environment, encounters several challenges: slow convergence rate, susceptibility to local optima, and an excessive number of turning points in the planned paths. To address these limitations, an improved ant colony algorithm has been developed. First, the heuristic function is enhanced by incorporating artificial potential field (APF) attraction, which introduces the influence of the target point’s attraction on the heuristic function. This modification accelerates convergence and improves the optimization performance of the algorithm. Second, an additional pheromone increment, calculated from the difference in pheromone levels between the best and worst paths of the previous generation, is introduced during the pheromone update process. This adjustment adaptively enhances the path length optimality. Lastly, a triangle pruning method is applied to eliminate unnecessary turning points, reducing the number of turns the logistics robot must execute and ensuring a more direct and efficient path. To validate the effectiveness of the improved algorithm, extensive simulation experiments were conducted in two grid-based environments of varying complexity. Several performance indicators were utilized to compare the conventional ACO algorithm, a previously improved version, and the newly proposed algorithm. MATLAB simulation results demonstrated that the improved ant colony algorithm significantly outperforms the other methods in terms of path length, number of iterations, and the reduction of inflection points, confirming its superiority in logistics robot path planning.

Task-Level Customized Pruning for Image Classification on Edge Devices

Convolutional neural networks (CNNs) are widely utilized in image classification. Nevertheless, CNNs typically require substantial computational resources, posing challenges for deployment on resource-constrained edge devices and limiting the spread of AI-driven applications. While various pruning approaches have been proposed to mitigate this issue, they often overlook a critical fact that edge devices are typically tasked with handling only a subset of classes rather than the entire set. Moreover, the specific combinations of subcategories that each device must discern vary, highlighting the need for fine-grained task-specific adjustments. Unfortunately, these oversights result in pruned models that still contain unnecessary category redundancies, thereby impeding the potential for further model optimization and lightweight design. To bridge this gap, we propose a task-level customized pruning (TLCP) method via utilizing task-level information, i.e., class combination information relevant to edge devices. Specifically, TLCP first introduces channel control gates to assess the importance of each convolutional channel for individual classes. These class-level control gates are then aggregated through linear combinations, resulting in a pruned model customized to the specific tasks of edge devices. Experiments on various customized tasks demonstrate that TLCP can significantly reduce the number of parameters, by up to 33.9% on CIFAR-10 and 14.0% on CIFAR-100, compared to other baseline methods, while maintaining almost the same inference accuracy.

Prediction model optimization of gas turbine remaining useful life based on transfer learning and simultaneous distillation pruning algorithm

For the application of deep learning (DL) models in the field of remaining useful life (RUL) prediction and predictive maintenance (PdM) of complex equipment, the insufficient training data and large model are the two major problems. To address these issues, a model training method based on transfer learning and a simultaneous distillation pruning algorithm were proposed. By introducing prior knowledge, three transfer learning modes are devised to reduce the demand of training data. Additionally, the simultaneous distillation pruning algorithm was devised to make the model lightweight, and an iterative pruning method was adopted to trim the large neural network model. By analyzing the performance of different transfer learning modes, the effectiveness of the proposed method can be demonstrated. The number of model parameters and the performance before and after pruning were compared. The results demonstrated that, without significant alterations to the prediction performance, the proposed model exhibited the capability to markedly reduce the number of model parameters. Based on the proposed methods, the challenges of insufficient data and efficiency encountered by DL models could be effectively addressed.

Pruning convolutional neural networks for inductive conformal prediction

Neural network pruning is a popular approach to reducing model storage size and inference time by removing redundant parameters in the neural network. However, the uncertainty of predictions from pruned models is unexplored. In this paper we study neural network pruning in the context of conformal predictors (CP). The conformal prediction framework built on top of machine learning algorithms supplements their predictions with reliable uncertainty measure in the form of prediction sets, under the independent and identically distributed assumption on the data. Convolutional neural networks (CNNs) have complicated architectures and are widely used in various applications nowadays. Therefore, we focus on pruning CNNs and, in particular, filter-level pruning. We first propose a brute force method that estimates the contribution of a filter to the CP’s predictive efficiency and removes those with the least contribution. Given the computation inefficiency of the brute force method, we also propose the Taylor expansion to approximate the filter’s contribution. Furthermore, we improve the global pruning method by protecting the most important filters within each layer from being pruned. In addition, we explore the ConfTr loss function which is optimized to yield maximal CP efficiency in the context of neural network pruning. We have conducted extensive experimental studies and compared the results regarding the trade-offs between predictive efficiency, computational efficiency, and network sparsity. These results are instructive for deploying pruned neural networks with applications using conformal prediction where reliable predictions and reduced computational cost are relevant, such as in safety-critical applications.

Analysing the Metagenomic Dynamics of Soil Microbiota Affected by Tea Pruning and Skiffing Methods in Tea Plantations of Dibrugarh, Assam, India

Analysing the Metagenomic Dynamics of Soil Microbiota Affected by Tea Pruning and Skiffing Methods in Tea Plantations of Dibrugarh, Assam, India

HCPrune: A Hierarchical Classification Pruning Method Based on Receptive Field Size

Abstract Model pruning, as an optimisation algorithm for convolutional neural network (CNN) models, can effectively compress and accelerate various complex CNN models. Recent network pruning algorithms usually focus on removing unimportant and redundant convolutional kernels or channels in the network. This paper propose a hierarchical pruning algorithm, HCPrune, which classifies and then prunes the convolutional layers of a CNN model to improve the model prediction accuracy while reducing the number of parameters and FLOPs. In this paper, the CNN convolutional layers are divided in terms of the size relationship between the receptive field of each layer of the network and the resolution of the input image, and use the VGG and Resnet series of networks based on some popular pruning methods to conduct experiments on the CIFAR10 and CIFAR100 datasets and ensure that the same number of parameters and FLOPs compared with the basic pruning methods. The VGG16 network improves the accuracy by 0.42% in CIFAR10 and 0.47% in CIFAR100, and the ResNet164 improves the accuracy by about 0.2% in the same configuration.

RGP: Neural Network Pruning Through Regular Graph With Edges Swapping.

Deep learning technology has found a promising application in lightweight model design, for which pruning is an effective means of achieving a large reduction in both model parameters and float points operations (FLOPs). The existing neural network pruning methods mostly start from the consideration of the importance of model parameters and design parameter evaluation metrics to perform parameter pruning iteratively. These methods were not studied from the perspective of network model topology, so they might be effective but not efficient, and they require completely different pruning for different datasets. In this article, we study the graph structure of the neural network and propose a regular graph pruning (RGP) method to perform a one-shot neural network pruning. Specifically, we first generate a regular graph and set its node-degree values to meet the preset pruning ratio. Then, we reduce the average shortest path-length (ASPL) of the graph by swapping edges to obtain the optimal edge distribution. Finally, we map the obtained graph to a neural network structure to realize pruning. Our experiments demonstrate that the ASPL of the graph is negatively correlated with the classification accuracy of the neural network and that RGP has a strong precision retention capability with high parameter reduction (more than 90%) and FLOPs reduction (more than 90%) (the code for quick use and reproduction is available at https://github.com/Holidays1999/Neural-Network-Pruning-through-its-RegularGraph-Structure).

Neighbor patches merging reduces spatial redundancy to accelerate vision transformer

Vision Transformers (ViTs) deliver outstanding performance but often require substantial computational resources. Various token pruning methods have been developed to enhance throughput by removing redundant tokens; however, these methods do not address the peak memory consumption, which remains equivalent to that of the unpruned networks. In this study, we introduce Neighbor Patches Merging (NEPAM), a method that significantly reduces the maximum memory footprint of ViTs while pruning tokens. NEPAM targets spatial redundancy within images and prunes redundant patches at the onset of the model, thereby achieving the optimal throughput-accuracy trade-off without fine-tuning. Experimental results demonstrate that NEPAM can accelerate the inference speed of the Vit-Base-Patch16-384 model by 25% with a negligible accuracy loss of 0.07% and a notable 18% reduction in memory usage. When applied to VideoMAE, NEPAM doubles the throughput with a 0.29% accuracy loss and a 48% reduction in memory usage. These findings underscore the efficacy of NEPAM in mitigating computational requirements while maintaining model performance.

Efficient Multitask Reinforcement Learning Without Performance Loss.

We propose an iterative sparse Bayesian policy optimization (ISBPO) scheme as an efficient multitask reinforcement learning (RL) method for industrial control applications that require both high performance and cost-effective implementation. Under continual learning scenarios in which multiple control tasks are sequentially learned, the proposed ISBPO scheme preserves the previously learned knowledge without performance loss (PL), enables efficient resource use, and improves the sample efficiency of learning new tasks. Specifically, the proposed ISBPO scheme continually adds new tasks to a single policy neural network while completely preserving the control performance of previously learned tasks through an iterative pruning method. To create a free-weight space for adding new tasks, each task is learned through a pruning-aware policy optimization method called the sparse Bayesian policy optimization (SBPO), which ensures efficient allocation of limited policy network resources for multiple tasks. Furthermore, the weights allocated to the previous tasks are shared and reused in new task learning, thereby improving sample efficiency and the performance of new task learning. Simulations and practical experiments demonstrate that the proposed ISBPO scheme is highly suitable for sequentially learning multiple tasks in terms of performance conservation, efficient resource use, and sample efficiency.

Polarization Pruning: Reliability Enhancement of Hafnia-Based Ferroelectric Devices for Memory and Neuromorphic Computing.

Ferroelectric (FE) materials are key to advancing electronic devices owing to their non-volatile properties, rapid state-switching abilities, and low-energy consumption. FE-based devices are used in logic circuits, memory-storage devices, sensors, and in-memory computing. However, the primary challenge in advancing the practical applications of FE-based memory is its reliability. To address this problem, a novel polarization pruning (PP) method is proposed. The PP is designed to eliminate weakly polarized domains by applying an opposite-sign pulse immediately after a program or erase operation. Significant improvements in the reliability of ferroelectric devices are achieved by reducing the depolarization caused by weakly polarized domains and mitigating the fluctuations in the ferroelectric dipole. These enhancements include a 25% improvement in retention, a 50% reduction in read noise, a 45% decrease in threshold voltage variation, and a 72% improvement in linearity. The proposed PP method significantly improves the memory storage efficiency and performance of neuromorphic systems.

Saliency and location aware pruning of deep visual detectors for autonomous driving

Despite the remarkable achievements of deep neural networks, their high computational complexity limits their wide use in many real-world embedded applications, such as autonomous driving perception. While current neural network pruning approaches can reduce model complexity to various extents, they often adopt local or ad hoc importance measures that are not directly related to the final task. More importantly, most of them focus on classification tasks and do not take location information into consideration during pruning. To address these issues, we present a novel channel importance measure that incorporates detection-related saliency and location awareness, specifically designed for the pruning of self-driving visual detectors. Our comprehensive experiments on the KITTI and COCO_traffic datasets demonstrate that our pruning method achieves significant reductions in model size and computational operations with little performance degradation. Notably, it outperforms other state-of-the-art methods across various pruning rates and base detectors. Our pruned YOLOX-S model with 40.2% fewer parameters even improves the original model’s mAP by 1.8% on KITTI. Moreover, we experimentally highlight the potential of our pruning approach in effectively detecting small-scale objects.

Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization

Abstract Despite the remarkable performance of generative large language models (LLMs) on abstractive summarization, they face two significant challenges: their considerable size and tendency to hallucinate. Hallucinations are concerning because they erode reliability and raise safety issues. Pruning is a technique that reduces model size by removing redundant weights, enabling more efficient sparse inference. Pruned models yield downstream task performance comparable to the original, making them ideal alternatives when operating on a limited budget. However, the effect that pruning has upon hallucinations in abstractive summarization with LLMs has yet to be explored. In this paper, we provide an extensive empirical study across five summarization datasets, two state-of-the-art pruning methods, and five instruction-tuned LLMs. Surprisingly, we find that hallucinations are less prevalent from pruned LLMs than the original models. Our analysis suggests that pruned models tend to depend more on the source document for summary generation. This leads to a higher lexical overlap between the generated summary and the source document, which could be a reason for the reduction in hallucination risk.1

Effective sentence-level relation extraction model using entity-centric dependency tree.

The syntactic information of a dependency tree is an essential feature in relation extraction studies. Traditional dependency-based relation extraction methods can be categorized into hard pruning methods, which aim to remove unnecessary information, and soft pruning methods, which aim to utilize all lexical information. However, hard pruning has the potential to overlook important lexical information, while soft pruning can weaken the syntactic information between entities. As a result, recent studies in relation extraction have been shifting from dependency-based methods to pre-trained language model (LM) based methods. Nonetheless, LM-based methods increasingly demand larger language models and additional data. This trend leads to higher resource consumption, longer training times, and increased computational costs, yet often results in only marginal performance improvements. To address this problem, we propose a relation extraction model based on an entity-centric dependency tree: a dependency tree that is reconstructed by considering entities as root nodes. Using the entity-centric dependency tree, the proposed method can capture the syntactic information of an input sentence without losing lexical information. Additionally, we propose a novel model that utilizes entity-centric dependency trees in conjunction with language models, enabling efficient relation extraction without the need for additional data or larger models. In experiments with representative sentence-level relation extraction datasets such as TACRED, Re-TACRED, and SemEval 2010 Task 8, the proposed method achieves F1-scores of 74.9%, 91.2%, and 90.5%, respectively, which are state-of-the-art performances.

Intermediate-grained kernel elements pruning with structured sparsity

Neural network pruning provides a promising prospect for the deployment of neural networks on embedded or mobile devices with limited resources. Although current structured strategies are unconstrained by specific hardware architecture in the phase of forward inference, the decline in classification accuracy of structured methods is beyond the tolerance at the level of general pruning rate. This inspires us to develop a technique that satisfies high pruning rate with a small decline in accuracy and has the general nature of structured pruning. In this paper, we propose a new pruning method, namely KEP (Kernel Elements Pruning), to compress deep convolutional neural networks by exploring the significance of elements in each kernel plane and removing unimportant elements. In this method, we apply a controllable regularization penalty to constrain unimportant elements by adding a prior knowledge mask and obtain a compact model. In the calculation procedure of forward inference, we introduce a sparse convolution operation which is different from the sliding window to eliminate invalid zero calculations and verify the effectiveness of the operation for further deployment on FPGA. A massive variety of experiments demonstrate the effectiveness of KEP on two datasets: CIFAR-10 and ImageNet. Specially, with few indexes of non-zero weights introduced, KEP has a significant improvement over the latest structured methods in terms of parameter and float-point operation (FLOPs) reduction, and performs well on large datasets.

Towards compressed and efficient CNN architectures via pruning

Convolutional Neural Networks (CNNs) use convolutional kernels to extract important low-level to high-level features from data. The performance of CNNs improves as they grow deep thereby learning better representations of the data. However, such deep CNNs are compute and memory-intensive, making deployment on resource-constrained devices challenging. To address this, the CNNs are compressed by adopting pruning strategies that remove redundant convolutional kernels from each layer while maintaining accuracy. Existing pruning methods that are based on feature map importance, only prune the convolutional layers uniformly and do not consider fully connected layers. Also, current techniques do not take into account class labels while pruning the less important feature maps and do not explore the need for retraining after pruning. This paper presents pruning techniques to prune convolutional and fully connected layers. This paper proposes a novel class-specific pruning strategy based on finding feature map importance in terms of entropy for convolutional layers and the number of incoming zeros to neurons for fully connected layers. The class-specific approach helps to have a different pruning threshold for every convolutional layer and ensures that the pruning threshold is not influenced by any particular class. A study on the need for retraining the entire network or a part of the network after pruning is also carried out. For Intel image, CIFAR10 and CIFAR100 datasets the proposed pruning method has compressed AlexNet by 83.2%, 87.19%, and 79.7%, VGG-16 by 83.7%, 85.11%, and 84.06% and ResNet-50 by 62.99%, 62.3% and 58.34% respectively.

SFP: Similarity-based filter pruning for deep neural networks

Convolutional neural networks have exhibited exceptional performance in various artificial intelligence domains, particularly in large-scale image processing tasks. However, the proliferation of network parameters and computational requirements has emerged as a significant bottleneck for the practical deployment of CNNs. In this paper, we propose a novel similarity-based filter pruning (SFP) approach for compressing convolutional neural networks, which is different from the traditional pruning method. The existing pruning methods eliminate the unimportant parameters but ignore the duplication of the reserved convolutional kernels. In the proposed SFP, kernels are clustered first according to their similarity, then the unimportant and redundant kernels are pruned in each class, which is more efficient than traditional pruning methods only based on the importance criterion. Furthermore, this paper introduces the concept of Kernel Dispersion to evaluate sparsity across distinct network layers, and proposes Distillation Fine-Tuning with Variable Temperature Coefficient to expedite convergence and enhance accuracy. The performance of the proposed similarity-based filter pruning approach is evaluated on different datasets, including CIFAR10, CIFAR100, ImageNet, and VOC. The experimental results indicate that the proposed SFP achieves approximately 1% higher accuracy at a comparable pruning rate compared to traditional state-of-the-art pruning methods.

Spatio-Temporal Pruning for Training Ultra-Low-Latency Spiking Neural Networks in Remote Sensing Scene Classification

In remote sensing scene classification (RSSC), restrictions on real-time processing on power consumption, performance, and resources necessitate the compression of neural networks. Unlike artificial neural networks (ANNs), spiking neural networks (SNNs) convey information through spikes, offering superior energy efficiency and biological plausibility. However, the high latency of SNNs restricts their practical application in RSSC. Therefore, there is an urgent need to research ultra-low-latency SNNs. As latency decreases, the performance of the SNN significantly deteriorates. To address this challenge, we propose a novel spatio-temporal pruning method that enhances the feature capture capability of ultra-low-latency SNNs. Our approach integrates spatial fundamental structures during the training process, which are subsequently pruned. We conduct a comprehensive evaluation of the impacts of these structures across classic network architectures, such as VGG and ResNet, demonstrating the generalizability of our method. Furthermore, we develop an ultra-low-latency training framework for SNNs to validate the effectiveness of our approach. In this paper, we successfully achieve high-performance ultra-low-latency SNNs with a single time step for the first time in RSSC. Remarkably, our SNN with one time step achieves at least 200 times faster inference time while maintaining a performance comparable to those of other state-of-the-art methods.