Pruning Strategy Research Articles

Mining contextually meaningful subgraphs from a vertex-attributed graph.

Networks have emerged as a natural data structure to represent relations among entities. Proteins interact to carry out cellular functions and protein-Protein interaction network analysis has been employed for understanding the cellular machinery. Advances in genomics technologies enabled the collection of large data that annotate proteins in interaction networks. Integrative analysis of interaction networks with gene expression and annotations enables the discovery of context-specific complexes and improves the identification of functional modules and pathways. Extracting subnetworks whose vertices are connected and have high attribute similarity have applications in diverse domains. We present an enumeration approach for mining sets of connected and cohesive subgraphs, where vertices in the subgraphs have similar attribute profile. Due to the large number of cohesive connected subgraphs and to overcome the overlap among these subgraphs, we propose an algorithm for enumerating a set of representative subgraphs, the set of all closed subgraphs. We propose pruning strategies for efficiently enumerating the search tree without missing any pattern or reporting duplicate subgraphs. On a real protein-protein interaction network with attributes representing the dysregulation profile of genes in multiple cancers, we mine closed cohesive connected subnetworks and show their biological significance. Moreover, we conduct a runtime comparison with existing algorithms to show the efficiency of our proposed algorithm.

Generating the assembly instructions of helicopter subassemblies using the hierarchical pruning strategy and large language model

Assembly instructions are process documents in detail describing the operation steps, materials, tools, fixtures, and assembly sequences in assembly procedures. Due to assembly instructions including numerous contents, and the content being easy for workers to understand, process designers need to spend lots of time thinking and authoring assembly instructions to ensure that workers can complete the assembly task according to the assembly instructions. Focusing on the difficulties of the variety of assembly instructions and the process factors implicit in the standard languages of assembly instructions, a method of assembly instruction generation for helicopter subassemblies is proposed. First, a data representation model of multi-source heterogeneous knowledge and information based on knowledge graphs is designed and established. Then, a hierarchical pruning VF3 algorithm is presented to reuse assembly instructions according to hybrid similarity. Finally, a process factor revision model based on RoBERTa-BiLSTM-CRF is proposed to generate revised assembly instructions. Helicopter subassemblies, which contain 11,240 assembly procedures, are used to evaluate the performance of the method for generating assembly instructions. The proposed method greatly reduces the time cost of assembly instruction authoring and promotes the intelligent development of assembly process design.

Pruned tree-structured temporal convolutional networks for quality variable prediction of industrial process

In real industrial processes, the rapid and accurate acquisition of quality variables is essential. Therefore, this paper proposes a pruned tree-structured temporal convolutional network (PT-TCN) for efficient and accurate variables prediction. First, a novel tree network is developed, utilizing dilated causal convolution blocks as nodes to avoid the loss of local information. Each node extracts distinct local information, and by concatenating all tree nodes, the network can capture a comprehensive range of temporal scales. Then, to avoid the increased complexity caused by the tree structure, we design an online two-stage pruning strategy to compress the tree network without introducing additional computations. During the training process, blocks are initially pruned based on the correlation assessment between quality variables and tree nodes. Subsequently, weight normalization layers are employed to evaluate the importance of output channels in blocks, thereby enabling intra-block channel pruning. The effectiveness of PT-TCN is verified on Tennessee Eastman benchmark process. In addition, experiments on the real zinc flotation process demonstrate that the proposed PT-TCN improves in R2 and MAE by 1.32% and 1.26% respectively in predicting quality variables, and it can reduce 91.8% parameters of the initial tree-structured TCN without sacrificing accuracy.

MemoriaNova: Optimizing Memory-Aware Model Inference for Edge Computing

In recent years, deploying deep learning models on edge devices has become pervasive, driven by the increasing demand for intelligent edge computing solutions across various industries. From industrial automation to intelligent surveillance and healthcare, edge devices are being leveraged for real-time analytics and decision-making. Existing methods face two challenges when deploying machine learning models on edge devices. The first challenge is handling the execution order of operators with a simple strategy, which can lead to a potential waste of memory resources when dealing with directed acyclic graph structure models. The second challenge is that they usually process operators of a model one by one to optimize the inference latency, which may lead to the optimization problem getting trapped in local optima. We present MemoriaNova, comprising BTSearch and GenEFlow, to solve these two problems. BTSearch is a graph state backtracking algorithm with efficient pruning and hashing strategies designed to minimize memory overhead during inference and enlarge latency optimization search space. GenEFlow, based on genetic algorithms, integrates latency modeling and memory constraints to optimize distributed inference latency. This innovative approach considers a comprehensive search space for model partitioning, ensuring robust and adaptable solutions. We implement BTSearch and GenEFlow and test them on eleven deep-learning models with different structures and scales. The results show that BTSearch can reach 12% memory optimization compared with the widely used random execution strategy. At the same time, GenEFlow reduces inference latency by 33.9% in distributed systems with four-edge devices.

UCC: A unified cascade compression framework for vision transformer models

In recent years, Vision Transformer (ViT) and its variants have dominated many computer vision tasks. However, the high computational consumption and training data requirements of ViT make it challenging to be deployed directly on resource-constrained devices and environments. Model compression is an effective approach to accelerate deep learning networks, but existing methods for compressing ViT models are limited in their scopes and struggle to strike a balance between performance and computational cost. In this paper, we propose a novel Unified Cascaded Compression Framework (UCC) to compress ViT in a more precise and efficient manner. Specifically, we first analyze the frequency information within tokens and prune them based on a joint score of their both spatial and spectral characteristics. Subsequently, we propose a similarity-based token aggregation scheme that combines the abundant contextual information contained in all pruned tokens with the host tokens according to their weights. Additionally, we introduce a novel cumulative cascaded pruning strategy that performs bottom-up cascaded pruning of tokens based on cumulative scores, avoiding information loss caused by individual idiosyncrasies of blocks. Finally, we design a novel two-level distillation strategy, incorporating imitation and exploration, to ensure the diversity of knowledge and better performance recovery. Extensive experiments demonstrate that our unified cascaded compression framework outperforms most existing state-of-the-art approaches, compresses the floating-point operations of ViT-Base as well as DeiT-Base models by 22 % and 54.1 %, and improves the recognition accuracy of the models by 3.74 % and 1.89 %, respectively, significantly reducing model computational consumption while enhancing performance, which enables efficient end-to-end training of compact ViT models.

A Multilinear HJB-POD Method for the Optimal Control of PDEs on a Tree Structure

Optimal control problems driven by evolutionary partial differential equations arise in many industrial applications and their numerical solution is known to be a challenging problem. One approach to obtain an optimal feedback control is via the Dynamic Programming principle. Nevertheless, despite many theoretical results, this method has been applied only to very special cases since it suffers from the curse of dimensionality. Our goal is to mitigate this crucial obstruction developing a version of dynamic programming algorithms based on a tree structure and exploiting the compact representation of the dynamical systems based on tensors notations via a model reduction approach. Here, we want to show how this algorithm can be constructed for general nonlinear control problems and to illustrate its performances on a number of challenging numerical tests introducing novel pruning strategies that improve the efficacy of the method. Our numerical results indicate a large decrease in memory requirements, as well as computational time, for the proposed problems. Moreover, we prove the convergence of the algorithm and give some hints on its implementation.

Structured pruning of neural networks for constraints learning

In recent years, the integration of Machine Learning (ML) models with Operation Research (OR) tools has gained popularity in applications such as cancer treatment, algorithmic configuration, and chemical process optimization. This integration often uses Mixed Integer Programming (MIP) formulations to represent the chosen ML model, that is often an Artificial Neural Networks (ANNs) due to their widespread use. However, ANNs frequently contain a large number of parameters, resulting in MIP formulations impractical to solve. In this paper we showcase the effectiveness of a ANN pruning, when applied to models prior to their integration into MIPs. We discuss why pruning is more suitable in this context than other ML compression techniques, and we highlight the potential of appropriate pruning strategies via experiments on MIPs used to construct adversarial examples to ANNs. Our results demonstrate that pruning offers remarkable reductions in solution times without hindering the quality of the final decision, enabling the resolution of previously unsolvable instances.

Model optimization and acceleration method based on meta-learning and model pruning for laser vision weld tracking system

Purpose This paper aims to propose a lightweight, high-accuracy object detection model designed to enhance seam tracking quality under strong arcs and splashes condition. Simultaneously, the model aims to reduce computational costs. Design/methodology/approach The lightweight model is constructed based on Single Shot Multibox Detector (SSD). First, a neural architecture search method based on meta-learning and genetic algorithm is introduced to optimize pruning strategy, reducing human intervention and improving efficiency. Additionally, the Alternating Direction Method of Multipliers (ADMM) is used to perform structural pruning on SSD, effectively compressing the model with minimal loss of accuracy. Findings Compared to state-of-the-art models, this method better balances feature extraction accuracy and inference speed. Furthermore, seam tracking experiments on this welding robot experimental platform demonstrate that the proposed method exhibits excellent accuracy and robustness in practical applications. Originality/value This paper presents an innovative approach that combines ADMM structural pruning and meta-learning-based neural architecture search to significantly enhance the efficiency and performance of the SSD network. This method reduces computational cost while ensuring high detection accuracy, providing a reliable solution for welding robot laser vision systems in practical applications.

Robust filter pruning guided by deep frequency-features for edge intelligence

With the popularity of mobile applications, smart terminals require the deployment of lightweight deep neural networks (DNNs). However, those lightweight DNNs deployed on resource-constrained edge devices, have serious security vulnerabilities to adversarial examples. Pruning methods have been widely used to obtain lightweight DNNs, which mainly focus on improving classification accuracy, but ignore robustness. To this end, we present a Fourier space-based robust pruning algorithm (FSRP). We find that a robust network pays more attention to the low-frequency region of the adversarial examples’ feature maps, but ignores the high-frequency region. According to this finding, we design a filter robust indicator, a ratio of low-frequency components to high-frequency components of the feature map, to guide the pruning process. With this new robust pruning criterion, we adopt a strategy of local pruning that removes the filters with low robustness layer by layer in the model. Extensive experiments on CIFAR10/CIFAR100 and ImageNet show that the robust pruning accuracy is significantly improved under the FSRP robust pruning criterion. On CIFAR10, a pruning ratio of 90% on the VGG16 network still shows a 14.1% improvement in robust accuracy under auto attacks (AA).

Research on Lightweight Method of Insulator Target Detection Based on Improved SSD.

Aiming at the problems of a large volume, slow processing speed, and difficult deployment in the edge terminal, this paper proposes a lightweight insulator detection algorithm based on an improved SSD. Firstly, the original feature extraction network VGG-16 is replaced by a lightweight Ghost Module network to initially achieve the lightweight model. A Feature Pyramid structure and Feature Pyramid Network (FPN+PAN) are integrated into the Neck part and a Simplified Spatial Pyramid Pooling Fast (SimSPPF) module is introduced to realize the integration of local features and global features. Secondly, multiple Spatial and Channel Squeeze-and-Excitation (scSE) attention mechanisms are introduced in the Neck part to make the model pay more attention to the channels containing important feature information. The original six detection heads are reduced to four to improve the inference speed of the network. In order to improve the recognition performance of occluded and overlapping targets, DIoU-NMS was used to replace the original non-maximum suppression (NMS). Furthermore, the channel pruning strategy is used to reduce the unimportant weight matrix of the model, and the knowledge distillation strategy is used to fine-adjust the network model after pruning, so as to ensure the detection accuracy. The experimental results show that the parameter number of the proposed model is reduced from 26.15 M to 0.61 M, the computational load is reduced from 118.95 G to 1.49 G, and the mAP is increased from 96.8% to 98%. Compared with other models, the proposed model not only guarantees the detection accuracy of the algorithm, but also greatly reduces the model volume, which provides support for the realization of visible light insulator target detection based on edge intelligence.

A Clustering Pruning Method Based on Multidimensional Channel Information

Pruning convolutional neural networks offers a promising solution to mitigate the computational complexity challenges encountered during application deployment. However, prevalent pruning techniques primarily concentrate on model parameters or feature mapping analysis to devise static pruning strategies, often overlooking the underlying feature extraction capacity of convolutional kernels. To address this, the study first quantitatively expresses the feature extraction capability of convolutional channels from three aspects: global features, distribution metrics, and directional metrics. It explores the multi-dimensional information of the channels, calculates the overall expectation, variance, and cosine distance from the unit vector as the quantitative results of the channels. Subsequently, a clustering algorithm is employed to categorize the multidimensional information. This approach ensures that convolutional channels grouped within each cluster possess similar feature extraction capabilities. An enhanced differential evolutionary algorithm is utilized to optimize the number of clustering centers across all convolutional layers, ensuring optimal grouping. The final step involves achieving channel sparsification through the calculation of crowding distances for each sample within its designated cluster. This preserves a diverse subset of channels that are critical for maintaining model accuracy. Extensive empirical evaluations conducted on three benchmark image classification datasets demonstrate the efficacy of this method. For instance, on the ImageNet dataset, the ResNet-50 model experiences a substantial reduction in FLOPs by 58.43% while incurring a minimal decrease in TOP-1 accuracy of only 1.15%.

Towards compressed and efficient CNN architectures via pruning

Convolutional Neural Networks (CNNs) use convolutional kernels to extract important low-level to high-level features from data. The performance of CNNs improves as they grow deep thereby learning better representations of the data. However, such deep CNNs are compute and memory-intensive, making deployment on resource-constrained devices challenging. To address this, the CNNs are compressed by adopting pruning strategies that remove redundant convolutional kernels from each layer while maintaining accuracy. Existing pruning methods that are based on feature map importance, only prune the convolutional layers uniformly and do not consider fully connected layers. Also, current techniques do not take into account class labels while pruning the less important feature maps and do not explore the need for retraining after pruning. This paper presents pruning techniques to prune convolutional and fully connected layers. This paper proposes a novel class-specific pruning strategy based on finding feature map importance in terms of entropy for convolutional layers and the number of incoming zeros to neurons for fully connected layers. The class-specific approach helps to have a different pruning threshold for every convolutional layer and ensures that the pruning threshold is not influenced by any particular class. A study on the need for retraining the entire network or a part of the network after pruning is also carried out. For Intel image, CIFAR10 and CIFAR100 datasets the proposed pruning method has compressed AlexNet by 83.2%, 87.19%, and 79.7%, VGG-16 by 83.7%, 85.11%, and 84.06% and ResNet-50 by 62.99%, 62.3% and 58.34% respectively.

Leveraging discriminative data: A pathway to high-performance, stable One-shot Network Pruning at Initialization

One-shot Network Pruning at Initialization (OPaI) is acknowledged as a highly cost-effective strategy for network pruning. However, it has been observed that OPaI models tend to suffer from reduced accuracy stability as target sparsity increases. This study introduces a novel approach by incorporating Discriminative Data (DD) into OPaI, significantly improving performance at higher sparsity levels while maintaining the “one-shot” nature. Our approach achieves state-of-the-art (SOTA) performance, challenging the previously held belief of OPaI’s data independence. Through detailed ablation studies, we thoroughly investigate the influence of data on OPaI, particularly focusing on how DD addresses a common failure in OPaI known as “layer collapse”. Furthermore, our experiments demonstrate that leveraging DD from various pre-trained models can markedly boost pruning performance across different models without requiring changes to the existing model architectures or pruning methodologies. These significant improvements highlight our method’s high generalizability and stability, paving new paths for advancing pruning strategies. Our code is publicly available at: https://github.com/Nonac/DDOPaI.

A path splitting and pruning strategy on list decoder for PAC codes

AbstractRecently, Arıkan proposed a polarization‐adjusted convolutional (PAC) codes, demonstrating their superior error correction performance over polar codes at short block lengths. It was confirmed that PAC codes approached the optimal performance achievable with limited code length. This paper proposes a novel low‐complexity list decoding algorithm for PAC codes, incorporating path splitting and pruning strategies based on a set of highly reliable information bits. Simulation results reveal that the proposed algorithm significantly reduces sorting complexity and average list size, all while incurring negligible performance loss. Unlike previous pruning algorithms designed for polar codes, the proposed strategy eliminates the need to individually assess the reliability of decoding paths in each decoding process. Instead, the algorithm minimizes redundant decoding paths through a high‐reliability information bit set, constructed using Monte Carlo experiments.

Insulator Defect Detection Based on YOLOv5s-KE

To tackle the issue of low detection accuracy in insulator images caused by intricate backgrounds and small defect sizes, as well as the requirement for real-time detection on embedded and mobile devices, this research introduces the YOLOv5s-KE model. Integrating multiple strategies, YOLOv5s-KE aims to boost detection accuracy significantly. Initially, an enhanced anchor generation method utilizing the K-means++ algorithm is proposed to generate more appropriate anchor boxes for insulator defects. Moreover, an attention mechanism is integrated into both the backbone and neck networks to enhance the model’s capacity to focus on defect features and resist interference. To improve the detection of small defects, the EIoU loss function is implemented in place of the original CIoU loss function. In order to meet the real-time detection needs on embedded and mobile devices, the model is further refined through the integration of Ghost convolution for lightweight feature extraction and a linear transformation to reduce the computational burden of standard convolution. A channel pruning strategy is deployed to optimize the sparsely trained network, diminishing redundancy, and improving model generalization. Additionally, the CARAFE operator replaces the original upsampling operator to minimize model parameters and elevate detection speed. Experimental outcomes demonstrate that YOLOv5s-KE achieves a detection accuracy of 92.3% on the Chinese transmission line insulator dataset, marking a 5.2% enhancement over the original YOLOv5s. The streamlined version of YOLOv5s-KE achieves a detection speed of 94.3 frames per second, indicating an improvement of 30.1 frames per second compared to the original model. Model parameters are condensed to 9.6 M, resulting in a detection accuracy of 91.1%. This study underscores the precision and efficiency of the proposed approach, suggesting that the advanced strategies explored introduce novel possibilities for insulator defect detection.

IPHM: Incremental periodic high-utility mining algorithm in dynamic and evolving data environments

Periodic high-utility itemset (PHUI) mining can extend beyond the conventional approach of high-utility itemset mining by uncovering recurring customer purchase behaviors common in real-life scenarios (e.g., buying apples and oranges every three days or weekly). Such behaviors, particularly in market basket databases, signify stable patterns that ensure long-term profitability. Existing PHUI mining algorithms assume a static database and incur significant costs when handling incremental databases, as each batch of new transactions necessitates reprocessing the entire dataset. To overcome this challenge, we introduce the Incremental Periodic High-Utility Itemset Miner (IPHM), a method for efficiently extracting periodic high-utility itemsets in incremental database environments. We propose an innovative incremental utility-list structure tailored for incremental database scenarios. Effective pruning strategies are employed to expedite the construction and update of incremental utility-lists and to discard unpromising candidates. As demonstrated by the experimental results, the algorithm is efficacious and efficient, highlighting its practical applicability in dynamic data environments. The algorithm shows a remarkable ability to quickly adapt to database changes, making it highly suitable for applications in market basket analysis where frequent updates are common.

Streamlining YOLOv7 for Rapid and Accurate Detection of Rapeseed Varieties on Embedded Device.

Real-time seed detection on resource-constrained embedded devices is essential for the agriculture industry and crop yield. However, traditional seed variety detection methods either suffer from low accuracy or cannot directly run on embedded devices with desirable real-time performance. In this paper, we focus on the detection of rapeseed varieties and design a dual-dimensional (spatial and channel) pruning method to lighten the YOLOv7 (a popular object detection model based on deep learning). We design experiments to prove the effectiveness of the spatial dimension pruning strategy. And after evaluating three different channel pruning methods, we select the custom ratio layer-by-layer pruning, which offers the best performance for the model. The results show that using custom ratio layer-by-layer pruning can achieve the best model performance. Compared to the YOLOv7 model, this approach results in mAP increasing from 96.68% to 96.89%, the number of parameters reducing from 36.5 M to 9.19 M, and the inference time per image on the Raspberry Pi 4B reducing from 4.48 s to 1.18 s. Overall, our model is suitable for deployment on embedded devices and can perform real-time detection tasks accurately and efficiently in various application scenarios.

LPulse: An efficient algorithm for service function chain placement and routing with delay guarantee

Modern network services increasingly depend on the effective orchestration of the Service Function Chain (SFC) with stringent end-to-end delay guarantees. To achieve this, the Delay-Constrained Service Function Chain Placement and Routing (DC-SFCPR) problem must be addressed. This problem involves the optimal selection of nodes for placing network functions and routes that adhere to a specific sequence of service functions, to minimize network bandwidth and CPU costs while strictly adhering to stringent end-to-end delay constraints. The DC-SFCPR problem is NP-hard and existing algorithms either fail to guarantee strict delay constraints or are computationally expensive, making them unsuitable for expanding network topologies. We propose the LPulse algorithm, designed to efficiently solve the DC-SFCPR problem. This algorithm utilizes a layered graph to embed the requirements of service functions, transforming the DC-SFCPR problem into a Delay-Constrained Shortest Path (DCSP) problem. The LPulse algorithm then applies Pulse, a depth-first search framework enhanced with efficient pruning strategies, and incorporates two novel acceleration strategies to solve the DCSP problem. We prove that LPulse ensures the optimality of solutions. Evaluations conducted across various topologies, with node scales ranging from 22 to 10,000, show that LPulse surpasses existing algorithms in both solution quality and speed. For instance, the number of cases meeting strict delay constraints with LPulse is 1.9× that of those solved by deep reinforcement learning algorithms; furthermore, its solving efficiency is 4.9× that of the highest-performing existing optimal algorithm, the LagrangianKsp algorithm.

Semantic community query in a large‐scale attributed graph based on an attribute cohesiveness optimization strategy

AbstractThe task of a semantic community query is to obtain a subgraph based on a given query vertex (or vertex set) and other query parameters in an attributed graph such that belongs to , contains and satisfies a predefined community cohesiveness model. In most cases, existing community query models based on the network structure for traditional attributed networks usually lack community semantics. However, the features of vertex attributes, especially the attributes of the query vertices, which are closely related to the community semantics, are rarely considered in an attributed graph. Existing community query algorithms based on both structure cohesiveness and attribute cohesiveness usually do not take the attributes of the query vertex as an important factor of the community cohesiveness model, which leads to weak semantics of the communities. This paper proposes a semantic community query method named in a large‐scale attributed graph. First, the k‐core structure model is adopted as the structure cohesiveness of our community query model to obtain a subgraph of the original graph. Second, we define attribute cohesiveness based on the average distance between the query vertices and other vertices in terms of attributes in the community to prune the subgraph and obtain the semantic community. In order to improve the community query efficiency in large‐scale attributed graphs, applies two heuristic pruning strategies. The experimental results show that our method outperforms the existing community query methods in multiple evaluation metrics and is ideal for querying semantic communities in large‐scale attributed graphs.

An Improved Medical Image Classification Algorithm Based on Adam Optimizer

Due to the complexity and illegibility of medical images, it brings inconvenience and difficulty to the diagnosis of medical personnel. To address these issues, an optimization algorithm called GSL(Gradient sine linear) based on Adam algorithm improvement is proposed in this paper, which introduces gradient pruning strategy, periodic adjustment of learning rate, and linear interpolation strategy. The gradient trimming technique can scale the gradient to prevent gradient explosion, while the periodic adjustment of the learning rate and linear interpolation strategy adjusts the learning rate according to the characteristics of the sinusoidal function, accelerating the convergence while reducing the drastic parameter fluctuations, improving the efficiency and stability of training. The experimental results show that compared to the classic Adam algorithm, this algorithm can demonstrate better classification accuracy, the GSL algorithm achieves an accuracy of 78% and 75.2% on the MobileNetV2 network and ShuffleNetV2 network under the Gastroenterology dataset; and on the MobileNetV2 network and ShuffleNetV2 network under the Glaucoma dataset, an accuracy of 84.72% and 83.12%. The GSL optimizer achieved significant performance improvement on various neural network structures and datasets, proving its effectiveness and practicality in the field of deep learning, and also providing new ideas and methods for solving the difficulties in medical image recognition.