Pruning Approaches Research Articles

Model Compression Using Progressive Channel Pruning

In this work, we propose a simple but effective channel pruning framework called Progressive Channel Pruning (PCP) to accelerate Convolutional Neural Networks (CNNs). In contrast to the existing channel pruning methods that prune the channels only once per layer in a layer-by-layer fashion, our new progressive framework iteratively prunes a small number of channels from several selected layers, which consists of a three-step attempting-selecting-pruning pipeline in each iteration. In the attempting step, we attempt to prune a pre-defined number of channels from one layer by using any existing channel pruning methods and estimate the accuracy drop for this layer based on the labelled samples in the validation set. In the selecting step, based on the estimated accuracy drops for all layers, we propose a greedy strategy to automatically select a set of layers that will lead to less overall accuracy drop after pruning these layers. In the pruning step, we prune a small number of channels from these selected layers. We further extend our PCP framework to prune channels for the deep transfer learning methods like Domain Adversarial Neural Network (DANN), in which we effectively reduce the data distribution mismatch in the channel pruning process by using both labelled samples from the source domain and pseudo-labelled samples from the target domain. Our comprehensive experiments on two benchmark datasets demonstrate that our PCP framework outperforms the existing channel pruning approaches under both supervised learning and transfer learning settings.

Acceleration of Deep Convolutional Neural Networks Using Adaptive Filter Pruning

While convolutional neural networks (CNNs) have achieved remarkable performance on various supervised and unsupervised learning tasks, they typically consist of a massive number of parameters. This results in significant memory requirements as well as a computational burden. Consequently, there is a growing need for filter-level pruning approaches for compressing CNN based models that not only reduce the total number of parameters but reduce the overall computation as well. We present a new min-max framework for the filter-level pruning of CNNs. Our framework jointly prunes and fine-tunes CNN model parameters, with an adaptive pruning rate, while maintaining the model's predictive performance. Our framework consists of two modules: (1) An adaptive filter pruning (AFP) module, which minimizes the number of filters in the model; and (2) A pruning rate controller (PRC) module, which maximizes the accuracy during pruning. In addition, we also introduce orthogonality regularization in training of CNNs to reduce redundancy across filters of a particular layer. In the proposed approach, we prune the least important filters and, at the same time, reduce the redundancy level in the model by using orthogonality constraints during training. Moreover, unlike most previous approaches, our approach allows directly specifying the desired error tolerance instead of the pruning level. We perform extensive experiments for object classification (LeNet, VGG, MobileNet, and ResNet) and object detection (SSD, and Faster-RCNN) over benchmarked datasets such as MNIST, CIFAR, GTSDB, ImageNet, and MS-COCO. We also present several ablation studies to validate the proposed approach. Our compressed models can be deployed at run-time, without requiring any special libraries or hardware. Our approach reduces the number of parameters of VGG-16 by an impressive factor of 17.5X, and the number of FLOPS by 6.43X, with no loss of accuracy, significantly outperforming other state-of-the-art filter pruning methods.

Deep network compression based on partial least squares

Modern visual pattern recognition methods are based on convolutional networks since they are able to learn complex patterns directly from the data. However, convolutional networks are computationally expensive in terms of floating point operations (FLOPs), energy consumption and memory requirements, which hinder their deployment on low-power and resource-constrained systems. To address this problem, many works have proposed pruning strategies, which remove neurons (i.e., filters) in convolutional networks to reduce their computational cost. Despite achieving remarkable results, existing pruning approaches are ineffective since the accuracy of the network is degraded. This loss in accuracy is an effect of the criterion used to remove filters, as it may result in the removal of the filters with high influence to the classification ability of the network. Motivated by this, we propose an approach that eliminates filters based on the relationship of their outputs with the class label, on a low-dimensional space. This relationship is captured using Partial Least Squares (PLS), a discriminative feature projection method. Due to the nature of PLS, our method focuses on keeping discriminative filters. As a consequence, we are able to remove up to 60% of FLOPs while improving network accuracy. We show that our criterion is superior to existing pruning criteria, which include state-of-the-art feature selection techniques and handcrafted approaches. Compared to state-of-the-art pruning strategies, our method achieves the best tradeoff between drop/improvement in accuracy and FLOPs reduction.

Channel Pruning Guided by Classification Loss and Feature Importance

In this work, we propose a new layer-by-layer channel pruning method called Channel Pruning guided by classification Loss and feature Importance (CPLI). In contrast to the existing layer-by-layer channel pruning approaches that only consider how to reconstruct the features from the next layer, our approach additionally take the classification loss into account in the channel pruning process. We also observe that some reconstructed features will be removed at the next pruning stage. So it is unnecessary to reconstruct these features. To this end, we propose a new strategy to suppress the influence of unimportant features (i.e., the features will be removed at the next pruning stage). Our comprehensive experiments on three benchmark datasets, i.e., CIFAR-10, ImageNet, and UCF-101, demonstrate the effectiveness of our CPLI method.

Dynamic Network Pruning with Interpretable Layerwise Channel Selection

Dynamic network pruning achieves runtime acceleration by dynamically determining the inference paths based on different inputs. However, previous methods directly generate continuous decision values for each weight channel, which cannot reflect a clear and interpretable pruning process. In this paper, we propose to explicitly model the discrete weight channel selections, which encourages more diverse weights utilization, and achieves more sparse runtime inference paths. Meanwhile, with the help of interpretable layerwise channel selections in the dynamic network, we can visualize the network decision paths explicitly for model interpretability. We observe that there are clear differences in the layerwise decisions between normal and adversarial examples. Therefore, we propose a novel adversarial example detection algorithm by discriminating the runtime decision features. Experiments show that our dynamic network achieves higher prediction accuracy under the similar computing budgets on CIFAR10 and ImageNet datasets compared to traditional static pruning methods and other dynamic pruning approaches. The proposed adversarial detection algorithm can significantly improve the state-of-the-art detection rate across multiple attacks, which provides an opportunity to build an interpretable and robust model.

Compression of recurrent neural networks for efficient language modeling

Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long–Short Term Memory models. We make particular attention to the high-dimensional output problem caused by the very large vocabulary size. We focus on effective compression methods in the context of their exploitation on devices: pruning, quantization, and matrix decomposition approaches (low-rank factorization and tensor train decomposition, in particular). For each model we investigate the trade-off between its size, suitability for fast inference and perplexity. We propose a general pipeline for applying the most suitable methods to compress recurrent neural networks for language modeling. It has been shown in the experimental study with the Penn Treebank (PTB) dataset that the most efficient results in terms of speed and compression–perplexity balance are obtained by matrix decomposition techniques.

Improved discrete artificial fish swarm algorithm combined with margin distance minimization for ensemble pruning

Ensemble pruning aims to achieve a good result in classification using a smaller size of classifiers by finding the optimal sub-ensemble. Diversity and accuracy of classifiers are widely recognized to be two key factors for a successful ensemble. There is a tradeoff between the diversity and the accuracy of classifiers, which makes the ensemble perform the best. Existing ensemble pruning approaches always find the tradeoff using diversity measures or heuristic algorithms separately. Those pruning approaches based on diversity measures cannot exactly find the tradeoff; Those algorithms based on heuristic algorithms are not also to exhaustively search for it extracted from an initial pool of classifiers with a medium-scale or large-scale size. To address the issue, Improved Discrete Artificial Fish swarm algorithm combined with Margin distance minimization for Ensemble Pruning (IDAFMEP) is proposed using a combination of diversity measure and heuristic algorithm. First, the classifiers in a constructed initial pool are pre-pruned using Margin Distance Minimization (MDM), which can downsize the classifiers who perform badly, and markedly alleviate the computational complexity of ensemble pruning. Second, the final ensemble is efficiently achieved from the retaining classifiers after pre-pruning based on MDM using the proposed Improved discrete Artificial Fish Swarm Algorithm (IDAFSA). Experimental results on 29 datasets from the UCI Machine Learning Repository demonstrate that IDAFMEP can achieve better results than the original ensemble and other state-of-the-art pruning approaches, and that its validity and effectiveness. It provides a new research idea for ensemble pruning.

A novel decision tree classification based on post-pruning with Bayes minimum risk

Pruning is applied in order to combat over-fitting problem where the tree is pruned back with the goal of identifying decision tree with the lowest error rate on previously unobserved instances, breaking ties in favour of smaller trees with high accuracy. In this paper, pruning with Bayes minimum risk is introduced for estimating the risk-rate. This method proceeds in a bottom-up fashion converting a parent node of a subtree to a leaf node if the estimated risk-rate of the parent node for that subtree is less than the risk-rates of its leaf. This paper proposes a post-pruning method that considers various evaluation standards such as attribute selection, accuracy, tree complexity, and time taken to prune the tree, precision/recall scores, TP/FN rates and area under ROC. The experimental results show that the proposed method produces better classification accuracy and its complexity is not much different than the complexities of reduced-error pruning and minimum-error pruning approaches. The experiments also demonstrate that the proposed method shows satisfactory performance in terms of precision score, recall score, TP rate, FP rate and area under ROC.

Optimizing multi-sensor deployment via ensemble pruning for wearable activity recognition

With the rapid development of sensor types and processers, most wearable activity recognition systems tend to making use of multiple homogeneous or heterogeneous sensors to obtain plethora information. However, in a realistic environment, it is difficult to configure an appropriate multi-sensor deployment to gain a tradeoff among computational complexity, accuracy and subject personality. In this paper, a multi-sensor fusion with ensemble pruning system (MSF-EP) is designed to connect with multi-sensor based wearable activity recognition system. As a result, the multi-sensor configuration problem is transformed to multiple ensemble classifier pruning problem. With respect to ensemble pruning for MSF-EP system, two popular order-based ensemble pruning approaches are utilized firstly: reduce-error pruning (RE) and complementarily pruning (Comp). Then, in light of the proposed MSF-EP system, two new ensemble pruning criteria are proposed: mRMR pruning and discriminative pruning (Disc). The mRMR pruning measure is based on mutual information and is a composite criterion of classifier redundancy and relevance with regard to the selected classifier set. Taking into account the discriminations of misclassified instances and correctly classified instances in classifier subset selected respectively, another ensemble pruning measure Disc, which combining RE pruning and Comp pruning, is presented in the form of mutual information also. Finally, the proposed pruning learner with lower error is selected as final ensemble classifier. Through the algorithm, the number and type of multi-sensor are appropriately decided to optimize the multi-sensor fusion without regrading the accuracy performance. The system conducts experimental studies on two real-world activity recognition data sets and the results show the superiority of system over other ensemble algorithms.

Forest Pruning Based on Branch Importance.

A forest is an ensemble with decision trees as members. This paper proposes a novel strategy to pruning forest to enhance ensemble generalization ability and reduce ensemble size. Unlike conventional ensemble pruning approaches, the proposed method tries to evaluate the importance of branches of trees with respect to the whole ensemble using a novel proposed metric called importance gain. The importance of a branch is designed by considering ensemble accuracy and the diversity of ensemble members, and thus the metric reasonably evaluates how much improvement of the ensemble accuracy can be achieved when a branch is pruned. Our experiments show that the proposed method can significantly reduce ensemble size and improve ensemble accuracy, no matter whether ensembles are constructed by a certain algorithm such as bagging or obtained by an ensemble selection algorithm, no matter whether each decision tree is pruned or unpruned.

Resolving inconsistencies and redundancies in declarative process models

Declarative process models define the behaviour of business processes as a set of constraints. Declarative process discovery aims at inferring such constraints from event logs. Existing discovery techniques verify the satisfaction of candidate constraints over the log, but completely neglect their interactions. As a result, the inferred constraints can be mutually contradicting and their interplay may lead to an inconsistent process model that does not accept any trace. In such a case, the output turns out to be unusable for enactment, simulation or verification purposes. In addition, the discovered model contains, in general, redundancies that are due to complex interactions of several constraints and that cannot be cured using existing pruning approaches. We address these problems by proposing a technique that automatically resolves conflicts within the discovered models and is more powerful than existing pruning techniques to eliminate redundancies. First, we formally define the problems of constraint redundancy and conflict resolution. Second, we introduce techniques based on the notion of automata-product monoid, which guarantees the consistency of the discovered models and, at the same time, keeps the most interesting constraints in the pruned set. The level of interestingness is dictated by user-specified prioritisation criteria. We evaluate the devised techniques on a set of real-world event logs.

Baseband Equivalent Volterra Series for Behavioral Modeling and Digital Predistortion of Power Amplifiers Driven With Wideband Carrier Aggregated Signals

This paper presents a new baseband equivalent (BBE) Volterra-series model suitable for the behavioral modeling and linearization of wideband RF power amplifiers (PAs). Starting with passband Volterra series, and following a number of signal transformations, a discrete expression relating the envelopes of the output and input signals to the carrier frequency was derived. The new BBE Volterra series reduces the number of kernels, and hence, avoids resorting to the pruning approaches commonly applied in the literature to the classical low-pass equivalent (LPE) Volterra formulation. The proposed baseband Volterra series has similar modeling and linearization performance to the full classical LPE Volterra series. It also outperforms the pruned LPE Volterra models, which use a dynamic deviation reduction approach, in terms of both linearization performance and complexity. The proposed BBE Volterra series was successfully used to linearize different PAs (200-W LDMOS Doherty and 45-W broadband GaN PA) driven with wideband and intra-band carrier aggregated signals (mixed LTE and WCDMA signals).

Dynamic ensemble pruning based on multi-label classification

Dynamic (also known as instance-based) ensemble pruning selects a (potentially) different subset of models from an ensemble during prediction based on the given unknown instance with the goal of maximizing prediction accuracy. This paper models dynamic ensemble pruning as a multi-label classification task, by considering the members of the ensemble as labels. Multi-label training examples are constructed by evaluating whether ensemble members are accurate or not on the original training set via cross-validation. We show that classification accuracy is maximized when learning algorithms that optimize example-based precision are used in the multi-label classification task. Results comparing the proposed framework against state-of-the-art dynamic ensemble pruning approaches in a variety of datasets using a heterogeneous ensemble of 200 classifiers show that it leads to significantly improved accuracy.

A spectral clustering based ensemble pruning approach

This paper introduces a novel bagging ensemble classifier pruning approach. Most investigated pruning approaches employ heuristic functions to rank classifiers in the ensemble, and select part of them from the ranked ensemble, so redundancy may exist in the selected classifiers. Based on the idea that the selected classifiers should be accurate and diverse, we define classifier similarity according to the predictive accuracy and the diversity, and introduce a Spectral Clustering based classifier selection approach (SC). SC groups the classifiers into two clusters based on the classifier similarity, and retains one cluster of classifiers in the ensemble. Experimental results show that SC is competitive in terms of classification accuracy.

Sparse least square support vector machine via coupled compressive pruning

Among the support vector machines, Least Square Support Vector Machine (LSSVM) is computationally attractive for reducing a set of inequality constraints to linear equations. Several pruning algorithms have been developed to obtain reduced support vectors to improve the generalization performance of LSSVM. However, most of the pruning algorithms iteratively select the support vectors, which is of high computation complexity. In this paper, inspired by the recently developed compressive sampling theory, a one-step compressive pruning strategy is proposed to construct a sparse LSSVM without the remarkable reduction of accuracy. It is a fast, universal and information-preserved pruning approach that can avoid the intensive computations in iterative retraining. Some experiments on pattern recognition and function approximation are taken to compare the proposed method with the available pruning approaches, and the results show the feasibility of the proposed method and the superiority to its counterparts.

Variance Sensitivity Analysis of Parameters for Pruning of a Multilayer Perceptron: Application to a Sawmill Supply Chain Simulation Model

Simulation is a useful tool for the evaluation of a Master Production/Distribution Schedule (MPS). The goal of this paper is to propose a new approach to designing a simulation model by reducing its complexity. According to the theory of constraints, a reduced model is built using bottlenecks and a neural network exclusively. This paper focuses on one step of the network model design: determining the structure of the network. This task may be performed by using the constructive or pruning approaches. The main contribution of this paper is twofold; it first proposes a new pruning algorithm based on an analysis of the variance of the sensitivity of all parameters of the network and then uses this algorithm to reduce the simulation model of a sawmill supply chain. In the first step, the proposed pruning algorithm is tested with two simulation examples and compared with three classical pruning algorithms from the literature. In the second step, these four algorithms are used to determine the optimal structure of the network used for the complexity-reduction design procedure of the simulation model of a sawmill supply chain.

Skeleton pruning as trade-off between skeleton simplicity and reconstruction error

Skeletons can be viewed as a compact shape representation in that each shape can be completely reconstructed from its skeleton. However, the usefulness of a skeletal representation is strongly limited by its instability. Skeletons suffer from contour noise in that small contour deformation may lead to large structural changes in the skeleton. A large number of skeleton computation and skeleton pruning approaches has been proposed to address this issue. Our approach differs fundamentally in the fact that we cast skeleton pruning as a trade-off between skeleton simplicity and shape reconstruction error. An ideal skeleton of a given shape should be the skeleton with a simplest possible structure that provides a best possible reconstruction of a given shape. To quantify this trade-off, we propose that the skeleton simplicity corresponds to model simplicity in the Bayesian framework, and the shape reconstruction accuracy is expressed as goodness of fit to the data. We also provide a simple algorithm to approximate the maximum of the Bayesian posterior probability which defines an order for iteratively removing the end branches to obtain the pruned skeleton. Presented experimental results obtained without any parameter tuning clearly demonstrate that the resulting skeletons are stable to boundary deformations and intra class shape variability.

A competitive ensemble pruning approach based on cross-validation technique

Ensemble pruning is crucial for the consideration of both efficiency and predictive accuracy of an ensemble system. This paper proposes a new Competitive technique for Ensemble Pruning based on Cross-Validation (CEPCV). The data to be learnt by neural computing models are mostly drifting with time and environment, therefore a dynamic ensemble pruning method is indispensable for practical applications, while the proposed CEPCV method is just the kind of dynamic ensemble pruning method, which can realize on-line ensemble pruning and take full advantage of potentially valuable information. The algorithm naturally inherits the predominance of cross-validation technique, which implies that those networks regarded as winners in selective competitions and chosen into the pruned ensemble have the “strongest” generalization capability. It is essentially based on the strategy of “divide and rule, collect the wisdom”, and might alleviate the local minima problem of many conventional ensemble pruning approaches only at the cost of a little greater computational cost, which is acceptable to most applications of ensemble learning. The comparative experiments among the four ensemble pruning algorithms, including: CEPCV and the state-of-the-art Directed Hill Climbing Ensemble Pruning (DHCEP) algorithm and two baseline methods, i.e. BSM, which chooses the Best Single Model in the initial ensemble based on their performances on the pruning set, and ALL, which reserves all network members of the initial ensemble, on ten benchmark classification tasks, demonstrate the effectiveness and validity of CEPCV.

An evolutionary constructive and pruning algorithm for artificial neural networks and its prediction applications

We propose a method for designing artificial neural networks (ANNs) for prediction problems based on an evolutionary constructive and pruning algorithm (ECPA). The proposed ECPA begins with a set of ANNs with the simplest possible structure, one hidden neuron connected to an input node, and employs crossover and mutation operators to increase the complexity of an ANN population. Additionally, cluster-based pruning (CBP) and age-based survival selection (ABSS) are proposed as two new operators for ANN pruning. The CBP operator retains significant neurons and prunes insignificant neurons on a probability basis and therefore prevents the exponential growth of an ANN. The ABSS operator can delete old ANNs with potentially complex structures and then introduce new ANNs with simple structures; thus, the ANNs are less likely to be trapped in a fully connected topology. The ECPA framework incorporates constructive and pruning approaches in an attempt to efficiently evolve compact ANNs. As a demonstration of the method, ECPA is applied to three prediction problems: the Mackey-Glass time series, the number of sunspots, and traffic flow. The numerical results show that ECPA makes the design of ANNs more feasible and practical for real-world applications.

Probabilistic Reverse Nearest Neighbor Queries on Uncertain Data

Uncertain data are inherent in various important applications and reverse nearest neighbor (RNN) query is an important query type for many applications. While many different types of queries have been studied on uncertain data, there is no previous work on answering RNN queries on uncertain data. In this paper, we formalize probabilistic reverse nearest neighbor query that is to retrieve the objects from the uncertain data that have higher probability than a given threshold to be the RNN of an uncertain query object. We develop an efficient algorithm based on various novel pruning approaches that solves the probabilistic RNN queries on multidimensional uncertain data. The experimental results demonstrate that our algorithm is even more efficient than a sampling-based approximate algorithm for most of the cases and is highly scalable.