Conventional Pruning Research Articles

Regression Trees With Fused Leaves.

We propose a novel regression tree method named "TreeFuL," an abbreviation for 'Tree with Fused Leaves.' TreeFuL innovatively combines recursive partitioning with fused regularization, offering a distinct approach to the conventional pruning method. One of TreeFuL's noteworthy advantages is its capacity for cross-validated amalgamation of non-neighboring terminal nodes. This is facilitated by a leaf coloring scheme that supports tree shearing and node amalgamation. As a result, TreeFuL facilitates the development of more parsimonious tree models without compromising predictive accuracy. The refined model offers enhanced interpretability, making it particularly well-suited for biomedical applications of decision trees, such as disease diagnosis and prognosis. We demonstrate the practical advantages of our proposed method through simulation studies and an analysis of data collected in an obesity study.

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

Neural network compression techniques, such as knowledge distillation (KD) and network pruning, have received increasing attention. Recent work `Prune, then Distill' reveals that a pruned student-friendly teacher network can benefit the performance of KD. However, the conventional teacher-student pipeline, which entails cumbersome pre-training of the teacher and complicated compression steps, makes pruning with KD less efficient. In addition to compressing models, recent compression techniques also emphasize the aspect of efficiency. Early pruning demands significantly less computational cost in comparison to the conventional pruning methods as it does not require a large pre-trained model. Likewise, a special case of KD, known as self-distillation (SD), is more efficient since it requires no pre-training or student-teacher pair selection. This inspires us to collaborate early pruning with SD for efficient model compression. In this work, we propose the framework named Early Pruning with Self-Distillation (EPSD), which identifies and preserves distillable weights in early pruning for a given SD task. EPSD efficiently combines early pruning and self-distillation in a two-step process, maintaining the pruned network's trainability for compression. Instead of a simple combination of pruning and SD, EPSD enables the pruned network to favor SD by keeping more distillable weights before training to ensure better distillation of the pruned network. We demonstrated that EPSD improves the training of pruned networks, supported by visual and quantitative analyses. Our evaluation covered diverse benchmarks (CIFAR-10/100, Tiny-ImageNet, full ImageNet, CUB-200-2011, and Pascal VOC), with EPSD outperforming advanced pruning and SD techniques.

CrossPrune: Cooperative pruning for camera–LiDAR fused perception models of autonomous driving

Deep neural network pruning is effective in enabling high-performance perception models to be deployed on autonomous driving platforms with limited computation and memory resources. With their rapid development, state-of-the-art autonomous driving perception (ADP) models have advocated the use of multimodal sensors to extract diverse feature categories. However, existing pruning studies in the ADP area focus on single-modal models and neglect multimodal models. Compared with conventional pruning, multimodal pruning presents a new type of redundancy, namely modal-wise redundancy that is caused by multimodal branches extracting similar perception information. When a specific type of information is extracted by more than one modal branch, although this extraction is deemed essential from an individual single-modal standpoint, it becomes redundant when viewed from a modal-wise perspective. Therefore, modal branches must be handled cooperatively to eliminate modal-wise redundancy while concurrently preserving the original perception accuracy. Building on this, we propose CrossPrune, a modal cooperative pruning framework designed for camera–LiDAR fused perception in autonomous driving. The primary objective of CrossPrune is to effectively eliminate redundancy and achieve nondestructive pruning for multimodal ADP models. This was accomplished by approaching the problem as a multi-objective optimization task, encompassing both weight pruning and the restriction of feature distortions caused by pruning. Experiments conducted on the nuScenes and KITTI datasets demonstrated that CrossPrune attained superior pruning ratios while minimizing accuracy loss, surpassing the performance of the baselines. The key results indicated that the proposed CrossPrune achieved relative improvements of 9.6% in mAP and 11.5% in NDS under 89.8% pruning sparsity.

Variable Scale Pruning for Transformer Model Compression in End-to-End Speech Recognition

Transformer models are being increasingly used in end-to-end speech recognition systems for their performance. However, their substantial size poses challenges for deploying them in real-world applications. These models heavily rely on attention and feedforward layers, with the latter containing a vast number of parameters that significantly contribute to the model’s memory footprint. Consequently, it becomes pertinent to consider pruning these layers to reduce the model’s size. In this article, our primary focus is on the feedforward layers. We conduct a comprehensive analysis of their parameter count and distribution. Specifically, we examine the weight distribution within each layer and observe how the weight values progress across the transformer model’s blocks. Our findings demonstrate a correlation between the depth of the feedforward layers and the magnitude of their weights. Consequently, layers with higher weight values require less pruning. Building upon this insight, we propose a novel pruning algorithm based on variable rates. This approach sets the pruning rate according to the significance and location of each feedforward layer within the network. To evaluate our new pruning method, we conduct experiments on various datasets. The results reveal its superiority over conventional pruning techniques, such as local pruning and global pruning.

Optimization Based Layer-Wise Pruning Threshold Method for Accelerating Convolutional Neural Networks

Among various network compression methods, network pruning has developed rapidly due to its superior compression performance. However, the trivial pruning threshold limits the compression performance of pruning. Most conventional pruning threshold methods are based on well-known hard or soft techniques that rely on time-consuming handcrafted tests or domain experience. To mitigate these issues, we propose a simple yet effective general pruning threshold method from an optimization point of view. Specifically, the pruning threshold problem is formulated as a constrained optimization program that minimizes the size of each layer. More importantly, our pruning threshold method together with conventional pruning works achieves a better performance across various pruning scenarios on many advanced benchmarks. Notably, for the L1-norm pruning algorithm with VGG-16, our method achieves higher FLOPs reductions without utilizing time-consuming sensibility analysis. The compression ratio boosts from 34% to 53%, which is a huge improvement. Similar experiments with ResNet-56 reveal that, even for compact networks, our method achieves competitive compression performance even without skipping any sensitive layers.

Ergonomic Evaluation of Pruning in Simulated Greenhouse conditions

Greenhouse is a space-efficient structure for farming. The workers stretch their bodies to perform operations away from the body and at different heights. The present study aimed to assess the physiological and postural discomfort because of the varied working heights and horizontal distances and also attempts to suggest suitable interventions to overcome them. The current study investigated the physiological and subjective measures of exertion while working at different heights (elbow height, shoulder height and above shoulder height) and horizontal distances (30, 45, 60, and 75 cm) with two different pruning tools under simulated laboratory conditions. All the physiological parameters and responses against different conditions were recorded from six male participants and analyzed as per experimental requirements. The outcome of the present study demonstrated that physiological cost and subjective measure of exertion varied significantly with the variation in heights and horizontal distances. Heart rate, oxygen consumption, energy expenditure, and perceived exertion were significantly higher “above the shoulder height” than elbow and shoulder height. Similarly, all the measured physiological parameters and Rating of Perceived Exertion (RPE) exhibited a gradual increase with increasing the distance from the worker’s body. The study also revealed a differential effect of tools used while performing the pruning work. Further, the multiple regression analysis helped to predict Relative Cardiac Cost from RPE and other evaluated physiological parameters. Overall, findings from the current study concluded that conventional pruning activities need workspace optimization and ergonomic intervention to select pruning tools and achieve the desired productivity with minimum Musculoskeletal Disorders (MSDs).

LWMD: A Comprehensive Compression Platform for End-to-End Automatic Speech Recognition Models

Recently end-to-end (E2E) automatic speech recognition (ASR) models have achieved promising performance. However, existing models tend to adopt increasing model sizes and suffer from expensive resource consumption for real-world applications. To compress E2E ASR models and obtain smaller model sizes, we propose a comprehensive compression platform named LWMD (light-weight model designing), which consists of two essential parts: a light-weight architecture search (LWAS) framework and a differentiable structured pruning (DSP) algorithm. On the one hand, the LWAS framework adopts the neural architecture search (NAS) technique to automatically search light-weight architectures for E2E ASR models. By integrating different architecture topologies of existing models together, LWAS designs a topology-fused search space. Furthermore, combined with the E2E ASR training criterion, LWAS develops a resource-aware search algorithm to select light-weight architectures from the search space. On the other hand, given the searched architectures, the DSP algorithm performs structured pruning to reduce parameter numbers further. With a Gumbel re-parameter trick, DSP builds a stronger correlation between the pruning criterion and the model performance than conventional pruning methods. And an attention-similarity loss function is further developed for better performance. On two mandarin datasets, Aishell-1 and HKUST, the compression results are well evaluated and analyzed to demonstrate the effectiveness of the LWMD platform.

Manejo da poda de amora-preta para aumentar a sazonalidade da colheita

Abstract The objective of this work was to evaluate the pruning management of the BRS Tupy and Brazos blackberry cultivars, in order to extend their harvest seasonality and fruit yield. The experiment was performed in an altitude subtropical region in the state of Minas Gerais, Brazil. Bushes were grown at 3.0x0.5 m spacing. The treatments consisted of the four following pruning seasons: one conventional pruning, with suppression of the produced stems in February and reduction of the stems in July; and three drastic pruning performed in the first two weeks of January, March, and May, with the application of 10% urea and 3% hydrogenated cyanamide five months later. The experimental design was in randomized complete blocks, in a 2x4 factorial arrangement, with two cultivars and four pruning managements, with four blocks. Fruit phenology, yield, and physicochemical quality were evaluated in two production cycles. It is possible to extend the harvest season of blackberry up to five months, in an altitude subtropical region in Lavras, in the state of Minas Gerais, Brazil, with drastic pruning in January and no irrigation. Drastic pruning in January or March increases the yield of the BRS Tupy and Brazos blackberry cultivars. The extended harvest season does not affect the quality of blackberries and does not create challenges for harvesting operations.

Progressive Local Filter Pruning for Image Retrieval Acceleration

This paper focuses on network pruning for image retrieval acceleration. Prevailing image retrieval works target at the discriminative feature learning, while little attention is paid to how to accelerate the model inference, which should be taken into consideration in real-world practice. The challenge of pruning image retrieval models is that the middle-level feature should be preserved as much as possible. Such different requirements of the retrieval and classification model make the traditional pruning methods not that suitable for our task. To solve the problem, we propose a new Progressive Local Filter Pruning (PLFP) method for image retrieval acceleration. Specifically, layer by layer, we analyze the local geometric properties of each filter and select the one that can be replaced by the neighbors. Then we progressively prune the filter by gradually changing the filter weights. In this way, the representation ability of the model is preserved. To verify this, we evaluate our method on two widely-used image retrieval datasets,i.e., Oxford5k and Paris6K, and one person re-identification dataset,i.e., Market-1501. The proposed method arrives with superior performance to the conventional pruning methods, suggesting the effectiveness of the proposed method for image retrieval.

Compression of deep neural networks: bridging the gap between conventional-based pruning and evolutionary approach

Recently, many studies have been carried out on model compression to handle the high computational cost and high memory footprint brought by the implementation of deep neural networks. In this paper, model compression of convolutional neural networks is constructed as a multiobjective optimization problem with two conflicting objectives, reducing the model size and improving the performance. A novel structured pruning method called Conventional-based and Evolutionary Approaches Guided Multiobjective Pruning (CEA-MOP) is proposed to address this problem, where the power of conventional pruning methods is effectively exploited for the evolutionary process. A delicate balance in pruning rate and model accuracy has been automated achieved by a multiobjective optimization evolutionary model. First, an ensemble framework integrates pruning metrics to establish a codebook for further evolutionary operations. Then, an efficient coding method is developed to shorten the length of chromosome, thus ensuring its superior scalability. Finally, sensitivity analysis is automatically carried out to determine the upper bound of pruning rate for each layer. Notably, on CIFAR-10, CEA-MOP reduces more than 50% FLOPs on ResNet-110 and improves the relative accuracy. Moreover, on ImageNet, CEA-MOP reduces more than 50% FLOPs on ResNet-101 with negligible top-1 accuracy drop.

LAP: Latency-aware automated pruning with dynamic-based filter selection

Model pruning is widely used to compress and accelerate convolutional neural networks (CNNs). Conventional pruning techniques only focus on how to remove more parameters while ensuring model accuracy. This work not only covers the optimization of model accuracy, but also optimizes the model latency during pruning. When there are multiple optimization objectives, the difficulty of algorithm design increases exponentially. So latency sensitivity is proposed to effectively guide the determination of layer sparsity in this paper. We present the latency-aware automated pruning (LAP) framework which leverages the reinforcement learning to automatically determine the layer sparsity. Latency sensitivity is used as a prior knowledge and involved into the exploration loop. Rather than relying on a single reward signal such as validation accuracy or floating-point operations (FLOPs), our agent receives the feedback on the accuracy error and latency sensitivity. We also provide a novel filter selection algorithm to accurately distinguish important filters within a layer based on their dynamic changes. Compared to the state-of-the-art compression policies, our framework demonstrated superior performances for VGGNet, ResNet, and MobileNet on CIFAR-10, ImageNet, and Food-101. Our LAP allowed the inference latency of MobileNet-V1 to achieve approximately 1.64 times speedup on the Titan RTX GPU, with no loss of ImageNet Top-1 accuracy. It significantly improved the pareto optimal curve on the accuracy and latency trade-off.

Machine Learning-Based Pruning Technique for Low Power Approximate Computing

Approximate Computing is a low power achieving technique that offers an additional degree of freedom to design digital circuits. Pruning is one of the types of approximate circuit design technique which removes logic gates or wires in the circuit to reduce power consumption with minimal insertion of error. In this work, a novel machine learning (ML) -based pruning technique is introduced to design digital circuits. The machine-learning algorithm of the random forest decision tree is used to prune nodes selectively based on their input pattern. In addition, an error compensation value is added to the original output to reduce an error rate. Experimental results proved the efficiency of the proposed technique in terms of area, power and error rate. Compared to conventional pruning, proposed ML pruning achieves 32% and 26% of the area and delay reductions in 8*8 multiplier implementation. Low power image processing algorithms are essential in various applications like image compression and enhancement algorithms. For real-time evaluation, proposed ML optimized pruning is applied in discrete cosine transform (DCT). It is a basic element of image and video processing applications. Experimental results on benchmark images show that proposed pruning achieves a very good peak signal-to-noise ratio (PSNR) value with a considerable amount of energy savings compared to other methods.

Virtual pruning of 3D trees as a tool for managing shading effects in agroforestry systems

Light is a limiting resource for crops within integrated land use systems especially those including woody perennials. The amount of available light at ground level can be modified by artificially pruning the overstory. Aiming to increase the understanding of light management strategies, we simulated the pruning of wild cherry trees and compared the shading effects of the resulting tree structures over a complete growing season, with fine spatiotemporal resolution. Original 3D-tree structures were retrieved employing terrestrial laser scanning and quantitative structure models, and subjected to two pruning treatments at low and high intensities. By using the ‘shadow model’, the analogous tree structures created diverse shaded scenarios varying in size and intensity of insolation reduction. Conventional pruning treatments reduced the crown structure to the uppermost portion of the tree bole, reducing the shading effects, and thus, shrinking the shaded area on the ground by up to 38%, together with the shading intensity. As an alternative, the selective removal of branches reduced the shading effects, while keeping a more similar spatial distribution compared to the unpruned tree. Hence, the virtual pruning of tree structures can support designing and selecting adequate tending operations for the management of light distribution in agroforestry systems. The evidence assembled in this study is highly relevant for agroecosystems and can be strategically used for maintaining, planning and designing integrated tree-crop agricultural systems.

A novel weight pruning strategy for light weight neural networks with application to the diagnosis of skin disease

In recent years, deep learning-based models have achieved significant advances in manifold computer vision problems, but tedious parameter tuning has complicated their application to computer-aided diagnostic (CAD) systems. As such, this study introduces a novel pruning strategy to improve the accuracy of five lightweight deep convolutional neural network (DCNN) architectures applied to the classification of skin disease. Unlike conventional pruning methods (such as optimal brain surgeon), the proposed technique does not change the model size yet improves performance after fine tuning. This training approach, intended to improve accuracy without increasing model complexity, is experimentally verified using 1167 pathological images. The clinical data included 11 different skin disease types collected over the past ten years, with varying image quantities in each category. A novel hierarchical pruning method, based on standard deviation, is then developed and used to prune parameters in each convolution layer according to the different weight distributions. This training strategy achieves an 83.5% Top-1 accuracy using a pruned MnasNet (12.5 MB), which is 1.8% higher than that of unpruned InceptionV3 (256 MB). Comparative experiments using other networks (MobileNetV2, SqueezeNet, ShuffleNetV2, Xception, ResNet50, DenseNet121) and dataset (HAM10000) also demonstrate consistent improvements when adopting the proposed model training technique. This distinctive robustness across various network types and simple deployment demonstrates the potential of this methodology for generalization to other computer vision tasks.

Effects of water-saving pruning on the growth and water consumption of jujube (Ziziphus jujuba Mill.) in a loess hilly region with deep soil desiccation

Abstract In order to study whether jujube trees can grow normally under rain-fed conditions in loess hilly areas, we planted jujube trees (Ziziphus jujuba Mill.) 4 years after felling a 23-year-old apple orchard. The growth process of the jujube trees and the variation in soil water content (SWC) were monitored for three consecutive years following planting in order to study the effects of the water-saving pruning (WSP) technique. Results showed the following. (1) The soil at a depth of 0–1,000 cm had been desiccated when the area was an apple orchard. (2) Under rain-fed conditions, the jujube trees with the WSP technique were always able to maintain normal growth while the jujube trees with conventional pruning method had a normal growth stage of only 4 years. The water use efficiency of the jujube trees with the WSP technique was much higher than that of the jujube trees with conventional pruning. We recommend WSP in jujube orchard management, because the jujube trees with WSP could maintain normal growth in the deep dried soils of the loess hilly region, as WSP can reduce the water consumption of the jujube trees and may have a positive effect on soil moisture restoration.

A novel GWO optimized pruning technique for inexact circuit design

Improving area efficiency of modern computing systems is the main task in today's digital design. circuit design innovations are necessary to keep the power budget under control. Inexact or approximate circuit design is a computing solution that trades circuit precision for cost reduction. In this work present a new pruning technique which using grey wolf optimizer for pruning. This optimization mimics the social behavior of grey wolves whereas it follows the leadership hierarchy and attacking strategy. The probabilistic pruning complexity cost is optimized at each computation to maintain a tradeoff between minimum area, error and area delay product. The parameters like Area, delay and area delay product measures are utilized to analyze the potency of GWO rule with existing one. The observed results convey that the GWO based pruning provides promising results compared to conventional pruning techniques

Modification and Performance Evaluation of Date Palm Leaf and Leaf-end Pruning Machine

Modification of date palm leaf and leaf-end pruning machine was carried out according to the following aspects: Considering the conventional pruning method, Geometric dimension of the usual garden tool (sickle) or local named Acfea, Morphological form of date palm tree, Designed machine dimension.In this study, the main aims were focused on: Agronomical specifications, - machine net weight, - on farm operation, - application efficiency. To design and develop this machine, a lightweight Hydro-motor as well as proper size frame, handle, proper, hydraulic control valve and a disk-type cutting blade were used. The machine is able to pruning the leaves and leaves-end with depth of up to 10 cm. In general, two changes were considered for on farm application: as a part of date palm service machine and /or a self-operating machine.In the first phase, because of the lack of enough space for bucket of machine to reach the date palm trunk, pruning operation can be perform from a distance from 1 to 2 meter from the main tree trunk; therefore, the designed machine was equipped with a 3-meter length boom so it can be controlled from the lifting cabin. In the second phase, for the trees that are not higher than 4 meters, the machine can be used as a self-operating machine by ladder.The machine was equipped with a short and convenient handle, which will help the pruner to reach the tree trunk for pruning the leaves and leaves’ end.The prototype pruning machine was installed to the tractor MF-285 and site conditions to test and evaluate performs the machine technical and ergonomically parameters was successful. Field operation of tractor mounted leaf and leaf-end pruning machine was performed in Date Palm and Tropical Fruits Research Institute. 95% of the confidence interval of pruning leaf and leaf-end were achieved between 8 to 14 s and 10 to 19 s, respectively. The total pruning time was gained between 9.9 to 13.8 min for each tree.

Improved pruning algorithm for Gaussian mixture probability hypothesis density filter

With the increment of the number of Gaussian components, the computation cost increases in the Gaussian mixture probability hypothesis density (GM-PHD) filter. Based on the theory of Chen et al, we propose an improved pruning algorithm for the GM-PHD filter, which utilizes not only the Gaussian components' means and covariance, but their weights as a new criterion to improve the estimate accuracy of the conventional pruning algorithm for tracking very closely proximity targets. Moreover, it solves the end-less while-loop problem without the need of a second merging step. Simulation results show that this improved algorithm is easier to implement and more robust than the formal ones.

Modification and Performance Evaluation of Date Palm Leaf and Leaf-end Pruning Machine

Modification of date palm leaf and leaf-end pruning machine was carried out according to the following aspects: Considering the conventional pruning method, Geometric dimension of the usual garden tool (sickle) or local named Acfea, Morphological form of date palm tree, Designed machine dimension. In this study, the main aims were focused on: Agronomical specifications, - machine net weight, - on farm operation, - application efficiency. To design and develop this machine, a lightweight Hydro-motor as well as proper size frame, handle, proper, hydraulic control valve and a disk-type cutting blade were used. The machine is able to pruning the leaves and leaves-end with depth of up to 10 cm. In general, two changes were considered for on farm application: as a part of date palm service machine and /or a self-operating machine. In the first phase, because of the lack of enough space for bucket of machine to reach the date palm trunk, pruning operation can be perform from a distance from 1 to 2 meter from the main tree trunk; therefore, the designed machine was equipped with a 3-meter length boom so it can be controlled from the lifting cabin. In the second phase, for the trees that are not higher than 4 meters, the machine can be used as a self-operating machine by ladder. The machine was equipped with a short and convenient handle, which will help the pruner to reach the tree trunk for pruning the leaves and leaves’ end. The prototype pruning machine was installed to the tractor MF-285 and site conditions to test and evaluate performs the machine technical and ergonomically parameters was successful. Field operation of tractor mounted leaf and leaf-end pruning machine was performed in Date Palm and Tropical Fruits Research Institute. 95% of the confidence interval of pruning leaf and leaf-end were achieved between 8 to 14 s and 10 to 19 s, respectively. The total pruning time was gained between 9.9 to 13.8 min for each tree.

Blackberry and redberry production in crop and intercrop in Pouso Alegre, southern Minas Gerais, Brazil

ABSTRACT: The cultivation of mulberry ( Rubus sp.) is highly disseminated in cold climates, but some cultivars adapt to warm weather environments. The objective of the present study was to evaluate the production of blackberry and redberry in season and off season in Pouso Alegre, south of Minas Gerais. The experimental design was a randomized block, with twelve cultivars ('Arapaho', 'Xavante', 'Brazos', 'Tupy' 'Comanche', 'Choctaw', 'Guarani', 'Caingangue', 'Cherokee', 'Chicasaw' and 'Clone' and a redberry, three blocks and an experimental unit of four plants with a spacing of 0.5x3.0m and a total density of 6,666 plants per hectare. Conventional pruning was performed to produce the crop (October-December) and for off-season production (April-July). In 'Brazos' there was increased production in the two seasons evaluated against other mulberry trees, conversely, in the off season, there was a positive effect of pruning carried out in January only in 'Tupy', 'Choctaw', 'Comanche' and 'Clone', with 'Tupy' presenting the highest productivity off season. In 'Brazos' and 'Tupy' there was the best balance between soluble solids and acidity in the crop and red berry in season. 'Brazos' and 'Cherokee' had the highest average content of total sugars. Due to the lack of supply of fruit, pruning mulberry trees in January is an alternative to increase the income of family-based farms.