Deep Neural Network Compression Research Articles

PRF: deep neural network compression by systematic pruning of redundant filters

PRF: deep neural network compression by systematic pruning of redundant filters

Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian Approach

Due to deep neural networks (DNNs) a large number of parameters, DNNs increase the demand for computing and storage during training, reasoning and deployment, especially when DNNs stack deeper and wider. Tensor decomposition can not only compress DNN models but also reduce parameters and storage requirements while maintaining high accuracy and performance. About tensor ring (TR) decomposition of tensor decomposition, there are two problems: (1) The practice of setting the TR rank to be equal in TR decomposition results in an unreasonable rank configuration. (2) The training time of selecting rank through iterative processes is time-consuming. To address the two problems, a TR network compression method by Variational Bayesian (TR-VB) is proposed based on the Global Analytic Solution of Empirical Variational Bayesian Matrix Factorization (GAS of EVBMF). The method consists of three steps: (1) rank selection, (2) TR decomposition, and (3) fine-tuning to recover accumulated loss of accuracy. Experimental results show that, for a given network, TR-VB gives the best results in terms of Top-1 accuracy, parameters, and training time under different compression levels. Furthermore, TR-VB validated on CIFAR-10/100 public benchmarks achieves state-of-the-art performance.

Deep Convolutional Neural Network Compression based on the Intrinsic Dimension of the Training Data

Selecting the optimal deep learning architecture for a particular task and dataset remains an ongoing challenge. Typically, this decision-making process involves exhaustive searches for neural network architectures or multi-phase optimization, which includes initial training, compression or pruning, and fine-tuning steps. In this study, we introduce an approach utilizing a deep reinforcement learning-based agent to dynamically compress a deep convolutional neural network throughout its training process. We integrate the concept of the intrinsic dimension of the training data to provide the agent with insights into the task's complexity. The agent employs two distinct ranking criteria, L1-norm-based and attention-based measures, to selectively prune filters from each layer as it determines necessary. In the experiments, we used the CIFAR-10 dataset and its subsets (2-class and 5-class subsets) to model the task complexity and showed that the agent learns different policies depending on the intrinsic dimension. The agent, on average, pruned off 78.48%, 77.9%, and 83.12% filters from all the layers of VGG-16 network for CIFAR-10 full, 5-class, and 2-class subsets respectively.

DeepStream: Video Streaming Enhancements using Compressed Deep Neural Networks

InIn HTTP Adaptive Streaming (HAS), each video is divided into smaller segments, and each segment is encoded at multiple pre-defined bitrates to construct a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bitrate ladder</i> . To optimize bitrate ladders, per-title encoding approaches encode each segment at various bitrates and resolutions to determine the convex hull. From the convex hull, an optimized bitrate ladder is constructed, resulting in an increased Quality of Experience (QoE) for end-users. With the ever-increasing efficiency of deep learning-based video enhancement approaches, they are more and more employed at the client-side to increase the QoE, specifically when GPU capabilities are available. Therefore, scalable approaches are needed to support end-user devices with both CPU and GPU capabilities (denoted as CPU-only and GPU-available end-users, respectively) as a new dimension of a bitrate ladder. To address this need, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepStream</i> , a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable content-aware</i> per-title encoding approach to support both CPU-only and GPU-available end-users. ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> ) To support <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">backward compatibility</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepStream</i> constructs a bitrate ladder based on any existing per-title encoding approach. Therefore, the video content will be provided for legacy end-user devices with CPU-only capabilities as a base layer (BL). ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ii</i> ) For high-end end-user devices with GPU capabilities, an enhancement layer (EL) is added on top of the base layer comprising lightweight video super-resolution deep neural networks (DNNs) for each bitrate-resolution pair of the bitrate ladder. A content-aware video super-resolution approach leads to higher video quality, however, at the cost of bitrate overhead. To reduce the bitrate overhead for streaming content-aware video super-resolution DNNs, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepCABAC</i> , context-adaptive binary arithmetic coding for DNN compression, is used. Furthermore, the similarity among ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> ) segments within a scene and ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ii</i> ) frames within a segment are used to reduce the training costs of DNNs. Experimental results show bitrate savings of 34% and 36% to maintain the same PSNR and VMAF, respectively, for GPU-available end-users, while the CPU-only users get the desired video content as usual.

Smart-DNN+: A Memory-efficient Neural Networks Compression Framework for the Model Inference

Deep Neural Networks (DNNs) have achieved remarkable success in various real-world applications. However, running a Deep Neural Network (DNN) typically requires hundreds of megabytes of memory footprints, making it challenging to deploy on resource-constrained platforms such as mobile devices and IoT. Although mainstream DNNs compression techniques such as pruning, distillation, and quantization can reduce the memory overhead of model parameters during DNN inference, they suffer from three limitations: (i) low model compression ratio for the lightweight DNN structures with little redundancy, (ii) potential degradation in model inference accuracy, and (iii) inadequate memory compression ratio is attributable to ignoring the layering property of DNN inference. To address these issues, we propose a lightweight memory-efficient DNN inference framework called Smart-DNN+, which significantly reduces the memory costs of DNN inference without degrading the model quality. Specifically, ① Smart-DNN+ applies a layerwise binary-quantizer with a remapping mechanism to greatly reduce the model size by quantizing the typical floating-point DNN weights of 32-bit to the 1-bit signs layer by layer. To maintain model quality, ② Smart-DNN+ employs a bucket-encoder to keep the compressed quantization error by encoding the multiple similar floating-point residuals into the same integer bucket IDs. When running the compressed DNN in the user’s device, ③ Smart-DNN+ utilizes a partially decompressing strategy to greatly reduce the required memory overhead by first loading the compressed DNNs in memory and then dynamically decompressing the required materials for model inference layer by layer. Experimental results on popular DNNs and datasets demonstrate that Smart-DNN+ achieves lower 0.17%–0.92% memory costs at lower runtime overheads compared with the states of the art without degrading the inference accuracy. Moreover, Smart-DNN+ potentially reduces the inference runtime up to 2.04× that of conventional DNN inference workflow.

Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression

Recently, the deep neural network (DNN) has become one of the most advanced and powerful methods used in classification tasks. However, the cost of DNN models is sometimes considerable due to the huge sets of parameters. Therefore, it is necessary to compress these models in order to reduce the parameters in weight matrices and decrease computational consumption, while maintaining the same level of accuracy. In this paper, in order to deal with the compression problem, we first combine the loss function and the compression cost function into a joint function, and optimize it as an optimization framework. Then we combine the CUR decomposition method with this joint optimization framework to obtain the low-rank approximation matrices. Finally, we narrow the gap between the weight matrices and the low-rank approximations to compress the DNN models on the image classification task. In this algorithm, we not only solve the optimal ranks by enumeration, but also obtain the compression result with low-rank characteristics iteratively. Experiments were carried out on three public datasets under classification tasks. Comparisons with baselines and current state-of-the-art results can conclude that our proposed low-rank joint optimization compression algorithm can achieve higher accuracy and compression ratios.

Transforming Large-Size to Lightweight Deep Neural Networks for IoT Applications

Deep Neural Networks (DNNs) have gained unprecedented popularity due to their high-order performance and automated feature extraction capability. This has encouraged researchers to incorporate DNN in different Internet of Things (IoT) applications in recent years. However, the colossal requirement of computation, energy, and storage of DNNs make their deployment prohibitive on resource-constrained IoT devices. Therefore, several compression techniques have been proposed in recent years to reduce the energy, storage, and computation requirements of the DNN. These techniques have utilized a different perspective for compressing a DNN with minimal accuracy compromise. This encourages us to comprehensively overview DNN compression techniques for the IoT. This article presents a comprehensive overview of existing literature on compressing the DNN that reduces energy consumption, storage, and computation requirements for IoT applications. We divide the existing approaches into five broad categories—network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous—based upon the mechanism incorporated for compressing the DNN. The article discusses the challenges associated with each category of DNN compression techniques and presents some prominent applications using IoT in conjunction with a compressed DNN. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.

Structured Dynamic Precision for Deep Neural Networks Quantization

Deep Neural Networks (DNNs) have achieved remarkable success in various Artificial Intelligence applications. Quantization is a critical step in DNNs compression and acceleration for deployment. To further boost DNN execution efficiency, many works explore to leverage the input-dependent redundancy with dynamic quantization for different regions. However, the sensitive regions in the feature map are irregularly distributed, which restricts the real speed up for existing accelerators. To this end, we propose an algorithm-architecture co-design, named Structured Dynamic Precision (SDP). Specifically, we propose a quantization scheme in which the high-order bit part and the low-order bit part of data can be masked independently. And a fixed number of term parts are dynamically selected for computation based on the importance of each term in the group. We also present a hardware design to enable the algorithm efficiently with small overheads, whose inference time mainly scales with the precision proportionally. Evaluation experiments on extensive networks demonstrate that compared to the state-of-the-art dynamic quantization accelerator DRQ, our SDP can achieve 29% performance gain and 51% energy reduction for the same level of model accuracy.

Automatic Compression of Neural Network with Deep Reinforcement Learning Based on Proximal Gradient Method

In recent years, the model compression technique is very effective for deep neural network compression. However, many existing model compression methods rely heavily on human experience to explore a compression strategy between network structure, speed, and accuracy, which is usually suboptimal and time-consuming. In this paper, we propose a framework for automatically compressing models through the actor–critic structured deep reinforcement learning (DRL) which interacts with each layer in the neural network, where the actor network determines the compression strategy and the critic network ensures the decision accuracy of the actor network through predicted values, thus improving the compression quality of the network. To enhance the prediction performance of the critic network, we impose the L1 norm regularizer on the weights of the critic network to obtain a distinct activation output feature on the representation, thus enhancing the prediction accuracy of the critic network. Moreover, to improve the decision performance of the actor network, we impose the L1 norm regularizer on the weights of the actor network to improve the decision accuracy of the actor network by removing the redundant weights in the actor network. Furthermore, to improve the training efficiency, we use the proximal gradient method to optimize the weights of the actor network and the critic network, which can obtain an effective weight solution and thus improve the compression performance. In the experiment, in MNIST datasets, the proposed method has only a 0.2% loss of accuracy when compressing more than 70% of neurons. Similarly, in CIFAR-10 datasets, the proposed method compresses more than 60% of neurons, with only 7.1% accuracy loss, which is superior to other existing methods. In terms of efficiency, the proposed method also cost the lowest time among the existing methods.

Evaluating Robustness to Noise and Compression of Deep Neural Networks for Keyword Spotting

Keyword Spotting (KWS) has been the subject of research in recent years given the increase of embedded systems for command recognition such as Alexa, Google Home, and Siri. Performance, model size, processing time, and robustness to noise are fundamental in these systems. Furthermore, applications in embedded systems demand computationally efficient models that can be implemented in current technology. In this work, an approach for keyword recognition is evaluated using three deep learning models namely LeNet-5, SqueezeNet, and EfficientNet-B0. We evaluate transfer learning, pruning and quantization strategies in training and test using noisy and clean speech signals. In addition, compression techniques such as pruning and quantization were assessed in terms of the size reduction of the model footprint and the accuracy obtained in each case. Using the Google’s Speech Commands dataset and additive babble noise signal, our keyword recognition approach achieves an accuracy of 94.6% using an unstructured pruning of 80% of the parameters of the original SqueezeNet network with a reduction of 70% in the model size.

Knowledge Distillation With Feature Self Attention

With the rapid development of deep learning technology, the size and performance of the network continuously grow, making network compression essential for commercial applications. In this paper, we propose a Feature Self Attention (FSA) module that extracts correlation information between the hidden features of a network and a new method for distilling the correlation features to compress the model. FSA does not require a special module or network to match features between the teacher model and the student model. By removing the multi-head structure and the repeated self-attention blocks in the existing self-attention mechanism, it minimizes the addition of parameters. Based on ResNet-18, 34, the added parameters are only 2.00M and the training speed is also the fastest in comparison to benchmark models. It was demonstrated through experiments that the use of interrelationship loss between features can be beneficial for training student models, indicating the importance of considering correlation information in deep neural network compression. And it was verified through training from scratch on the vanilla without the pre-trained weight of the student model.

Knowledge distillation: A novel approach for deep feature selection

High dimensional data in hyperspectral remote sensing leads to computational, analytical, and storage complexities. Dimensionality reduction serves as an efficient tool to remove redundant, irrelevant, and highly correlated features. Recently, deep learning approaches have received remarkable progress in hyperspectral data analysis. In this paper, a new end-to-end deep learning framework based on a teacher-student network inspired by knowledge distillation is proposed for deep feature selection. Initially, a complicated teacher deep neural network is employed on complex high dimensional data to learn its corresponding best low dimensional representation. Then, the knowledge from the network is transferred to a simple student network that performs feature selection. Hence, it eventually leads to deep neural network compression which is of prime concern in hyperspectral remote sensing. Limited studies have been carried out to explore the benefits of knowledge distillation on hyperspectral data. The proposed method could be employed to choose deep features for both supervised and unsupervised tasks. Experimental results reveal the performance of the proposed scheme using limited features. In comparison to 1D and simple autoencoder models, the 2D model based on convolutional autoencoder delivers greater classification accuracies, with a classification accuracy value of 96.15% for the Indian Pines dataset and 97.82% for the Pavia University dataset. A similar trend is reported with unsupervised learning as well. Furthermore, the proposed model has a low degree of sensitivity to parameter selection.

Deep neural networks compression: A comparative survey and choice recommendations

The state-of-the-art performance for several real-world problems is currently reached by deep and, in particular, convolutional neural networks (CNN). Such learning models exploit recent results in the field of deep learning, leading to highly performing, yet very large neural networks with typically millions to billions of parameters. As a result, such models are often redundant and excessively oversized, with a detrimental effect on the environment in terms of unnecessary energy consumption and a limitation to their deployment on low-resource devices. The necessity for compression techniques able to reduce the number of model parameters and their resource demand is thereby increasingly felt by the research community. In this paper we propose the first extensive comparison, to the best of our knowledge, of the main lossy and structure-preserving approaches to compress pre-trained CNNs, applicable in principle to any existing model. Our study is intended to provide a first and preliminary guidance to choose the most suitable compression technique when there is the need to reduce the occupancy of pre-trained models. Both convolutional and fully-connected layers are included in the analysis. Our experiments involved two pre-trained state-of-the-art CNNs (proposed to solve classification or regression problems) and five benchmarks, and gave rise to important insights about the applicability and performance of such techniques w.r.t.the type of layer to be compressed and the category of problem tackled.

FasterAI: A Lightweight Library for Neural Networks Compression

FasterAI is a PyTorch-based library, aiming to facilitate the use of deep neural network compression techniques, such as sparsification, pruning, knowledge distillation, or regularization. The library is built with the purpose of enabling quick implementation and experimentation. More particularly, compression techniques are leveraging callback systems of libraries, such as fastai and Pytorch Lightning to propose a user-friendly and high-level API. The main asset of FasterAI is its lightweight, yet powerful, simplicity of use. Indeed, because it has been developed in a very granular way, users can create thousands of unique experiments by using different combinations of parameters, with only a single line of additional code. This allows FasterAI to be suited for practical usage, as it contains the most common compression techniques available out-of-the-box, but also for research, as implementing a new compression technique usually boils down to writing a single line of code. In this paper, we propose an in-depth presentation of the different compression techniques available in FasterAI. As a proof of concept and to better grasp how the library is used, we present results achieved by applying each technique on a ResNet-18 architecture, trained on CALTECH-101.

Joint matrix decomposition for deep convolutional neural networks compression

Deep convolutional neural networks (CNNs) with a large number of parameters require intensive computational resources, and thus are hard to be deployed in resource-constrained platforms. Decomposition-based methods, therefore, have been utilized to compress CNNs in recent years. However, since the compression factor and performance are negatively correlated, the state-of-the-art works either suffer from severe performance degradation or have relatively low compression factors. To overcome this problem, we propose to compress CNNs and alleviate performance degradation via joint matrix decomposition, which is different from existing works that compressed layers separately. The idea is inspired by the fact that there are lots of repeated modules in CNNs. By projecting weights with the same structures into the same subspace, networks can be jointly compressed with larger ranks. In particular, three joint matrix decomposition schemes are developed, and the corresponding optimization approaches based on Singular Value Decomposition are proposed. Extensive experiments are conducted across three challenging compact CNNs for different benchmark data sets to demonstrate the superior performance of our proposed algorithms. As a result, our methods can compress the size of ResNet-34 by 22× with slighter accuracy degradation compared with several state-of-the-art methods.

DiverGet: a Search-Based Software Testing approach for Deep Neural Network Quantization assessment

Quantization is one of the most applied Deep Neural Network (DNN) compression strategies, when deploying a trained DNN model on an embedded system or a cell phone. This is owing to its simplicity and adaptability to a wide range of applications and circumstances, as opposed to specific Artificial Intelligence (AI) accelerators and compilers that are often designed only for certain specific hardware (e.g., Google Coral Edge TPU). With the growing demand for quantization, ensuring the reliability of this strategy is becoming a critical challenge. Traditional testing methods, which gather more and more genuine data for better assessment, are often not practical because of the large size of the input space and the high similarity between the original DNN and its quantized counterpart. As a result, advanced assessment strategies have become of paramount importance. In this paper, we present DiverGet, a search-based testing framework for quantization assessment. DiverGet defines a space of metamorphic relations that simulate naturally-occurring distortions on the inputs. Then, it optimally explores these relations to reveal the disagreements among DNNs of different arithmetic precision. We evaluate the performance of DiverGet on state-of-the-art DNNs applied to hyperspectral remote sensing images. We chose the remote sensing DNNs as they're being increasingly deployed at the edge (e.g., high-lift drones) in critical domains like climate change research and astronomy. Our results show that DiverGet successfully challenges the robustness of established quantization techniques against naturally-occurring shifted data, and outperforms its most recent concurrent, DiffChaser, with a success rate that is (on average) four times higher.

Realistic acceleration of neural networks with fine-grained tensor decomposition

As the modern deep neural networks (DNNs) have become more and more large-scale and expensive, the topic of DNN compression grows into a hot direction nowadays. Among variant compression methods, tensor decomposition seems to be the most promising and low-cost one because of its solid mathematical foundations and regular data structure. However, most of the existing tensor decompositions are not very good at accelerating DNNs, because there are always necessary transpositions on tensor modes to make the input data calculate with the decomposed factor tensors correctly, and transposition will bring extra memory and time cost for the realistic system without doubt. In this paper, we select a relatively novel Kronecker CANDECOMP/PARAFAC (KCP) tensor decomposition which has fine-grained factor tensors, and propose the transposition-free algorithm to calculate the contractions between the input data and the neural weight in KCP format. The theoretically analysis of computation complexity indicates that the proposed method is much more efficient than the existing algorithms. We further prove that the training complexity of KCP-DNN based on the proposed transposition-free algorithm can also be faster than the traditional ones, and make a comprehensive comparison of space and computation complexity including training and inference stages to show the superiority of our method. As a series of related works pay more attention to the recurrent neural networks (RNNs), we follow these existing practices and focus on the KCP-RNN to make a comprehensive comparison with them, and the experimental results show our KCP-RNN with transposition-free algorithm has systematically advantages including accuracy, space complexity, computation complexity, and realistic running time. Besides, some advanced characteristics of KCP-DNN such as collocation of ranks, have also been discussed.

The Compression Techniques Applied on Deep Learning Model

In recent years, the penetration rate of smartphones has gradually completed, artificial intelligence is the cutting-edge technology that can trigger disruptive changes. Deep learning neural networks are also starting to appear on mobile devices. In order to obtain better performance, more complex networks need to be designed, and the corresponding models, computation and storage space are increasing, however, the challenges of resource allocation and energy consumption still exist in mobile. The techniques for compressing deep learning models are quite important, and this paper studies a series of related literatures. This paper reviews deep learning-based deep neural network compression techniques and introduces the key operational points of knowledge extraction and network model on the learning performance of Resolution-Aware Knowledge Distillation. In this paper, a low-rank decomposition algorithm is evaluated based on sparse parameters and rank using the extended BIC for tuning parameter selection. This paper discusses the reduction of redundancy in the fully connected and constitutive layers of the training network model by pruning strategies.Moreover, this paper presents the quantization techniques and a neural network that quantifies weights and activations by applying differentiable nonlinear functions.

Multi-Knowledge Aggregation and Transfer for Semantic Segmentation

As a popular deep neural networks (DNN) compression technique, knowledge distillation (KD) has attracted increasing attentions recently. Existing KD methods usually utilize one kind of knowledge in an intermediate layer of DNN for classification tasks to transfer useful information from cumbersome teacher networks to compact student networks. However, this paradigm is not very suitable for semantic segmentation, a comprehensive vision task based on both pixel-level and contextual information, since it cannot provide rich information for distillation. In this paper, we propose a novel multi-knowledge aggregation and transfer (MKAT) framework to comprehensively distill knowledge within an intermediate layer for semantic segmentation. Specifically, the proposed framework consists of three parts: Independent Transformers and Encoders module (ITE), Auxiliary Prediction Branch (APB), and Mutual Label Calibration (MLC) mechanism, which can take advantage of abundant knowledge from intermediate features. To demonstrate the effectiveness of our proposed approach, we conduct extensive experiments on three segmentation datasets: Pascal VOC, Cityscapes, and CamVid, showing that MKAT outperforms the other KD methods.

Compression of deep neural networks: bridging the gap between conventional-based pruning and evolutionary approach

Recently, many studies have been carried out on model compression to handle the high computational cost and high memory footprint brought by the implementation of deep neural networks. In this paper, model compression of convolutional neural networks is constructed as a multiobjective optimization problem with two conflicting objectives, reducing the model size and improving the performance. A novel structured pruning method called Conventional-based and Evolutionary Approaches Guided Multiobjective Pruning (CEA-MOP) is proposed to address this problem, where the power of conventional pruning methods is effectively exploited for the evolutionary process. A delicate balance in pruning rate and model accuracy has been automated achieved by a multiobjective optimization evolutionary model. First, an ensemble framework integrates pruning metrics to establish a codebook for further evolutionary operations. Then, an efficient coding method is developed to shorten the length of chromosome, thus ensuring its superior scalability. Finally, sensitivity analysis is automatically carried out to determine the upper bound of pruning rate for each layer. Notably, on CIFAR-10, CEA-MOP reduces more than 50% FLOPs on ResNet-110 and improves the relative accuracy. Moreover, on ImageNet, CEA-MOP reduces more than 50% FLOPs on ResNet-101 with negligible top-1 accuracy drop.