Convolutional Neural Networks Compression Research Articles

Real-Time CNN Training and Compression for Neural-Enhanced Adaptive Live Streaming.

We propose a real-time convolutional neural network (CNN) training and compression method for delivering high-quality live video even in a poor network environment. The server delivers a low-resolution video segment along with the corresponding CNN for super resolution (SR), after which the client applies the CNN to the segment in order to recover high-resolution video frames. To generate a trained CNN corresponding to a video segment in real-time, our method rapidly increases the training accuracy by promoting the overfitting property of the CNN while also using curriculum-based training. In addition, assuming that the pretrained CNN is already downloaded on the client side, we transfer only residual values between the updated and pretrained CNN parameters. These values can be quantized with low bits in real time while minimizing the amount of loss, as the distribution range is significantly narrower than that of the updated CNN. Quantitatively, our neural-enhanced adaptive live streaming pipeline (NEALS) achieves higher SR accuracy and a lower CNN compression loss rate within a constrained training time compared to the state-of-the-art CNN training and compression method. NEALS achieves 15 to 48% higher quality of the user experience compared to state-of-the-art neural-enhanced live streaming systems.

High Performance CNN Accelerators based on Hardware and Algorithm Co-Optimization

The efficacy of convolutional neural networks (CNNs) has led to their extensive application in picture recognition and categorization. Nevertheless, embedded systems face challenges when it comes to storing the massive volumes of weight data required by CNNs in their on-chip memory. Pruning a convolutional neural network (CNN) model can reduce its size with no impact on accuracy; however, when used with a parallel architecture, the resulting model will be slower to execute. A CNN compression method that is hardware-centric is introduced in this paper. A model of a deep neural network (DNN) will include "pruning layers (P-layers)" and "no-pruning layers (NP-layers)" attached to it. For effective and parallel processing, an NP-layer employs a regular distribution for its weights. Pruning causes a P-layer to have an irregular form, yet the high compression ratio it yields makes the shape undesirable. Using uniform and gradual quantization methods, we can achieve a processing efficiency/compression ratio trade-off with a small loss of precision. Integrating multiple parallel finite impulse response (FIR) filters into an NP-layer distributed convolutional architecture, we further improve the regular model. The P-layer irregular sparse model is suggested to use an ADF-based processing element. The suggested compression strategy and hardware architecture can be utilised by a hardware/algorithm co-optimization (HACO) method to run an NP→P hybrid compressed CNN model on FPGAs. The VGG-16 network achieves a compression ratio of 27.5× with a top-5 accuracy loss of 0.44% by using a hardware accelerator on a single FPGA chip without off-chip memory. A result of 83.0 FPS, or 1.8 times quicker, was achieved when the compressed VGG-16 model was implemented on a Xilinx VCU118 evaluation board for image applications. Index Terms - CNN, Accelerators, Co-Optimization.

Global Instance Relation Distillation for convolutional neural network compression

Global Instance Relation Distillation for convolutional neural network compression

Efficient Framework to Manipulate Data Compression and Classification of Power Quality Disturbances for Distributed Power System

There is some risk of power quality disturbances at many stages of production, transformation, distribution, and energy consumption. The cornerstone for dealing with power quality problems is the characterization of power quality disturbances (PQDs). However, past research has focused on a narrow topic: noise disruption, overfitting, and training time. A new strategy is suggested to address this problem that combines efficient one-dimensional dataset compression with the convolutional neural network (CNN) classification algorithm. First, three types of compression algorithms: wavelet transform, autoencoder, and CNN, are proposed to be evaluated. According to the IEEE-1159 standard, the synthetic dataset was built with fourteen different PQD types. Furthermore, the PQD classification procedure integrated compressed data with the CNN classification algorithm. Finally, the suggested method demonstrates that combining CNN compression and classification methods can efficiently recognize PQDs. Even in noisy environments, PQD signal processing achieved up to 98.25% accuracy and managed the overfitting.

Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian Approach

Due to deep neural networks (DNNs) a large number of parameters, DNNs increase the demand for computing and storage during training, reasoning and deployment, especially when DNNs stack deeper and wider. Tensor decomposition can not only compress DNN models but also reduce parameters and storage requirements while maintaining high accuracy and performance. About tensor ring (TR) decomposition of tensor decomposition, there are two problems: (1) The practice of setting the TR rank to be equal in TR decomposition results in an unreasonable rank configuration. (2) The training time of selecting rank through iterative processes is time-consuming. To address the two problems, a TR network compression method by Variational Bayesian (TR-VB) is proposed based on the Global Analytic Solution of Empirical Variational Bayesian Matrix Factorization (GAS of EVBMF). The method consists of three steps: (1) rank selection, (2) TR decomposition, and (3) fine-tuning to recover accumulated loss of accuracy. Experimental results show that, for a given network, TR-VB gives the best results in terms of Top-1 accuracy, parameters, and training time under different compression levels. Furthermore, TR-VB validated on CIFAR-10/100 public benchmarks achieves state-of-the-art performance.

Compression of Convolutional Neural Networks With Divergent Representation of Filters.

Convolutional neural networks (CNNs) have made remarkable achievements in many tasks. However, most of them are hardly applied to embedded systems directly because of the requirement of huge memory space and computing power. In this article, we propose a pruning framework, namely, FiltDivNet, to accelerate and compress CNN models for their applicability to small or portable devices. The correlations among filters are taken into account and measured by the goodness of fit. On this basis, a hybrid-cluster pruning strategy is designed with dynamic pruning ratios for different clusters in CNN models. It aims at representing its filters in their diversity by removing redundant ones cluster by cluster. In addition, a new loss function with adaptive sparsity constraints is introduced for the retraining and fine-tuning in the FiltDivNet. Finally, some comparative experiments based on classical CNN models are carried out to demonstrate its effectiveness in compression performance and its adaptability with different CNN architectures.

Deep Convolutional Neural Network Compression based on the Intrinsic Dimension of the Training Data

Selecting the optimal deep learning architecture for a particular task and dataset remains an ongoing challenge. Typically, this decision-making process involves exhaustive searches for neural network architectures or multi-phase optimization, which includes initial training, compression or pruning, and fine-tuning steps. In this study, we introduce an approach utilizing a deep reinforcement learning-based agent to dynamically compress a deep convolutional neural network throughout its training process. We integrate the concept of the intrinsic dimension of the training data to provide the agent with insights into the task's complexity. The agent employs two distinct ranking criteria, L1-norm-based and attention-based measures, to selectively prune filters from each layer as it determines necessary. In the experiments, we used the CIFAR-10 dataset and its subsets (2-class and 5-class subsets) to model the task complexity and showed that the agent learns different policies depending on the intrinsic dimension. The agent, on average, pruned off 78.48%, 77.9%, and 83.12% filters from all the layers of VGG-16 network for CIFAR-10 full, 5-class, and 2-class subsets respectively.

Maximum output discrepancy computation for convolutional neural network compression

Network compression methods minimize the number of network parameters and computation costs while maintaining desired network performance. However, the safety assurance of many compression methods is based on a large amount of experimental data, whereas unforeseen incidents beyond the experiment data may result in unsafe consequences. In this work, we developed a discrepancy computation method for two convolutional neural networks by giving a concrete value to characterize the maximum output difference between the two networks after compression. Using Imagestar-based reachability analysis, we propose a novel method to merge the two networks to compute the difference. We illustrate reachability computation for each layer in the merged network, such as the convolution, max pooling, fully connected, and ReLU layers. We apply our method to a numerical example to prove its correctness. Furthermore, we implement our developed methods on the VGG16 model with the Quantization Aware Training (QAT) compression method; the results show that our approach can efficiently compute the accurate maximum output discrepancy between the original neural network and the compressed neural network.

Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity

Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy.

Hybrid multi-objective evolutionary model compression with convolutional neural networks

Deep learning has been utilized in the fields of image processing, natural language processing and speech recognition. For improving the structure of deep learning, how to compress Convolutional Neural Networks has become a major focus topic. The study proposes a sparsification-based Convolutional Neural Networks hybrid multi-objective evolutionary model compression method based on hybrid multi-objective evolution, in which the process improves the sparsity of the model by sparse pruning and introduces a time-domain residual module to provide more robust generalization performance, and finally the function of the research method is tested. The test outcomes indicate that the Convolutional Neural Networks model generated by the research method drops to the lowest value in the first 10 iterations; in the sparsity test, the total weight density of the model generated by the research method is about 36%; when the latency test is conducted, the latency of the research method is between 3 ms and 14 ms when the weight sparsity is 60%. The above results show that the research method has better model compression effect and can generate Convolutional Neural Networks models with both model accuracy and model size, which can provide new technical support for model compression of convolutional neural networks.

EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI

Convolutional neural networks (CNNs) have demonstrated encouraging results in image classification tasks. However, the prohibitive computational cost of CNNs hinders the deployment of CNNs onto resource-constrained embedded devices. To address this issue, we propose, a comprehensive compression framework to reduce the computational overhead of CNNs. In, we first introduce dynamic image cropping, where we design a lightweight foreground predictor to accurately crop the most informative foreground object of input images for inference, which avoids redundant computation on background regions. Subsequently, we present compound shrinking to collaboratively compress the three dimensions (depth, width, and resolution) of CNNs according to their contribution to accuracy and model computation. Dynamic image cropping and compound shrinking together constitute a multi-dimensional CNN compression framework, which is able to comprehensively reduce the computational redundancy in both input images and neural network architectures, thereby improving the inference efficiency of CNNs. Further, we present a dynamic inference framework to efficiently process input images with different recognition difficulties, where we cascade multiple models with different complexities from our compression framework and dynamically adopt different models for different input images, which further compresses the computational redundancy and improves the inference efficiency of CNNs, facilitating the deployment of advanced CNNs onto embedded hardware. Experiments on ImageNet-1K demonstrate that reduces the computation of ResNet-50 by 48.8% while improving the top-1 accuracy by 0.8%. Meanwhile, we improve the accuracy by 4.1% with similar computation compared to HRank. the state-of-the-art compression framework. The source code and models are available at

A priori compression of convolutional neural networks for wave simulators

Convolutional neural networks are now seeing widespread use in a variety of fields, including image classification, facial and object recognition, medical imaging analysis, and many more. In addition, there are applications such as physics-informed simulators in which accurate forecasts in real time with a minimal lag are required. The present neural network designs include millions of parameters, which makes it difficult to install such complex models on devices that have limited memory. Compression techniques might be able to resolve these issues by decreasing the size of CNN models that are created by reducing the number of parameters that contribute to the complexity of the models. We propose a compressed tensor format of convolutional layer, a priori, before the training of the neural network. 3-way kernels or 2-way kernels in convolutional layers are replaced by one-way fiters. The overfitting phenomena will be reduced also. The time needed to make predictions or time required for training using the original Convolutional Neural Networks model would be cut significantly if there were fewer parameters to deal with. In this paper we present a method of a priori compressing convolutional neural networks for finite element (FE) predictions of physical data. Afterwards we validate our a priori compressed models on physical data from a FE model solving a 2D wave equation. We show that the proposed convolutional compression technique achieves equivalent performance in the prediction error as classical convolutional layers with fewer trainable parameters(around 20%) and lower memory footprint.

Kernel-wise difference minimization for convolutional neural network compression in metaverse.

Convolutional neural networks have achieved remarkable success in computer vision research. However, to further improve their performance, network models have become increasingly complex and require more memory and computational resources. As a result, model compression has become an essential area of research in recent years. In this study, we focus on the best-case scenario for Huffman coding, which involves data with lower entropy. Building on this concept, we formulate a compression with a filter-wise difference minimization problem and propose a novel algorithm to solve it. Our approach involves filter-level pruning, followed by minimizing the difference between filters. Additionally, we perform filter permutation to further enhance compression. Our proposed algorithm achieves a compression rate of 94× on Lenet-5 and 50× on VGG16. The results demonstrate the effectiveness of our method in significantly reducing the size of deep neural networks while maintaining a high level of accuracy. We believe that our approach holds great promise in advancing the field of model compression and can benefit various applications that require efficient neural network models. Overall, this study provides important insights and contributions toward addressing the challenges of model compression in deep neural networks.

Resource-Efficient Convolutional Networks: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques

Convolutional neural networks (CNNs) are used in our daily life, including self-driving cars, virtual assistants, social network services, healthcare services, and face recognition, among others. However, deep CNNs demand substantial compute resources during training and inference. The machine learning community has mainly focused on model-level optimizations such as architectural compression of CNNs, whereas the system community has focused on implementation-level optimization. In between, various arithmetic-level optimization techniques have been proposed in the arithmetic community. This article provides a survey on resource-efficient CNN techniques in terms of model-, arithmetic-, and implementation-level techniques, and identifies the research gaps for resource-efficient CNN techniques across the three different level techniques. Our survey clarifies the influence from higher- to lower-level techniques based on our resource efficiency metric definition and discusses the future trend for resource-efficient CNN research.

FSConv: Flexible and separable convolution for convolutional neural networks compression

Because of limited computation resources, convolutional neural networks (CNNs) are difficult to deploy on mobile devices. To overcome this issue, many methods have successively reduced parameters in CNNs with the idea of removing redundancy among feature maps. We observe similarities between feature maps at the same layer but not complete consistency. Intuitively, the difference between similar feature maps is an essential ingredient for the success of CNNs. Therefore, we propose a flexible and separable convolution (FSConv) in a different perspective to embrace redundancy while requiring less computation, which can implicitly cluster feature maps into different clusters without introducing similarity measurements. Our proposed model extracts intrinsic information from the representative part through ordinary convolution in each cluster and reveals tiny hidden details from the redundant part through groupwise/depthwise convolution. Experimental results demonstrate that FSConv-equipped networks always perform better than previous state-of-the-art CNNs compression algorithms. Code is available at https://github.com/Clarkxielf/FSConv-Flexible-and-Separable-Convolution-for-Convolutional-Neural-Networks-Compression.

Compressing convolutional neural networks with cheap convolutions and online distillation

Visual impairment assistance systems play a vital role in improving the standard of living for visually impaired people (VIP). With the development of deep learning technologies and assistive devices, many assistive technologies for VIP have achieved remarkable success in environmental perception and navigation. In particular, convolutional neural network (CNN)-based models have surpassed the level of human recognition and achieved a strong generalization ability. However, the large memory and computation consumption in CNNs have been one of the main barriers to deploying them into resource-limited systems for visual impairment assistance applications. To this end, most cheap convolutions (e.g., group convolution, depth-wise convolution, and shift convolution) have recently been used for memory and computation reduction but with a specific architecture design. Furthermore, it results in a low discriminability of the compressed networks by directly replacing the standard convolution with these cheap ones. In this paper, we propose to use knowledge distillation to improve the performance of compact student networks with cheap convolutions. In our case, the teacher is a network with the standard convolution, while the student is a simple transformation of the teacher architecture without complicated redesigning. In particular, we introduce a novel online distillation method, which online constructs the teacher network without pre-training and conducts mutual learning between the teacher and student network, to improve the performance of the student model. Extensive experiments demonstrate that the proposed approach achieves superior performance to simultaneously reduce memory and computation overhead of cutting-edge CNNs on different datasets, including CIFAR-10/100 and ImageNet ILSVRC 2012, compared to the previous CNN compression and acceleration methods. The codes are publicly available at https://github.com/EthanZhangYC/OD-cheap-convolution.

E-UPQ: Energy-Aware Unified Pruning-Quantization Framework for CIM Architecture

The wide adoption of convolutional neural networks (CNNs) in many applications has given rise to unrelenting computational demand and memory requirements. Computing-in-Memory (CIM) architecture has demonstrated great potential to break the memory wall and effectively execute CNN workloads. Ongoing research focuses on pruning or quantizing CNNs to achieve higher efficiency on CIM. However, prior works preclude the possibility of integrating both techniques in the same framework. On the other hand, directly incorporating energy estimation during the model compression process has not been well explored in the literature. In this paper, we present an Energy-aware Unified Pruning-Q uantization (E-UPQ) mechanism, a novel framework for automated compression (pruning + quantization) of CNNs while considering the energy-accuracy trade-off. Specifically, E-UPQ interweaves pruning and quantization seamlessly by viewing pruning as a special case of “0-bit” quantization during the mixed-precision search. In addition, E-UPQ introduces a set of trainable parameters to incorporate energy information during the compression process, closing the gap between compression policy and energy optimization. Experimental results evaluated on DNN+NeuroSim show that E-UPQ reduces energy consumption by up to 79.3% and 66.6% for VGG-16 and ResNet-18, respectively, compared with the state-of-the-art work, while achieving similar accuracy on CIFAR-100. Layer-wise analysis and ablation studies are provided to validate the effectiveness of the E-UPQ. We also present the corresponding CIM architecture to support the proposed E-UPQ framework.

A Hardware-Friendly High-Precision CNN Pruning Method and Its FPGA Implementation

To address the problems of large storage requirements, computational pressure, untimely data supply of off-chip memory, and low computational efficiency during hardware deployment due to the large number of convolutional neural network (CNN) parameters, we developed an innovative hardware-friendly CNN pruning method called KRP, which prunes the convolutional kernel on a row scale. A new retraining method based on LR tracking was used to obtain a CNN model with both a high pruning rate and accuracy. Furthermore, we designed a high-performance convolutional computation module on the FPGA platform to help deploy KRP pruning models. The results of comparative experiments on CNNs such as VGG and ResNet showed that KRP has higher accuracy than most pruning methods. At the same time, the KRP method, together with the GSNQ quantization method developed in our previous study, forms a high-precision hardware-friendly network compression framework that can achieve "lossless" CNN compression with a 27× reduction in network model storage. The results of the comparative experiments on the FPGA showed that the KRP pruning method not only requires much less storage space, but also helps to reduce the on-chip hardware resource consumption by more than half and effectively improves the parallelism of the model in FPGAs with a strong hardware-friendly feature. This study provides more ideas for the application of CNNs in the field of edge computing.

Compressing convolutional neural networks with hierarchical Tucker-2 decomposition

Convolutional neural networks (CNNs) play a crucial role and achieve top results in computer vision tasks but at the cost of high computational cost and storage complexity. One way to solve this problem is the approximation of the convolution kernel using tensor decomposition methods. In this way, the original kernel is replaced with a sequence of kernels in a lower-dimensional space. This study proposes a novel CNN compression technique based on the hierarchical Tucker-2 (HT-2) tensor decomposition and makes an important contribution to the field of neural network compression based on low-rank approximations. We demonstrate the effectiveness of our approach on many CNN architectures on CIFAR-10 and ImageNet datasets. The obtained results show a significant reduction in parameters and FLOPS with a minor drop in classification accuracy. Compared to different state-of-the-art compression methods, including pruning and matrix/tensor decomposition, the HT-2, as a new alternative, outperforms most of the cited methods. The implementation of the proposed approach is very straightforward and can be easily coded in every deep learning library.

Joint matrix decomposition for deep convolutional neural networks compression

Deep convolutional neural networks (CNNs) with a large number of parameters require intensive computational resources, and thus are hard to be deployed in resource-constrained platforms. Decomposition-based methods, therefore, have been utilized to compress CNNs in recent years. However, since the compression factor and performance are negatively correlated, the state-of-the-art works either suffer from severe performance degradation or have relatively low compression factors. To overcome this problem, we propose to compress CNNs and alleviate performance degradation via joint matrix decomposition, which is different from existing works that compressed layers separately. The idea is inspired by the fact that there are lots of repeated modules in CNNs. By projecting weights with the same structures into the same subspace, networks can be jointly compressed with larger ranks. In particular, three joint matrix decomposition schemes are developed, and the corresponding optimization approaches based on Singular Value Decomposition are proposed. Extensive experiments are conducted across three challenging compact CNNs for different benchmark data sets to demonstrate the superior performance of our proposed algorithms. As a result, our methods can compress the size of ResNet-34 by 22× with slighter accuracy degradation compared with several state-of-the-art methods.