Abstract
As a classic and well-performed deep convolutional neural network, DenseNet links every layer to each of its preceding layers via skip connections. However, the dense connectivity of the links leads to much redundance, consuming lots of computational resources. In this paper, to automatically prune redundant skip connections in DenseNet, we introduce a novel reinforcement learning method called automatic DenseNet sparsification (ADS). In ADS, we use adjacent matrix to represent dense connections in DenseNet, and design an agent using recurrent neural networks (RNNs) to sparsify the matrix, i. e. removing redundant skip connections in DenseNet. The validation accuracies of the sparsified DenseNets are used as rewards to update the agent, which promotes the agent to generate sparsified DenseNets with high performance. Extensive experiments demonstrate the effectiveness of ADS: The performance of the sparsified DenseNet surpasses not only the original DenseNet but related models; Moreover, the sparsified DenseNet has strong transferability when it is applied to new tasks. More importantly, ADS is very efficient. For the compression of a 40-layer DenseNet, it takes less than 1 day on a single GPU.
Highlights
In recent years, convolutional neural networks (CNNs) are widely applied in many pattern recognition and computer vision tasks, such as image classification, object tracking and image super-resolution [1]–[4]
1) EXPERIMENTAL RESULTS OBTAINED BY automatic DenseNet sparsification (ADS) The CIFAR-10 dataset consists of 60,000 images belonging to 10 classes
In this paper, we present a novel method called automatic DenseNet sparsification (ADS) to prune unimportant skip connections in DenseNet based on reinforcement learning
Summary
Convolutional neural networks (CNNs) are widely applied in many pattern recognition and computer vision tasks, such as image classification, object tracking and image super-resolution [1]–[4]. Following DenseNet, some deep learning approaches using dense connections have been proposed. Compared with some previous reinforcement learning algorithms, which validate the generated deep models by training them from scratch and require huge computational resources (e.g. 104 − 105 GPU hours) [14], [15], ADS is very efficient by employing the weight inheritance technique. It only takes less than one day on a single GPU for compressing a 40-layer DenseNet. The contributions of this paper can be summarized as follows:.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
