Abstract
In recent years, deep neural networks (DNNs) have been widely applied in many areas, such as computer vision and pattern recognition. However, we observe that most of the DNNs include redundant layers. Hence, in this paper, we introduce a novel method named incremental layers resection (ILR) to resect the redundant layers in DNNs, while preserving their learning performances. ILR uses a multistage learning strategy to incrementally resect the inconsequential layers. In each stage, it preserves the data representations learned by the original network, while connecting the two nearby layers of each resected one. Particularly, based on a teacher-student knowledge transfer framework, we have designed the layer-level learning and overall learning procedures to enforce the resected network performing similarly with the original one. Extensive experiments demonstrate that, compared to the original networks, the compressed ones by ILR need only about half of the storage space and have higher inference speed. More importantly, they even deliver higher classification accuracy than the original networks.
Highlights
In recent years, deep neural networks (DNNs) have attracted much attention in the areas related to artificial intelligence, such as computer vision and pattern recognition
Deep convolutional neural networks (CNNs) have obtained great successes in the applications of image classification and object detection, since AlexNet [2] won the champion of the ImageNet Large Scale Visual Recognition Competition (ILSVRC) in 2012
2) Compared with previous knowledge distillation (KD) approaches, where the structures of the teacher and student networks may quite different, the student network inherits the structure of the teacher network and only inconsequential layers are removed during the incremental layers resection (ILR) learning process
Summary
Deep neural networks (DNNs) have attracted much attention in the areas related to artificial intelligence, such as computer vision and pattern recognition. In this paper, we propose a new network compression method called incremental layers resection (ILR), to remove the redundant layers in DNNs. ILR combines the ideas of weight pruning and KD. ILR combines the ideas of weight pruning and KD It removes the inconsequential layers, and transfers the knowledge of the original network to the compressed one. 1) Compared with previous weight pruning approaches, which mainly remove the redundant connections, ILR focuses on layer-level resection. 2) Compared with previous KD approaches, where the structures of the teacher and student networks may quite different, the student network inherits the structure of the teacher network and only inconsequential layers are removed during the ILR learning process. As the redundant layers are removed step by step, the original network is incrementally compressed, without any hurt to its performance.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
