Abstract
Neural Networks (NN), although successfully applied to several Artificial Intelligence tasks, are often unnecessarily over-parametrized. In edge/fog computing, this might make their training prohibitive on resource-constrained devices, contrasting with the current trend of decentralizing intelligence from remote data centres to local constrained devices. Therefore, we investigate the problem of training effective NN models on constrained devices having a fixed, potentially small, memory budget. We target techniques that are both resource-efficient and performance effective while enabling significant network compression. Our Dynamic Hard Pruning (DynHP) technique incrementally prunes the network during training, identifying neurons that marginally contribute to the model accuracy. DynHP enables a tunable size reduction of the final neural network and reduces the NN memory occupancy during training. Freed memory is reused by a dynamic batch sizing approach to counterbalance the accuracy degradation caused by the hard pruning strategy, improving its convergence and effectiveness. We assess the performance of DynHP through reproducible experiments on three public datasets, comparing them against reference competitors. Results show that DynHP compresses a NN up to 10 times without significant performance drops (up to 3.5% additional error w.r.t. the competitors), reducing up to 80% the training memory occupancy.
Highlights
In the last years, AI solutions have been successfully adopted in a variety of different tasks
The available memory is reused to minimise the accuracy degradation caused by the hard pruning strategy by adaptively adjusting the size of the data provided to the neural network to improve its convergence and final effectiveness
We proposed Dynamic Hard Pruning (DynHP), a new resource-efficient Neural networks (NN) learning technique that achieves performances comparable to conventional neural network training algorithms while enabling significant levels of network compression
Summary
AI solutions have been successfully adopted in a variety of different tasks. Neural networks (NN) are among the most successful technologies that achieve state-of-the-art performance in several application fields, including image recognition, computer vision, natural language processing, and speech recognition. It has been proven that NNs may suffer over parametrization [12, 27, 34] so that they can be pruned significantly without any loss of accuracy. They can over-fit and even memorize random patterns in the data [36] if not properly regularized. We first introduce the notation used in the rest of the paper, we discuss the state of the art techniques for soft pruning neural networks (during training) that inspired our solution.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
