Improving Differentiable Neural Architecture Search with Sparse Connections and Model Pruning

Abstract

Neural networks have contributed to many breakthroughs across several disciplines. Their ease of use and scalability have motivated the development of many techniques in computer vision, natural language processing, audio analysis, etc. The neural network architecture plays a dominant role in its performance, and there have been many advances on designs and strategies for defining efficient neural networks. However, manually tuning neural architectures requires a significant amount of time and expert knowledge. To overcome the difficulty of manually setting up the architecture for a neural network, Neural Architecture Search (NAS) has gained popularity. NAS methods involve three general dimensions, namely search space, search strategies, and performance estimation strategies [1]. Different approaches vary in these dimensions.