Abstract
Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.
Highlights
A UTOMATIC machine learning (AutoML) has become a favorable solution for developing deep learning (DL) systems without any human efforts
LESSONS LEARNED This study has reviewed various Gradient descent (GD)-based neural architecture search (NAS) approaches from different directions, and here we summarize the lessons learned from this survey
Gradient descent is a better solution for architecture search in NAS approaches and ignoring it will increase architecture search cost in terms of GPU days
Summary
A UTOMATIC machine learning (AutoML) has become a favorable solution for developing deep learning (DL) systems without any human efforts. The model generation stage is either created by machine learning experts or by an automatic design process. The search space construction stage explores a large set of possible network architectures that can match or outperform expert-designed architectures. As NAS has been recognized as the core technology of neural architecture designing in next-generation, researchers have focused on extending their knowledge to automatic architecture design processes Along this line, Elsken et al [9] presented a survey on NAS, where they have addressed the elementary ideas of NAS, different search approaches and performance estimation strategies, and future directions of NAS. We present an in-depth study that explores architecture optimization strategies to generate candidate architectures with good performance and helps readers obtain possible research ideas and further directions, which inspires us to write this survey article
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.