DE-DARTS: Neural architecture search with dynamic exploration

Abstract

Neural architecture search (NAS) methods automatically find optimal neural networks without human assistance. Numerous algorithms for NAS have been studied to find architectures with gradient-based search. Differentiable architecture search (DARTS), one of the key papers of gradient-based search, dramatically reduced search cost, and showed outstanding performance through continuous relaxation and meta-learning based approximation. However, one of the issues with DARTS is that the gradient-based search process is biased due to the nested bi-level optimization structure, and the greedy behavior of the gradient descent. As a result, there is a problem that search spaces are limited to a limited set of architectures. To overcome the bias of the gradient-based search in network architecture search (NAS), we used a dynamic search method. This technique allows gradient-based search to have exploration. In this paper, we present a novel approach, namely, Dynamic-Exploration DARTS (DE-DARTS). For effective exploration, we use dynamic attention networks (DANs) in DE-DARTS, which change model architectures based on input data. As our DANs are activated early in the search, more diverse architectures are considered, depending on input data at the beginning of search. Our algorithm is evaluated in multiple image classification datasets including CIFAR-10, and ImageNet, and shows improved performance.