Abstract
Neural architecture search (NAS) is an automated method searching for the optimal network architecture by optimizing the combinations of edges and operations. For efficiency, recent differentiable architecture search methods adopt a one-shot network, containing all the candidate operations in each edge, instead of sampling and training individual architectures. However, a recent study doubts the effectiveness of differentiable methods by showing that random search can achieve comparable performance with differentiable methods using the same search cost. Therefore, there is a need to reduce the search cost even for previous differentiable methods. For more efficient differentiable architecture search, we propose a differentiable architecture search based on coordinate descent (DARTS-CD) that searches for optimal operation over only one sampled edge per training step. DARTS-CD is proposed based on the coordinate descent algorithm, which is an efficient learning method for resolving large-scale problems by updating only a subset of parameters. In DARTS-CD, one edge is randomly sampled, in which all the operations are performed, whereas only one operation is applied to the other edges. Weight update is also performed only at the sampled edge. By optimizing each edge separately, as in the coordinate descent that optimizes each coordinate individually, DARTS-CD converges much faster than DARTS while using the network architecture similar to that used for evaluation. We experimentally show that DARTS-CD performs comparably to the state-of-the-art efficient architecture search algorithms, with an extremely low search cost of 0.125 GPU days (1/12 of the search cost of DARTS) on CIFAR-10 and CIFAR-100. Furthermore, a warm-up regularization method is introduced to improve the exploration capability, which further enhances the performance.
Highlights
Over the past few years, deep neural networks have shown remarkable performance in many computer vision tasks such as object recognition [1]–[5], object detection [6]–[8], and semantic segmentation [9]–[11]
Based on state-of-the-art convolutional neural network (CNN) architectures [4], [5], the network is divided into multiple stages, each of which consists of repeated cell structures, called normal cells
differentiable architecture search (DARTS)-CD operates within only 3h on a single Titan X GPU, which allows a fast search of architectures for a new task
Summary
Over the past few years, deep neural networks have shown remarkable performance in many computer vision tasks such as object recognition [1]–[5], object detection [6]–[8], and semantic segmentation [9]–[11]. When a new task or a new dataset arises, human experts’ trial-and-error search methods are time-consuming. To resolve this issue, recent studies have attempted to automate the process of architecture search. State-of-the-art methods based on reinforcement learning [12] or evolutionary algorithms [13] still require thousands of GPU days for the search. They are not considered an appropriate tool for finding the optimal architecture for a new task or a new dataset
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
