Att-DARTS: Differentiable Neural Architecture Search for Attention

Abstract

Neural architecture search (NAS) is a promising method to automatically identify neural network architectures. Differentiable architecture search (DARTS) is method that significantly reduces search time and finds architectures that can achieve state-of-the-art performance. For computer vision tasks, DARTS searches convolutional neural networks (CNNs) via stacking convolution layers and pooling operations. Recent studies on neural architectures indicate that attention modules can improve the performances of CNNs by discarding information of no interest, while existing NAS methods have put little focus on it. In this study, we propose Att-DARTS, which searches attention modules as well as convolution and pooling operations simultaneously. In our experiments on CIFAR-10 and CIFAR-100 datasets, we demonstrate that Att-DARTS can find architectures that achieve lower classification error rates and require fewer parameters compared to those found by DARTS.