Scalable NAS with factorizable architectural parameters

Abstract

Neural Architecture Search (NAS) replaces manually designed networks with automatically searched networks and has become a hot topic in machine learning and computer vision. One key factor of NAS, scaling up the search space by adding operators, could help bring about more possibilities for effective architectures. However, existing works require high search costs and are prone to make confused selections caused by competition among similar operators. This paper presents a scalable NAS that utilizes a factorized method, which improves existing art in the following aspects. (1) Our work explores a broader search space. We construct an ample search space through the cartesian product of activation operators and regular operators. (2) Our work experiences a limited computation burden even though searching in a more extensive search space. We factorize a search space into two subspaces and adopt different architectural parameters to control corresponding subspaces. (3) Our work avoids competition among similarly combined operators (e.g., ReLU &sep-conv-3x3 and ReLU6 &sep-conv-3x3 and PReLU &sep-conv-3x3). That is because our architectural parameters are optimized sequentially. Experimental results show that our approach achieves SOTA on CIFAR10 (2.38% test error) and ImageNet (23.8% test error). Furthermore, the excellent performance of 35.8% (AP) and 55.7% (AP50) on COCO further proves the superiority of our factorized method.