DARTS-PT-CORE: Collaborative and Regularized Perturbation-based Architecture Selection for differentiable NAS

Abstract

DARTS-PT is a well-known differentiable NAS method that measures the operation strength through its contribution to the supernet performance, extracting architecture from the underlying supernet. However, persistent issues of degraded architecture in DARTS-PT have been identified in recent studies. In response, we undertake a comprehensive analysis of this performance degradation issue and identify two primary contributing factors: the unfavorable competition among correlated operations during the operation selection process and the unfair advantage of parameter-free operations within DARTS-PT supernet. Building upon these findings, we propose DARTS-PT-CORE, a novel architecture selection algorithm that incorporates a collaborative operation competition mechanism and a regularization technique in the perturbation-based architecture selection approach. Our method aims to mitigate the negative effects of competition among correlated operations, yielding more reliable operation contribution scores. Furthermore, our regularization technique addresses the unfair advantage of parameter-free operations, facilitating a more balanced architecture selection process. Extensive experiments conducted on various datasets and search spaces indicate that DARTS-PT-CORE outperforms other state of-the-art methods. Specifically, in the DARTS search space, DARTS-PT-CORE achieves 2.43% test error on CIFAR10 and 16.23% test error on CIFAR100, while the search time is less than 0.8 GPU days. When transferring to ImageNet, DARTS-PT-CORE achieves 24.97% top-1 error. Such results underscore the effectiveness of our method in enhancing the reliability and balance of architecture selection in differentiable NAS.