Momentum recursive DARTS

Abstract

DARTS has emerged as a popular method for neural architecture search (NAS) owing to its efficiency and simplicity. It employs gradient-based bi-level optimization to iteratively optimize the upper-level architecture parameters and lower-level super-network weights. The key challenge in DARTS is the accurate estimation of gradients for two-level object functions, leading to significant errors in gradient approximation. To address this issue, we propose a new approach, MR-DARTS, that incorporates a momentum term and a recursive scheme to improve gradient estimation. Specifically, we leverage historical information by using a running average of past observed gradients to enhance the quality of current gradient estimation in both upper-level and lower-level functions. Our theoretical analysis shows that the variance of our estimated gradient decreases with each iteration. By utilizing momentum and a recursive scheme, MR-DARTS effectively controls the error in stochastic gradient updates that result from inaccurate gradient estimation. Furthermore, we utilize the Neumann series approximation and Hessian Vector Product scheme to reduce computational requirements and memory usage. We evaluate our proposed method on several benchmarks and demonstrate its effectiveness through comprehensive experiments.