Action Command Encoding for Surrogate-Assisted Neural Architecture Search

Abstract

With the development of neural architecture search, the performance of deep neural networks has been considerably enhanced with less human expertise. While the existing work mainly focuses on the development of optimizers, the design of encoding scheme is still in its infancy. This article thus proposes a novel encoding scheme for neural architecture search, termed action command encoding (ACEncoding). Inspired by the gene expression process, ACEncoding defines several action commands to indicate the addition and clone of layers, connections, and local modules, where an architecture grows from empty according to multiple action commands. ACEncoding provides a compact and rich search space that can be explored by various optimizers efficiently. Furthermore, a surrogate-assisted performance evaluator is tailored for ACEncoding, termed sequence-to-rank (Seq2Rank). By integrating the Seq2Seq model with RankNet, Seq2Rank embeds the variable-length encoding of ACEncoding into a continuous space, and then predicts the rankings of architectures based on the continuous representation. In the experiments, ACEncoding brings improvement to neural architecture search with existing encoding schemes and Seq2Rank shows better accuracy than existing performance evaluators. The neural architectures obtained by ACEncoding and Seq2Rank have competitive test errors and complexities on image classification tasks, and also show high transferability between different data sets.