Surrogate-Assisted Particle Swarm Optimization for Evolving Variable-Length Transferable Blocks for Image Classification.

Abstract

Deep convolutional neural networks (CNNs) have demonstrated promising performance on image classification tasks, but the manual design process becomes more and more complex due to the fast depth growth and the increasingly complex topologies of CNNs. As a result, neural architecture search (NAS) has emerged to automatically design CNNs that outperform handcrafted counterparts. However, the computational cost is immense, e.g., 22400 GPU-days and 2000 GPU-days for two outstanding NAS works named NAS and NASNet, respectively, which motivates this work. A new effective and efficient surrogate-assisted particle swarm optimization (PSO) algorithm is proposed to automatically evolve CNNs. This is achieved by proposing a novel surrogate model, a new method of creating a surrogate data set, and a new encoding strategy to encode variable-length blocks of CNNs, all of which are integrated into a PSO algorithm to form the proposed method. The proposed method shows its effectiveness by achieving the competitive error rates of 3.49% on the CIFAR-10 data set, 18.49% on the CIFAR-100 data set, and 1.82% on the SVHN data set. The CNN blocks are efficiently learned by the proposed method from CIFAR-10 within 3 GPU-days due to the acceleration achieved by the surrogate model and the surrogate data set to avoid the training of 80.1% of CNN blocks represented by the particles. Without any further search, the evolved blocks from CIFAR-10 can be successfully transferred to CIFAR-100, SVHN, and ImageNet, which exhibits the transferability of the block learned by the proposed method.