A multi-perspective revisit to the optimization methods of Neural Architecture Search and Hyper-parameter optimization for non-federated and federated learning environments

Abstract

The use of Neural Architecture Search (NAS) techniques is becoming more widespread. NAS methods have been essential for automating and accelerating the laborious and erroneous process of designing and tuning new Deep Learning (DL) approaches. The past few years have seen increasing number of research works on NAS and associated hyper-parameter optimization (HPO). They have arguably had the biggest influence on classification and object detection tasks, where they have produced state-of-the-art results. Despite the notable success to date, it is still difficult and not always practicable to apply NAS to real-world situations. In general, the structure of Convolution Neural Networks (CNNs) is too sophisticated to be used in platforms with limited resources, such as IoT, mobile, and embedded systems. In this paper, we revisit several state-of-the-art NAS and HPO methods from a multi-perspective view both in non-federated and distributed federated learning environments. Earlier attempts to critically analyze the components of NAS or HPO are only limited to non-federated environment, where NAS and HPO related computations need to be performed on a single CPU/ multiple GPUs and where all the data was readily accessible for performing the task of learning. In this article, we extend the discussion and critically analyze the important components of NAS and HPO methods in federated learning environment, where computations need to be performed on multiple computational units in federation and where only local data can be used by a single computational unit. Due to unavailability of entire data in federated learning environment, the earlier NAS methods (i.e., non-federated NAS methods) and the earlier HPO methods (i.e., non-federated HPO methods) cannot be used directly without modifications. We discuss these modifications made by researchers in non-federated NAS and HPO methods to be used in federated learning environment and provide discussion for future avenues.