Genetic CFL: Hyperparameter Optimization in Clustered Federated Learning.

Shaashwat Agrawal,Thippa Reddy Gadekallu,Mamoun Alazab,Quoc-Viet Pham,Sagnik Sarkar,Praveen Kumar Reddy Maddikunta,Rodolfo E Haber

doi:10.1155/2021/7156420

Abstract

Federated learning (FL) is a distributed model for deep learning that integrates client-server architecture, edge computing, and real-time intelligence. FL has the capability of revolutionizing machine learning (ML) but lacks in the practicality of implementation due to technological limitations, communication overhead, non-IID (independent and identically distributed) data, and privacy concerns. Training a ML model over heterogeneous non-IID data highly degrades the convergence rate and performance. The existing traditional and clustered FL algorithms exhibit two main limitations, including inefficient client training and static hyperparameter utilization. To overcome these limitations, we propose a novel hybrid algorithm, namely, genetic clustered FL (Genetic CFL), that clusters edge devices based on the training hyperparameters and genetically modifies the parameters clusterwise. Then, we introduce an algorithm that drastically increases the individual cluster accuracy by integrating the density-based clustering and genetic hyperparameter optimization. The results are bench-marked using MNIST handwritten digit dataset and the CIFAR-10 dataset. The proposed genetic CFL shows significant improvements and works well with realistic cases of non-IID and ambiguous data. An accuracy of 99.79% is observed in the MNIST dataset and 76.88% in CIFAR-10 dataset with only 10 training rounds.

Highlights

Federated learning (FL) [1, 2] has risen as a groundbreaking subdomain of machine learning (ML) that enables Internet of ings (IoT) devices to contribute their real-time data and processing to train ML models
E limited feasibility of ML in industrial and IoT applications is overturned by the introduction of FL
We propose three novel contributions to lessen the empirical risk in FL, as shown in Figure 1: (i) Clustering of clients solely based on model hyperparameters to increase the learning efficiency per unit training of the model (ii) Implementation of density-based clustering, i.e., DBSCAN, on the hyperparameters for proper analysis of devices properties (iii) Introduction of genetic evolution of hyperparameters per cluster for finer tuning of individual device models and better aggregation

Summary

Introduction

Federated learning (FL) [1, 2] has risen as a groundbreaking subdomain of machine learning (ML) that enables Internet of ings (IoT) devices to contribute their real-time data and processing to train ML models. Computational power and properties of data (ambiguity, size, and complexity) vary drastically, and diversely trained client models are hard to aggregate. In a realistic scenario of thousands of edge devices, the updated global model may not converge at all Existing aggregating algorithms such as FedAvg and FedMA [14] focus more on integration of weights of the local models. We introduce a new algorithm, namely, Genetic CFL, that clusters hyperparameters of a model to drastically increase the adaptability of FL in realistic environments. Hyperparameters such as batch size and learning rate are core features of any MFL model.

Related Work

Genetic CFL Architecture

Findings

Conclusion