Communication Cost Reduction with Partial Structure in Federated Learning

Dongseok Kang,Chang Wook Ahn

doi:10.3390/electronics10172081

Abstract

Federated learning is a distributed learning algorithm designed to train a single server model on a server using different clients and their local data. To improve the performance of the server model, continuous communication with clients is required, and since the number of clients is very large, the algorithm must be designed in consideration of the cost required for communication. In this paper, we propose a method for distributing a model with a structure different from that of the server model, distributing a model suitable for clients with different data sizes, and training a server model using the reconstructed model trained by the client. In this way, the server model deploys only a subset of the sequential model, collects gradient updates, and selectively applies updates to the server model. This method of delivering the server model at a lower cost to clients who only need smaller models can reduce the communication cost of training server models compared to standard methods. An image classification model was designed to verify the effectiveness of the proposed method via three data distribution situations and two datasets, and it was confirmed that training was accomplished only with a cost 0.229 times smaller than the standard method.

Highlights

As we become better at using data to enable the use of more complex artificial neural network models, interest in data utilization has attracted the attention of all of society and is starting to be utilized on all devices [1,2]
In the most extreme case of data distribution difference (90, 10), FederatedPartial required 0.612 times less cost than the standard method, but it required 0.672 times less cost in the less localized data (10, 90). This is because FederatedPartial has a greater effect when the data distribution difference between clients is large, which can be interpreted as the relationship between the data and model size of each client
FederatedPartial distributes server models of various sizes to clients and shows that the training efficiency of the central server model can be increased by distributing a server model suitable for clients that need a smaller model to increase the client’s learning efficiency

Summary

Introduction

As we become better at using data to enable the use of more complex artificial neural network models, interest in data utilization has attracted the attention of all of society and is starting to be utilized on all devices [1,2]. As various devices begin to participate in training, research is being conducted on how to use a computing model in an environment where computing devices and data are distributed, as opposed to a centralized system with the typical deep learning models [3]. There has been a series of active studies on federated learning, which considers privacy more than existing distributed computing models [4,5,6]. In federated learning, the focus is on the privacy of data resources, the availability of distributed devices, and the cost of communicating with a central server.

Objectives

Methods

Findings

Discussion

Conclusion