Quality-Aware Distributed Computation and User Selection for Cost-Effective Federated Learning

Abstract

Federated learning (FL) allows machine learning (ML) models to be trained on distributed edge computing devices without the need of collecting data from a large number of users. In distributed stochastic gradient descent which is a typical method of FL, the quality of a local parameter update is measured by the variance of the update, determined by the data sampling size (a.k.a. mini-batch size) used to compute the update. In this paper, we study quality-aware distributed computation for FL, which controls the quality of users' local updates via their mini-batch sizes. We first characterize the dependency of learning accuracy bounds on the quality of users' local updates over the learning process. It reveals that the impacts of local updates' quality on learning accuracy increase with the number of rounds in the learning process. Based on these insights, we develop cost-effective dynamic distributed learning algorithms that adaptively select participating users and their mini-batch sizes, based on users' communication and computation costs. Our result shows that for the case of IID data, it is optimal to select a set of users with the lowest communication costs in each round, and select more users with a larger total mini-batch size in a later round. We evaluate the proposed algorithms using simulation results, which demonstrate their efficiency.