Hierarchical federated learning with local model embedding

Abstract

Federated learning can synergize the local model training with private data samples from geo-distributed users. Nevertheless, the unification process of a comprehensive global model through periodical parameter sharing can be time-consuming at a high cost. On the one hand, the data samples are collected from users with diverse preferences, and the data distribution can be non-independent and identically distributed (non-IID). On the other hand, the consequent model trained locally needs to communicate with a remote parameter server periodically for parameter synchronization, which leads to overwhelming communication and synchronization overhead given heterogeneous device capacities and network conditions of end-users. Generally, a hierarchical system design with a clustered group is ideal for accommodating diversity. Actually, it is still challenging to maintain the relationship of the local model training without knowing the data samples in advance for privacy concerns. Therefore, we present hierarchical federated model embedding to formulate the relationship between local data distributions. Initially, the local models are embedded through the global shared dataset to obtain feature latent representation vectors. The cloud server groups the clients according to the vectors, making clients with similar data distribution train collaboratively in a same group. Then, these vectors are used to train the predictor on the cloud server, which is utilized for efficient group assignment when new clients join the system. Compared with the baseline, the accuracy of the group model can be improved by 1.22%∼5.63% and that of the global model can be improved by 3.97%∼14.25% on different datasets.