Abstract
This study develops a federated learning (FL) framework overcoming largely incremental communication costs due to model sizes in typical frameworks without compromising model performance. To this end, based on the idea of leveraging an unlabeled open dataset, we propose a distillation-based semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models among mobile devices, instead of model parameter exchange employed by the typical frameworks. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size. The exchanged model outputs are used to label each sample of the open dataset, which creates an additionally labeled dataset. Based on the new dataset, local models are further trained, and model performance is enhanced owing to the data augmentation effect. We further highlight that in DS-FL, the heterogeneity of the devices' dataset leads to ambiguous of each data sample and lowing of the training convergence. To prevent this, we propose entropy reduction averaging, where the aggregated model outputs are intentionally sharpened. Moreover, extensive experiments show that DS-FL reduces communication costs up to 99% relative to those of the FL benchmark while achieving similar or higher classification accuracy.
Highlights
F EDERATED Learning (FL) [1], [2], [3], [4] is an emerging machine learning (ML) framework to perform datadriven analysis or decision making, leveraging privacysensitive data from mobile devices
This prohibits the use of large-sized models, when the mobile devices are connected to wireless networks while competing for limited radio resources, which can be a crucial bottleneck for building practical ML models
Motivated by the inconvenience mentioned above, we aim to answer the following question: How should an FL framework be designed scalable according to the model sizes in terms of communication efficiency while achieving model performance comparable to that of the benchmark FL designed in [4]? Concisely, our answer is leveraging an unlabeled open data shared among the clients to enhance the model performance of model output exchange methods
Summary
F EDERATED Learning (FL) [1], [2], [3], [4] is an emerging machine learning (ML) framework to perform datadriven analysis or decision making, leveraging privacysensitive data from mobile devices. In FL, mobile devices collaboratively train their local ML model through the periodical exchange and aggregation of ML model parameters or gradients at central servers rather than exchanging their raw data. The periodical model parameter exchange in typical FL entails communication overhead that scales up according to the model size. This prohibits the use of large-sized models, when the mobile devices are connected to wireless networks while competing for limited radio resources, which can be a crucial bottleneck for building practical ML models. Distillation” step can result in a similar model to the one trained in the previous “1. The models trained in FD exhibit similar performance to the local model training, which is lower than the benchmark 1, i.e., FL, with model parameter exchange
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
