Abstract
Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.
Highlights
F EDERATED learning (FL) [1]–[5] is a distributed training framework, which allows multiple clients to jointly train a single deep learning model on their combined data in a communication-efficientManuscript received October 2, 2019; revised February 25, 2020 and June 12, 2020; accepted August 8, 2020
We present clustered FL (CFL), a novel algorithmic framework that is able to deal with federated multitask learning (FMTL) problems that satisfy Assumption 2
We presented CFL, a framework for FMTL that can improve any existing FL framework by enabling the participating clients to learn more specialized models
Summary
F EDERATED learning (FL) [1]–[5] is a distributed training framework, which allows multiple clients (typically, mobile or the IoT devices) to jointly train a single deep learning model on their combined data in a communication-efficient. Presence of Adversaries: A special case of incongruence is given if a subset of the client population behaves in an adversarial manner In this scenario, the adversaries could deliberately alter their local data distribution in order to encode arbitrary behavior into the jointly trained model, affecting the model decisions on all other clients and causing potential harm [6]. 2) We derive a computationally efficient tool based on the cosine similarity between the clients’ gradient updates that provably allows us to infer whether two members of the client population have different data generating distributions, making it possible for us to infer the clustering structure C (see Section II-A). 5) We investigate several practical concerns (varying client populations, training with formal privacy guarantees, and communication of weight-updates instead of gradients) and demonstrate that CFL can seamlessly adapt to these conditions/constraints (see Section IV).
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: IEEE Transactions on Neural Networks and Learning Systems
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
