Personalized Federated Learning on long-tailed data via knowledge distillation and generated features

Abstract

Personalized Federated Learning (PFL) offers a novel paradigm for distributed learning, which aims to learn a personalized model for each client through collaborative training of all distributed clients in a privacy-preserving manner. However, the performance of personalized models is often compromised by data heterogeneity and the challenges of long-tailed distributions, both of which are common in real-world applications. In this paper, we explore the joint problem of data heterogeneity and long-tailed distribution in PFL and propose a corresponding solution called Personalized Federated Learning with Distillation and generated Features (PFLDF). Specifically, we employ a lightweight generator trained on the server to generate a balanced feature set for each client that can supplement local minority class information with global class information. This augmentation mechanism is a robust countermeasure against the adverse effects of data imbalance. Subsequently, we use knowledge distillation to transfer the knowledge of the global model to personalized models to improve their generalization performance. Extensive experimental results show the superiority of PFLDF compared to other state-of-the-art PFL methods with long-tailed data distribution.