DKD-pFed: A novel framework for personalized federated learning via decoupling knowledge distillation and feature decorrelation

Abstract

Listen

Translate article

Personalized federated learning (PFL) aims to collaboratively train models for clients with highly heterogeneous data distributions. Although Knowledge Distillation (KD) facilitates personalized learning by transferring knowledge from the teacher to the student, it still faces the following challenges. First, the student model cannot fully learn the knowledge that the teacher model intends to transmit. Second, the distillation efficiency depends on the initial performance of the student model, which limits the overall model performance. To address these issues, we propose DKD-pFed, a novel personalized federated learning framework that decouples knowledge distillation and incorporates feature decorrelation. The main idea is to separate the logit distillation into two parts, enabling them to function more efficiently and flexibly. Feature decorrelation is employed to enhance the performance of the student model by mitigating the issue of performance degradation caused by feature dimension collapse in client models under highly heterogeneous data conditions. Extensive experiments on the CIFAR-10, CIFAR-100, and FMNIST datasets demonstrate that our DKD-pFed achieves average improvements of 16.49%, 17.76%, and 8.86%, respectively, compared with other state-of-the-art models when the Dirichlet distribution parameter is set to 0.1. The code is available at https://github.com/TuringSu/DKD_pFed.