PFL-DKD: Modeling decoupled knowledge fusion with distillation for improving personalized federated learning

Abstract

We develop a novel framework for personalized federated learning (PFL) utilizing a decoupled version of knowledge distillation (DKD). Unlike traditional PFL methods, the proposed PFL-DKD creates a dynamically connected network among local clients and categorizes them according to their knowledge, storage, and computational capabilities. The developed decoupling of knowledge distillation into target class (TC) and latent class (LC) enables knowledge-rich clients to efficiently transfer their expertise to knowledge-poor clients. To further enhance our innovative PFL-DKD approach, we extend it to PFL-FDKD by introducing a ”logit fusion” that seamlessly aggregates knowledge and experiences from neighboring clients. Both our theoretical analyses and extensive experiments reveal that PFL-DKD outperforms existing centralized and decentralized PFL approaches, making significant strides in mitigating the challenges associated with heterogeneous data and system configurations. The details of our implementation with the codebase are in PFL-DKD.