Bidirectional Decoupled Distillation for Heterogeneous Federated Learning

Abstract

Federated learning enables multiple devices to collaboratively train a high-performance model on the central server while keeping their data on the devices themselves. However, due to the significant variability in data distribution across devices, the aggregated global model’s optimization direction may differ from that of the local models, making the clients lose their personality. To address this challenge, we propose a Bidirectional Decoupled Distillation For Heterogeneous Federated Learning (BDD-HFL) approach, which incorporates an additional private model within each local client. This design enables mutual knowledge exchange between the private and local models in a bidirectional manner. Specifically, previous one-way federated distillation methods mainly focused on learning features from the target class, which limits their ability to distill features from non-target classes and hinders the convergence of local models. To solve this limitation, we decompose the network output into target and non-target class logits and distill them separately using a joint optimization of cross-entropy and decoupled relative-entropy loss. We evaluate the effectiveness of BDD-HFL through extensive experiments on three benchmarks under IID, Non-IID, and unbalanced data distribution scenarios. Our results show that BDD-HFL outperforms state-of-the-art federated distillation methods across five baselines, achieving at most 3% improvement in average classification accuracy on the CIFAR-10, CIFAR-100, and MNIST datasets. The experiments demonstrate the superiority and generalization capability of BDD-HFL in addressing personalization challenges in federated learning.