FedNN: Federated learning on concept drift data using weight and adaptive group normalizations

Abstract

Federated Learning (FL) allows a global model to be trained without sharing private raw data. The major challenge in FL is client-wise data heterogeneity leading to different model convergence speed and accuracy. Despite the recent progress of FL, most methods verify their accuracy on prior probability shift (label distribution skew) dataset, while the concept drift problem (i.e., where each client has distinct styles of input while sharing the same labels) has not been explored. In real scenarios, concept drift is of paramount concern in FL since the client’s data is collected under extremely different conditions making FL optimization more challenging. Significant differences in inputs among clients exacerbate the heterogeneity of clients’ parameters compared to prior probability shift, ultimately resulting in failures for previous FL approaches. To address the challenge of concept drift, we use Weight Normalization (WN) and Adaptive Group Normalization (AGN) to alleviate conflicts during global model updates. WN re-parameterizes weights to have zero mean and unit variance while AGN adaptively selects the optimal mean and standard deviation for feature normalization based on the dataset. These two components significantly contribute to having consistent activations after global model updates reducing heterogeneity in concept drift data. Comprehensive experiments on seven datasets (with concept drift) demonstrate that our method outperforms five state-of-the-art FL methods and shows faster convergence speed compared to the previous FL methods.