Heterogeneous Clients Research Articles

A Robust Client Selection Mechanism for Federated Learning Environments

There is a exponential growth of data usage, specially due to the proliferation of connected applications with personalized models for different applications. In this context, Federated Learning (FL) emerges as a promising solution to enable collaborative model training while preserving the privacy and autonomy of participating clients. In a typical FL scenario, clients exhibit significant heterogeneity in terms of data distribution and hardware configurations. In this way, randomly sampling clients in each training round may not fully exploit the local updates from heterogeneous clients, resulting in lower model accuracy, slower convergence rate, degraded fairness, etc. In addition, malicious users could disseminate incorrect weights, which may decrease the accuracy of aggregated models and increase the time for convergence in FL. In this article, we introduce Resilience-aware Client Selection Mechanism for non-IID data and malicious clients in FL environment, called RICA. The proposed mechanism employs data size and entropy as criteria for client selection. In addition, RICA relies Centroid-Based Kernel Alignment (CKA) to identify and exclude potentially malicious clients. Our evaluation shows an improvement of 125% in Accuracy values in a scenario of malicious clients, which means the RICA+CKA demonstrates a more stable and resilient approach, reaching 90% accuracy in a few rounds compared to the default average approach, reached only around 30%. Therefore, results of the behavior of RICA+CKA in different datasets show the evaluation of different numbers of clients reaching around 90% while the other approach does not pass the 50% Accuracy.

Robust Federated Learning for Heterogeneous Clients and Unreliable Communications

Robust Federated Learning for Heterogeneous Clients and Unreliable Communications

Adaptive Heterogeneous Client Sampling for Federated Learning Over Wireless Networks

Adaptive Heterogeneous Client Sampling for Federated Learning Over Wireless Networks

Neural-aware Decoupling Fusion based Personalized Federated Learning for Intelligent Sensing

Personalized federated learning (PFL) framework that targets individual models for optimization provides better privacy and flexibility for clients. However, in challenging intelligent sensing applications, the heterogeneous client’s data distributions make the aggregation of local models in the server unstable or even hard to converge. To deal with the performance degradation caused by the above problem, existing PFL methods focus more on how to fine-tune the global model but ignore the impact of the global model fusion algorithm on the results. In this paper, we propose a new explainable neural-aware decoupling fusion based PFL framework, p-FedADF , to address the above challenges. It contains two carefully designed modules. The local decoupling module deployed on the client, utilizes the architecture disentangle technique to decouple the feature extractors in the client’s local model into sub-network according to data categories. It obtains the inference process of feature extraction for different categories of data by training. The global aggregation module, deployed on the server, aligns the sub-networks positions for multiple clients and implements a fine-grained generic feature extractor aggregation. In addition, we provide a mask encoding scheme to reduce the communication overhead of transmitting the sub-network sets between the server and clients. Our p-FedADF obtains 1.6%, 0.2%, 2.3%, and 4.5% improvement on a real-world dataset and three benchmark datasets, compared to state-of-the-art (SOTA) methods.

Structural graph federated learning: Exploiting high-dimensional information of statistical heterogeneity

With the recent progress in graph-federated learning (GFL), it has demonstrated a promising performance in effectively addressing challenges associated with heterogeneous clients. Although the majority of advances in GFL have been focused on techniques for elucidating the intricate relationships among clients, existing GFL methods have two limitations. First, current methods comprising the use of low-dimensional graphs fail to accurately depict the associations between clients, thereby compromising the performance of GFL. Second, these methods may disclose additional information when sharing client-side hidden representations. This paper presents a structural GFL (SGFL) framework and a suite of novel optimization methods. SGFL addresses the limitations of existing GFL approaches with three original contributions. Firstly, our approach advocates the dynamic construction of federated learning (FL) graphs by leveraging the high-dimensional information inherent among clients, while enabling the discovery of hierarchical communities within clients. Secondly, we present SG-FedX, a novel federated stochastic gradient optimization algorithm that mitigates the effects of heterogeneity by intelligently using a global representation. Furthermore, SG-FedX introduces a strict sharing mechanism that protects client privacy more effectively by refraining from sharing client information beyond the model parameters. Our comparative evaluations, conducted against ten representative FL algorithms under challenging non-independently-and-identically-distributed settings, demonstrated the superior performance of SG-FedX. It was noted that, in the cross-dataset scenarios, SG-FedX outperformed the second-best baseline by 8.12% and 7.91% in personalization and generalization performance, respectively.

Federated Best Arm Identification With Heterogeneous Clients

Federated Best Arm Identification With Heterogeneous Clients

Mitigating data imbalance and generating better prototypes in heterogeneous Federated Graph Learning

Federated Graph Learning (FGL) opens up new possibilities for machine learning in complex networks and distributed training, enabling multiple clients to collaborate on training Graph Neural Networks (GNNs) while preserving privacy. However, there may be differences between participating clients in terms of data distribution, computational power, and model architecture, which requires us to be able to mitigate the problem of heterogeneity across participating clients. Most of the existing model-heterogeneous Federated Learning frameworks are based on non-graph data, which is not sufficiently effective in FGL scenarios. Moreover we find that the model-heterogeneous Federated Learning approach based on prototype learning generates a single global prototype that does not provide good guidelines for each client in the case of multiple heterogeneous clients. Also, the problem of class imbalance of local data in FGL scenarios affects the overall utility. To this end, we propose FGPL, a model heterogeneous FGL framework based on prototype learning and local data augmentation. Specifically, on the one hand, locally, we mitigate the problem of local data imbalance by generating better local prototypes through a node generation method based on global prototypes and local structures. On the other hand, we cope with the domain bias problem by generating targeted global prototypes, and use a comparative learning approach to provide more personalized and correct guidelines for each client’s training. We demonstrate with extensive experiments that our approach can greatly improve the prediction of FGL. The code associated with this study is available at https://github.com/1447884149/FGPL.

Enhancing Federated Learning With Server-Side Unlabeled Data by Adaptive Client and Data Selection

Federated learning (FL) has been widely applied to collaboratively train deep learning (DL) models on massive end devices (i.e., clients). Due to the limited storage capacity and high labeling cost, the data on each client may be insufficient for model training. Conversely, in cloud datacenters, there exist large-scale unlabeled data, which are easy to collect from public access (e.g., social media). Herein, we propose the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ada-FedSemi</i> system, which leverages both on-device labeled data and in-cloud unlabeled data to boost the performance of DL models. In each round, local models are aggregated to produce pseudo-labels for the unlabeled data, which are utilized to enhance the global model. Considering that the number of participating clients and the quality of pseudo-labels will have a significant impact on the training performance, we introduce a multi-armed bandit (MAB) based online algorithm to adaptively determine the participating fraction and confidence threshold. Besides, to alleviate the impact of stragglers, we assign local models of different depths for heterogeneous clients. Extensive experiments on benchmark models and datasets show that given the same resource budget, the model trained by Ada-FedSemi achieves 3% <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 14.8% higher test accuracy than that of the baseline methods. When achieving the same test accuracy, Ada-FedSemi saves up to 48% training cost, compared with the baselines. Under the scenario with heterogeneous clients, the proposed HeteroAda-FedSemi can further speed up the training process by 1.3× <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim 1.5\times$</tex-math></inline-formula> .

FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning

Recently, Heterogeneous Federated Learning (HtFL) has attracted attention due to its ability to support heterogeneous models and data. To reduce the high communication cost of transmitting model parameters, a major challenge in HtFL, prototype-based HtFL methods are proposed to solely share class representatives, a.k.a, prototypes, among heterogeneous clients while maintaining the privacy of clients’ models. However, these prototypes are naively aggregated into global prototypes on the server using weighted averaging, resulting in suboptimal global knowledge which negatively impacts the performance of clients. To overcome this challenge, we introduce a novel HtFL approach called FedTGP, which leverages our Adaptive-margin-enhanced Contrastive Learning (ACL) to learn Trainable Global Prototypes (TGP) on the server. By incorporating ACL, our approach enhances prototype separability while preserving semantic meaning. Extensive experiments with twelve heterogeneous models demonstrate that our FedTGP surpasses state-of-the-art methods by up to 9.08% in accuracy while maintaining the communication and privacy advantages of prototype-based HtFL. Our code is available at https://github.com/TsingZ0/FedTGP.

Robust Distributed Gradient Aggregation Using Projections onto Gradient Manifolds

We study the distributed gradient aggregation problem where individual clients contribute to learning a central model by sharing parameter gradients constructed from local losses. However, errors in some gradients, caused by low-quality data or adversaries, can degrade the learning process when naively combined. Existing robust gradient aggregation approaches assume that local data represent the global data-generating distribution, which may not always apply to heterogeneous (non-i.i.d.) client data. We propose a new algorithm that can robustly aggregate gradients from potentially heterogeneous clients. Our approach leverages the manifold structure inherent in heterogeneous client gradients and evaluates gradient anomaly degrees by projecting them onto this manifold. This algorithm is implemented as a simple and efficient method that accumulates random projections within the subspace defined by the nearest neighbors within a gradient cloud. Our experiments demonstrate consistent performance improvements over state-of-the-art robust aggregation algorithms.

Federated Learning with Efficient Aggregation via Markov Decision Process in Edge Networks

Federated Learning (FL), as an emerging paradigm in distributed machine learning, has received extensive research attention. However, few works consider the impact of device mobility on the learning efficiency of FL. In fact, it is detrimental to the training result if heterogeneous clients undergo migration or are in an offline state during the global aggregation process. To address this issue, the Optimal Global Aggregation strategy (OGAs) is proposed. The OGAs first models the interaction between clients and servers of the FL as a Markov Decision Process (MDP) model, which jointly considers device mobility and data heterogeneity to determine local participants that are conducive to global aggregation. To obtain the optimal client participation strategy, an improved σ-value iteration method is utilized to solve the MDP, ensuring that the number of participating clients is maintained within an optimal interval in each global round. Furthermore, the Principal Component Analysis (PCA) is used to reduce the dimensionality of the original features to deal with the complex state space in the MDP. The experimental results demonstrate that, compared with other existing aggregation strategies, the OGAs has the faster convergence speed and the higher training accuracy, which significantly improves the learning efficiency of the FL.

Tree Learning: Towards Promoting Coordination in Scalable Multi-Client Training Acceleration

Iteration based collaborative learning (CL) paradigms, such as federated learning (FL) and split learning (SL), faces challenges in training neural models over the rapidly growing yet resource-constrained edge devices. Such devices have difficulty in accommodating a full-size large model for FL or affording an excessive waiting time for the mandatory synchronization step in SL. To deal with such challenge, we propose a novel CL framework which adopts an tree-aggregation structure with an adaptive partition and ensemble strategy to achieve optimal synchronization and fast convergence at scale. To find the optimal split point for heterogeneous clients, we also design a novel partitioning algorithm by minimizing the idleness during communication and achieving the optimal synchronization between clients. In addition, a parallelism paradigm is proposed to unleash the potential of optimum synchronization between the clients and server to boost the distributed training process without losing model accuracy for edge devices. Furthermore, we theoretically prove that our framework can achieve better convergence rate than state-of-the-art CL paradigms. We conduct extensive experiments and show that our framework is 4.6× in training speed as compared with the traditional methods, without compromising training accuracy.

Incentive Design for Heterogeneous Client Selection: A Robust Federated Learning Approach

Incentive Design for Heterogeneous Client Selection: A Robust Federated Learning Approach

Achieving Linear Speedup in Asynchronous Federated Learning With Heterogeneous Clients

Achieving Linear Speedup in Asynchronous Federated Learning With Heterogeneous Clients

PipeSFL: A Fine-Grained Parallelization Framework for Split Federated Learning on Heterogeneous Clients

PipeSFL: A Fine-Grained Parallelization Framework for Split Federated Learning on Heterogeneous Clients

Personalized Federated Transfer Learning for Cycle-Life Prediction of Lithium-Ion Batteries in Heterogeneous Clients With Data Privacy Protection

Personalized Federated Transfer Learning for Cycle-Life Prediction of Lithium-Ion Batteries in Heterogeneous Clients With Data Privacy Protection

Optimal Mechanism Design for Heterogeneous Client Sampling in Federated Learning

Optimal Mechanism Design for Heterogeneous Client Sampling in Federated Learning

CroSSHeteroFL: Cross-Stratified Sampling Composition-Fitting to Federated Learning for Heterogeneous Clients

<b>CroSSHeteroFL</b>: Cross-Stratified Sampling Composition-Fitting to Federated Learning for Heterogeneous Clients

Accelerating Federated Learning via Sequential Training of Grouped Heterogeneous Clients

Accelerating Federated Learning via Sequential Training of Grouped Heterogeneous Clients

Federated Learning With Heterogeneous Client Expectations: A Game Theory Approach

Federated Learning With Heterogeneous Client Expectations: A Game Theory Approach