non-IID Data Research Articles

Clipper: Online Joint Client Sampling and Power Allocation for Wireless Federated Learning

Communication overhead is a main bottleneck in federated learning (FL) especially in the wireless environment due to the limited data rate and unstable radio channels. The communication challenge necessitates holistic selection of participating clients that accounts for both the computation needs and communication cost, as well as judicious allocation of the limited transmission resource. Meanwhile, the random unpredictable nature of both the training data samples and the communication channels requires an online optimization approach that adapts to the changing system state over time. In this work, we consider a general framework of online joint client sampling and power allocation for wireless FL under time-varying communication channels. We formulate it as a stochastic network optimization problem that admits a Lyapunov-typed solution approach. This leads to per-training-round subproblems with a special bi-convex structure, which we leverage to propose globally optimal solutions, culminating in a meta algorithm that provides strong performance guarantees. We further study three specific FL problems covering multiple scenarios, namely with IID or non-IID data, whether robustness against data drift is required, and with unbiased or biased client sampling. We derive detailed algorithms for each of these problems. Simulation with standard classification tasks demonstrate that the proposed communication-aware algorithms outperform their counterparts under a wide range of learning and communication scenarios.

Asynchronous Federated Learning-Based Indoor Localization With Non-IID Data

Asynchronous Federated Learning-Based Indoor Localization With Non-IID Data

FedVeca: Federated Vectorized Averaging on Non-IID Data With Adaptive Bi-Directional Global Objective

FedVeca: Federated Vectorized Averaging on Non-IID Data With Adaptive Bi-Directional Global Objective

Backdoor Attacks in Peer-to-Peer Federated Learning

Most machine learning applications rely on centralized learning processes, opening up the risk of exposure of their training datasets. While federated learning (FL) mitigates to some extent these privacy risks, it relies on a trusted aggregation server for training a shared global model. Recently, new distributed learning architectures based on Peer-to-Peer Federated Learning (P2PFL) offer advantages in terms of both privacy and reliability. Still, their resilience to poisoning attacks during training has not been investigated. In this paper, we propose new backdoor attacks for P2PFL that leverage structural graph properties to select the malicious nodes, and achieve high attack success, while remaining stealthy. We evaluate our attacks under various realistic conditions, including multiple graph topologies, limited adversarial visibility of the network, and clients with non-IID data. Finally, we show the limitations of existing defenses adapted from FL and design a new defense that successfully mitigates the backdoor attacks, without an impact on model accuracy.

Federated split learning via dynamic aggregation and homomorphic encryption on non-IID data

Federated split learning via dynamic aggregation and homomorphic encryption on non-IID data

Advanced Optimization Techniques for Federated Learning on Non-IID Data

Federated learning enables model training on multiple clients locally, without the need to transfer their data to a central server, thus ensuring data privacy. In this paper, we investigate the impact of Non-Independent and Identically Distributed (non-IID) data on the performance of federated training, where we find a reduction in accuracy of up to 29% for neural networks trained in environments with skewed non-IID data. Two optimization strategies are presented to address this issue. The first strategy focuses on applying a cyclical learning rate to determine the learning rate during federated training, while the second strategy develops a sharing and pre-training method on augmented data in order to improve the efficiency of the algorithm in the case of non-IID data. By combining these two methods, experiments show that the accuracy on the CIFAR-10 dataset increased by about 36% while achieving faster convergence by reducing the number of required communication rounds by 5.33 times. The proposed techniques lead to improved accuracy and faster model convergence, thus representing a significant advance in the field of federated learning and facilitating its application to real-world scenarios.

A Robust Client Selection Mechanism for Federated Learning Environments

There is a exponential growth of data usage, specially due to the proliferation of connected applications with personalized models for different applications. In this context, Federated Learning (FL) emerges as a promising solution to enable collaborative model training while preserving the privacy and autonomy of participating clients. In a typical FL scenario, clients exhibit significant heterogeneity in terms of data distribution and hardware configurations. In this way, randomly sampling clients in each training round may not fully exploit the local updates from heterogeneous clients, resulting in lower model accuracy, slower convergence rate, degraded fairness, etc. In addition, malicious users could disseminate incorrect weights, which may decrease the accuracy of aggregated models and increase the time for convergence in FL. In this article, we introduce Resilience-aware Client Selection Mechanism for non-IID data and malicious clients in FL environment, called RICA. The proposed mechanism employs data size and entropy as criteria for client selection. In addition, RICA relies Centroid-Based Kernel Alignment (CKA) to identify and exclude potentially malicious clients. Our evaluation shows an improvement of 125% in Accuracy values in a scenario of malicious clients, which means the RICA+CKA demonstrates a more stable and resilient approach, reaching 90% accuracy in a few rounds compared to the default average approach, reached only around 30%. Therefore, results of the behavior of RICA+CKA in different datasets show the evaluation of different numbers of clients reaching around 90% while the other approach does not pass the 50% Accuracy.

Federated Diverse Self-ensembling Learning Approach for Data Heterogenity in Drive Vision

Federated learning (FL) has developed as an efficient framework that can be used to train models across isolated data sources while also protecting the privacy of the data. In FL a common method is to construct local and global models together, with the global model (server) informing the local models and the local models (clients) updating the global model. Most present works assume clients have labeled datasets and the server has no data for supervised learning (SL) problems. In reality, clients lack the competence and drive to identify their data, while the server may host a tiny amount. How to reasonably use serverlabeled and client-unlabeled data is crucial in semi-supervised learning (SSL) and Cclientdata heterogeneity is widespread in FL. However, inadequate high-quality labels and non-IID client data, especially in autonomous driving, decrease model performance across domains and interact negatively. To solve this Semi-Supervised Federated Learning (SSFL) problem, we come up with a new FL algorithm called FedDSL in this work. We use self-ensemble learning and complementary negative learning in our method to make clients’ unsupervised learning on unlabeled data more accurate and efficient. It also coordinates the model training on both the server side and the clients’ side. In an important distinction to earlier research that kept some subset of labels at each client, our method is the first to implement SSFL for clients with 0% labeled non-IID data. Our contributions include the effectiveness of self-ensemble learning by using confidence score vector for calculating only for the current model performing data filtering and initiated negative learning by showing the data filtering performance in the beginning rounds. Our approach has been rigorously validated on two significant autonomous driving datasets, BDD100K and Cityscapes, proving to be highly effective. We have achieved state-of-the-art results and the metric that is utilized to evaluate the effectiveness of each detection task is mean average precision (mAP@0.5). Astonishingly FedDSL performs nearly as well as fully-supervised centralized training approaches, despite the fact that it only uses 25% of the labels in the Global model.

Federated influencer learning for secure and efficient collaborative learning in realistic medical database environment.

Enhancing deep learning performance requires extensive datasets. Centralized training raises concerns about data ownership and security. Additionally, large models are often unsuitable for hospitals due to their limited resource capacities. Federated learning (FL) has been introduced to address these issues. However, FL faces challenges such as vulnerability to attacks, non-IID data, reliance on a central server, high communication overhead, and suboptimal model aggregation. Furthermore, FL is not optimized for realistic hospital database environments, where data are dynamically accumulated. To overcome these limitations, we propose federated influencer learning (FIL) as a secure and efficient collaborative learning paradigm. Unlike the server-client model of FL, FIL features an equal-status structure among participants, with an administrator overseeing the overall process. FIL comprises four stages: local training, qualification, screening, and influencing. Local training is similar to vanilla FL, except for the optional use of a shared dataset. In the qualification stage, participants are classified as influencers or followers. During the screening stage, the integrity of the logits from the influencer is examined. If the integrity is confirmed, the influencer shares their knowledge with the others. FIL is more secure than FL because it eliminates the need for model-parameter transactions, central servers, and generative models. Additionally, FIL supports model-agnostic training. These features make FIL particularly promising for fields such as healthcare, where maintaining confidentiality is crucial. Our experiments demonstrated the effectiveness of FIL, which outperformed several FL methods on large medical (X-ray, MRI, and PET) and natural (CIFAR-10) image dataset in a dynamically accumulating database environment, with consistently higher precision, recall, Dice score, and lower standard deviation between participants. In particular, in the PET dataset, FIL achieved about a 40% improvement in Dice score and recall.

Bidirectional Corrective Model-Contrastive Federated Adversarial Training

When dealing with non-IID data, federated learning confronts issues such as client drift and sluggish convergence. Therefore, we propose a Bidirectional Corrective Model-Contrastive Federated Adversarial Training (BCMCFAT) framework. On the client side, we design a category information correction module to correct biases caused by imbalanced local data by incorporating the local client’s data distribution information. Through local adversarial training, more robust local models are obtained. Secondly, we propose a model-based adaptive correction algorithm in the server that leverages a self-attention mechanism to handle each client’s data distribution information and introduces learnable aggregation tokens. Through the self-attention mechanism, model contrast learning is conducted on each client to obtain aggregation weights of corrected client models, thus addressing the issues of accuracy degradation and slow convergence caused by client drift. Our algorithm achieves the best natural accuracy on the CIFAR-10, CIFAR-100, and SVHN datasets and demonstrates excellent adversarial defense performance against FGSM, BIM, and PGD attacks.

Scenario-aware clustered federated learning for vehicle trajectory prediction with non-IID data

In recent years, Federated Learning (FL) has attracted much attention in Vehicle Trajectory Prediction (VTP) as it can resolve the critical issues of insufficient data, data privacy, and training efficiency. However, compared with centralized training, the model trained by FL may have insufficient prediction performance. This important issue comes from a statistical heterogeneity distribution of the local data in the participating clients, that is, non-IID. Therefore, this paper introduces a Clustered Federated Learning (CFL) approach for the VTP model to mitigate the influence of non-IID data. The proposed approach consists of federated trajectory clustering and federated VTP model training. In federated trajectory clustering, the optimal trajectory scenario discriminator is produced using federated K-means clustering without direct access to private data. In the federated VTP model training, multiple VTP models for specific trajectory scenarios are trained to deal with the influence of non-IID data. Experimental results reveal that our approach outperforms the state-of-the-art FL method on both NGSIM and HighD datasets, achieving up to 13.82% convergence acceleration and 12.47% RMSE reduction.

Heterogeneous Federated Learning for Non-IID Smartwatch Data Classification

Heterogeneous Federated Learning for Non-IID Smartwatch Data Classification

Fed-RHLP: Enhancing Federated Learning with Random High-Local Performance Client Selection for Improved Convergence and Accuracy

We introduce the random high-local performance client selection strategy, termed Fed-RHLP. This approach allows opportunities for higher-performance clients to contribute more significantly by updating and sharing their local models for global aggregation. Nevertheless, it also enables lower-performance clients to participate collaboratively based on their proportional representation determined by the probability of their local performance on the roulette wheel (RW). Improving symmetry in federated learning involves IID Data: symmetry is naturally present, making model updates easier to aggregate and Non-IID Data: asymmetries can impact performance and fairness. Solutions include data balancing, adaptive algorithms, and robust aggregation methods. Fed-RHLP enhances federated learning by allowing lower-performance clients to contribute based on their proportional representation, which is determined by their local performance. This fosters inclusivity and collaboration in both IID and Non-IID scenarios. In this work, through experiments, we demonstrate that Fed-RHLP offers accelerated convergence speed and improved accuracy in aggregating the final global model, effectively mitigating challenges posed by both IID and Non-IID Data distribution scenarios.

Bidirectional Decoupled Distillation for Heterogeneous Federated Learning.

Federated learning enables multiple devices to collaboratively train a high-performance model on the central server while keeping their data on the devices themselves. However, due to the significant variability in data distribution across devices, the aggregated global model's optimization direction may differ from that of the local models, making the clients lose their personality. To address this challenge, we propose a Bidirectional Decoupled Distillation For Heterogeneous Federated Learning (BDD-HFL) approach, which incorporates an additional private model within each local client. This design enables mutual knowledge exchange between the private and local models in a bidirectional manner. Specifically, previous one-way federated distillation methods mainly focused on learning features from the target class, which limits their ability to distill features from non-target classes and hinders the convergence of local models. To solve this limitation, we decompose the network output into target and non-target class logits and distill them separately using a joint optimization of cross-entropy and decoupled relative-entropy loss. We evaluate the effectiveness of BDD-HFL through extensive experiments on three benchmarks under IID, Non-IID, and unbalanced data distribution scenarios. Our results show that BDD-HFL outperforms state-of-the-art federated distillation methods across five baselines, achieving at most 3% improvement in average classification accuracy on the CIFAR-10, CIFAR-100, and MNIST datasets. The experiments demonstrate the superiority and generalization capability of BDD-HFL in addressing personalization challenges in federated learning.

Tackling Non-IID Data and Data Poisoning in Federated Learning using Adversarial Synthetic Data

Federated learning (FL) involves joint model training by various devices while preserving the privacy of their data. However, it presents a challenge of dealing with heterogeneous data located on participating devices. This issue can further be complicated by the appearance of malicious clients, aiming to sabotage the training process by poisoning local data. In this context, a problem of differentiating between poisoned and non-IID data appears. To address it, a technique utilizing data-free synthetic data generation is proposed, using a reverse concept of adversarial attack. Adversarial inputs allow for improving the training process by measuring clients' coherence, favoring trustworthy participants. Experimental results, obtained from the image classification tasks for MNIST, EMNIST, and CIFAR-10 datasets are reported and analyzed.

Differentially Private Auction Design for Federated Learning With non-IID Data

Differentially Private Auction Design for Federated Learning With non-IID Data

An adaptive asynchronous federated learning framework for heterogeneous Internet of things

Federated learning (FL) is a distributed machine learning framework that enables the training of shared models without the need to share local data. However, FL faces challenges in heterogeneous Internet of Things (IoT) environments, including communication bottleneck, staleness, and non- independent and identically distributed (non-IID) data. To tackle these challenges, we present ASAFL, an Asynchronous Federated Learning framework with an Adaptive Scheduling Strategy. Firstly, we quantify the potential contribution of client models relative to the server model. Secondly, we design a client upload strategy to reduce the uploading of redundant models with low contribution. Finally, we propose a server model update method based on the contributions to address model divergence caused by staleness and non-IID data. Furthermore, our theoretical analysis guarantees ASAFL's convergence, and experiments show it reduces communication overhead by over 70% compared to traditional asynchronous FL.

Adaptive Federated Learning in Non-IID Data Environment

Adaptive Federated Learning in Non-IID Data Environment

Concurrent vertical and horizontal federated learning with fuzzy cognitive maps

Data privacy is a major concern in industries such as healthcare or finance. The requirement to safeguard privacy is essential to prevent data breaches and misuse, which can have severe consequences for individuals and organisations. Federated learning is a distributed machine learning approach where multiple participants collaboratively train a model without compromising the privacy of their data. However, a significant challenge arises from the differences in feature spaces among participants, known as non-IID data. This research introduces a novel federated learning framework employing fuzzy cognitive maps, designed to comprehensively address the challenges posed by diverse data distributions and non-identically distributed features in federated settings. The proposal is tested through several experiments using four distinct federation strategies: constant-based, accuracy-based, AUC-based, and precision-based weights. The results demonstrate the effectiveness of the approach in achieving the desired learning outcomes while maintaining privacy and confidentiality standards.

Federated Many-Task Bayesian Optimization

Bayesian optimization is a powerful surrogate-assisted algorithm for solving expensive black-box optimization problems. While Bayesian optimization was developed for centralized optimization, the availability of massive distributed data has attracted increased interests in exploring federated Bayesian optimization that can use data on multiple clients without leaking the raw data. However, existing federated Bayesian optimization (FBO) approaches assume that either all clients jointly solve the same optimization task, or only one client solves one target optimization task by transferring knowledge from others in a federated way, making them unsuited for many real-world applications. In this paper, we consider FBO for the scenario where multiple related local black-box tasks associated with different clients are jointly optimized by sharing knowledge between tasks without leaking the data privacy. An efficient federated many-task Bayesian optimization framework is proposed to address not independent and identically distributed (non-IID) data while protecting the data privacy in the federated setting. A novel federated knowledge transfer paradigm is developed for dynamic many-task model aggregation according to a dissimilarity matrix. The dissimilarity is measured based on the rank of the predictions and only the hyperparameters in the local Gaussian process models are shared. In addition, a federated ensemble acquisition function is constructed by integrating the predictions of two surrogates using the global and local hyperparameters, respectively, to effectively search for the optimal solution. Experimental results show that our proposed method has reliable performance on both benchmark problems and a real machine learning problem also in the presence of non-IID data.