non-IID Data Research Articles

Transfer learning via random forests: A one-shot federated approach

A one-shot federated transfer learning method using random forests (FTRF) is developed to improve the prediction accuracy at a target data site by leveraging information from auxiliary sites. Both theoretical and numerical results show that the proposed federated transfer learning approach is at least as accurate as the model trained on the target data alone regardless of possible data heterogeneity, which includes imbalanced and non-IID data distributions across sites and model mis-specification. FTRF has the ability to evaluate the similarity between the target and auxiliary sites, enabling the target site to autonomously select more similar site information to enhance its predictive performance. To ensure communication efficiency, FTRF adopts the model averaging idea that requires a single round of communication between the target and the auxiliary sites. Only fitted models from auxiliary sites are sent to the target site. Unlike traditional model averaging, FTRF incorporates predicted outcomes from other sites and the original variables when estimating model averaging weights, resulting in a variable-dependent weighting to better utilize models from auxiliary sites to improve prediction. Five real-world data examples show that FTRF reduces the prediction error by 2-40% compared to methods not utilizing auxiliary information.

Federated learning using model projection for multi-center disease diagnosis with non-IID data

Multi-center disease diagnosis aims to build a global model for all involved medical centers. Due to privacy concerns, it is infeasible to collect data from multiple centers for training (i.e., centralized learning). Federated Learning (FL) is a decentralized framework that enables multiple clients (e.g., medical centers) to collaboratively train a global model while retaining patient data locally for privacy. However, in practice, the data across medical centers are not independently and identically distributed (Non-IID), causing two challenging issues: (1) catastrophic forgetting at clients, i.e., the local model at clients will forget the knowledge received from the global model after local training, causing reduced performance; and (2) invalid aggregation at the server, i.e., the global model at the server may not be favorable to some clients after model aggregation, resulting in a slow convergence rate. To mitigate these issues, an innovative Federated learning using Model Projection (FedMoP) is proposed, which guarantees: (1) the loss of local model on global data does not increase after local training without accessing the global data so that the performance will not be degenerated; and (2) the loss of global model on local data does not increase after aggregation without accessing local data so that convergence rate can be improved. Extensive experimental results show that our FedMoP outperforms state-of-the-art FL methods in terms of accuracy, convergence rate and communication cost. In particular, our FedMoP also achieves comparable or even higher accuracy than centralized learning. Thus, our FedMoP can ensure privacy protection while outperforming centralized learning in accuracy and communication cost.

IOFL: Intelligent-Optimization-Based Federated Learning for Non-IID Data

IOFL: Intelligent-Optimization-Based Federated Learning for Non-IID Data

Decentralized and robust privacy-preserving model using blockchain-enabled Federated Deep Learning in intelligent enterprises

In Federated Deep Learning (FDL), multiple local enterprises are allowed to train a model jointly. Then, they submit their local updates to the central server, and the server aggregates the updates to create a global model. However, trained models usually perform worse than centralized models, especially when the training data distribution is non-independent and identically distributed (non-IID). Because non-IID data harms the accuracy and performance of the model. Second, due to the centrality of federated learning (FL) and the untrustworthiness of enterprises, traditional FL solutions are vulnerable to security and privacy attacks. Therefore, to tackle this issue, we propose FEDANIL, a secure blockchain-enabled Federated Deep LeArning Model that improves enterprise models’ decentralized, performance, and tamper-proof properties, including two main phases. The first phase is proposed to address the non-IID challenge (label and feature distribution skew). In this phase, local models with similar data distributions are grouped into homogeneous clusters using the cosine similarity (CS) and affinity propagation (AP) techniques. Then, for each homogeneous cluster, Wasserstein Generative Adversarial Networks (WGAN) are used to deal with label and feature distribution skew. The second phase was adopted to address security and privacy concerns against poisoning and inference attacks via three steps. In the first step, data poisoning attacks are prevented by using CS. Then, in the second step, collude attacks were prevented by randomly selecting enterprises in the consortium blockchain. Finally, in the third step, model poisoning, membership inference, and reconstruction attacks have been prevented using the CKKS Fully Homomorphic Encryption (CKKS-FHE) technique and consortium blockchain. Extensive experiments were conducted using the Sent140, Fashion-MNIST, FEMNIST, and CIFAR-10 new real-world datasets to evaluate FedAnil’s robustness and performance. The simulation results demonstrate that FEDANIL satisfies FDL privacy-preserving requirements. In terms of convergence analysis, the model parameter obtained with the FEDANIL converges to the optimum of the model parameter. In addition, it performs better in terms of accuracy (more than 11, 15, and 24%) and computation overhead (less than 8, 10, and 15%) compared with baseline approaches, namely SHIELDFL, RVPFL, and RFA, respectively. The FEDANIL source code can be found on GitHub.11Code available on GitHub Repository: https://github.com/rezafotohi/FedAnil. For any questions about the code, please contact Fotohi.reza@gmail.com.

SF-CABD: Secure Byzantine fault tolerance federated learning on Non-IID data

Federated learning facilitates collaborative learning among multiple parties while ensuring client privacy. The vulnerability of federated learning to diverse Byzantine attacks stems from the opacity inherent in its local training processes. This issue becomes more pronounced when data among clients lacks independent and identical distribution. Concurrently, the advent of gradient inversion attacks introduces an escalating risk to the confidentiality and integrity of the federated learning system. This work introduces a framework to tackle the difficulties associated with Non-IID data within federated learning. Initially, it classifies diverse data using Euclidean distance to alleviate intra-cluster heterogeneity. A privacy-preserving Byzantine fault tolerance strategy utilizing cosine similarity is implemented within each cluster. We adopt normalizing and introduce historical momentum to the inter-cluster aggregation. The integration of homomorphic encryption ensures that clients’ gradient information participates in training under ciphertext, safeguarding against gradient inversion attacks. Lastly, we conduct a comparative analysis against classical and latest algorithms under various conditions. These results validate the effectiveness of our design in enhancing robustness in Non-IID data.

Asynchronous Federated Learning with Grey Wolf Optimization for the Heterogeneity IoT Devices

The Internet of Things (IoT) creates new options for real-time data collection and Machine Learning (ML) model development. Nonetheless, it's feasible that a particular IoT device does not have sufficient computational power to train and implement a complete learning model. However, there are significant communication costs and data security and privacy concerns associated with sending actual data to a centralised server with a lot of computational power. Federated Learning (FL) is a potential way to train ML models using low-powered devices and Edge Servers (ES) since it is a distributed ML architecture. However, the vast majority of the works in existence make the unsustainable assumption of a synchronized parameters update manner with similar IoT nodes and reliable communications networks. To increase training efficiency and accelerate the speed for heterogeneous IoT devices in an unreliable network environment, we designed an Asynchronous Federated Learning strategy with Grey Wolf Optimization (AFL-GWO) in this research. In particular, we develop a Lightweight Node Selection (LNS) technique and propose an AFL-GWO model to efficiently complete learning tasks. To ensure that diverse IoT nodes with varying computational capabilities and network connectivity are represented in the global learning aggregate, the proposed technique makes node selections on an iterative basis. We show through extensive experiments that our suggested AFL-GWO system outperforms the state-of-the-art techniques on identically and independently distributed (IID) and non-IID data distribution in a variety of contexts.

Edge Federated Optimization for Heterogeneous Data

This study focuses on optimizing federated learning in heterogeneous data environments. We implement the FedProx and a baseline algorithm (i.e., the FedAvg) with advanced optimization strategies to tackle non-IID data issues in distributed learning. Model freezing and pruning techniques are explored to showcase the effective operations of deep learning models on resource-constrained edge devices. Experimental results show that at a pruning rate of 10%, the FedProx with structured pruning in the MIT-BIH and ST databases achieved the best F1 scores, reaching 96.01% and 77.81%, respectively, which achieves a good balance between system efficiency and model accuracy compared to those of the FedProx with the original configuration, reaching F1 scores of 66.12% and 89.90%, respectively. Similarly, with layer freezing technique, unstructured pruning method, and a pruning rate of 20%, the FedAvg algorithm effectively balances classification performance and degradation of pruned model accuracy, achieving F1 scores of 88.75% and 72.75%, respectively, compared to those of the FedAvg with the original configuration, reaching 56.82% and 85.80%, respectively. By adopting model optimization strategies, a practical solution is developed for deploying complex models in edge federated learning, vital for its efficient implementation.

Two-stage sampling with predicted distribution changes in federated semi-supervised learning

Federated semi-supervised learning (FSSL) involves training a model in a federated environment using a few labeled samples and many unlabeled samples. Compared with semi-supervised learning, FSSL faces more complex data situations, especially when data are non-independently and identically distributed (non-IID), adding more challenges to the learning process. The previous method addresses the aforementioned issues by enlarging the training sample space through multiple client random sampling and reweighting the parameters. Although it achieves high accuracy, it sacrifices communication efficiency. In this study, we propose PDCFed, a two-stage sampling method that uses the Predicted Distribution Changes of samples after different data augmentations. We evaluate the credibility of the samples based on the maximum probability predicted by weak augmentation. When the samples are in a less reliable space, they are further sampled after adjusting the predicted distribution changes using a Gaussian function. To enhance the model’s generalization ability, an entropy penalty term is incorporated after unsupervised training loss. Extensive experiments demonstrate that this method outperforms existing methods on three datasets with non-IID data and significantly improves communication efficiency.

"Federated Learning: Advancements, Applications, and Future Directions for Collaborative Machine Learning in Distributed Environments"

Federated Learning (FL) has become widely recognized as a feasible method for training machine learning models on decentralized devices, ensuring the preservation of data privacy. This study offers an extensive overview of the latest progress in federated learning methods, their applications, and the challenges they entail. We begin by introducing the concept of federated learning and its significance in distributed environments. Next, we delve into a range of methodologies aimed at improving the effectiveness, scalability, and confidentiality of federated learning. These encompass optimization algorithms, communication protocols, and mechanisms designed to uphold privacy. Moreover, we investigate the broad spectrum of applications where federated learning finds utility, spanning healthcare, the Internet of Things (IoT), and edge computing. This exploration illuminates tangible scenarios and advantages in real-world settings. Top of Form
 Additionally, we analyze the challenges and limitations inherent in federated learning, including communication overhead, non-IID data distribution, and model heterogeneity. We review recent research efforts aimed at addressing these challenges, such as federated averaging variants, adaptive client selection, and robust aggregation techniques. Finally, we outline future research directions and potential avenues for the advancement of federated learning, emphasizing the need for standardized benchmarks, federated learning frameworks, and interdisciplinary collaborations.

Empowering precise advertising with Fed-GANCC: A novel federated learning approach leveraging Generative Adversarial Networks and group clustering.

In the realm of targeted advertising, the demand for precision is paramount, and the traditional centralized machine learning paradigm fails to address this necessity effectively. Two critical challenges persist in the current advertising ecosystem: the data privacy concerns leading to isolated data islands and the complexity in handling non-Independent and Identically Distributed (non-IID) data and concept drift due to the specificity and diversity in user behavior data. Current federated learning frameworks struggle to overcome these hurdles satisfactorily. This paper introduces Fed-GANCC, an innovative federated learning framework that synergizes Generative Adversarial Networks (GANs) and Group Clustering. The framework incorporates a user data augmentation algorithm predicated on adversarial generative networks to enrich user behavior data, curtail the impact of non-uniform data distribution, and enhance the applicability of the global machine learning model. Unlike traditional approaches, our framework offers user data augmentation algorithms based on adversarial generative networks, which not only enriches user behavior data but also reduces the challenges posed by non-uniform data distribution, thereby enhancing the applicability of the global machine learning (ML) model. The effectiveness of Fed-GANCC is distinctly showcased through experimental results, outperforming contemporary methods like FED-AVG and FED-SGD in terms of accuracy, loss value, and receiver operating characteristic (ROC) indicators within the same computing time. Experimental results vindicate the effectiveness of Fed-GANCC, revealing substantial enhancements in accuracy, loss value, and receiver operating characteristic (ROC) metrics compared to FED-AVG and FED-SGD given the same computational time. These outcomes underline Fed-GANCC's exceptional prowess in mitigating issues such as isolated data islands, non-IID data, and concept drift. With its novel approach to addressing the prevailing challenges in targeted advertising such as isolated data islands, non-IID data, and concept drift, the Fed-GANCC framework stands as a benchmark, paving the way for future advancements in federated learning solutions tailored for the advertising domain. The Fed-GANCC framework promises to offer pivotal insights for the future development of efficient and advanced federated learning solutions for targeted advertising.

Fed-mRMR: A lossless federated feature selection method

Feature selection has become a mandatory task in data mining, due to the overwhelming amount of features in Big Data problems. To handle this high-dimensional data and avoid the well-known curse of dimensionality, we need to pre-select an optimal subset of features to reduce redundant computations. Federated learning is a machine learning technique based on training an algorithm over many decentralized edge devices holding local rather than global data on a centralized server. Application of this technique is extending to fields such as self-driving cars, medicine and health, and Industry 4.0, where data privacy is compulsory. Feature selection through federated learning is a complicated task since suboptimal features calculated by feature selection methods may be different in heterogeneous datasets from different nodes. In this paper, we propose a lossless federated version of the classic minimum redundancy maximum relevance (mRMR) feature selection algorithm, called federated mRMR (fed-mRMR), which, without losing any effectiveness of the original mRMR method, is applicable to federated learning approaches and capable of dealing with data that are not independent and identically distributed (non-IID data).Implementation can be found at: https://github.com/jorgehermo9/fed-mrmr

Defending Against Data Poisoning Attack in Federated Learning With Non-IID Data

Defending Against Data Poisoning Attack in Federated Learning With Non-IID Data

Ensemble Federated Learning With Non-IID Data in Wireless Networks

Ensemble Federated Learning With Non-IID Data in Wireless Networks

DRL-empowered joint batch size and weighted aggregation adjustment mechanism for federated learning on non-IID data

To address the accuracy degradation as well as prolonged convergence time due to the inherent data heterogeneity among end-devices in federated learning (FL), we introduce the joint batch size and weighted aggregation adjustment problem, which is non-convex problem. To adjust optimal hyperparameters, we develop deep reinforcement learning (DRL) to empower a mechanism known as Batch size and Weighted aggregation Adjustment (BWA). Experimental evaluation demonstrates that BWA not only outperforms methods optimized solely from either a local training or server perspective but also achieves higher accuracy, with an increase of up to 5.53% compared to FedAvg, and additionally accelerates convergence speeds.

Joint Client Scheduling and Wireless Resource Allocation for Heterogeneous Federated Edge Learning With Non-IID Data

Joint Client Scheduling and Wireless Resource Allocation for Heterogeneous Federated Edge Learning With Non-IID Data

Analyzing the robustness of decentralized horizontal and vertical federated learning architectures in a non-IID scenario

Federated learning (FL) enables participants to collaboratively train machine and deep learning models while safeguarding data privacy. However, the FL paradigm still has drawbacks that affect its trustworthiness, as malicious participants could launch adversarial attacks against the training process. Previous research has examined the robustness of horizontal FL scenarios under various attacks. However, there is a lack of research evaluating the robustness of decentralized vertical FL and comparing it with horizontal FL architectures affected by adversarial attacks. Therefore, this study proposes three decentralized FL architectures: HoriChain, VertiChain, and VertiComb. These architectures feature different neural networks and training protocols suitable for horizontal and vertical scenarios. Subsequently, a decentralized, privacy-preserving, and federated use case with non-IID data to classify handwritten digits is deployed to assess the performance of the three architectures. Finally, a series of experiments computes and compares the robustness of the proposed architectures when they are affected by different data poisoning methods, including image watermarks and gradient poisoning adversarial attacks. The experiments demonstrate that while specific configurations of both attacks can undermine the classification performance of the architectures, HoriChain is the most robust one.

Personalized federated learning for dealing with computational latency and non-IID data scenarios

Federated learning (FL) is widely used because it is effective at enhancing data privacy. However, there will be many problems in the FL training process, such as poor performance of training models and the model converging too slowly, as the data is typically heterogeneous and the computing capabilities of the participants device are different. Here, we proposed an optimized FL model paradigm, that applies model arithmetic prediction to prevent the training process's inefficiency due to the participants' limited computational resources. The proposed formula for participant selection is based on posterior probabilities and correlation coefficients, which have been validated to reduce data noise and enhance the effect of central model aggregation. In addition, high-quality participant models are selected based on posterior probability, combined with correlation coefficients, which allows the server model to aggregate as many better-performing participant models as possible, meanwhile avoiding the impact of participants with too much data noise. During the aggregation step, the model loss values and the participant training delay are used to weight factors for participant devices, which accelerates FL convergence and improves model performance. Data heterogeneity and non-IID are fully taken into consideration in the method we proposed. Finally, these results have been verified by extensive experimental, we demonstrate better performance in the presence of non-IID data, especially affective computing. Compared with previous research, reduces training latency by 4 seconds, and the model accuracy is increased by 10% on average.

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

Federated learning has emerged as a promising approach for collaborative model training across distributed devices. Federated learning faces challenges such as Non-Independent and Identically Distributed (non-IID) data and communication challenges. This study aims to provide in-depth knowledge in the federated learning environment by identifying the most used techniques for overcoming non-IID data challenges and techniques that provide communication-efficient solutions in federated learning. The study highlights the most used non-IID data types, learning models, and datasets in federated learning. A systematic mapping study was performed using six digital libraries, and 193 studies were identified and analyzed after the inclusion and exclusion criteria were applied. We identified that enhancing the aggregation method and clustering are the most widely used techniques for non-IID data problems (used in 18% and 16% of the selected studies), and a quantization technique was the most common technique in studies that provide communication-efficient solutions in federated learning (used in 27% and 15% of the selected studies). Additionally, our work shows that label distribution skew is the most used case to simulate a non-IID environment, specifically, the quantity label imbalance. The supervised learning model CNN model is the most commonly used learning model, and the image datasets MNIST and Cifar-10 are the most widely used datasets when evaluating the proposed approaches. Furthermore, we believe the research community needs to consider the client’s limited resources and the importance of their updates when addressing non-IID and communication challenges to prevent the loss of valuable and unique information. The outcome of this systematic study will benefit federated learning users, researchers, and providers.

Federated Learning Enabled IDS for Internet of Things on non-IID Data

Federated Learning Enabled IDS for Internet of Things on non-IID Data

Exploiting Label Skews in Federated Learning with Model Concatenation

Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data. However, non-IID data has been a key challenge in FL, which could significantly degrade the accuracy of the final model. Among different non-IID types, label skews have been challenging and common in image classification and other tasks. Instead of averaging the local models in most previous studies, we propose FedConcat, a simple and effective approach that concatenates these local models as the base of the global model to effectively aggregate the local knowledge. To reduce the size of the global model, we adopt the clustering technique to group the clients by their label distributions and collaboratively train a model inside each cluster. We theoretically analyze the advantage of concatenation over averaging by analyzing the information bottleneck of deep neural networks. Experimental results demonstrate that FedConcat achieves significantly higher accuracy than previous state-of-the-art FL methods in various heterogeneous label skew distribution settings and meanwhile has lower communication costs. Our code is publicly available at https://github.com/sjtudyq/FedConcat.