Distributed Machine Learning Framework Research Articles

FedUB: Federated Learning Algorithm Based on Update Bias

Federated learning, as a distributed machine learning framework, aims to protect data privacy while addressing the issue of data silos by collaboratively training models across multiple clients. However, a significant challenge to federated learning arises from the non-independent and identically distributed (non-iid) nature of data across different clients. non-iid data can lead to inconsistencies between the minimal loss experienced by individual clients and the global loss observed after the central server aggregates the local models, affecting the model’s convergence speed and generalization capability. To address this challenge, we propose a novel federated learning algorithm based on update bias (FedUB). Unlike traditional federated learning approaches such as FedAvg and FedProx, which independently update model parameters on each client before direct aggregation to form a global model, the FedUB algorithm incorporates an update bias in the loss function of local models—specifically, the difference between each round’s local model updates and the global model updates. This design aims to reduce discrepancies between local and global updates, thus aligning the parameters of locally updated models more closely with those of the globally aggregated model, thereby mitigating the fundamental conflict between local and global optima. Additionally, during the aggregation phase at the server side, we introduce a metric called the bias metric, which assesses the similarity between each client’s local model and the global model. This metric adaptively sets the weight of each client during aggregation after each training round to achieve a better global model. Extensive experiments conducted on multiple datasets have confirmed the effectiveness of the FedUB algorithm. The results indicate that FedUB generally outperforms methods such as FedDC, FedDyn, and Scaffold, especially in scenarios involving partial client participation and non-iid data distributions. It demonstrates superior performance and faster convergence in tasks such as image classification.

Privacy preserving and secure robust federated learning: A survey

SummaryFederated learning (FL) has emerged as a promising solution to address the challenges posed by data silos and the need for global data fusion. It offers a distributed machine learning framework with privacy‐preserving features, allowing model training without the need to collect user data. However, FL also presents significant security and privacy threats that hinder its widespread adoption. The requirements of privacy and security in FL are inherently conflicting. Privacy necessitates the concealment of individual client updates, while security requires the disclosure of client updates to detect anomalies. While most existing research focused on the privacy and security aspects of FL, very few studies have addressed the compatibility of these two demands. In this work, we aim to bridge this gap by proposing a comprehensive defense scheme that ensures privacy, security, and compatibility in FL. We categorize the existing literature into two key directions: privacy defense and security defense. Privacy defense includes methods based on additive masks, differential privacy, homomorphic encryption, and trusted execution environment, whereas security defense encompasses distance‐, performance‐, clustering‐, and similarity‐based anomaly detection techniques and statistical information‐based anomaly update bypassing techniques when the server is trusted and privacy‐compatible anomaly update detection techniques when the server is not trusted. In addition, this article presents decentralized FL solutions based on blockchain. For each direction, we discuss specific technical solutions, their advantages, and disadvantages. By evaluating various defense methods, we identify the most suitable approach to address the primary challenge of “achieving a secure and robust FL system against malicious adversaries while protecting users' privacy.” We then propose a theoretical reference framework for end‐to‐end protection of privacy and security in FL for the key problem, which summarizes the attack surface of FL systems from the client to the server under the security model where the client and server are malicious. Leveraging the strengths and characteristics of existing schemes, our proposed framework integrates multiple techniques to strike a balance between privacy, usability, and efficiency. This framework serves as a valuable reference and provides insights for future work in the field. Finally, we also provide recommendations for future research directions in this field.

A Novel Federated Learning Framework Based on Conditional Generative Adversarial Networks for Privacy Preserving in 6G

With the rapid development of 6G networks, data transmission speed has significantly increased, making data privacy protection issues even more crucial. The federated learning (FL) is a distributed machine learning framework with privacy protection and secure encryption technology, aimed at enabling dispersed participants to collaborate on model training without disclosing private data to other participants. Nonetheless, recent research indicates that the exchange of shared gradients may lead to information disclosure, and thus FL still needs to address privacy concerns. Additionally, FL relies on a large number of diverse training data to forge efficient models, but in reality, the training data available to clients are limited, and data imbalance issues lead to over fitting in existing federated learning models. To alleviate these issues, we introduce a Novel Federated Learning Framework based on Conditional Generative Adversarial Networks (NFL-CGAN). NFL-CGAN divides the local networks of each client into private and public modules. The private module contains an extractor and a discriminator to protect privacy by retaining them locally. Conversely, the public module is shared with the server to aggregate the shared knowledge of clients, thereby improving the performance of each client local network. Comprehensive experimental analyses demonstrate that NFL-CGAN surpasses traditional FL baseline methods in data classification, showcasing its superior efficacy. Moreover, privacy assessments also verified robust and reliable privacy protection capabilities of NFL-CGAN.

Privacy‐preserving task offloading in mobile edge computing: A deep reinforcement learning approach

AbstractAs machine learning (ML) technologies continue to evolve, there is an increasing demand for data. Mobile crowd sensing (MCS) can motivate more users in the data collection process through reasonable compensation, which can enrich the data scale and coverage. However, nowadays, users are increasingly concerned about their privacy and are unwilling to easily share their personal data. Therefore, protecting privacy has become a crucial issue. In ML, federated learning (FL) is a widely known privacy‐preserving technique where the model training process is performed locally by the data owner, which can protect privacy to a large extent. However, as the model size grows, the weak computing power and battery life of user devices are not sufficient to support training a large number of models locally. With mobile edge computing (MEC), user can offload some of the model training tasks to the edge server for collaborative computation, allowing the edge server to participate in the model training process to improve training efficiency. However, edge servers are not fully trusted, and there is still a risk of privacy leakage if data is directly uploaded to the edge server. To address this issue, we design a local differential privacy (LDP) based data privacy‐preserving algorithm and a deep reinforcement learning (DRL) based task offloading algorithm. We also propose a privacy‐preserving distributed ML framework for MEC and model the cloud‐edge‐mobile collaborative training process. These algorithms not only enable effective utilization of edge computing to accelerate machine learning model training but also significantly enhance user privacy and save device battery power. We have conducted experiments to verify the effectiveness of the framework and algorithms.

Clustered Federated Learning in Heterogeneous Environment.

Federated learning (FL) is a distributed machine learning framework that allows resource-constrained clients to train a global model jointly without compromising data privacy. Although FL is widely adopted, high degrees of systems and statistical heterogeneity are still two main challenges, which leads to potential divergence and nonconvergence. Clustered FL handles the problem of statistical heterogeneity straightly by discovering the geometric structure of clients with various data generation distributions and getting multiple global models. The number of clusters contains prior knowledge about the clustering structure and has a significant impact on the performance of clustered FL methods. Existing clustered FL methods are inadequate for adaptively inferring the optimal number of clusters in environments with high systems' heterogeneity. To address this issue, we propose an iterative clustered FL (ICFL) framework in which the server dynamically discovers the clustering structure by successively performing incremental clustering and clustering in one iteration. We focus on the average connectivity within each cluster and give incremental clustering and clustering methods that are compatible with ICFL based on mathematical analysis. We evaluate ICFL in experiments on high degrees of systems and statistical heterogeneity, multiple datasets, and convex and nonconvex objectives. Experimental results verify our theoretical analysis and show that ICFL outperforms several clustered FL baseline methods.

CRS: A Privacy-Preserving Two-Layered Distributed Machine Learning Framework for IoV

Nowadays, vehicles can provide many valuable data (such as the videos recorded by dashcams) for analytical model building. Integrating vehicular ad-hoc networks with Internet of Things (IoT), Internet of Vehicles (IoV) has a promising future. In IoV, vehicles maintain their own communication, computing, and learning capabilities. Thus, instead of sending the data to a central server for model training, which leads to a high communication overhead, vehicles can train the data locally. However, it is still a challenge to preserve the privacy while keeping both the communication and computation overheads of vehicles acceptable. In this paper, we present a distributed machine learning framework with a two-layered architecture. The architecture uniquely involves vehicle clusters, road-side units, and a central server, which provides a basic guarantee to the vehicle privacy and also limits the overhead. By carefully adopting cryptographic tools and techniques, the framework has the following properties: 1) it preserves the privacy of the local inputs and model weight vectors to all parties; 2) it protects the identities and trajectories of vehicles; 3) packet loss is handled in the application layer; 4) the evaluation shows that it is lightweight for vehicles. Compared with other existing works, the proposed framework is more suitable for IoV.

Secure and efficient multi-key aggregation for federated learning

Federated learning (FL) is a distributed machine learning framework that aims to provide privacy for local datasets while learning a global machine learning model. However, the updates exchanged in FL may indirectly reveal information about the local training datasets. To protect the confidentiality of updates, various solutions using cryptographic schemes such as secret sharing, differential privacy, and homomorphic encryption have been developed. The solutions using secret sharing often have high communication costs, while those using differential privacy require a trade-off between accuracy and privacy. Homomorphic encryption has been used to address these challenges to reduce communication costs and provide provable security. However, existing solutions based on homomorphic encryption require clients to use the same public-private key pair, which may lead to local updates disclosure. To address the existing challenges, we propose a secure and efficient multi-key aggregation protocol (MKAgg) that utilizes homomorphic encryption. The protocol allows clients to drop out at any point during the process. We construct MKAgg based on a two-server model and adopt a proxy re-encryption scheme with additively homomorphic properties to implement secure and efficient ciphertext transformation and calculation. We provide security proof to demonstrate that our MKAgg protocol meets the required security standards. Furthermore, we perform an efficiency analysis of MKAgg and evaluate its performance on various datasets. The results affirm that MKAgg is both effective and efficient for aggregation in the multi-key setting. We then apply MKAgg in FL and develop a multi-key privacy-preserving neural network scheme called MKPNFL. We analyze the security of MKPNFL and conduct tests using real-world datasets. The results demonstrate that MKPNFL is secure and practical for real-world applications.

FedRight: An effective model copyright protection for federated learning

Federated learning (FL), an effective distributed machine learning framework, implements model training and meanwhile protects local data privacy. It has been applied to a broad variety of practical areas due to their great performance and appreciable profits. Who really owns the model, and how to protect the copyright has become a real problem. Intuitively, the existing property rights protection methods in centralized scenarios (e.g., watermark embedding and model fingerprints) are possible solutions for FL. But they are still challenged by the distributed nature of FL in aspects of the no data sharing, parameter aggregation, and federated training settings. For the first time we formalize the problem of copyright protection for FL, and propose FedRight to protect model copyright based on model fingerprints, i.e., extracting model features by generating adversarial examples as model fingerprints. FedRight outperforms previous works in four key aspects: (i) Validity - it extracts model features to generate transferable fingerprints to train a detector to verify the copyright of the model. (ii) Fidelity - it is with imperceptible impact on the federated training, thus promises good main task performance. (iii) Robustness - it is empirically robust against malicious attack on copyright protection, i.e., fine-tuning, model pruning and adaptive attacks. (iv) Black-box - it is valid in black-box forensic scenario where only application programming interface calls to the model are available. Extensive evaluations across 3 datasets and 9 model structures demonstrate FedRight's superior fidelity, validity and robustness.

Federated Optimization Under Intermittent Client Availability

Federated learning is a new distributed machine learning framework, where numerous heterogeneous clients collaboratively train a model without sharing training data. In this work, we consider a practical and ubiquitous issue when deploying federated learning in mobile environments: intermittent client availability, where the set of eligible clients may change during the training process. Such intermittent client availability would seriously deteriorate the performance of the classical federated averaging algorithm (FedAvg). Thus, we propose a simple distributed nonconvex optimization algorithm, called federated latest averaging (FedLaAvg), which leverages the latest gradients of all clients, even when the clients are not available, to jointly update the global model in each iteration. Our theoretical analysis shows that FedLaAvg achieves guaranteed convergence and a sublinear speedup with respect to the total number of clients. We implement FedLaAvg along with several baselines and evaluate them over the benchmarking MNIST and Sentiment140 data sets. The evaluation results demonstrate that FedLaAvg achieves more stable training than FedAvg in both convex and nonconvex settings and reaches a sublinear speedup. Source code and online supplement are available at the IJOC GitHub site ( http://dx.doi.org/10.1287/ijoc.2022.0057.cd , https://github.com/INFORMSJoC/2022.0057 ). History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Leaning. Funding: This work was supported by the National Key R&D Program of China [Grant 2022ZD0119100], the National Natural Science Foundation of China (NSFC) [Grants 61972252, 61972254, 62072303, 62025204, 62132018, 62202296, and 62202297], the Alibaba Innovation Research (AIR) Program, and the Tencent Rhino Bird Key Research Project. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0057 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0057 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Data heterogeneous federated learning algorithm for industrial entity extraction

Entity extraction is an important part to realize digital transformation in the industrial field. Building an entity extraction model in the industrial field requires a lot of data. The parties in industry often cannot share data due to commercial competition and security and privacy issues, thus forming “Data Island”. Federated learning provides a solution to this problem. Federated learning is a distributed machine learning framework that allows each party to train locally and independently using their own private data. The model parameters or gradient information of each party will be aggregated to the central server, thus forming a model jointly trained by all parties. This approach can not only protect the security and privacy of data from all parties, but also fully utilize their data resources. Federated learning can effectively solve the problem of data island, but it still faces some problems and challenges, among which the most typical problem is data heterogeneity. To address the data islanding problem and data heterogeneity problem faced by industrial entity extraction, this paper uses a federated learning framework to solve the data islanding problem and proposes the FedDP algorithm. This algorithm assigns weights based on the data quality performance of each participant. Participants with relatively good data quality performance have higher weights in the aggregation stage, while participants with relatively poor data quality performance have lower weights in the aggregation stage, thus optimizing the performance of federated learning in heterogeneous data scenarios.

Communication-Efficient and Privacy-Preserving Verifiable Aggregation for Federated Learning.

Federated learning is a distributed machine learning framework, which allows users to save data locally for training without sharing data. Users send the trained local model to the server for aggregation. However, untrusted servers may infer users' private information from the provided data and mistakenly execute aggregation protocols to forge aggregation results. In order to ensure the reliability of the federated learning scheme, we must protect the privacy of users' information and ensure the integrity of the aggregation results. This paper proposes an effective secure aggregation verifiable federated learning scheme, which has both high communication efficiency and privacy protection function. The scheme encrypts the gradients with a single mask technology to securely aggregate gradients, thus ensuring that malicious servers cannot deduce users' private information from the provided data. Then the masked gradients are hashed to verify the aggregation results. The experimental results show that our protocol is more suited for bandwidth-constraint and offline-users scenarios.

FedRAD: Heterogeneous Federated Learning via Relational Adaptive Distillation.

As the development of the Internet of Things (IoT) continues, Federated Learning (FL) is gaining popularity as a distributed machine learning framework that does not compromise the data privacy of each participant. However, the data held by enterprises and factories in the IoT often have different distribution properties (Non-IID), leading to poor results in their federated learning. This problem causes clients to forget about global knowledge during their local training phase and then tends to slow convergence and degrades accuracy. In this work, we propose a method named FedRAD, which is based on relational knowledge distillation that further enhances the mining of high-quality global knowledge by local models from a higher-dimensional perspective during their local training phase to better retain global knowledge and avoid forgetting. At the same time, we devise an entropy-wise adaptive weights module (EWAW) to better regulate the proportion of loss in single-sample knowledge distillation versus relational knowledge distillation so that students can weigh losses based on predicted entropy and learn global knowledge more effectively. A series of experiments on CIFAR10 and CIFAR100 show that FedRAD has better performance in terms of convergence speed and classification accuracy compared to other advanced FL methods.

Federated Learning Incentive Mechanism Design via Shapley Value and Pareto Optimality

Federated learning (FL) is a distributed machine learning framework that can effectively help multiple players to use data to train federated models while complying with their privacy, data security, and government regulations. Due to federated model training, an accurate model should be trained, and all federated players should actively participate. Therefore, it is crucial to design an incentive mechanism; however, there is a conflict between fairness and Pareto efficiency in the incentive mechanism. In this paper, we propose an incentive mechanism via the combination of the Shapley value and Pareto efficiency optimization, in which a third party is introduced to supervise the federated payoff allocation. If the payoff can reach Pareto optimality, the federated payoff is allocated by the Shapley value method; otherwise, the relevant federated players are punished. Numerical and simulation experiments show that the mechanism can achieve fair payoff allocation and Pareto optimality payoff allocation. The Nash equilibrium of this mechanism is formed when Pareto optimality payoff allocation is achieved.

Configure Your Federation: Hierarchical Attention-enhanced Meta-Learning Network for Personalized Federated Learning

Federated learning, as a distributed machine learning framework, enables clients to conduct model training without transmitting their data to the server, which is used to solve the dilemma of data silos and data privacy. It can work well on clients having similar data characteristics and distribution. However, it has some limitations where the dataset of clients may be different in distribution, quantity, and concept in many application scenarios. Personalized federated learning is a new federated learning paradigm that aims to guarantee client personalized models’ effectiveness when collaborating with the cloud server. Intuitively, providing further facilitated collaborations for the clients with similar data characteristics and distribution can benefit personalized model building. However, due to the invisibility of client data, it is challenging to extract client characteristics and define collaborative relationships among them from a fine-grained view. Moreover, a reasonable collaborative training approach needs to be designed for a distributed server–client framework. In this article, we design a Hierarchical Attention-enhanced Meta-learning Network (HAM) to address this issue. The main advantage of HAM is that it utilizes the meta-learning approach of taking model parameters as features and learns to learn an extra model for each client to analyze similarities according to their local dataset automatically. According to its two-layers framework, HAM can reasonably achieve a tradeoff between clients’ personality and commonality and provides a hybrid model with useful information from all clients. Considering there are two networks (HAM and base network) that need to learn for each client during the federated training process, we then provide an alternative learning approach to train them in an end-to-end fashion. To further clarify the approach, we describe the personalized federated learning settings framework as FedHAM where the HAM network is distributed deployed in each client. Extensive experiments based on two datasets prove that our method outperforms state-of-the-art baselines under different evaluation metrics.

UAV-Assisted Online Machine Learning Over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach

We investigate training machine learning (ML) models across a set of geo-distributed, resource-constrained clusters of devices through unmanned aerial vehicles (UAV) swarms. The presence of time-varying data heterogeneity and computational resource inadequacy among device clusters motivate four key parts of our methodology: (i) stratified UAV swarms of leader, worker, and coordinator UAVs, (ii) hierarchical nested personalized federated learning ( HN-PFL), a distributed ML framework for personalized model training across the worker-leader-core network hierarchy, (iii) cooperative UAV resource pooling to address computational inadequacy of devices by conducting model training among the UAV swarms, and (iv) model/concept drift to model time-varying data distributions. In doing so, we consider both micro (i.e., UAV-level) and macro (i.e., swarm-level) system design. At the micro-level, we propose network-aware HN-PFL, where we distributively orchestrate UAVs inside swarms to optimize energy consumption and ML model performance with performance guarantees. At the macro-level, we focus on swarm trajectory and learning duration design, which we formulate as a sequential decision making problem tackled via deep reinforcement learning. Our simulations demonstrate the improvements achieved by our methodology in terms of ML performance, network resource savings, and swarm trajectory efficiency.

PPVerifier: A Privacy-Preserving and Verifiable Federated Learning Method in Cloud-Edge Collaborative Computing Environment

Federated Learning is a distributed machine learning framework, which mainly adopts cloud-edge collaborative computing mode and supports multiple participants to train models without directly sharing local data. However, participants’ sensitive information may still be leaked through their gradients. Besides, incorrect aggregated results returned by the aggregation server may reduce the effect of joint modeling. This paper proposes a privacy-preserving and verifiable federated learning method called PPVerifier to support privacy protection and verification of aggregated results in cloud-edge collaborative computing environment. By integrating Paillier homomorphic encryption and random number generation technique, all gradients and their ciphertexts can be protected. Meanwhile, an additive secret sharing scheme is introduced to resist potential collusion attacks among the aggregation server, malicious participants, and edge nodes. Moreover, a verification scheme based on discrete logarithm is proposed, which can not only verify the correctness of aggregated results, but also discover lazy aggregation servers, and the verification overhead can be reduced by over half compared with bilinear aggregate signature method. Finally, theoretical analysis and experiments conducted on MNIST dataset prove that our proposed method can achieve gradient protection and correctness verification of the aggregated results with higher efficiency.

AsyFed: Accelerated Federated Learning With Asynchronous Communication Mechanism

As a new distributed machine learning framework for privacy protection, Federated Learning (FL) enables substantial Internet of Things (IoT) devices (e.g., mobile phones, tablets, etc.) to participate in collaborative training of a machine learning model. FL can protect the data privacy of IoT devices without exposing their raw data. However, the diversity of IoT devices may degrade the overall training process due to the straggler issue. To tackle this problem, we propose a gear-based asynchronous federated learning (AsyFed) architecture. It adds a gear layer between the clients and the FL server as a mediator to store the model parameters. The key insight is that we group these clients with similar training abilities into the same gear. The clients within the same gear conduct synchronous training. These gears then communicate with the global FL server asynchronously. Besides, we propose a T-step mechanism to reduce the weight from the slow gear when they are communicating with the FL server. Extensive experiment evaluations indicate that AsyFed outperforms FedAvg (baseline synchronous FL scheme) and some state-of-the-art asynchronous FL methods in terms of training accuracy or speed under different data distributions. The only negligible overhead is that we leverage the extra layer (gear layer) to preserve part of the model parameters.

Adaptive client and communication optimizations in Federated Learning

Federated Learning (FL) is a very effective distributed machine learning framework that enables a large number of devices to jointly train models without sharing raw data. However, the process of iterative learning and data communication in FL can be time-consuming, depending heavily on the number of clients participating in training and the number of local iterations between two consecutive global aggregations (communication period). In this paper, we analyze the runtime of the FL framework and propose a joint optimization problem, which considers both the number of local clients involved in global training and the communication period to minimize the error-runtime convergence of FedAvg. By theoretical analysis, we obtain the optimal solutions of this optimization problem in closed forms under the assumption of different probability distributions of the client’s local computation time. In addition, we design an adaptive control algorithm that can dynamically select the number of clients and communication period during FL training to speed up the convergence of the model. Experimental results validate our theoretical analysis and show that the proposed algorithm has outstanding performance compared with other related FL control algorithms.

Communication Analysis and Privacy in CAI Based on Data Mining and Federated Learning

Due to the fact that client data do not need to leave the local area, a distributed machine learning framework can aggregate training from several clients while preserving data privacy. In this essay, the development of CAI both domestically and internationally is reviewed and summarized, and the current state of CAI is examined. Communication analysis has so far been a key academic and theoretical area in federated learning, and some theoretical contributions have become the crucial theoretical foundations for understanding, defending, and guiding various human social behaviors. The major objective of knowledge distillation based on model responses is to provide students the ability to rapidly replicate the teacher’s model’s output. The experimental results demonstrate that the optimized Smith Regan model adopts the “Smith Logan” teaching design model, selects the courseware structure and record preservation as the teaching content in the fundamental CAI courseware design, and optimizes the teaching design from the perspectives of learning environment analysis, learner characteristics analysis, etc. Based on this, the model’s accuracy and robustness are increased by 7.34%.

Hierarchical federated learning with mobile edge computing in the Internet of Vehicles

Federated Learning is a distributed machine learning framework, which can be used in the Internet of Vehicles to train deep learning models without directly accessing the original data of mobile edge vehicle nodes. ECS can access massive data, but it has the characteristics of high latency and high communication overhead. However, mobile edge computing (MEC) platform can directly and efficiently communicate with mobile edge vehicle nodes. Combining the advantages of the two, a three-layer federated learning system of edge car network edge server cloud server is used. This system is supported by the HierFedProx algorithm and aggregates the model output of the edge car to the edge server to improve the model learning efficiency and reduce the global communication frequency. The experimental results show that the system can reduce the training time and improve the accuracy of the model compared with the federated learning without introducing the edge server.