Distributed Learning Method Research Articles

RNEP: Random Node Entropy Pairing for Efficient Decentralized Training with Non-IID Local Data

The proliferation of edge devices and advancements in Internet of Things technology have created a vast array of distributed data sources, necessitating machine learning models that can operate closer to the point of data generation. Traditional centralized machine learning approaches often struggle with real-time big data applications, such as climate prediction and traffic simulation, owing to high communication costs. In this study, we introduce random node entropy pairing, a novel distributed learning method for artificial neural networks tailored to distributed computing environments. This method reduces communication overhead and addresses data imbalance by enabling nodes to exchange only the weights of their local models with randomly selected peers during each communication round, rather than sharing the entire dataset. Our findings indicate that this approach significantly reduces communication costs while maintaining accuracy, even when learning from non-IID local data. Furthermore, we explored additional learning strategies that enhance system accuracy by leveraging the characteristics of this method. The results demonstrate that random node entropy pairing is an effective and efficient solution for distributed learning in environments where communication costs and data distribution present significant challenges.

An Overview of Data Contribution Evaluation Methods for Federated Learning

With the rapid development of artificial intelligence, especially machine learning technology, the demand for data is also increasing. As a distributed learning method, federated learning (FL) allows total multiple parties to co-train models locally while protecting data privacy. Due to heterogeneity in data, devices, and other factors, the contribution of participants to model quality also varies. If these differences are ignored, it poses a challenge to achieving a fair FL environment. Therefore, fairly assessing the contribution levels of each data provider becomes a critical issue in FL. This plays a significant role in fairly motivating participants and promoting the sustainable development of FL. This paper reviews existing evaluation methods from the perspective of data contribution assessment in FL, discusses ongoing challenges in this area, and explores future research directions.

Challenges and Countermeasures of Federated Learning Data Poisoning Attack Situation Prediction

Federated learning is a distributed learning method used to solve data silos and privacy protection in machine learning, aiming to train global models together via multiple clients without sharing data. However, federated learning itself introduces certain security threats, which pose significant challenges in its practical applications. This article focuses on the common security risks of data poisoning during the training phase of federated learning clients. First, the definition of federated learning, attack types, data poisoning methods, privacy protection technology and data security situational awareness are summarized. Secondly, the system architecture fragility, communication efficiency shortcomings, computing resource consumption and situation prediction robustness of federated learning are analyzed, and related issues that affect the detection of data poisoning attacks are pointed out. Thirdly, a review is provided from the aspects of building a trusted federation, optimizing communication efficiency, improving computing power technology and personalized the federation. Finally, the research hotspots of the federated learning data poisoning attack situation prediction are prospected.

Efficient knowledge management for heterogeneous federated continual learning on resource-constrained edge devices

Federated learning (FL) is a promising and privacy-preserving distributed learning method that is widely deployed on edge devices. However, in practical applications, the data collected by edge devices exhibits temporal variations. This leads to the problem of FL models adapting to new data while forgetting knowledge from old data, resulting in a catastrophic forgetting issue. Continual learning methods can be used to address this problem. However, when deploying these methods in FL on edge devices, it is challenging to adapt to the limited resources and heterogeneous data of the deployed devices, which reduces the efficiency and effectiveness of federated continual learning (FCL). Therefore, this article proposes a resource-efficient heterogeneous FCL framework. This framework divides the global model into an adaptation part for new knowledge and a preservation part for old knowledge. The preservation part is used to address the catastrophic forgetting problem. Only the adaptation part is trained when learning new knowledge on a new task, reducing resource consumption. Additionally, the framework mitigates the impact of heterogeneous data through an aggregation method based on feature representation. Experimental results show that our method performs well in mitigating catastrophic forgetting in a resource-efficient manner.

Semantic labeling of social big media using distributed online robust classification

Semantic labeling for image datasets is of significant importance in a wide range of social media. However, social datasets with massive amounts of data require effective technologies to increase the quality of classification. In this study, we propose a novel online robust classification using distributed learning method, in which the diffusion method is used over adaptive networks in order to the parallelization of the training process. The loss function of the suggested method is derived from the logarithm cosine function, being more suitable for impulsive noises. Also, the convergence of the proposed method is discussed theoretically. The Extensive experimental results show that the proposed method outperforms the other robust state-of-the-art classification algorithms in the presence of various scenarios, such as: free noise synthetic data sets, pure Gaussian noise, mixture of two Gaussian noises, impulse noise, and Alpha–Beta noise which are added to the synthetic data sets. In addition, the experiments were repeated for two different types of real data sets with real noises and outliers, i.e., UCI and 500PX social media data sets. The obtained results over UCI and 500px data sets on social media with real outliers and mislabeled samples confirmed the acceptable performance of the proposed method.

FedCSCD-GAN: A secure and collaborative framework for clinical cancer diagnosis via optimized federated learning and GAN

Digital technologies present unrivaled opportunities to improve healthcare services worldwide. Medical devices and hospitals are now using innovative techniques to diagnose cancer patients. Despite the vast amount of data generated, stored, and communicated to the cloud and edge devices, patient data privacy remains a crucial concern. Federated learning (FL) is a revolutionary distributed learning method with significant potential in medical data processing. However, data privacy, data quality, and model performance issues can make it challenging to develop a resilient model. As a solution to the above-mentioned challenges, a secure and collaborative FedCSCD-GAN framework is proposed for clinical cancer diagnosis under the federated learning and GAN optimization principle. By leveraging the collective intelligence of distributed data sources, this framework seeks to improve cancer diagnosis accuracy while maintaining appropriate security measures for sensitive patient data. In the proposed system, the quasi-identifiers (QIden) are identifiedas independent QIden attributes in health data, while the others areclassified as confidential information (CI). Consequently, to enhance security and privacy f- differential privacy anonymization is performed on the QIden attributes, and the resulting data is mixed with the CI attribute. Using anonymized data, the Cramer GAN was trained using Cramer distance for efficiency, and privacy was assessed. Notably, the suggested architecture obtains diagnosis accuracy of 97.80 % for lung cancer, 96.95 % for prostate cancer, and 97 % for breast cancer. Based on the experimental analysis, the proposed architecture detects cancer accurately and addresses security and collaboration issues. This paradigm has the potential to transform healthcare and improve patient outcomes globally.

FLGQM: Robust Federated Learning Based on Geometric and Qualitative Metrics

Federated learning is a distributed learning method that seeks to train a shared global model by aggregating contributions from multiple clients. This method ensures that each client’s local data are not shared with others. However, research has revealed that federated learning is vulnerable to poisoning attacks launched by compromised or malicious clients. Many defense mechanisms have been proposed to mitigate the impact of poisoning attacks, but there are still some limitations and challenges. The defense methods are either performing malicious model removal from the geometric perspective to measure the geometric direction of the model or adding an additional dataset to the server for verifying local models. The former is prone to failure when facing advanced poisoning attacks, while the latter goes against the original intention of federated learning as it requires an independent dataset; thus, both of these defense methods have some limitations. To solve the above problems, we propose a robust federated learning method based on geometric and qualitative metrics (FLGQM). Specifically, FLGQM aims to metricize local models in both geometric and qualitative aspects for comprehensive defense. Firstly, FLGQM evaluates all local models from both direction and size aspects based on similarity calculated by cosine and the Euclidean distance, which we refer to as geometric metrics. Next, we introduce a union client set to assess the quality of all local models by utilizing the union client’s local dataset, referred to as quality metrics. By combining the results of these two metrics, FLGQM is able to use information from multiple views for accurate poisoning attack identification. We conducted experimental evaluations of FLGQM using the MNIST and CIFAR-10 datasets. The experimental results demonstrate that, under different kinds of poisoning attacks, FLGQM can achieve similar performance to FedAvg in non-adversarial environments. Therefore, FLGQM has better robustness and poisoning attack defense performance.

Personalized Fair Split Learning for Resource-Constrained Internet of Things.

With the flourishing development of the Internet of Things (IoT), federated learning has garnered significant attention as a distributed learning method aimed at preserving the privacy of participant data. However, certain IoT devices, such as sensors, face challenges in effectively employing conventional federated learning approaches due to limited computational and storage resources, which hinder their ability to train complex local models. Additionally, in IoT environments, devices often face problems of data heterogeneity and uneven benefit distribution between them. To address these challenges, a personalized and fair split learning framework is proposed for resource-constrained clients. This framework first adopts a U-shaped structure, dividing the model to enable resource-constrained clients to offload subsets of the foundational model to a central server while retaining personalized model subsets locally to meet the specific personalized requirements of different clients. Furthermore, to ensure fair benefit distribution, a model-aggregation method with optimized aggregation weights is used. This method reasonably allocates model-aggregation weights based on the contributions of clients, thereby achieving collaborative fairness. Experimental results demonstrate that, in three distinct data heterogeneity scenarios, employing personalized training through this framework exhibits higher accuracy compared to existing baseline methods. Simultaneously, the framework ensures collaborative fairness, fostering a more balanced and sustainable cooperation among IoT devices.

Fed-Inforce-Fusion: A federated reinforcement-based fusion model for security and privacy protection of IoMT networks against cyber-attacks

Internet of Medical Things (IoMT) has emerged as a combination of sensors, healthcare devices, and Internet of Things (IoT) to deliver better and intelligent healthcare services. However, security and privacy protection issues have hindered its wide integration resulting in lack of quality IoMT dataset. Federated Learning (FL) is an emerging distributed learning method that can be helpful for IoMT networks and smart healthcare systems (SHS) due to enhanced security and privacy. The use of intrusion detection systems (IDS) has become vital for contemporary critical networks to protect against evolving cyber-attacks. Existing security models are often challenging and computationally expensive to apply to resource-limited medical IoT devices. To this end, this research work proposes a privacy-preserving FL-based IDS model named Fed-Inforce-Fusion to identify cyber-attacks from IoMT networks. Specifically, the proposed Fed-Inforce-Fusion model uses reinforcement learning technique to learn the latent relationships of medical data. Next, a FL-based system is designed allowing distributed SHS nodes to collaboratively train a comprehensive IDS model in a privacy preserving manner. Then, a fusion/aggregation strategy is adopted to improve the performance of detection model and reduce communication overhead by allowing participating clients to involve in the federation process dynamically. Theoretical study and experiment analysis show that the proposed Fed-Inforce-Fusion is better and effective in detecting complex attack vectors in comparison to existing benchmark IDS methods. Thus, proposing its efficacy and suitability in real-world IoMT networks.

Membership Mappings for Practical Secure Distributed Deep Learning

<underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Problem</u> : We consider the problem of privacy-preserving distributed deep learning where data privacy is protected by fully homomorphic encryption. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Aim</u> : The aim is to develop a method for practical and scalable distributed deep learning with fully homomorphic encrypted data. The method must address the issue arising from the large computational cost associated with fully homomorphic encrypted data to offer a practical and scalable solution. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Methods</u> : An approach that leverages fuzzy-based membership-mappings for data representation learning is considered for distributed deep learning with fully homomorphic encrypted data. The method introduces globally convergent and robust variational membership-mappings to build local deep models. The local models are combined in a robust and flexible manner by means of fuzzy attributes to build a global model such that the global model can be homomorphically evaluated in an efficient manner. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Results</u> : The membership-mappings based privacy-preserving distributed deep learning method is accurate, practical, and scalable. This is verified through numerous experiments which include demonstrations using MNIST and Freiburg Groceries datasets, and a biomedical application related to the detection of mental stress on individuals. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conclusion</u> : The study develops globally convergent and robust variational membership-mappings for their application to accurate, practical, and scalable privacy-preserving distributed deep learning.

Dynamic Network-Assisted D2D-Aided Coded Distributed Learning

Today, numerous machine learning (ML) applications offer continuous data processing and real-time data analytics at the edge of wireless networks. Distributed real-time ML solutions are highly susceptible to the so-called straggler effect caused by resource heterogeneity, which can be mitigated by various computation offloading mechanisms that severely impact communication efficiency, especially in large-scale scenarios. To reduce the communication overhead, we leverage device-to-device (D2D) connectivity, which enhances spectrum utilization and allows for efficient data exchange between proximate devices. In particular, we design a novel D2D-aided coded distributed learning method named D2D-CDL for efficient load balancing across devices. The proposed solution captures system dynamics, including <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">data</i> (time-varying learning model, irregular intensity of data arrivals), <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">device</i> (diverse computational resources and volume of training data), and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deployment</i> (different locations and D2D graph connectivity). To decrease the number of communication rounds, we derive an optimal compression rate, which minimizes the processing time. The resulting optimization problem provides suboptimal compression parameters that improve the total training time. Our proposed method is particularly beneficial for real-time collaborative applications, where users continuously generate training data thus yielding a model drift.

Cooperative Control of Uncertain Multiagent Systems via Distributed Gaussian Processes

For single agent systems, probabilistic machine learning techniques such as Gaussian process regression have been shown to be suitable methods for inferring models of unknown nonlinearities, which can be employed to improve the performance of control laws. While this approach can be extended to the cooperative control of multi-agent systems, it leads to a decentralized learning of the unknown nonlinearity, i.e., each agent independently infers a model. However, a decentralized learning approach can potentially lead to poor control performance, since the models of individual agents are often accurate in merely a small region of the state space. In order to overcome this issue, we propose a novel method for the distributed aggregation of Gaussian process models, and extend probabilistic error bounds for Gaussian process regression to the proposed approach. Based on this distributed learning method, we develop a cooperative tracking control law for leader-follower consensus of multi-agent systems with partially unknown, higher-order, control-affine dynamics, and analyze its stability using Lyapunov theory. The effectiveness of the proposed methods is demonstrated in numerical evaluations.

Utility Analysis about Log Data Anomaly Detection Based on Federated Learning

Logs that record system information are managed in anomaly detection, and more efficient anomaly detection methods have been proposed due to their increase in complexity and scale. Accordingly, deep learning models that automatically detect system anomalies through log data learning have been proposed. However, in existing log anomaly detection models, user logs are collected from the central server system, exposing the data collection process to the risk of leaking sensitive information. A distributed learning method, federated learning, is a trend proposed for artificial intelligence learning regarding sensitive information because it guarantees the anonymity of the collected user data and collects only weights learned from each local server in the central server. In this paper, we executed an experiment regarding system log anomaly detection using federated learning. The results demonstrate the feasibility of applying federated learning in deep-learning-based system-log anomaly detection compared to the existing centralized learning method. Moreover, we present an efficient deep-learning model based on federated learning for system log anomaly detection.

Wireless Traffic Prediction Based on a Gradient Similarity Federated Aggregation Algorithm

Wireless traffic prediction is critical to the intelligent operation of cellular networks, such as load balancing, congestion control, value-added service promotion, etc. However, the BTS data in each region has certain differences and privacy, and centralized prediction needs to transmit a large amount of traffic data, which will not only cause bandwidth consumption, but may also cause privacy leakage. Federated learning is a kind of distributed learning method with multi-client joint training and no sharing between clients. Based on existing related research, this paper proposes a gradient similarity-based federated aggregation algorithm for wireless traffic prediction (Gradient Similarity-based Federated Aggregation for Wireless Traffic Prediction) (FedGSA). First of all, this method uses a global sharing enhanced data strategy to overcome the data heterogeneity challenge of multi-client collaborative training in federated learning. Secondly, the sliding window scheme is used to construct the dual channel training data to improve the feature learning ability of the model; In addition, to improve the generalization ability of the final global model, a two-layer aggregation scheme based on gradient similarity is proposed. The personalized model is generated by comparing the gradient similarity of each client model, and the central server aggregates the personalized model to finally generate the global model. Finally, the FedGSA algorithm is applied to wireless network traffic prediction. Experiments are conducted on two real traffic datasets. Compared with the mainstream Federated Averaging (FedAvg) algorithm, FedGSA performs better on both datasets and obtains better prediction results on the premise of ensuring the privacy of client traffic data.

Cooperative Artificial Intelligence for underwater robotic swarm

Underwater Robots such as Autonomous Underwater Vehicles (AUVs) and Remotely Operated Vehicles (ROVs) has played an important role in many tasks, such as marine environmental monitoring, underwater resource exploration, oil and gas industries, hydrographic surveys, military missions, etc. Underwater robotic swarm is a team of cooperative underwater robots which focuses on controlling multiple underwater robots to work in an organic group. In contrast to a single underwater robot, underwater robotic swarm represents higher operation efficiency and better stability while executing complex tasks. However, it needs higher intelligence to realize complementary cooperation than a single robot. It is beneficial to researchers to present a comprehensive survey of the state of the art of cooperative research for underwater robotic swarm. We observe that the research of Artificial Intelligence (AI) for multiple underwater robots is still in an early stage. In this paper, we study different collaborative operation mode in detail, such as formation control, task allocation, path planning, obstacle avoidance, flocking control etc. We propose different classification frameworks for these research topics and it also can be used to compare different methods and help engineers choose suitable methods for various applications. To achieve better cooperative performance of underwater robots, there are several key factors, including multi-source heterogeneous sensing, cooperative communication and navigation, information fusion and decision. Moreover, cooperative AI for underwater robotic swarm has different kinds of interesting and helpful applications. Finally, several possible applied AI methods including meta-heuristic algorithms, deep learning method and distributed learning method are accomplishing to cooperation of underwater robotic swarm.

Federated Learning for Intelligent Transmission with Space-Air-Ground Integrated Network toward 6G

The future intelligent devices requires ultra-low communication delay and high QoS requirement for the following beyond-5G network. Space-air-ground Integrated Network (SAGIN) integrated with satellite networks in space, aerial networks, and terrestrial networks is an advanced network framework to expand the networking overage and improve connectivity for intelligent applications. However, due to the heterogeneous structure and high dynamic of SAGIN, the resource management and transmission strategy should be carefully designed to adjust to the unbalanced resources and varying environments. Federated learning is an innovative distributed learning method to intelligently manage the resource scheduling problem in SAGIN with security and guarantee of user privacy. In this paper, we introduce the advantages and potential usage directions of using federated learning in SAGIN in terms of different optimization objectives. To better illustrate the potential deployment of federated learning in SAGIN, we further provide a case study of a federated reinforcement learning-based traffic offloading approach in SAGIN.

EasyFL: A Low-Code Federated Learning Platform for Dummies

Academia and industry have developed several platforms to support the popular privacy-preserving distributed learning method -- Federated Learning (FL). However, these platforms are complex to use and require a deep understanding of FL, which imposes high barriers to entry for beginners, limits the productivity of researchers, and compromises deployment efficiency. In this paper, we propose the first low-code FL platform, EasyFL, to enable users with various levels of expertise to experiment and prototype FL applications with little coding. We achieve this goal while ensuring great flexibility and extensibility for customization by unifying simple API design, modular design, and granular training flow abstraction. With only a few lines of code, EasyFL empowers them with many out-of-the-box functionalities to accelerate experimentation and deployment. These practical functionalities are heterogeneity simulation, comprehensive tracking, distributed training optimization, and seamless deployment. They are proposed based on challenges identified in the proposed FL life cycle. Compared with other platforms, EasyFL not only requires just three lines of code (at least 10x lesser) to build a vanilla FL application but also incurs lower training overhead. Besides, our evaluations demonstrate that EasyFL expedites distributed training by 1.5x. It also improves the efficiency of deployment. We believe that EasyFL will increase the productivity of researchers and democratize FL to wider audiences.

Confederated learning in healthcare: Training machine learning models using disconnected data separated by individual, data type and identity for Large-Scale health system Intelligence

BackgroundA patient’s health information is generally fragmented across silos because it follows how care is delivered: multiple providers in multiple settings. Though it is technically feasible to reunite data for analysis in a manner that underpins a rapid learning healthcare system, privacy concerns and regulatory barriers limit data centralization for this purpose. ObjectivesMachine learning can be conducted in a federated manner on patient datasets with the same set of variables but separated across storage. But federated learning cannot handle the situation where different data types for a given patient are separated vertically across different organizations and when patient ID matching across different institutions is difficult. We call methods that enable machine learning model training on data separated by two or more dimensions “confederated machine learning”, which we aim to develop in this study. MethodsWe propose and evaluate confederated learning for training machine learning models to stratify the risk of several diseases among silos when data are horizontally separated by individual, vertically separated by data type, and separated by identity without patient ID matching. The confederated learning method can be intuitively understood as a distributed learning method with representation learning, generative model, imputation method and data augmentation elements. ResultsOur confederated learning method achieves AUCROC (Area Under The Curve Receiver Operating Characteristics) of 0.787 for diabetes prediction, 0.718 for psychological disorders prediction, and 0.698 for Ischemic heart disease prediction using nationwide health insurance claims. ConclusionOur proposed confederated learning method successfully trained machine learning models on health insurance data separated by two or more dimensions.

Weighted Averaging Federated Learning Based on Example Forgetting Events in Label Imbalanced Non-IID

Federated learning, a data privacy-focused distributed learning method, trains a model by aggregating local knowledge from clients. Each client collects and utilizes its own local dataset to train a local model. Local models in the connected federated learning network are uploaded to the server. In the server, local models are aggregated into a global model. During the process, no local data is transmitted in or out of any client. This procedure may protect data privacy; however, federated learning has a worse case of example forgetting problem than centralized learning. The problem manifests in lower performance in testing. We propose federated weighted averaging (FedWAvg). FedWAvg identifies forgettable examples in each client and utilizes that information to rebalance local models via weighting. By weighting clients with more forgettable examples, such clients are better represented and global models can acquire more knowledge from normally neglected clients. FedWAvg diminishes the example forgetting problem and achieve better performance. Our experiments on SVHN and CIFAR-10 datasets demonstrate that our proposed method gets improved performance compared to existing federated learning algorithm in non-IID settings, and that our proposed method can palliate the example forgetting problem.

Deep Reinforcement Learning-Based Cloud-Edge Collaborative Mobile Computation Offloading in Industrial Networks

With the rapid development of mobile industrial applications and due to the limited coverage of static edge servers, traditional edge computing technology has great limitations in dynamic environmental applications. This paper proposes a deep reinforcement learning-based cloud-edge collaborative mobile computation offloading mechanism for satisfying the dynamic service requirements in industrial networks. Specifically, a three-layer network model of digital twins and a decentralized network of task resources are first constructed to handle the mobility of user terminals and the relevance of tasks. Then, based on the comprehensive consideration of mobility, associated tasks, computing resources and offloading decisions, an optimization problem is formulated to minimize the weighted sum of the execution delay and energy consumption of all tasks for all users. Additionally, a deep reinforcement learning-based cloud-edge collaborative mobile computation offloading (DRL-CCMCO) algorithm is proposed to solve this optimization problem. Based on the differences in each edge cloud, this algorithm sets the priority of the shared experience pool and selects the most effective experience samples to complete better learning and training. It also utilizes a distributed learning method to learn the probability of an approximate reward distribution and optimizes network parameters through cloud-edge collaboration to achieve faster optimal offloading decision. Finally, a large number of simulation results show that the proposed algorithm has the characteristics of fast convergence and high stability, and it can obtain the optimal offloading decision with the lowest total cost.