Distributed Learning Framework Research Articles

An interpretable framework for high-precision flatness prediction in strip cold rolling

Flatness plays a crucial role in determining the quality of products in strip cold rolling. Data driven methods have shown promise in flatness prediction by effectively capturing the nonlinearities and strong coupling present in cold rolling processing, surpassing the capability of conventional methods. However, existing data driven models remain restricted by a lack of rolling theory guidance, a black-box nature of predictive processes, and gradient conflict of multi-channel flatness. To overcome these limitations, this paper proposes an interpretable mechanism guided multi-channel distributed meta learning framework for flatness prediction. Initially, significant physic-based parameters, such as theoretical rolling force deviation and tension deviation, and controller parameters are calculated to guide data driven modeling. Subsequently, a distributed meta learning framework is modeled for multi-channel flatness to eliminate gradient conflict. Furthermore, eXplainable Artificial Intelligence (XAI) technique is implemented to ensure the transparent predictive processes of multi-channel flatness. The analysis results present that theoretical parameters and controller parameters effectively improve the performance of flatness prediction. In addition, the comparative results demonstrate that the proposed framework outperforms the existing flatness prediction methods and other state-of-the-art machine learning methods by 4.24%. Importantly, the XAI-based explanation of the proposed framework effectively enhances the credibility of data driven flatness prediction.

Multi-task federated learning-based system anomaly detection and multi-classification for microservices architecture

The microservices architecture is extensively utilized in cloud-based application development, characterized by the construction of applications through a series of functionally independent, small, autonomous services. This architectural approach is renowned for its attributes such as high cohesion, availability, low coupling, and exceptional scalability. The detection of runtime system point anomalies in microservices architectures is crucial for enhancing the Quality of Service(QoS). Furthermore, identifying the classes of detected anomalies is critical in practical applications. However, given the highly dynamic nature of microservices systems as a distributed computing architecture, conducting real-time system anomaly detection on distributed independent microservices poses a challenging task. To address these challenges, we propose the System Anomaly Detection and Multi-Classification based on Multi-Task Feature Fusion Federated Learning (SADMC-MT-FF-FL) framework. Initially, we introduce a distributed learning framework based on Multi-task Federated Learning (MT-FL) to construct multi-classification anomaly detection models for each microservice. Secondly, to identify complex system anomaly patterns and features during the runtime of microservices, we develop a feature extractor based on External Attention Mechanism and Multi-channel Residual Structure (EA-MRS). Finally, we design a Local–Global Feature-based Parallel Knowledge Transfer (LGF-PKT) framework, utilizing parallel knowledge transfer to parallelize weight updates for local and global features. To validate the effectiveness of our approach, we conducted comprehensive comparative experiments on the microservices benchmark platforms Sock-Shop and Train-Ticket. The experimental results on anomaly detection for multiclassification systems demonstrate that SADMC-MT-FF-FL outperforms the best baseline method by 28.3% and 27.8% for Macro F1 and Micro F1 on Train-Ticket, and by 8.8% and 8.6% on Sock-Shop, respectively. Additionally, we conducted comparison experiments on three public datasets, SWaT, SMD, and SKAB. The F1 scores were 0.5% higher than those of the centralized methods on SMD, respectively, 6% and 2.8% higher than those of the federated learning based method on SWaT and SKAB. Source codes are available at: https://github.com/icc-lab-xhu1/SADMC-MT-FF-FL.

More communication-efficient distributed sparse learning

In a modern distributed learning framework, the speeds of intra-worker calculation and inter-worker communication may differ by 1,000 times. It is advisable to perform expensive calculations on worker machines and communicate as few rounds as possible. In this paper, we propose three novel distributed sparse learning algorithms (EDSL-ET, EDSL-IBCD, and EDSL-IBCD-ET) in high dimensions, which are efficient in both computation and communication. Algorithm EDSL-ET utilizes the Topk sparse technique to reduce communication costs and adopts error feedback to guarantee convergence. Algorithm EDSL-IBCD greatly reduces communication costs by introducing an independent block coordinate descent method. Algorithm EDSL-IBCD-ET takes advantage of EDSL-ET and EDSL-IBCD algorithms for extra-high dimensional feature learning, which is the best one whether in communication, computation, and stability. We give theoretical guarantees of the three algorithms under mild conditions, which match the performance of the centralized algorithm. Extensive experiments on simulated and real data validate our theoretical analysis and demonstrate that the proposed algorithms perform well with just a few rounds of communications.

Revisiting Gradient Pruning: A Dual Realization for Defending against Gradient Attacks

Collaborative learning (CL) is a distributed learning framework that aims to protect user privacy by allowing users to jointly train a model by sharing their gradient updates only. However, gradient inversion attacks (GIAs), which recover users' training data from shared gradients, impose severe privacy threats to CL. Existing defense methods adopt different techniques, e.g., differential privacy, cryptography, and perturbation defenses, to defend against the GIAs. Nevertheless, all current defense methods suffer from a poor trade-off between privacy, utility, and efficiency. To mitigate the weaknesses of existing solutions, we propose a novel defense method, Dual Gradient Pruning (DGP), based on gradient pruning, which can improve communication efficiency while preserving the utility and privacy of CL. Specifically, DGP slightly changes gradient pruning with a stronger privacy guarantee. And DGP can also significantly improve communication efficiency with a theoretical analysis of its convergence and generalization. Our extensive experiments show that DGP can effectively defend against the most powerful GIAs and reduce the communication cost without sacrificing the model's utility.

GLDL: Graph Label Distribution Learning

Label Distribution Learning (LDL), as a more general learning setting than generic single-label and multi-label learning, has been commonly used in computer vision and many other applications. To date, existing LDL approaches are designed and applied to data without considering the interdependence between instances. In this paper, we propose a Graph Label Distribution Learning (GLDL) framework, which explicitly models three types of relationships: instance-instance, label-label, and instance-label, to learn the label distribution for networked data. A label-label network is learned to capture label-to-label correlation, through which GLDL can accurately learn label distributions for nodes. Dual graph convolution network (GCN) Co-training with heterogeneous message passing ensures two GCNs, one focusing on instance-instance relationship and the other one targeting label-label correlation, are jointly trained such that instance-instance relationship can help induce label-label correlation and vice versa. Our theoretical study derives the error bound of GLDL. For verification, four benchmark datasets with label distributions for nodes are created using common graph benchmarks. The experiments show that considering dependency helps learn better label distributions for networked data, compared to state-of-the-art LDL baseline. In addition, GLDL not only outperforms simple GCN and graph attention networks (GAT) using distribution loss but is also superior to its variant considering label-label relationship as a static network. GLDL and its benchmarks are the first research endeavors to address LDL for graphs. Code and benchmark data are released for public access.

Chronic Poisoning: Backdoor Attack against Split Learning

Split learning is a computing resource-friendly distributed learning framework that protects client training data by splitting the model between the client and server. Previous work has proved that split learning faces a severe risk of privacy leakage, as a malicious server can recover the client's private data by hijacking the training process. In this paper, we first explore the vulnerability of split learning to server-side backdoor attacks, where our goal is to compromise the model's integrity. Since the server-side attacker cannot access the training data and client model in split learning, the traditional poisoning-based backdoor attack methods are no longer applicable. Therefore, constructing backdoor attacks in split learning poses significant challenges. Our strategy involves the attacker establishing a shadow model on the server side that can encode backdoor samples and guiding the client model to learn from this model during the training process, thereby enabling the client to acquire the same capability. Based on these insights, we propose a three-stage backdoor attack framework named SFI. Our attack framework minimizes assumptions about the attacker's background knowledge and ensures that the attack process remains imperceptible to the client. We implement SFI on various benchmark datasets, and extensive experimental results demonstrate its effectiveness and generality. For example, success rates of our attack on MNIST, Fashion, and CIFAR10 datasets all exceed 90%, with limited impact on the main task.

Multi‐dimensional resource management with deep deterministic policy gradient for digital twin‐enabled Industrial Internet of Things in 6 generation

AbstractIn the era of sixth generation mobile networks (6G), industrial big data is rapidly generated due to the increasing data‐driven applications in the Industrial Internet of Things (IIoT). Effectively processing such data, for example, knowledge learning, on resource‐limited IIoT devices becomes a challenge. To this end, we introduce a cloud‐edge‐end collaboration architecture, in which computing, communication, and storage resources are flexibly coordinated to alleviate the issue of resource constraints. To achieve better performance in hyper‐connected experience, real‐time communication, and sustainable computing, we construct a novel architecture combining digital twin (DT)‐IIoT with edge networks. In addition, considering the energy consumption and delay issues in distributed learning, we propose a deep reinforcement learning‐based method called deep deterministic policy gradient with double actors and double critics (D4PG) to manage the multi‐dimensional resources, that is, CPU cycles, DT models, and communication bandwidths, enhancing the exploration ability and improving the inaccurate value estimation of agents in continuous action spaces. In addition, we introduce a synchronization threshold for distributed learning framework to avoid the synchronization latency caused by stragglers. Extensive experimental results prove that the proposed architecture can efficiently conduct knowledge learning, and the intelligent scheme can also improve system efficiency by managing multi‐dimensional resources.

A Robust Learning Framework for Smart Grids in Defense Against False-Data Injection Attacks

With the rapid development of the application of smart grids in different sectors, security management has become a major concern due to cyber attack risks. Correctly and accurately estimating the real status of a smart grid under false-data injection attacks (FDIAs) is currently an emerging concern. In response to that concern, this work proposes a distributed robust learning framework for the inference of the working status under data integrity attacks. The proposed paradigm incorporates the technology median-of-means that enables identifying the correct state against various kinds of FDIAs that can efficiently prevent misleading information during the decision-making process in control centers. Compared with existing defense methods, our method is entirely data driven without training data, highly accurate, and reliable for wide-spectrum FDIAs. More important, it is capable of defending large-scale power electronic networks due to its distributed learning framework. Extensive experimental results demonstrate that our approach can provide efficient protection for Photovoltaic (PV) systems from FDIAs.

A Study of Federated Learning with Internet of Things for Data Privacy and Security using Privacy Preserving Techniques

Abstract: Privacy leakage that occurs when many IoT devices are utilized for training centralized models, a new distributed learning framework known as federated learning was created, where devices train models together while keeping their private datasets local. In a federated learning setup, a central aggregator coordinates the efforts of several clients working together to solve machine learning issues. The privacy of each device's data is protected by this setup's decentralized training data. Federated learning reduces traditional centralized machine learning systems' systemic privacy issues and costs by emphasizing local processing and model transfer. Client information is stored locally and cannot be copied or shared. By utilizing a centralized server, federated learning enables each participant's device to collect data locally for training purposes before sending the resulting model to the server for aggregate and subsequent distribution. As a means of providing a comprehensive review and encouraging further research into the topic, we introduce the works of federated learning from five different vantage points: data partitioning, privacy method, machine learning model, communication architecture, and systems heterogeneity. Then, we organize the issues plaguing federated learning today and the potential avenues for a prospective study. Finally, we provide a brief overview of the features of existing federated knowledge and discuss how it is currently being used in the field.

Channel Adaptive and Sparsity Personalized Federated Learning for Privacy Protection in Smart Healthcare Systems.

With the booming development of Smart Healthcare Systems (SHSs), employing federated learning (FL) in SHS devices has become a research hotspot. FL, as a distributed learning framework, can train models without sharing the original data among users, and then protect the user privacy. Existing research has proposed many methods to improve the security and efficiency of FL, which may not fully consider the characteristics of SHSs. Specifically, the requirements of privacy protection and efficiency pose significant challenges to FL. Current studies have struggled to balance privacy security and efficiency, and the degradation of model training efficiency in SHSs can be critical to patient health. Therefore, to improve the privacy protection of healthcare data and ensure communication efficiency, this work proposes a novel personalized FL framework based on Communication quality and Adaptive Sparsification (pFedCAS). In order to achieve privacy protection, a control unit is proposed and introduced to adjust the sparsity of the local model adaptively. To further improve the training efficiency, a selection unit is added during global model aggregation to select suitable clients for parameter updates. Finally, we validate the proposed method operated on the HAM10000 dataset. Simulation results validate that pFedCAS can not only improve privacy protection, but also gain an improvement of 15% in training accuracy and a reduction of 30% in training costs based on communication quality. The simulation results also validate the excellent robustness of pFedCAS to non-iid data.

UDSem: A Unified Distributed Learning Framework for Semantic Communications over Wireless Networks

End-to-end semantic communications (ESC) rely on deep neural networks (DNN) to boost the communication efficiency by only transmitting the semantics of data. However, ESC is data-hungry for training millions of neural parameters, while uploading massive data generated by users/devices to a central server is usually impractical due to privacy issues, regulatory guidelines and network congestion, etc. Inspired by federated learning, where each device contributes to the parameters' update by independently computing the gradient based on its local data, we present UDSem, a unified distributed learning framework of semantic communications for texts and images over wireless networks. The key ingredients of our UDSem include 1) a flexible learning mechanism that can be tailored to a device based on its computing capabilities, 2) an efficient learning approach that splits the system into multiple modules and updates the parameters independently both on clients and the central server, 3) a mixed aggregation method that globally updates the neural parameters for each module of the ESC system. Equipped with these ingredients, our proposed UDSem is able to properly fill the gap between the training efficiency of ESC systems and the privacy requirement of users' data. Experimental results over two public benchmarks show the superiority of our UDSem in terms of convergence speed and semantic interpretation, potentially paving the way for distributed semantic communications in future wireless networks.

PASS: A Parameter Audit-Based Secure and Fair Federated Learning Scheme Against Free-Rider Attack

Federated Learning (FL) as a secure distributed learning framework gains interests in Internet of Things (IoT) due to its capability of protecting the privacy of participant data. However, traditional FL systems are vulnerable to Free-Rider (FR) attacks, which causes unfairness, privacy leakage and inferior performance to FL systems. The prior defense mechanisms against FR attacks assumed that malicious clients (namely, adversaries) declare less than 50% of the total amount of clients. Moreover, they aimed for Anonymous FR (AFR) attacks and lost effectiveness in resisting Selfish FR (SFR) attacks. In this paper, we propose a Parameter Audit-based Secure and fair federated learning Scheme (PASS) against FR attack. PASS has the following key features: (a) prevent from privacy leakage with less accuracy loss; (b) be effective in countering both AFR and SFR attacks; (c) work well no matter whether AFR and SFR adversaries occupy the majority of clients or not. Extensive experimental results validate that PASS: (a) has the same level as the State-Of-The-Art method in mean square error against privacy leakage; (b) defends against AFR and SFR attacks in terms of a higher defense success rate, lower false positive rate, and higher F1-score; (c) is still effective where adversaries exceed 50%, with F1-score 89% against AFR attack and F1-score 87% against SFR attack. Note that PASS produces no negative effect on FL accuracy when there is no FR adversary.

SIA: A sustainable inference attack framework in split learning

Split learning is a widely recognized distributed learning framework suitable for joint training scenarios with limited computing resources. However, recent research indicates that the malicious server can achieve high-quality reconstruction of the client’s data through feature space hijacking attacks, leading to severe privacy leakage concerns. In this paper, we further enhance this attack to enable efficient data reconstruction while maintaining acceptable performance on the main task. Another significant advantage of our attack framework lies in its ability to fool the state-of-the-art attack detection mechanism, thus minimizing the risk of attacker exposure and making sustainable attacks possible. Moreover, we adaptively refine and adjust the attack strategy, extending the data reconstruction attack for the first time to the more challenging scenario of vertically partitioned data in split learning. In addition, we introduce three training modes for the attack framework, allowing the attacker to choose according to their requirements freely. Finally, we conduct extensive experiments on three datasets and evaluate the attack performance of attack frameworks in different scenarios, parameter settings, and defense mechanisms. The results demonstrate our attack framework’s effectiveness, invisibility, and generality. Our research comprehensively highlights the potential privacy risks associated with split learning and sounds the alarm for secure applications of split learning.

Ambiguity-aware breast tumor cellularity estimation via self-ensemble label distribution learning.

In this work, we address the task of tumor cellularity (TC) estimation with a novel framework based on the label distribution learning (LDL) paradigm. We propose a self-ensemble label distribution learning framework (SLDL) to resolve the challenges of existing LDL-based methods, including difficulties for inter-rater ambiguity exploitation, proper and flexible label distribution generation, and accurate TC value recovery. The proposed SLDL makes four main contributions which have been demonstrated to be quite effective in numerous experiments. First, we propose an expertness-aware conditional VAE for diversified single-rater modeling and an attention-based multi-rater fusion strategy that enables effective inter-rater ambiguity exploitation. Second, we propose a template-based label distribution generation method that is tailored for the TC estimation task and constructs label distributions based on the annotation priors. Third, we propose a novel restricted distribution loss, significantly improving the TC value estimation by effectively regularizing the learning with unimodal loss and regression loss. Fourth, to the best of our knowledge, we are the first to simultaneously leverage inter-rater and intra-rater variability to address the label ambiguity issue in the breast tumor cellularity estimation tasks. The experimental results on the public BreastPathQ dataset demonstrate that the SLDL outperforms the existing methods by a large margin and achieves new state-of-the-art results in the TC estimation task. The code will be available from https://github.com/PerceptionComputingLab/ULTRA.

Modeling annotator preference and stochastic annotation error for medical image segmentation

Manual annotation of medical images is highly subjective, leading to inevitable annotation biases. Deep learning models may surpass human performance on a variety of tasks, but they may also mimic or amplify these biases. Although we can have multiple annotators and fuse their annotations to reduce stochastic errors, we cannot use this strategy to handle the bias caused by annotators’ preferences. In this paper, we highlight the issue of annotator-related biases on medical image segmentation tasks, and propose a Preference-involved Annotation Distribution Learning (PADL) framework to address it from the perspective of modeling an annotator’s preference and stochastic errors so as to produce not only a meta segmentation but also the annotator-specific segmentation. Under this framework, a stochastic error modeling (SEM) module estimates the meta segmentation map and average stochastic error map, and a series of human preference modeling (HPM) modules estimate each annotator’s segmentation and the corresponding stochastic error. We evaluated our PADL framework on two medical image benchmarks with different imaging modalities, which have been annotated by multiple medical professionals, and achieved promising performance on all five medical image segmentation tasks. Code is available at https://github.com/Merrical/PADL.

Federated Learning Based on Mutual Information Clustering for Wireless Traffic Prediction

Wireless traffic prediction can help operators accurately predict the usage of wireless networks, and it plays an important role in the load balancing and energy saving of base stations. Currently, most traffic prediction methods are centralized learning strategies, which need to transmit a large amount of traffic data and have timeliness and data privacy issues. Federated learning, as a distributed learning framework with no client data sharing and multi-client collaborative training, can solve such problems. We propose a federated learning wireless traffic prediction framework based on mutual information clustering (FedMIC). First, a sliding window scheme is used to construct the raw data into adjacent and periodic dual-traffic sequences and capture their traffic characteristics separately to enhance the client model learning capability. Second, clients with similar traffic data distributions are clustered together using a mutual information-based spectral clustering algorithm to facilitate the capture of the personalized features of each clustered model. Then, models are aggregated using a hierarchical aggregation architecture of intra-cluster model aggregation and inter-cluster model aggregation to address the statistical heterogeneity challenge of federated learning and to improve the prediction accuracy of models. Finally, an attention mechanism-based model aggregation algorithm is used to improve the generalization ability of the global model. Experimental results show that our proposed method minimizes the prediction error and has superior traffic prediction performance compared to traditional distributed machine learning methods and other federated learning methods.

A High-Performance Federated Learning Aggregation Algorithm Based on Learning Rate Adjustment and Client Sampling

Federated learning is a distributed learning framework designed to protect user privacy, widely applied across various domains. However, existing federated learning algorithms face challenges, including slow convergence, significant loss fluctuations during aggregation, and imbalanced client sampling. To address these issues, this paper introduces a high-performance federated learning aggregation algorithm. This algorithm combines a cyclic adaptive learning rate adjustment strategy with client-weighted random sampling, addressing the aforementioned problems. Weighted random sampling assigns client weights based on their sampling frequency, balancing client sampling rates and contributions to enhance model aggregation. Additionally, it adapts the learning rate based on client loss variations and communication rounds, accelerating model convergence and reducing communication costs. To evaluate this high-performance algorithm, experiments are conducted using well-known datasets MNIST and CIFAR-10. The results demonstrate significant improvements in convergence speed and loss stability. Compared to traditional federated learning algorithms, our approach achieves faster and more stable convergence while effectively reducing training costs.

Curriculum label distribution learning for imbalanced medical image segmentation.

Label distribution learning (LDL) has the potential to resolve boundary ambiguity in semantic segmentation tasks. However, existing LDL-based segmentation methods suffer from severe label distribution imbalance: the ambiguous label distributions contain a small fraction of the data, while the unambiguous label distributions occupy the majority of the data. The imbalanced label distributions induce model-biased distribution learning and make it challenging to accurately predict ambiguous pixels. In this paper, we propose a curriculum label distribution learning (CLDL) framework to address the above data imbalance problem by performing a novel task-oriented curriculum learning strategy. Firstly, the region label distribution learning (R-LDL) is proposed to construct more balanced label distributions and improves the imbalanced model learning. Secondly, a novel learning curriculum (TCL) is proposed to enable easy-to-hard learning in LDL-based segmentation by decomposing the segmentation task into multiple label distribution estimation tasks. Thirdly, the prior perceiving module (PPM) is proposed to effectively connect easy and hard learning stages based on the priors generated from easier stages. Benefiting from the balanced label distribution construction and prior perception, the proposed CLDL effectively conducts a curriculum learning-based LDL and significantly improves the imbalanced learning. We evaluated the proposed CLDL using the publicly available BRATS2018 and MM-WHS2017 datasets. The experimental results demonstrate that our method significantly improves different segmentation metrics compared to many state-of-the-art methods. The code will be available.1.

How to backdoor split learning

Split learning, a distributed learning framework, has garnered significant attention from academic and industrial communities. In contrast to federated learning, split learning offers a more flexible architecture for participants with limited computing resources. However, the security of split learning has been questioned due to the separation of data and model control rights from usage rights. Currently, most research work focuses on inference attacks in split learning. In this paper, we first reveal the vulnerability of split learning to backdoor attacks and present two backdoor attack frameworks from both the server and client perspectives. Regarding the client-side attacker, we insert backdoor samples into the training data by utilizing the client’s direct control over local data, and propose two methods for labeling backdoor samples that can be adapted to various application scenarios. Due to the server’s lack of control over the client in split learning, it is infeasible for server-side attackers to inject backdoor samples into training data. Our strategy involves leveraging the server’s control over the training process to shape the optimization direction of the client model, thereby enabling it to encode backdoor samples. Moreover, we introduce an auxiliary model into the attack framework to enhance the effectiveness of the backdoor attack. The auxiliary model can increase the distinction between backdoor samples and clean samples in the feature space to improve the sensitivity of the client model to backdoor samples. Extensive evaluations demonstrate the high attack accuracy of both proposed attack frameworks without causing any compromise to the performance of the main task. Our research uncovers the potential security risks and rings the alarm for the application of split learning.

Finite-time convergence rates of distributed local stochastic approximation

We consider a distributed learning framework, where there are a group of agents communicating with a centralized coordinator. The goal of the agents is to find the root of an operator composed of the local operators at the agents. Such a framework models many practical problems in different areas, including those in federated learning and reinforcement learning. For solving this problem, we study the popular distributed stochastic approximation. Over a series of time epoch, each agent runs a number of local stochastic approximation steps based on its own data, whose results are then aggregated at the centralized coordinator.Existing theoretical guarantees for the finite-time performance of local stochastic approximation are studied under the common assumption that the local data at each agent is sampled i.i.d. Such an assumption may not hold in many applications, where the data are temporally dependent, for example, they are sampled from some dynamical systems. In this paper, we study the setting where the data are generated from Markov random processes, which are often used to model the systems in stochastic control and reinforcement learning. Our main contribution is to characterize the finite-time performance of the local stochastic approximation under this setting. We provide explicit formulas for the rates of this method for both constant and time-varying step sizes when the local operators are strongly monotone. Our results show that these rates are within a logarithmic factor of the comparable bounds under independent data. We also provide a number of numerical simulations to illustrate our theoretical results by applying local SA in solving problems in robust identification and reinforcement learning over multi-agent systems.