Distributed Learning Framework Research Articles

Client Selection for Federated Bayesian Learning

Distributed Stein Variational Gradient Descent (DSVGD) is a non-parametric distributed learning framework for federated Bayesian learning, where multiple clients jointly train a machine learning model by communicating a number of non-random and interacting particles with the server. Since communication resources are limited, selecting the clients with most informative local learning updates can improve the model convergence and communication efficiency. In this paper, we propose two selection schemes for DSVGD based on Kernelized Stein Discrepancy (KSD) and Hilbert Inner Product (HIP). We derive the upper bound on the decrease of the global free energy per iteration for both schemes, which is then minimized to speed up the model convergence. We evaluate and compare our schemes with conventional schemes in terms of model accuracy, convergence speed, and stability using various learning tasks and datasets.

LDIA: Label distribution inference attack against federated learning in edge computing

With the popularity of IoT (Internet of Things) applications, edge computing has received lots of attention. To meet data privacy protection requirements of edge nodes and cope with their unbalanced data distribution, federated learning (FL), a distributed learning framework, is widely used in intelligent edge computing applications. However, recent studies have shown that FL still suffers from privacy leakage problems, including membership inference, data reconstruction, etc. However, these studies mainly focus on the feature information of private data. In this paper, we concern the user-level label privacy in FL. We propose LDIA, a label distribution inference attack against FL in edge computing, exploring the possibility that an honest but curious cloud server can infer the proportions of samples per label in the edge user’s private data. LDIA is inspired by the observation that parameter changes in the output layer of a model can reflect the label distribution of training data. We use a neural network to learn individual features of the output layer updates over different label distributions, and then perform inference from local models uploaded by users. Our comprehensive evaluation shows that LDIA is effective on various datasets in different settings, demonstrating the severe privacy leakage in FL-based edge computing.

Towards energy-efficient and time-sensitive task assignment in cross-silo federated learning

Federated learning (FL) is a novel distributed learning framework in which clients with models can collaboratively train a high-quality global model while maintaining the privacy of their local data. The heterogeneity of datasets makes the global aggregation and local training requirements of each client in the cross-silo scenario inconsistent. Although many efforts are dedicated to improving the efficiency of cross-silo FL, the task assignment design of cross-silo FL remains an open question. To optimize the efficiency of energy consumption and training time of cross-silo FL, we develop a novel Energy-efficient and Time-sensitive Task Assignment (ETTA) mechanism. Specifically, we model clients’ requirements for energy efficiency and training time in cross-silo FL as an optimization problem, and design a utility transfer function to maintain the honesty of clients and the stability of ETTA. In terms of energy efficiency maximization, we analyze the conditions of optimal solutions based on the equilibrium of marginal cost and income, and design an approximate algorithm for cross-silo FL scenarios that require lightweight mechanisms. In terms of training time minimization, we model this objective as a min–max problem and give its analytical solution with Pareto optimality. To ensure that the task assignment achieves incentive compatibility, we further design a utility transfer selection function to incentivize honest collaboration among selfish clients. Finally, we conduct extensive experiments on the performance of ETTA to demonstrate that it outperforms the state-of-the-art and can maintain stability in a dishonest environment.

Privacy-preserving distributed deep learning via LWE-based Certificateless Additively Homomorphic Encryption (CAHE)

Current privacy-preserving schemes proposed for distributed deep learning (DDL) are based on public key infrastructure (PKI) which has management of keys problem and also key escrow problem for those that may use identity-based public key cryptography (ID-PKC). This presents the possibility for the privacy guarantee of the distributed/collaborative deep learning system to be flawed due to the presence of a Trusted Third Party (TTP) (i.e. a certificate authority (CA) for public key cryptography (PKC) and a key generation center (KGC) for an ID-PKC), that must be trusted. Additionally, the presence of high communication cost associated with the transmission of homomorphically encrypted gradients from participants to the cloud server in distributed deep learning needs more attention. Furthermore, few schemes have been proposed for DDL that have post-quantum robustness. We, therefore, propose and design a new privacy-preserving distributed deep learning solution. First, a new LWE-based certificateless additively homomorphic encryption scheme called CAHE is proposed and designed. Secondly, partial sharing algorithms based on random selection and selection of the largest (or topmost) gradients in terms of magnitude are proposed. Thirdly, we put forward a new privacy-preserving distributed deep learning framework that combines the certificateless additively homomorphic encryption and partial sharing in the DDL. Our approach provides better privacy (i.e. not relying solely on the TTP) and also reduces communication cost for distributed deep learning. We note emphatically that this is the first time certificateless HE is being proposed, designed, and implemented in DDL. Results from our work indicate that the accuracy of the CAHE-based deep learning does not deteriorate (97.20%) while greatly preserving the privacy of participant’s dataset. With partial sharing, our solution records 97.17% and 97.12% accuracy for the selection of topmost gradients and random selection of gradients scenarios respectively, and with a great reduction in communication cost for transmitting encrypted gradients.

Hierarchical Adaptive Collaborative Learning: A Distributed Learning Framework for Customized Cloud Services in 6G Mobile Systems

Hierarchical Adaptive Collaborative Learning: A Distributed Learning Framework for Customized Cloud Services in 6G Mobile Systems

Dual Preference Distribution Learning for Item Recommendation

Recommender systems can automatically recommend users with items that they probably like. The goal of them is to model the user-item interaction by effectively representing the users and items. Existing methods have primarily learned the user’s preferences and item’s features with vectorized embeddings, and modeled the user’s general preferences to items by the interaction of them. In fact, users have their specific preferences to item attributes and different preferences are usually related. Therefore, exploring the fine-grained preferences as well as modeling the relationships among user’s different preferences could improve the recommendation performance. Toward this end, we propose a dual preference distribution learning framework (DUPLE) , which aims to jointly learn a general preference distribution and a specific preference distribution for a given user, where the former corresponds to the user’s general preference to items and the latter refers to the user’s specific preference to item attributes. Notably, the mean vector of each Gaussian distribution can capture the user’s preferences, and the covariance matrix can learn their relationship. Moreover, we can summarize a preferred attribute profile for each user, depicting his/her preferred item attributes. We then can provide the explanation for each recommended item by checking the overlap between its attributes and the user’s preferred attribute profile. Extensive quantitative and qualitative experiments on six public datasets demonstrate the effectiveness and explainability of the DUPLE method.

Federated learning for minimizing nonsmooth convex loss functions

Federated learning possesses a distributed learning framework for protecting privacy where local clients need to collaboratively train a shared model via a central server. Many existing methods in the literature of federated learning are analyzed under some smoothness conditions of loss functions and need access to gradient information to perform local optimization algorithms such as stochastic gradient descent or dual average. Some methods even need a strong convexity of the loss function. However, in many situations of real world applications such as readability of texts, first-order gradient information is difficult to obtain, and the strong smoothness and strong convexity of loss functions are not satisfied. We consider methods to overcome such situations. This paper aims at providing an understanding of federated learning in the situation that the loss functions are nonsmooth and gradient information is unavailable, also a strong convexity condition is not needed. Based on Nesterov's zeroth-order (gradient-free) techniques, we propose a zeroth-order stochastic federated learning method. Constant and decreasing step size strategies are considered. Moreover, a new type of approximating sequence is proposed in federated learning for strictly decreasing step sizes. Expected error bounds for the proposed approximating sequence and learning rates of the proposed method are derived under some selection rules of the step sizes and smoothing parameters.

Task-Agnostic Vision Transformer for Distributed Learning of Image Processing.

Recently, distributed learning approaches have been studied for using data from multiple sources without sharing them, but they are not usually suitable in applications where each client carries out different tasks. Meanwhile, Transformer has been widely explored in computer vision area due to its capability to learn the common representation through global attention. By leveraging the advantages of Transformer, here we present a new distributed learning framework for multiple image processing tasks, allowing clients to learn distinct tasks with their local data. This arises from a disentangled representation of local and non-local features using a task-specific head/tail and a task-agnostic Vision Transformer. Each client learns a translation from its own task to a common representation using the task-specific networks, while the Transformer body on the server learns global attention between the features embedded in the representation. To enable decomposition between the task-specific and common representations, we propose an alternating training strategy between clients and server. Experimental results on distributed learning for various tasks show that our method synergistically improves the performance of each client with its own data.

Privacy-Enhanced Federated Learning: A Restrictively Self-Sampled and Data-Perturbed Local Differential Privacy Method

As a popular distributed learning framework, federated learning (FL) enables clients to conduct cooperative training without sharing data, thus having higher security and enjoying benefits in processing large-scale, high-dimensional data. However, by sharing parameters in the federated learning process, the attacker can still obtain private information from the sensitive data of participants by reverse parsing. Local differential privacy (LDP) has recently worked well in preserving privacy for federated learning. However, it faces the inherent problem of balancing privacy, model performance, and algorithm efficiency. In this paper, we propose a novel privacy-enhanced federated learning framework (Optimal LDP-FL) which achieves local differential privacy protection by the client self-sampling and data perturbation mechanisms. We theoretically analyze the relationship between the model accuracy and client self-sampling probability. Restrictive client self-sampling technology is proposed which eliminates the randomness of the self-sampling probability settings in existing studies and improves the utilization of the federated system. A novel, efficiency-optimized LDP data perturbation mechanism (Adaptive-Harmony) is also proposed, which allows an adaptive parameter range to reduce variance and improve model accuracy. Comprehensive experiments on the MNIST and Fashion MNIST datasets show that the proposed method can significantly reduce computational and communication costs with the same level of privacy and model utility.

Review on application progress of federated learning model and security hazard protection

Federated learning is a new type of distributed learning framework that allows multiple participants to share training results without revealing their data privacy. As data privacy becomes more important, it becomes difficult to collect data from multiple data owners to make machine learning predictions due to the lack of data security. Data is forced to be stored independently between companies, creating “data silos”. With the goal of safeguarding data privacy and security, the federated learning framework greatly expands the amount of training data, effectively improving the shortcomings of traditional machine learning and deep learning, and bringing AI algorithms closer to our reality. In the context of the current international data security issues, federated learning is developing rapidly and has gradually moved from the theoretical to the applied level. The paper first introduces the federated learning framework, analyzes its advantages, reviews the results of federated learning applications in industries such as communication and healthcare, then analyzes the pitfalls of federated learning and discusses the security issues that should be considered in applications, and finally looks into the future of federated learning and the application layer.

Federated learning based on stratified sampling and regularization

Federated learning (FL) is a new distributed learning framework that is different from traditional distributed machine learning: (1) differences in communication, computing, and storage performance among devices (device heterogeneity), (2) differences in data distribution and data volume (data heterogeneity), and (3) high communication consumption. Under heterogeneous conditions, the data distribution of clients varies greatly, which leads to the problem that the convergence speed of the training model decreases and the training model cannot converge to the global optimal solution. In this work, an FL algorithm based on stratified sampling and regularization (FedSSAR) is proposed. In FedSSAR, a density-based clustering method is used to divide the overall client into different clusters, then, some available clients are proportionally extracted from different clusters to participate in training which realizes unbiased sampling for the overall client and reduces the aggregation weight variance of the client. At the same time, when calculating the model local loss function, we limit the update direction of the model by a regular term, so that heterogeneous clients are optimized in the globally optimal direction. We prove the convergence of FedSSAR theoretically and experimentally, and demonstrate the superiority of FedSSAR by comparing it with other FL algorithms on public datasets.

LDP‐Fed+: A robust and privacy‐preserving federated learning based classification framework enabled by local differential privacy

AbstractAs a distributed learning framework, Federated Learning (FL) allows different local learners/participants to collaboratively train a joint model without exposing their own local data, and offers a feasible solution to legally resolve data islands. However, among them, the data privacy and model security are two challenges. The former means that, if original data are used for trained FL models, various methods can be used to deduce the original data samples, thereby causing the leakage of data. The latter implies that unreliable/malicious participants may affect or destroy the joint FL model, through uploading wrong local model parameters. Therefore, this paper proposes a novel distributed FL training framework, namely LDP‐Fed+, which takes into account differential privacy protection and model security defense. Specifically, firstly, a local perturbation module is added at the local learner side, which perturbs the original data of local learners through feature extraction, binary encoding and decoding, and random response. Then, through using the perturbed data, local neural network model is trained to obtain the network parameters that meet local differential protection, to effectively deal with model inversion attacks. Secondly, a security defense module is added on the server side, which uses the auxiliary model and differential index mechanism to select an appropriate number of local disturbance parameters for aggregation to enhance model security defense and deal with membership inference attacks. The experimental results show that, compared with other federated learning models based on differential privacy, LDP‐Fed+ has stronger robustness for model security and higher accuracy for model training while ensuring strict privacy protection.

AWFC: Preventing Label Flipping Attacks Towards Federated Learning for Intelligent IoT

AbstractCentralized machine learning methods require the aggregation of data collected from clients. Due to the awareness of data privacy, however, the aggregation of raw data collected by Internet of Things (IoT) devices is not feasible in many scenarios. Federated learning (FL), a kind of distributed learning framework, can be running on multiple IoT devices. It aims to resolve the issues of privacy leakage by training a model locally on the client-side, other than on the server-side that aggregates all the raw data. However, there are still threats of poisoning attacks in FL. Label flipping attacks, typical data poisoning attacks in FL, aim to poison the global model by sending model updates trained by the data with mismatched labels. The central parameter aggregation server is hard to detect the label flipping attacks due to its inaccessibility to the client in a typical FL system. In this work, we are motivated to prevent label flipping poisoning attacks by observing the changes in model parameters that were trained by different single labels. We propose a novel detection method called average weight of each class in its associated fully connected layer. In this method, we detect label flipping attacks by identifying the differences of classes in the data based on the weight assignments in a fully connected layer of the neural network model and use the statistical algorithm to recognize the malicious clients. We conduct extensive experiments on benchmark data like Fashion-MNIST and Intrusion Detection Evaluation Dataset (CIC-IDS2017). Comprehensive experimental results demonstrated that our method has the detection accuracy over 90% for the identification of the attackers flipping labels.

Communication-Efficient Federated Edge Learning via Optimal Probabilistic Device Scheduling

Federated edge learning (FEEL) is a popular distributed learning framework that allows privacy-preserving collaborative model training via periodic learning-updates communication between edge devices and server. Due to the constrained bandwidth, only a subset of devices can be selected to upload their updates at each training iteration. This has led to an active research area in FEEL studying the optimal device scheduling policy for communication time minimization. However, owing to the difficulty in quantifying the exact communication time, prior work in this area can only tackle the problem partially and indirectly by minimizing either the iteration rounds or per-round latency, while the total communication time is determined by both metrics. To close this research gap, we make the first attempt in this paper to formulate and solve the communication time minimization problem. We first derive a tight bound to approximate the remaining communication time through cross-disciplinary effort that combines the learning theory for convergence rate analysis and communication theory for per-round latency analysis. Building on the novel analytical result, an optimized probabilistic device scheduling policy is derived in closed-form by solving the approximate communication time minimization problem. It is found that the optimized policy gradually turns its priority from suppressing the remaining communication rounds to reducing per-round latency as the training process evolves. Extensive experiments based on real-world dataset and a use case on collaborative 3D objective detection in autonomous driving are provided to verify the superiority of the proposed policy over three benchmark policies based on the indirect solution approaches.

Optimal Leader-Following Consensus Control of Multi-Agent Systems: A Neural Network Based Graphical Game Approach

In this article, the optimal leader-following consensus control problem of multi-agent systems is solved using a novel neural network-based (NN-based) integrated heuristic dynamic programming (IHDP) algorithm under a distributed learning framework. The optimal controller is designed to find a Nash equilibrium of the multi-agent differential graphical game. According to the principle of optimality, this problem can be transformed into solving a set of coupled Hamilton-Jacobi-Bellman (HJB) equations, which are difficult to solve directly because of the nonlinearity and strong coupling of the equations. The proposed algorithm combines policy iteration (PI) and value iteration (VI), and the Nash equilibrium of performance index among agents is achieved distributedly, i.e., all agents update their control inputs only use local information. The convergence of the algorithm is rigorously proved. Neural network (NN) approximation implementation is given for the unknown structures of local value functions of agents. The effectiveness of the algorithm is verified by numerical simulation.

CLC: A Consensus-based Label Correction Approach in Federated Learning

Federated learning (FL) is a novel distributed learning framework where multiple participants collaboratively train a global model without sharing any raw data to preserve privacy. However, data quality may vary among the participants, the most typical of which is label noise. The incorrect label would significantly damage the performance of the global model. In FL, the inaccessibility of raw data makes this issue more challenging. Previously published studies are limited to using a task-specific benchmark-trained model to evaluate the relevance between the benchmark dataset in the server and the local one on the participants’ side. However, such approaches have failed to exploit the cooperative nature of FL itself and are not practical. This paper proposes a Consensus-based Label Correction approach (CLC) in FL, which tries to correct the noisy labels using the developed consensus method among the FL participants. The consensus-defined class-wise information is used to identify the noisy labels and correct them with pseudo-labels. Extensive experiments are conducted on several public datasets in various settings. The experimental results prove the advantage over the state-of-art methods. The link to the source code is https://github.com/bixiao-zeng/CLC.git .

User-Centric Online Gossip Training for Autoencoder-Based CSI Feedback

Recently, the autoencoder framework has shown great potential in reducing the feedback overhead of the downlink channel state information (CSI). In this work, we further find that the user equipment in practical systems occasionally moves in a relatively stable area for a long time, and the corresponding communication environment is relatively stable. A user-centric online training strategy is proposed to further improve CSI feedback performance using the above characteristics. The key idea of the proposed method is to train a new encoder for a specific area without changes to the decoder at the base station. Given that the CSI training samples are insufficient, two data augmentation strategies, including random erasing and random phase shift, are introduced to improve the neural network generalization. In addition, the proposed user-centric online training framework is extended to the multi-user scenario for considerable performance improvement via gossip learning, which is a fully decentralized distributed learning framework and can use crowd intelligence. The simulation results show that the proposed user-centric online gossip training offers a more substantial increase in the feedback accuracy and can considerably improve autoencoder generalization.

GAN-Based Information Leakage Attack Detection in Federated Learning

Federated learning (FL) has been a popular distributed learning framework to reduce privacy risks by keeping private data locally. However, recent work (Hitaj 2017) has demonstrated that sharing model’s parameter updates still leaves FL vulnerable to internal attacks in its training phase. Existing works cannot detect such attacks well. To address this problem, we propose a novel and lightweight detection scheme which selects and analyzes just a few parameter updates of the last convolutional layer in the FL model. Extensive experiments demonstrate that our proposed detection scheme can accurately and efficiently detect the malicious participant in near real time for a scenario with a malicious participant.

Resource Allocation and Trajectory Design in UAV-Aided Cellular Networks Based on Multiagent Reinforcement Learning

In this article, we focus on a downlink cellular network, where multiple unmanned aerial vehicles (UAVs) serve as aerial base stations for ground users through frequency-division multiple access (FDMA). With user locations and channel parameters inaccessible, the UAVs coordinate to make a decision on resource allocation and trajectory design in a decentralized way. Aiming at optimizing both overall and fairness throughput, we model resource allocation and trajectory design as a decentralized partially observable Markov decision process (Dec-POMDP) and propose multiagent reinforcement learning (RL) as a solution. Specifically, we use parameterized deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (P-DQN) for the action space comprising both discrete and continuous actions and the QMIX framework is leveraged to aggregate each UAV’s local critics. For fairness throughput optimization, we introduce an entropy-like fairness indicator to the reward to make the total return decomposable. In addition, we further propose a novel distributed learning framework for overall throughput optimization such that each UAV can contribute its local gradient, and model training can be implemented in parallel without need of observation data sharing among the UAVs. Simulation results show that the proposed multiagent RL approach as well as the distributed learning framework are efficient in model training and present acceptable performance close to that achieved by deterministic optimization, which relies on convention optimization techniques with user locations and channel parameters explicitly known beforehand. For fairness throughput optimization, we also show that ground users achieve individual throughputs close to each other, which verifies the effectiveness of the proposed fairness indicator as the reward definition in the RL framework.

On the Design of Federated Learning in Latency and Energy Constrained Computation Offloading Operations in Vehicular Edge Computing Systems

With the advent of smart vehicles, several new latency-critical and data-intensive applications are emerged in Vehicular Networks (VNs). Computation offloading has emerged as a viable option allowing to resort to the nearby edge servers for remote processing within a requested service latency requirement. Despite several advantages, computation offloading over resource-limited edge servers, together with vehicular mobility, is still a challenging problem to be solved. In particular, in order to avoid additional latency due to out-of-coverage operations, Vehicular Users (VUs) mobility introduces a bound on the amount of data to be offloaded towards nearby edge servers. Therefore, several approaches have been used for finding the correct amount of data to be offloaded. Among others, Federated Learning (FL) has been highlighted as one of the most promising solving techniques, given the data privacy concerns in VNs and limited communication resources. However, FL consumes resources during its operation and therefore incurs an additional burden on resource-constrained VUs. In this work, we aim to optimize the VN performance in terms of latency and energy consumption by considering both the FL and the computation offloading processes while selecting the proper number of FL iterations to be implemented. To this end, we first propose an FL-inspired distributed learning framework for computation offloading in VNs, and then develop a constrained optimization problem to jointly minimize the overall latency and the energy consumed. An evolutionary Genetic Algorithm is proposed for solving the problem in-hand and compared with some benchmarks. The simulation results show the effectiveness of the proposed approach in terms of latency and energy consumption.