PriPrune: Quantifying and Preserving Privacy in Pruned Federated Learning

In this paper, the convergence time of federated learning (FL), when deployed over a realistic wireless network, is studied. In particular, with the considered model, wireless users transmit their local FL models (trained using their locally collected data) to a base station (BS). The BS, acting as a central controller, generates a global FL model using the received local FL models and broadcasts it back to all users. Due to the limited number of resource blocks (RBs) in a wireless network, only a subset of users can be selected and transmit their local FL model parameters to the BS at each learning step. Meanwhile, since each user has unique training data samples and the BS must wait to receive all users' local FL models to generate the global FL model, the FL performance and convergence time will be significantly affected by the user selection scheme. In consequence, it is necessary to design an appropriate user selection scheme that enables all users to execute an FL scheme and efficiently train it. This joint learning, wireless resource allocation, and user selection problem is formulated as an optimization problem whose goal is to minimize the FL convergence time while optimizing the FL performance. To address this problem, a probabilistic user selection scheme is proposed using which the BS will connect to the users, whose local FL models have large effects on its global FL model, with high probabilities. Given the user selection policy, the uplink RB allocation can be determined. To further reduce the FL convergence time, artificial neural networks (ANNs) are used to estimate the local FL models of the users that are not allocated any RBs for local FL model transmission, which enables the BS to include more users' local FL models to generate the global FL model so as to improve the FL convergence speed and performance. Simulation results show that the proposed ANN-based FL scheme can reduce the FL convergence time by up to 53.8%, compared to a standard FL algorithm.

Wireless federated learning (FL) is a collaborative machine learning (ML) framework in which wireless client-devices independently train their ML models and send the locally trained models to the FL server for aggregation. In this paper, we consider the coexistence of privacy-sensitive client-devices and privacy-insensitive yet computing-resource constrained client-devices, and propose an FL framework with a hybrid centralized training and local training. Specifically, the privacy-sensitive client-devices perform local ML model training and send their local models to the FL server. Each privacy-insensitive client-device can have two options, i.e., (i) conducting a local training and then sending its local model to the FL server, and (ii) directly sending its local data to the FL server for the centralized training. The FL server, after collecting the data from the privacy-insensitive client-devices (which choose to upload the local data), conducts a centralized training with the received datasets. The global model is then generated by aggregating (i) the local models uploaded by the client-devices and (ii) the model trained by the FL server centrally. Focusing on this hybrid FL framework, we firstly analyze its convergence feature with respect to the client-devices' selections of local training or centralized training. We then formulate a joint optimization of client-devices' selections of the local training or centralized training, the FL training configuration (i.e., the number of the local iterations and the number of the global iterations), and the bandwidth allocations to the client-devices, with the objective of minimizing the overall latency for reaching the FL convergence. Despite the non-convexity of the joint optimization problem, we identify its layered structure and propose an efficient algorithm to solve it. Numerical results demonstrate the advantage of our proposed FL framework with the hybrid local and centralized training as well as our proposed algorithm, in comparison with several benchmark FL schemes and algorithms.

High-accuracy low-cost privacy-preserving federated learning in IoT systems via adaptive perturbation

In this paper, the convergence time of federated learning (FL), when deployed over a realistic wireless network, is studied. In particular, a wireless network is considered in which wireless users transmit their local FL models (trained using their locally collected data) to a base station (BS). The BS, acting as a central controller, generates a global FL model using the received local FL models and broadcasts it back to all users. Due to the limited number of resource blocks (RBs) in a wireless network, only a subset of users can be selected to transmit their local FL model parameters to the BS at each learning step. Moreover, since each user has unique training data samples, the BS prefers to include all local user FL models to generate a converged global FL model. Hence, the FL training loss and convergence time will be significantly affected by the user selection scheme. Therefore, it is necessary to design an appropriate user selection scheme that can select the users who can contribute toward improving the FL convergence speed more frequently. This joint learning, wireless resource allocation, and user selection problem is formulated as an optimization problem whose goal is to minimize the FL convergence time and the FL training loss. To solve this problem, a probabilistic user selection scheme is proposed such that the BS is connected to the users whose local FL models have significant effects on the global FL model with high probabilities. Given the user selection policy, the uplink RB allocation can be determined. To further reduce the FL convergence time, artificial neural networks (ANNs) are used to estimate the local FL models of the users that are not allocated any RBs for local FL model transmission at each given learning step, which enables the BS to improve the global model, the FL convergence speed, and the training loss. Simulation results show that the proposed approach can reduce the FL convergence time by up to 56% and improve the accuracy of identifying handwritten digits by up to 3%, compared to a standard FL algorithm.

This paper aims to develop a differential private federated learning (FL) scheme with the least artificial noises added while minimizing the energy consumption of participating mobile devices. By observing that some communication efficient FL approaches and even the nature of wireless communications contribute to the differential privacy (DP) preservation of training data on mobile devices, in this paper, we propose to jointly leverage gradient compression techniques (i.e., gradient quantization and sparsification) and additive white Gaussian noises (AWGN) in wireless channels to develop a piggyback DP approach for FL over mobile devices. Even with the piggyback DP approach, information distortion caused by gradient compression and noise perturbation may slow down FL convergence, which in turn consumes more energy of mobile devices for local computing and model update communications. Thus, we theoretically analyze FL convergence and formulate an energy efficient FL optimization under piggyback DP, transmission power, and FL convergence constraints. Furthermore, we propose an efficient iterative algorithm where closed-form solutions for artificial DP noise and power control are derived. Extensive simulation and experimental results demonstrate the effectiveness of the proposed scheme in terms of energy efficiency and privacy preservation.

Federated learning (FL) has been widely acknowledged as a promising solution to training machine learning (ML) model training with privacy preservation. To reduce the traffic overheads incurred by FL systems, edge servers have been included between clients and the parameter server to aggregate clients’ local models. Recent studies on this edge-assisted hierarchical FL scheme have focused on ensuring or accelerating model convergence by coping with various factors, e.g., uncertain network conditions, unreliable clients, heterogeneous compute resources, etc. This paper presents our three new discoveries of the edge-assisted hierarchical FL scheme: 1) it wastes significant time during its two-phase training rounds; 2) it does not recognize or utilize model diversity when producing a global model; and 3) it is vulnerable to model poisoning attacks. To overcome these drawbacks, we propose FedEdge, a novel edge-assisted hierarchical FL scheme that accelerates model training with asynchronous local federated training and adaptive model aggregation. Extensive experiments are conducted on two widely-used public datasets. The results demonstrate that, compared with state-of-the-art FL schemes, FedEdge accelerates model convergence by 1.14 × −3.20 ×, and improves model accuracy by 2.14% - 6.63%.

Federated learning (FL) has attracted growing interest for enabling privacy-preserving machine learning on data stored at multiple users while avoiding moving the data off-device. However, while data never leaves users’ devices, privacy still cannot be guaranteed since significant computations on users’ training data are shared in the form of trained local models. These local models have recently been shown to pose a substantial privacy threat through different privacy attacks such as model inversion attacks. As a remedy, Secure Aggregation (SA) has been developed as a framework to preserve privacy in FL, by guaranteeing the server can only learn the global aggregated model update but not the individual model updates.While SA ensures no additional information is leaked about the individual model update beyond the aggregated model update, there are no formal guarantees on how much privacy FL with SA can actually offer; as information about the individual dataset can still potentially leak through the aggregated model computed at the server. In this work, we perform a first analysis of the formal privacy guarantees for FL with SA. Specifically, we use Mutual Information (MI) as a quantification metric and derive upper bounds on how much information about each user's dataset can leak through the aggregated model update. When using the FedSGD aggregation algorithm, our theoretical bounds show that the amount of privacy leakage reduces linearly with the number of users participating in FL with SA. To validate our theoretical bounds, we use an MI Neural Estimator to empirically evaluate the privacy leakage under different FL setups on both the MNIST and CIFAR10 datasets. Our experiments verify our theoretical bounds for FedSGD, which show a reduction in privacy leakage as the number of users and local batch size grow, and an increase in privacy leakage as the number of training rounds increases. We also observe similar dependencies for the FedAvg and FedProx protocol.

A blockchain-based federated learning approach with secure aggregation in a trusted execution environment for the Internet of Things (IoT). The proposed approach aims to address the privacy and security concerns associated with federated learning in IoT environments. The approach involves using a blockchain to store the learning model and to maintain a distributed ledger of transactions. The learning model is trained on local IoT devices using federated learning techniques, with each device contributing its local data. The aggregation of the model updates is performed securely within a trusted execution environment, using homomorphism encryption and secret sharing techniques. The proposed approach offers several advantages over traditional federated learning approaches, including improved privacy and security, increased scalability, and enhanced trustworthiness. It also enables the creation of a decentralized and democratic learning environment, where each device has an equal say in the learning process. The approach is evaluated using a real-world dataset, and the results demonstrate its effectiveness in terms of accuracy and privacy preservation. The paper concludes that the proposed approach has the potential to enable secure and scalable federated learning in IoT environments, with applications in healthcare, smart cities, and other domains. This paper offers a blockchain-based federated learning (FL) framework with an Intel Software Guard Extension (SGX)-based trusted execution environment (TEE) for safely aggregating local models in the Industrial Internet of Things (IoT) Local models in FL can be modified with by attackers. As a result, a global model derived from manipulated local models may be incorrect. As a result, the proposed system makes use of a blockchain network to secure model aggregation. Each blockchain node contains an SGX-enabled CPU that secures the FL-based aggregating processes required to construct a global model. Blockchain nodes may validate the aggregated model's validity, perform a blockchain consensus method to secure the model's integrity, and add it to the distributed ledger for tamper-proof storage. . Before utilising the aggregated model, each cluster can acquire it from the blockchain and validate its fidelity. To assess the performance of the proposed system, we ran many experiments using various CNN models and datasets

In the Bigdata era, healthcare informatics need exploration of health records to identify hidden patterns. Machine learning and Deep learning techniques provide classification, clustering and prediction tasks. Healthcare data are processed in a centralized architecture pose a single point of failure and difficult to collaborate with different distributions of data to design a robust system. The sensitive data of the healthcare system are private and fragmented, difficult to collaborate for efficient learning models. Federated learning (FL) is a distributed preservation of privacy learning paradigm to address the data sensitiveness and silos. The model is trained with different distributions of data with distributed models to provide the global model. The sensitive data present in the local device model are not shared with the global model but only the gradients are transmitted till the convergence of the model. The privacy-preserving mechanism is essential to protect the model from privacy attacks. Differential privacy preserving is immune to privacy attacks on aggregated data of FL. The striking features of blockchain like decentralization, provenance, immutability, and finality enables a single shared ledger of the patient data and its distribution among the stakeholders with the mitigation of privacy threats. Blockchain provides the secure transaction between the local health model and the global health model during its gradient updation. Differential privacy mechanism with blockchain provides secure E-health data maintenance and data analytics in the distributed healthcare 4.0 industry. This article identifies the current challenges in healthcare informatics and addresses those issues with enabling technologies like FL, blockchain and differential privacy preserving security mechanisms. Communication efficient FL, and fusion learning is identified for E-health data management. The extraction of knowledge structure in healthcare data provides insight into recent trends in the domain and its opportunities in the associated industries. This article in wholesome explores healthcare informatics with the application of artificial learning and security mechanisms.

Artificial Intelligence (AI) has been applied to solve various challenges of real-world problems in recent years. However, the emergence of new AI technologies has brought several problems, especially with regard to communication efficiency, security threats and privacy violations. Towards this end, Federated Learning (FL) has received widespread attention due to its ability to facilitate the collaborative training of local learning models without compromising the privacy of data. However, recent studies have shown that FL still consumes considerable amounts of communication resources. These communication resources are vital for updating the learning models. In addition, the privacy of data could still be compromised once sharing the parameters of the local learning models in order to update the global model. Towards this end, we propose a new approach, namely, Federated Optimisation (FedOpt) in order to promote communication efficiency and privacy preservation in FL. In order to implement FedOpt, we design a novel compression algorithm, namely, Sparse Compression Algorithm (SCA) for efficient communication, and then integrate the additively homomorphic encryption with differential privacy to prevent data from being leaked. Thus, the proposed FedOpt smoothly trade-offs communication efficiency and privacy preservation in order to adopt the learning task. The experimental results demonstrate that FedOpt outperforms the state-of-the-art FL approaches. In particular, we consider three different evaluation criteria; model accuracy, communication efficiency and computation overhead. Then, we compare the proposed FedOpt with the baseline configurations and the state-of-the-art approaches, i.e., Federated Averaging (FedAvg) and the paillier-encryption based privacy-preserving deep learning (PPDL) on all these three evaluation criteria. The experimental results show that FedOpt is able to converge within fewer training epochs and a smaller privacy budget.

Federated learning (FL) refers to a system of training and stabilizing local machine learning models at the global level by aggregating the learning gradients of the models. It reduces the concern of sharing the private data of participating entities for statistical analysis to be carried out at the server. It allows participating entities called clients or users to infer useful information from their raw data. As a consequence, the need to share their confidential information with any other entity or the central entity called server is eliminated. FL can be clearly interpreted as a privacy-preserving version of traditional machine learning and deep learning algorithms. However, despite this being an efficient distributed training scheme, the client’s sensitive information can still be exposed to various security threats from the shared parameters. Since data has always been a major priority for any user or organization, this article is primarily concerned with discussing the significant problems and issues relevant to the preservation of data privacy and the viability and feasibility of several proposed solutions in the FL context. In this work, we conduct a detailed study on FL, the categorization of FL, the challenges of FL, and various attacks that can be executed to disclose the users’ sensitive data used during learning. In this survey, we review and compare different privacy solutions for FL to prevent data leakage and discuss secret sharing (SS)-based security solutions for FL proposed by various researchers in concise form. We also briefly discuss quantum federated learning (QFL) and privacy-preservation techniques in QFL. In addition to these, a comparison and contrast of several survey works on FL is included in this work. We highlight the major applications based on FL. We discuss certain future directions pertaining to the open issues in the field of FL and finally conclude our work.

In this paper, the problem of training federated learning (FL) algorithms over a wireless network with mobile users is studied. In the considered model, several mobile users and a network base station (BS) cooperatively perform an FL algorithm. In particular, the wireless mobile users train their local FL models and send the trained local FL model parameters to the BS. The BS will then integrate the received local FL models to generate a global FL model and send it back to all users. Due to the limited training time at each iteration, the number of users that can transmit their local FL models to the BS will be affected by changes in the users’ locations and wireless channels. In this paper, this joint learning, user selection, and wireless resource allocation problem is formulated as an optimization problem whose goal is to minimize the FL loss function, which captures the FL performance, while meeting the transmission delay requirement. To solve this problem, a closed-form expression for the expected convergence rate of the FL algorithm is first derived to quantify the impact of the users’ mobility and wireless factors on FL. Then, based on the expected FL convergence rate, the user selection and uplink resource allocation is optimized at each FL iteration so as to minimize the FL loss function while satisfying the FL parameter transmission delay requirement. Simulation results show that the proposed approach can reduce the FL loss function value by up to 20% compared to a standard FL algorithm.

Federated Learning (FL) is highly useful for the applications which suffer silo effect and privacy preserving, such as healthcare, finance, education, etc. Existing FL approaches generally do not account for disparities in the quality of local data labels. However, the participants tend to suffer from label noise due to annotators’ varying skill-levels, biases or malicious tampering. In this chapter, we propose an alternative approach to address this challenge. It maintains a small set of benchmark samples on the FL coordinator and quantifies the credibility of the participants’ local data without directly observing them by computing the mutual cross-entropy between performance of the FL model on the local datasets and that of the participant’s local model on the benchmark dataset. Then, a credit-weighted orchestration is performed to adjust the weight assigned to participants in the FL model based on their credibility values. By experimentally evaluating on both synthetic data and real-world data, the results show that the proposed approach effectively identifies participants with noisy labels and reduces their impact on the FL model performance, thereby significantly outperforming existing FL approaches.

Federated learning (FL) is a promising framework for distributed machine learning that trains models without sharing local data while protecting privacy. FL exploits the concept of collaborative learning and builds privacy-preserving models. Nevertheless, the integral features of FL are fraught with problems, such as the disclosure of private information, the unreliability of uploading model parameters to the server, the communication cost, etc. Blockchain, as a decentralized technology, is able to improve the performance of FL without requiring a centralized server and also solves the above problems. In this paper, a systematic literature review on the integration of Blockchain in federated learning was considered with the analysis of the existing FL problems that can be compensated. Through carefully screening, most relevant studies are included and research questions cover the potential security and privacy attacks in traditional federated learning that can be solved by blockchain as well as the characteristics of Blockchain-based FL. In addition, the latest Blockchain-based approaches to federated learning have been studied in-depth in terms of security and privacy, records and rewards, and verification and accountability. Furthermore, open issues related to the combination of Blockchain and FL are discussed. Finally, future research directions for the robust development of Blockchain-based FL systems are proposed.

• We propose an FL model to predict a client’s (or loan requester’s) financial situation by considering variant local epochs for the data holders of the clients (e.g., banks, financial organizations). • We leverage FL strategy that consider customer’s local resources to assign computational task during training. Particularly, the local computational tasks of each FL client is assigned based on their data volume, bandwidth, and network availability. • We analyze our prediction model by considering various batch sizes and client numbers for the training phase. • To the end, we visualize the performance of our FL model comparing with a centralized model, and also with a mean local model, and the best local model in an FL process. In recent years, as economic stability is shaking, and the unemployment rate is growing high due to the COVID-19 effect, assigning credit scoring by predicting consumers’ financial conditions has become more crucial. The conventional machine learning (ML) and deep learning approaches need to share customer’s sensitive information with an external credit bureau to generate a prediction model that opens up the door of privacy leakage. A recently invented privacy-preserving distributed ML scheme referred to as Federated learning (FL) enables generating a target model without sharing local information through on-device model training on edge resources. In this paper, we propose an FL-based application to predict customers’ financial issues by constructing a global learning model that is evolved based on the local models of the distributed agents. The local models are generated by the network agents using their on-device data and local resources. We used the FL concept because the learning strategy does not require sharing any data with the server or any other agent that ensures the preservation of customers’ sensitive data. To that end, we enable partial works from the weak agents that eliminate the issue if the model convergence is retarded due to straggler agents. We also leverage asynchronous FL that cut off the extra waiting time during global model generation. We simulated the performance of our FL model considering a popular dataset, Give me Some Credit (Freshcorn, 2017). We evaluated our proposed method considering a a different number of stragglers and setting up various computational tasks (e.g., local epoch, batch size), and simulated the training loss and testing accuracy of the prediction model. Finally, we compared the F1-score of our proposed model with the existing centralized and decentralized approaches. Our results show that our proposed model achieves an almost identical F1-score as like centralized model even when we set up a skew-level of more than 80 % and outperforms the state-of-the-art FL models by obtaining an average of 5 ∼ 6 % higher accuracy when we have resource-constrained agents within a learning environment.