Inversion Attack Research Articles

How Much Privacy Does Federated Learning with Secure Aggregation Guarantee?

Federated learning (FL) has attracted growing interest for enabling privacy-preserving machine learning on data stored at multiple users while avoiding moving the data off-device. However, while data never leaves users’ devices, privacy still cannot be guaranteed since significant computations on users’ training data are shared in the form of trained local models. These local models have recently been shown to pose a substantial privacy threat through different privacy attacks such as model inversion attacks. As a remedy, Secure Aggregation (SA) has been developed as a framework to preserve privacy in FL, by guaranteeing the server can only learn the global aggregated model update but not the individual model updates.While SA ensures no additional information is leaked about the individual model update beyond the aggregated model update, there are no formal guarantees on how much privacy FL with SA can actually offer; as information about the individual dataset can still potentially leak through the aggregated model computed at the server. In this work, we perform a first analysis of the formal privacy guarantees for FL with SA. Specifically, we use Mutual Information (MI) as a quantification metric and derive upper bounds on how much information about each user's dataset can leak through the aggregated model update. When using the FedSGD aggregation algorithm, our theoretical bounds show that the amount of privacy leakage reduces linearly with the number of users participating in FL with SA. To validate our theoretical bounds, we use an MI Neural Estimator to empirically evaluate the privacy leakage under different FL setups on both the MNIST and CIFAR10 datasets. Our experiments verify our theoretical bounds for FedSGD, which show a reduction in privacy leakage as the number of users and local batch size grow, and an increase in privacy leakage as the number of training rounds increases. We also observe similar dependencies for the FedAvg and FedProx protocol.

Zero-Shot Machine Unlearning

Modern privacy regulations grant citizens the right to be forgotten by products, services and companies. In case of machine learning (ML) applications, this necessitates deletion of data not only from storage archives but also from ML models. Due to an increasing need for regulatory compliance required for ML applications, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">machine unlearning</i> is becoming an emerging research problem. The right to be forgotten requests come in the form of removal of a certain set or class of data from the already trained ML model. Practical considerations preclude retraining of the model from scratch after discarding the deleted data. The few existing studies use either the whole training data, or a subset of training data, or some metadata stored during training to update the model weights for unlearning. However, strict regulatory compliance requires time-bound deletion of data. Thus, in many cases, no data related to the training process or training samples may be accessible even for the unlearning purpose. We therefore ask the question: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">is it possible to achieve unlearning with zero training samples?</i> In this paper, we introduce the novel problem of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">zero-shot machine unlearning</i> that caters for the extreme but practical scenario where zero original data samples are available for use. We then propose two novel solutions for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">zero-shot machine unlearning</i> based on (a) error minimizing-maximizing noise and (b) gated knowledge transfer. These methods remove the information of the forget data from the model while maintaining the model efficacy on the retain data. The zero-shot approach offers good protection against the model inversion attacks and membership inference attacks. We introduce a new evaluation metric, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Anamnesis Index</i> (AIN) to effectively measure the quality of the unlearning method. The experiments show promising results for unlearning in deep learning models on benchmark vision data-sets. The source code is available here: https://github.com/ayu987/zero-shot-unlearning.

FedNets: Federated Learning on Edge Devices Using Ensembles of Pruned Deep Neural Networks

Federated Learning (FL) is an innovative area of machine learning that enables different clients to collaboratively generate a shared model while preserving their data privacy. In a typical FL setting, a central model is updated by aggregating the clients’ parameters of the respective artificial neural network. The aggregated parameters are then sent back to the clients. However, two main challenges are associated with the central aggregation approach. Firstly, most state-of-the-art strategies are not optimised to operate in the presence of certain types of non-iid (not independent and identically distributed) applications and datasets. Secondly, federated learning is vulnerable to various privacy and security concerns related to model inversion attacks, which can be used to access sensitive information from the training data. To address these issues, we propose a novel federated learning strategy FedNets based on ensemble learning. Instead of sharing the parameters of the clients over the network to update a single global model, our approach allows clients to have ensembles of diverse-lightweight models and collaborate by sharing ensemble members. FedNets utilises graph embedding theory to reduce the complexity of running Deep Neural Networks (DNNs) on resource-limited devices. Each Deep Neural Network (DNN) is treated as a graph, from which respective graph embeddings are generated and clustered to determine which part of the DNN should be shared with other clients. Our approach outperforms state-of-the-art FL algorithms such as Federated Averaging (Fed-Avg) and Adaptive Federated Optimisation (Fed-Yogi) in terms of accuracy; on the Federated CIFAR100 dataset (non-iid), FedNets demonstrates a remarkable 63% and 92% improvement in accuracy, respectively. Furthermore, FedNets does not compromise the client’s privacy, as it is safeguarded by the design of the method.

Preserving data privacy in federated learning through large gradient pruning

In federated learning, the server trains a global learning model based on gradient information shared by multiple clients; thus, protecting client data privacy. However, it has been shown that training data can be reconstructed from the shared gradients, which can result in serious privacy breaches (e.g., also known as gradient-based inversion attacks). Popular privacy-preserving methods include those that are perturbation-related, such as differential privacy. However, these methods can result in high utility loss. In this paper, we reveal that large magnitude gradients play an important role in the image reconstructing process, and thus propose two pruning based defense mechanisms (i.e., SLGP and RLGP) for different model architectures. As only very few gradients have been affected, the utility can be maintained. To demonstrate efficiency, We evaluate the impact of our mechanisms on preventing the reconstruction of input images on various model architectures and datasets using state-of-the-art attack methods. The reconstructed images obtained using the gradient processed by our method are unrecognizable while maintaining the original performance of the models.

LDP‐Fed+: A robust and privacy‐preserving federated learning based classification framework enabled by local differential privacy

AbstractAs a distributed learning framework, Federated Learning (FL) allows different local learners/participants to collaboratively train a joint model without exposing their own local data, and offers a feasible solution to legally resolve data islands. However, among them, the data privacy and model security are two challenges. The former means that, if original data are used for trained FL models, various methods can be used to deduce the original data samples, thereby causing the leakage of data. The latter implies that unreliable/malicious participants may affect or destroy the joint FL model, through uploading wrong local model parameters. Therefore, this paper proposes a novel distributed FL training framework, namely LDP‐Fed+, which takes into account differential privacy protection and model security defense. Specifically, firstly, a local perturbation module is added at the local learner side, which perturbs the original data of local learners through feature extraction, binary encoding and decoding, and random response. Then, through using the perturbed data, local neural network model is trained to obtain the network parameters that meet local differential protection, to effectively deal with model inversion attacks. Secondly, a security defense module is added on the server side, which uses the auxiliary model and differential index mechanism to select an appropriate number of local disturbance parameters for aggregation to enhance model security defense and deal with membership inference attacks. The experimental results show that, compared with other federated learning models based on differential privacy, LDP‐Fed+ has stronger robustness for model security and higher accuracy for model training while ensuring strict privacy protection.

A new approach to data differential privacy based on regression models under heteroscedasticity with applications to machine learning repository data

Generation of massive data in the digital age leads to possible violations of individual privacy. The search for personal data becomes an increasingly recurrent exposure today. The present work corresponds to the area of differential privacy, which guarantees data confidentiality and robustness against invasive identification attacks. This area stands out in the literature for its rigorous mathematical basis capable of quantifying the loss of privacy. A differentially private method based on regression models was developed to prevent inversion attacks while retaining model efficacy. In this paper, we propose a novel approach to improve the data privacy based on regression models under heteroscedasticity, a common aspect, but not studied, in practical situations of differential privacy. The influence of privacy restriction on the statistical performance of the estimators of model parameters is evaluated using Monte Carlo simulations, including a study of performance associated with test rejection rates for the proposed approach. The results of the numerical evaluation show high inferential distortion for stricter privacy restrictions. Empirical illustrations with real-world data are presented to show potential applications.

An Approximate Memory Based Defense Against Model Inversion Attacks to Neural Networks

Diverse and comprehensive training data is critical in building robust machine learning (ML) models. However, model inversion attacks (MIA) have demonstrated that an ML model can leak important information about its training dataset. This work examines the existing MIAs and proposes a hardware-oriented solution to protect the training data from such attacks. Our proposed solution – MIDAS: Model Inversion Defenses with an Approximate memory System – intentionally introduces memory faults to thwart MIA without compromising the original ML model. We use detailed SPICE simulations to build the DRAM fault model with voltage overscaling, implement the state-of-the-art MIAs, and evaluate our proposed solution. Our experiments demonstrate that MIDAS can effectively protect training data from run-time adversarial attacks. In terms of the Pearson Correlation Coefficient (PCC) similarity measure (between the original and the recovered training data), MIDAS reduces the PCC value for shallow and deep neural networks.

Recover User's Private Training Image Data by Gradient in Federated Learning.

Exchanging gradient is a widely used method in modern multinode machine learning system (e.g., distributed training, Federated Learning). Gradients and weights of model has been presumed to be safe to delivery. However, some studies have shown that gradient inversion technique can reconstruct the input images on the pixel level. In this study, we review the research work of data leakage by gradient inversion technique and categorize existing works into three groups: (i) Bias Attacks, (ii) Optimization-Based Attacks, and (iii) Linear Equation Solver Attacks. According to the characteristics of these algorithms, we propose one privacy attack system, i.e., Single-Sample Reconstruction Attack System (SSRAS). This system can carry out image reconstruction regardless of whether the label can be determined. It can extends gradient inversion attack from a fully connected layer with bias terms to attack a fully connected layer and convolutional neural network with or without bias terms. We also propose Improved R-GAP Alogrithm, which can utlize DLG algorithm to derive ground truth. Furthermore, we introduce Rank Analysis Index (RA-I) to measure the possible of whether the user’s raw image data can be reconstructed. This rank analysis derive virtual constraints from weights. Compared with the most representative attack algorithms, this reconstruction attack system can recover a user’s private training image with high fidelity and attack success rate. Experimental results also show the superiority of the attack system over some other state-of-the-art attack algorithms.

Private Data Inference Attacks against Cloud: Model, Technologies, and Research Directions

Machine learning models are established with a variety of data collected from individual users who are concerned about their privacy. Various cloud service providers (e.g., Amazon, Google, Alibaba) commercialize their models by selling them or providing limited access via APIs. However, existing studies have shown that such cloud-based models are susceptible to training data inference attacks (i.e., membership inference attacks, attribute inference attacks, and model inversion attacks) since the trained models remember information about the training data. In this article, we perform an exhaustive investigation on the current training data inference attacks against cloud-based models. According to the attack scenario, we divide the existing inference attacks into two categories: centralized and collaborative learning scenarios. We first give a comprehensive review of attacks in each category and then give a qualitative comparison of the existing attacks. Further, we quantitatively compare the performance of different attack strategies by experiments. Last but not least, we pinpoint some open issues and potential future directions that need to be investigated in this area.

An effective and practical gradient inversion attack

An effective and practical gradient inversion attack

MediSecFed: Private and Secure Medical Image Classification in the Presence of Malicious Clients

Deep learning demonstrates its efficacy and potential to solve challenging computer vision problems in medical and other industrial applications. Federated learning is a learning paradigm that facilitates collaborative learning in a federation of users without exchanging actual data with a single authority like a server. However, federated learning provides only a basic level of privacy and robustness and is vulnerable to model poisoning and model inversion attacks in hostile training environments. Hence, in this article, we propose MediSecFed—a secure framework for federated learning in a hostile environment. Compared to the widely used FedAvg, our method relies on simple and practical ideas from knowledge distillation and model inversion to ensure additional security and privacy features. Our approach achieves knowledge exchange among participating entities without sharing model parameters as FedAvg does, thus protecting the privacy of the local data from the server and significantly reducing communication costs. We evaluate our method on two chest X-ray datasets. Our method outperforms FedAvg by 15% on both datasets in a hostile environment. Our method will also continue to maintain good performance even if the number of malicious participating entities increases. Robustness to learn in a malicious environment while preserving privacy with reduced communication costs makes our method more desirable and efficient than that of FedAvg.

Perturbation-enabled Deep Federated Learning for Preserving Internet of Things-based Social Networks

Federated Learning (FL), as an emerging form of distributed machine learning (ML), can protect participants’ private data from being substantially disclosed to cyber adversaries. It has potential uses in many large-scale, data-rich environments, such as the Internet of Things (IoT), Industrial IoT, Social Media (SM), and the emerging SM 3.0. However, federated learning is susceptible to some forms of data leakage through model inversion attacks. Such attacks occur through the analysis of participants’ uploaded model updates. Model inversion attacks can reveal private data and potentially undermine some critical reasons for employing federated learning paradigms. This article proposes novel differential privacy (DP)-based deep federated learning framework. We theoretically prove that our framework can fulfill DP’s requirements under distinct privacy levels by appropriately adjusting scaled variances of Gaussian noise. We then develop a Differentially Private Data-Level Perturbation (DP-DLP) mechanism to conceal any single data point’s impact on the training phase. Experiments on real-world datasets, specifically the social media 3.0, Iris, and Human Activity Recognition (HAR) datasets, demonstrate that the proposed mechanism can offer high privacy, enhanced utility, and elevated efficiency. Consequently, it simplifies the development of various DP-based FL models with different tradeoff preferences on data utility and privacy levels.

An empirical analysis of image augmentation against model inversion attack in federated learning

Federated Learning (FL) is a technology that facilitates a sophisticated way to train distributed data. As the FL does not expose sensitive data in the training process, it was considered privacy-safe deep learning. However, a few recent studies proved that it is possible to expose the hidden data by exploiting the shared models only. One common solution for the data exposure is differential privacy that adds noise to hinder such an attack, however, it inevitably involves a trade-off between privacy and utility. This paper demonstrates the effectiveness of image augmentation as an alternative defense strategy that has less impact of the trade-off. We conduct comprehensive experiments on the CIFAR-10 and CIFAR-100 datasets with 14 augmentations and 9 magnitudes. As a result, the best combination of augmentation and magnitude for each image class in the datasets was discovered. Also, our results show that a well-fitted augmentation strategy can outperform differential privacy.

A Secure High-Order Gene Interaction Detecting Method for Infectious Diseases.

Infectious diseases pose a serious threat to human life, the Genome Wide Association Studies (GWAS) can analyze susceptibility genes of infectious diseases from the genetic level and carry out targeted prevention and treatment. The susceptibility genes for infectious diseases often act in combination with multiple susceptibility sites; therefore, high-order epistasis detection has become an important means. However, due to intensive computational burden and diversity of disease models, existing methods have drawbacks on low detection power, high computation cost, and preference for some types of disease models. Furthermore, these methods are exposed to repeated query and model inversion attacks in the process of iterative optimization, which may disclose Single Nucleotide Polymorphism (SNP) information associated with individual privacy. Therefore, in order to solve these problems, this paper proposed a safe harmony search algorithm for high-order gene interaction detection, termed as HS-DP. Firstly, the linear weighting method was used to integrate 5 objective functions to screen out high-order SNP sets with high correlation, including K2-Score, JS divergence, logistic regression, mutual information, and Gini. Then, based on the Differential Privacy (DP) theory, the function disturbance mechanism was introduced to protect the security of individual privacy information associated with the objective function, and we proved the rationality of the disturbance mechanism theoretically. Finally, the practicability and superiority of the algorithm were verified by experiments. Experimental results showed that the algorithm proposed in this paper could improve the detection accuracy to the greatest extent while guaranteeing privacy.

Toward Industrial Private AI: A Two-Tier Framework for Data and Model Security

With the advances in 5G and IoT devices, industries are vastly adopting artificial intelligence (AI) techniques for improving classification and prediction-based services. However, the use of AI also raises concerns regarding data privacy and security that can be misused or leaked. Private AI was recently coined to address the data security issue by combining AI with encryption techniques, but existing studies have shown that model inversion attacks can be used to reverse engineer the images from model parameters. In this regard, we propose a federated learning and encryption-based private (FLEP) AI framework that provides two-tier security for data and model parameters in an Industrial IoT environment. We propose a three-layer encryption method for data security and provided a hypothetical method to secure the model parameters. Experimental results show that the proposed method achieves better encryption quality at the expense of slightly increased execution time. We also highlight several open issues and challenges regarding the FLEP AI framework's realization.

Inversion Attacks against CNN Models Based on Timing Attack

Model confidentiality attacks on convolutional neural networks (CNN) are becoming more and more common. At present, model reverse attack is an important means of model confidentiality attacks, but all of these attacks require strong attack ability, meanwhile, the success rates of these attacks are low. We study the time leakage of CNN running on the SoC (system on-chip) system and propose a reverse method based on side-channel attack. It uses the SDK tool-profiler to collect the time leakage of different networks of various CNNs. According to the linear relationship between time leakage, calculation, and memory usage parameters, we take the profiling attack to establish a mapping library of time and the different networks. After that, the smallest difference between the measured time of unknown models and the theoretical time in the mapping library is considered to be the real parameters of the unknown models. Finally, we can reverse other layers even the entire model. Based on the experiments, the reverse success rate of common convolutional layers is above 78.5%, and the reverse success rates of different CNNs (such as AlexNet, ConvNet, LeNet, etc.) are all above 67.67%. Moreover, the results show that the success rate of our method is 10% higher than the traditional methods on average. In the adversarial sample attack, the success rate reached 97%.

A privacy-preserving and verifiable federated learning method based on blockchain

As a novel distributed learning mechanism, federated learning has drawn widespread attention by allowing multiple parties to train an accurate model collaboratively without collecting their raw data. However, it relies on a trustworthy central server that still suffers from severe security challenges, such as model inversion attack and single point of failure. Thereby, a privacy-preserving and verifiable federated learning method based on blockchain is proposed to enable fully decentralized and reliable federated learning in untrusted network. In our scheme, we propose a secure aggregation protocol to guarantee the confidentiality of gradients while supporting clients dropping out during the workflow, and design a novel blockchain structure enabling global gradient verification to defend against potential tampering attack. In addition, a gradient compression method is proposed to reduce the communication overhead. Security analysis shows that our scheme can preserve the privacy by adding pairwise random masks to the gradients, and prevent Sybil attack by reasonable threshold setting in verifiable secret sharing. Experimental results on two real-world datasets illustrate that when the clients’ dropout rate is less than 20%, our scheme can achieve almost the same accuracy as original federated learning, and performs better than similar blockchain-based federated learning methods in terms of computation overhead and communication overhead.

One Parameter Defense—Defending Against Data Inference Attacks via Differential Privacy

Machine learning models are vulnerable to data inference attacks, such as membership inference and model inversion attacks. In these types of breaches, an adversary attempts to infer a data record’s membership in a dataset or even reconstruct this data record using a confidence score vector predicted by the target model. However, most existing defense methods only protect against membership inference attacks. Methods that can combat both types of attacks require a new model to be trained, which may not be time-efficient. In this paper, we propose a differentially private defense method that handles both types of attacks in a time-efficient manner by tuning only one parameter, the privacy budget. The central idea is to modify and normalize the confidence score vectors with a differential privacy mechanism which preserves privacy and obscures membership and reconstructed data. Moreover, this method can guarantee the order of scores in the vector to avoid any loss in classification accuracy. The experimental results show the method to be an effective and timely defense against both membership inference and model inversion attacks with no reduction in accuracy.

Model Inversion Attack by Integration of Deep Generative Models: Privacy-Sensitive Face Generation From a Face Recognition System

Cybersecurity in front of attacks to a face recognition system is an emerging issue in the cloud era, especially due to its strong bonds with the privacy of the users registered to the system. A possible attack is the model inversion attack (MIA) which aims to reveal the identity of a targeted user by generating the most proper datapoint input to the system with maximum corresponding confidence score at the output. The generated data of a registered user can be maliciously used as a serious invasion of the user privacy. In literature, MIA processes are categorized into white-box and black-box scenarios which are respectively with and without information about the system structure, parameters, and partially about the users. This research work assumes the MIA under semi-white box scenario of availability of system model structure and parameters but not any user data information, and verifies it as a severe threat even for a deep-learning-based face recognition system despite its complex structure and the diversity of registered user data. The alert state is promoted by Deep MIA which is the integration of deep generative models in MIA, and <inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>-GAN integrated MIA-initilized by a face based seed (<inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>-GAN-MIA-FS) is proposed. As a novel MIA search strategy, a pre-trained deep generative model with capability of generating a face image from a random feature vector is used for narrowing down the image search space to the feature vectors space, which has much lower dimensions. This allows the MIA process to efficiently search for a low-dimensional feature vector whose corresponding face image maximizes the confidence score. We have experimentally evaluated the proposed method by two objective criteria and three subjective criteria in comparison to <inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>-GAN-integrated MIA initialized with a random seed (<inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>-GAN-MIA-RS), DCGAN-integrated MIA (DCGAN-MIA), and the conventional MIA. The evaluation results approve the efficiency and superiority of the proposed technique in generating natural looking face clones with high recognizability as the targeted users.

Masking Feedforward Neural Networks Against Power Analysis Attacks

Abstract Recent advances in machine learning have enabled Neural Network (NN) inference directly on constrained embedded devices. This local approach enhances the privacy of user data, as the inputs to the NN inference are not shared with third-party cloud providers over a communication network. At the same time, however, performing local NN inference on embedded devices opens up the possibility of Power Analysis attacks, which have recently been shown to be effective in recovering NN parameters, as well as their activations and structure. Knowledge of these NN characteristics constitutes a privacy threat, as it enables highly effective Membership Inference and Model Inversion attacks, which can recover information about the sensitive data that the NN model was trained on. In this paper we address the problem of securing sensitive NN inference parameters against Power Analysis attacks. Our approach employs masking, a countermeasure well-studied in the context of cryptographic algorithms. We design a set of gadgets, i.e., masked operations, tailored to NN inference. We prove our proposed gadgets secure against power attacks and show, both formally and experimentally, that they are composable, resulting in secure NN inference. We further propose optimizations that exploit intrinsic characteristics of NN inference to reduce the masking’s runtime and randomness requirements. We empirically evaluate the performance of our constructions, showing them to incur a slowdown by a factor of about 2–5.