Membership Privacy Research Articles

MemberShield: A framework for federated learning with membership privacy

Federated Learning (FL) allows multiple data owners to build high-quality deep learning models collaboratively, by sharing only model updates and keeping data on their premises. Even though FL offers privacy-by-design, it is vulnerable to membership inference attacks (MIA), where an adversary tries to determine whether a sample was included in the training data. Existing defenses against MIA cannot offer meaningful privacy protection without significantly hampering the model’s utility and causing a non-negligible training overhead. In this paper we analyze the underlying causes of the differences in the model behavior for member and non-member samples, which arise from model overfitting and facilitate MIAs. Accordingly, we propose MemberShield, a generalization-based defense method for MIAs that consists of: (i) one-time preprocessing of each client’s training data labels that transforms one-hot encoded labels to soft labels and eventually exploits them in local training, and (ii) early stopping the training when the local model’s validation accuracy does not improve on that of the global model for a number of epochs. Extensive empirical evaluations on three widely used datasets and four model architectures demonstrate that MemberShield outperforms state-of-the-art defense methods by delivering substantially better practical privacy protection against all forms of MIAs, while better preserving the target model utility. On top of that, our proposal significantly reduces training time and is straightforward to implement, by just tuning a single hyperparameter.

Dual defense: Combining preemptive exclusion of members and knowledge distillation to mitigate membership inference attacks

Membership inference (MI) attacks threaten user privacy through determining if a given data example has been used to train a target model. Existing MI defenses protect the membership privacy through preemptive exclusion of members techniques and knowledge distillation. Unfortunately, using either of these two defenses alone, the defense effect can still offers an unsatisfactory trade-off between membership privacy and utility.Given that the defense method that directly combines these two defenses is still very limited (e.g., the test accuracy of the target model is decreased by about 40% (in our experiments)), in this work, we propose a dual defense (DD) method that includes the preemptive exclusion of high-risk member samples module and the knowledge distillation module, which thwarts the access of the resulting models to the private training data twice to mitigate MI attacks. Our defense method can be divided into two steps: the preemptive exclusion of high-risk member samples (Step 1) and the knowledge distillation to obtain the protected student model (Step 2). We propose three types of exclusions: existing MI attacks-based exclusions, sample distances of members and nonmembers-based exclusions, and mutual information value-based exclusions, to preemptively exclude the high-risk member samples. During the knowledge distillation phase, we add ground-truth labeled data to the reference dataset to decrease the protected student model's dependency on soft labels, aiming to maintain or improve its test accuracy. Extensive evaluation shows that DD significantly outperforms state-of-the-art defenses and offers a better privacy-utility trade-off. For example, DD achieves ∼100% test accuracy improvement over the distillation for membership privacy (DMP) defense for ResNet50 trained on CIFAR100. DD simultaneously achieves the reductions in the attack effectiveness (e.g., the TPR@0.01%FPR of enhanced MI attacks decreased by 2.10% on the ImageNet dataset, the membership advantage (MA) of risk score-based attacks decreased by 56.30%) and improvements of the target models' test accuracies (e.g., by 42.80% on CIFAR100).

Differentially private federated learning with Laplacian smoothing

Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. However, an adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models. In this paper, we investigate a utility enhancement scheme based on Laplacian smoothing for differentially private federated learning (DP-Fed-LS), to improve the statistical precision of parameter aggregation with injected Gaussian noise without losing privacy budget. Our key observation is that the aggregated gradients in federated learning often enjoy a type of smoothness, i.e. sparsity in a graph Fourier basis with polynomial decays of Fourier coefficients as frequency grows, which can be exploited by the Laplacian smoothing efficiently. Under a prescribed differential privacy budget, convergence error bounds with tight rates are provided for DP-Fed-LS with uniform subsampling of heterogeneous non-iid data, revealing possible utility improvement of Laplacian smoothing in effective dimensionality and variance reduction, among others. Experiments over MNIST, SVHN, and Shakespeare datasets show that the proposed method can improve model accuracy with DP-guarantee and membership privacy under both uniform and Poisson subsampling mechanisms.

Secure Decentralized Aggregation to Prevent Membership Privacy Leakage in Edge-Based Federated Learning

Secure Decentralized Aggregation to Prevent Membership Privacy Leakage in Edge-Based Federated Learning

Defending against membership inference attacks: RM Learning is all you need

Large-capacity machine learning models are vulnerable to membership inference attacks that disclose the privacy of the training dataset. The privacy concerns posed by membership inference attacks have inspired many defense strategies. Unfortunately, they have various limitations: 1) failing to achieve a high-quality tradeoff between membership privacy and model utility. 2) Providing little flexibility to users.This paper proposes Repair-Membership Learning (RM Learning), a simple but effective defense mechanism. RM Learning mitigates membership inference attacks by giving each sample of the training dataset an appropriate soft label to reduce the loss gap between the training and testing datasets of the model. Meanwhile, RM Learning controls the distinguishability between the training and testing datasets by limiting the model's loss gap over them with an adjustable parameter λ to give users flexibility. We evaluated the RM Learning algorithm on four datasets with seven models and compared it with three state-of-the-art methods. For instance, we reduce the effectiveness of black-box and white-box membership inference attacks to about 50.0%, with a loss of less than 1.3% of testing accuracy, for the CIFAR10 dataset with the ResNet18 model.

SEDMA: Self-Distillation with Model Aggregation for Membership Privacy

Membership inference attacks (MIAs) are important measures to evaluate potential risks of privacy leakage from machine learning (ML) models. State-of-the-art MIA defenses have achieved favorable privacy-utility trade-offs using knowledge distillation on split training datasets. However, such defenses increase computational costs as a large number of the ML models must be trained on the split datasets. In this study, we proposed a new MIA defense, called SEDMA, based on self-distillation using model aggregation to mitigate the MIAs, inspired by the model parameter averaging as used in federated learning. The key idea of SEDMA is to split the training dataset into several parts and aggregate multiple ML models trained on each split for self-distillation. The intuitive explanation of SEDMA is that model aggregation prevents model over-fitting by smoothing information related to the training data among the multiple ML models and preserving the model utility, such as in federated learning. Through our experiments on major benchmark datasets (Purchase100, Texas100, and CIFAR100), we show that SEDMA outperforms state-of-the-art MIA defenses in terms of membership privacy (MIA accuracy), model accuracy, and computational costs. Specifically, SEDMA incurs at most approximately 3 - 5% model accuracy drop, while achieving the lowest MIA accuracy in state-of-the-art empirical MIA defenses. For computational costs, SEDMA takes significantly less processing time than a defense with the state-of-the-art privacy-utility trade-offs in previous defenses. SEDMA achieves both favorable privacy-utility trade-offs and low computational costs.

GradDiff: Gradient-based membership inference attacks against federated distillation with differential comparison

Membership inference attacks (MIAs) has demonstrated a great threat to federated learning (FL) and its extension federated distillation (FD). However, existing research on MIAs against FD is insufficient. In this paper, we propose a novel membership inference attack named GradDiff, which is a passive gradient-based MIA employing differential comparison. Additionally, to make full use of the federated training process, we also design the gradient drift attack (GradDrift), an active version of GradDiff, in which the attacker modifies the target model by gradient tuning and is able to obtain more information about membership privacy. We conduct extensive experiments on three real-world datasets to evaluate the effectiveness of the proposed attacks. The results show that our proposed attacks can outperform the existing baseline methods in terms of precision and recall. Besides, we perform a thorough investigation of the factors that may influence the performance of MIAs against FD.

Membership reconstruction attack in deep neural networks

To further enhance the reliability of Machine Learning (ML) systems, considerable efforts have been dedicated to developing privacy protection techniques. Recently, membership privacy has gained increasing attention, with a focus on determining whether a specific data point is present in the confidential training set of an ML model. However, most current attacks only prioritize attack accuracy and fail to extend their range to the evidence that contributes to the member/non-member classification. This limitation greatly reduces the practicality of Membership Inference Attacks (MIA), as real-world data typically includes multiple features, making it challenging to identify which features are involved in the sensitive training set. Therefore, this paper targets one of the fundamental challenges in membership inference attack: measuring the distance between an attack sample and a member sample. Specifically, we propose a novel threat model called Membership Reconstruction Attack (MRA), which aims to reconstruct the exact distribution of the target training set. MRA achieves this by marking each input dimension (e.g., pixels) according to its similarity to the target dataset in feature space. Our attack demonstrates its effectiveness across various settings, including different major datasets (MNIST, CIFAR-10, CIFAR-100) and different model architectures (AlexNet, ResNet, DenseNet, and generative models). Additionally, we evaluate MRA from the defenders' perspective and test several defense approaches against our attack.

Providing Membership Privacy to the Asynchronous Ratcheting Trees Protocol without losing Scalability

A secure messaging protocol, such as the Signal protocol, provides end-to-end encrypted asynchronous communication. This paper focuses on a secure group messaging (SGM) protocol, and proposes a method capable of hiding membership information from the viewpoint of non group members, which we call “membership privacy”. In this work, we add membership privacy to the Asynchronous Ratcheting Trees (ART) protocol (proposed by Cohn-Gordon et al., (ACM CCS 2018)). For hiding membership-related values in the setup phase, we employ a key-private and robust publickey encryption (Abdalla et al., TCC2010/JoC2018). Moreover, we introduce a group common key to encrypt membership information in the key update phase. Our modification achieves asymptotically the same efficiency of the ART protocol in the setup phase. Any additional cost for key update does not depend on the number of group members. Therefore, the proposed protocol can add membership privacy to the ART protocol with a quite small overhead. Specifically, one encryption and decryption of a symmetric-key encryption scheme and one execution of a key-derivation function for each key update are employed. Finally, we discuss how to extend our protocol to provide sender-specific authentication, dynamic groups, group-size hiding, and how to adopt our technique to the Messaging Layer Security (MLS) protocol. We note that, although Chase et al. (ACM CCS 2020) have considered the same notion, their proposal is an extension of Signal so called “Pairwise Signal” where a group message is repeatedly sent over individual Signal channels. Thus their protocol is not scalable.

Mitigating Membership Inference in Deep Learning Applications with High Dimensional Genomic Data.

The use of deep learning techniques in medical applications holds great promises for advancing health care. However, there are growing privacy concerns regarding what information about individual data contributors (i.e., patients in the training set) these deep models may reveal when shared with external users. In this work, we first investigate the membership privacy risks in sharing deep learning models for cancer genomics tasks, and then study the applicability of privacy-protecting strategies for mitigating these privacy risks.

Knowledge Cross-Distillation for Membership Privacy

AbstractAmembership inference attack (MIA)poses privacy risks for the training data of a machine learning model. With an MIA, an attacker guesses if the target data are a member of the training dataset. The state-of-the-art defense against MIAs, distillation for membership privacy (DMP), requires not only private data for protection but a large amount of unlabeled public data. However, in certain privacy-sensitive domains, such as medicine and finance, the availability of public data is not guaranteed. Moreover, a trivial method for generating public data by using generative adversarial networks significantly decreases the model accuracy, as reported by the authors of DMP. To overcome this problem, we propose a novel defense against MIAs that uses knowledge distillation without requiring public data. Our experiments show that the privacy protection and accuracy of our defense are comparable to those of DMP for the benchmark tabular datasets used in MIA research, Purchase100 and Texas100, and our defense has a much better privacy-utility trade-off than those of the existing defenses that also do not use public data for the image dataset CIFAR10.

Learning in Your “Pocket”: Secure Collaborative Deep Learning with Membership Privacy

Organizations tend to collaboratively train the deep learning model over their combined datasets for a common benefit (e.g., better-trained model or learning a complicated model). However, due to the consideration about privacy leakage, organizations cannot share their data directly, especially related to sensitive domains. In this paper, a privacy-preserving collaborative deep learning mechanism, namely Sigma, is designed to allow participating organizations to train a collective model without exposing their local training data to the others. Specifically, a single-server-aided private collaborative architecture is introduced to achieve the private collaborative learning, which protects organizations&#x2019; data even if <inline-formula><tex-math notation="LaTeX">$n-1$</tex-math></inline-formula> out of <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> participants colluded. We also design a practical protocol to perform the secure model training, which can resist the typical inference attack through the sharing information. After that, we propose a fair model releasing mechanism for participants and introduce differential privacy to prevent model stealing and membership inference attack. Furthermore, we prove that Sigma can ensure participants&#x2019; privacy preservation and analyze the communication overhead in theory. To evaluate the effectiveness and efficiency of Sigma, we conduct an experiment over two real-world datasets and the simulation results demonstrate that Sigma can efficiently achieve the collaborative model training and effectively resist the membership inference attack.

Demystifying Membership Inference Attacks in Machine Learning as a Service

Membership inference attacks seek to infer membership of individual training instances of a model to which an adversary has black-box access through a machine learning-as-a-service API. In providing an in-depth characterization of membership privacy risks against machine learning models, this paper presents a comprehensive study towards demystifying membership inference attacks from two complimentary perspectives. First, we provide a generalized formulation of the development of a black-box membership inference attack model. Second, we characterize the importance of model choice on model vulnerability through a systematic evaluation of a variety of machine learning models and model combinations using multiple datasets. Through formal analysis and empirical evidence from extensive experimentation, we characterize under what conditions a model may be vulnerable to such black-box membership inference attacks. We show that membership inference vulnerability is data-driven and corresponding attack models are largely transferable. Though different model types display different vulnerabilities to membership inference, so do different datasets. Our empirical results additionally show that (1) using the type of target model under attack within the attack model may not increase attack effectiveness and (2) collaborative learning exposes vulnerabilities to membership inference risks when the adversary is a participant. We also discuss countermeasure and mitigation strategies.

Membership Privacy for Machine Learning Models Through Knowledge Transfer

Large capacity machine learning (ML) models are prone to membership inference attacks (MIAs), which aim to infer whether the target sample is a member of the target model's training dataset. The serious privacy concerns due to the membership inference have motivated multiple defenses against MIAs, e.g., differential privacy and adversarial regularization. Unfortunately, these defenses produce ML models with unacceptably low classification performances. Our work proposes a new defense, called distillation for membership privacy (DMP), against MIAs that preserves the utility of the resulting models significantly better than prior defenses. DMP leverages knowledge distillation to train ML models with membership privacy. We provide a novel criterion to tune the data used for knowledge transfer in order to amplify the membership privacy of DMP. Our extensive evaluation shows that DMP provides significantly better tradeoffs between membership privacy and classification accuracies compared to state-of-the-art MIA defenses. For instance, DMP achieves ~100% accuracy improvement over adversarial regularization for DenseNet trained on CIFAR100, for similar membership privacy (measured using MIA risk): when the MIA risk is 53.7%, adversarially regularized DenseNet is 33.6% accurate, while DMP-trained DenseNet is 65.3% accurate. We have released our code at github.com/vrt1shjwlkr/AAAI21-MIA-Defense.

Quantifying Membership Privacy via Information Leakage

Machine learning models are known to memorize the unique properties of individual data points in a training set. This memorization capability can be exploited by several types of attacks to infer information about the training data, most notably, membership inference attacks. In this paper, we propose an approach based on information leakage for guaranteeing membership privacy. Specifically, we propose to use a conditional form of the notion of maximal leakage to quantify the information leaking about individual data entries in a dataset, i.e., the entrywise information leakage. We apply our privacy analysis to the Private Aggregation of Teacher Ensembles (PATE) framework for privacy-preserving classification of sensitive data and prove that the entrywise information leakage of its aggregation mechanism is Schur-concave when the injected noise has a log-concave probability density. The Schur-concavity of this leakage implies that increased consensus among teachers in labeling a query reduces its associated privacy cost. Finally, we derive upper bounds on the entrywise information leakage when the aggregation mechanism uses Laplace distributed noise.

Measuring Membership Privacy on Aggregate Location Time-Series

While location data is extremely valuable for various applications, disclosing it prompts serious threats to individuals' privacy. To limit such concerns, organizations often provide analysts with aggregate time-series that indicate, e.g., how many people are in a location at a time interval, rather than raw individual traces. In this paper, we perform a measurement study to understand Membership Inference Attacks (MIAs) on aggregate location time-series, where an adversary tries to infer whether a specific user contributed to the aggregates. We find that the volume of contributed data, as well as the regularity and particularity of users' mobility patterns, play a crucial role in the attack's success. We experiment with a wide range of defenses based on generalization, hiding, and perturbation, and evaluate their ability to thwart the attack vis-à-vis the utility loss they introduce for various mobility analytics tasks. Our results show that some defenses fail across the board, while others work for specific tasks on aggregate location time-series. For instance, suppressing small counts can be used for ranking hotspots, data generalization for forecasting traffic, hotspot discovery, and map inference, while sampling is effective for location labeling and anomaly detection when the dataset is sparse. Differentially private techniques provide reasonable accuracy only in very specific settings, e.g., discovering hotspots and forecasting their traffic, and more so when using weaker privacy notions like crowd-blending privacy. Overall, our measurements show that there does not exist a unique generic defense that can preserve the utility of the analytics for arbitrary applications, and provide useful insights regarding the disclosure of sanitized aggregate location time-series.

LocMIA: Membership Inference Attacks Against Aggregated Location Data

An increasing amount of users' locations are aggregated, and the statistical results about the collected data are further released to support mobile applications, such as point-of-interest recommendation and smart transportation. However, such statistical results cause users' membership privacy leakage. Unfortunately, most studies concerning data aggregation focused on privacy preservation and various attacks rather than the membership inference attacks. Moreover, literature about membership inference attacks mainly aimed at machine learning models and gene sequences rather than the locations in data aggregation. More importantly, these work concerning membership inference attacks assumed that adversaries know the exact data of victims, which is always impossible in practical scenarios. To this end, we propose LocMIA, a more invasive attack system that allows adversaries to launch membership inference attacks against aggregated location data without reliance on any prior knowledge of the locations of victims. The main idea of LocMIA is to train a binary classifier to infer whether a specific victim's location data is involved in the aggregation group, based solely on the data aggregation's output (i.e., the statistical results). Finally, experimental results on a real-world check-in data set prove the devastating privacy leaks caused by the proposed LocMIA.

Measuring Membership Privacy on Aggregate Location Time-Series

While location data is extremely valuable for various applications, disclosing it prompts serious threats to individuals' privacy. To limit such concerns, organizations often provide analysts with aggregate time-series that indicate, e.g., how many people are in a location at a time interval, rather than raw individual traces. In this paper, we perform a measurement study to understand Membership Inference Attacks (MIAs) on aggregate location time-series, where an adversary tries to infer whether a specific user contributed to the aggregates. We find that the volume of contributed data, as well as the regularity and particularity of users' mobility patterns, play a crucial role in the attack's success. We experiment with a wide range of defenses based on generalization, hiding, and perturbation, and evaluate their ability to thwart the attack vis-à-vis the utility loss they introduce for various mobility analytics tasks. Our results show that some defenses fail across the board, while others work for specific tasks on aggregate location time-series. For instance, suppressing small counts can be used for ranking hotspots, data generalization for forecasting traffic, hotspot discovery, and map inference, while sampling is effective for location labeling and anomaly detection when the dataset is sparse. Differentially private techniques provide reasonable accuracy only in very specific settings, e.g., discovering hotspots and forecasting their traffic, and more so when using weaker privacy notions like crowd-blending privacy. Overall, our measurements show that there does not exist a unique generic defense that can preserve the utility of the analytics for arbitrary applications, and provide useful insights regarding the disclosure of sanitized aggregate location time-series.

Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics

Bayesian deep learning is recently regarded as an intrinsic way to characterize the weight uncertainty of deep neural networks (DNNs). Stochastic Gradient Langevin Dynamics (SGLD) is an effective method to enable Bayesian deep learning on large-scale datasets. Previous theoretical studies have shown various appealing properties of SGLD, ranging from the convergence properties to the generalization bounds. In this paper, we study the properties of SGLD from a novel perspective of membership privacy protection (i.e., preventing the membership attack). The membership attack, which aims to determine whether a specific sample is used for training a given DNN model, has emerged as a common threat against deep learning algorithms. To this end, we build a theoretical framework to analyze the information leakage (w.r.t. the training dataset) of a model trained using SGLD. Based on this framework, we demonstrate that SGLD can prevent the information leakage of the training dataset to a certain extent. Moreover, our theoretical analysis can be naturally extended to other types of Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods. Empirical results on different datasets and models verify our theoretical findings and suggest that the SGLD algorithm can not only reduce the information leakage but also improve the generalization ability of the DNN models in real-world applications.

Investigating Statistical Privacy Frameworks from the Perspective of Hypothesis Testing

Abstract Over the last decade, differential privacy (DP) has emerged as the gold standard of a rigorous and provable privacy framework. However, there are very few practical guidelines on how to apply differential privacy in practice, and a key challenge is how to set an appropriate value for the privacy parameter ɛ. In this work, we employ a statistical tool called hypothesis testing for discovering useful and interpretable guidelines for the state-of-the-art privacy-preserving frameworks. We formalize and implement hypothesis testing in terms of an adversary’s capability to infer mutually exclusive sensitive information about the input data (such as whether an individual has participated or not) from the output of the privacy-preserving mechanism. We quantify the success of the hypothesis testing using the precision- recall-relation, which provides an interpretable and natural guideline for practitioners and researchers on selecting ɛ. Our key results include a quantitative analysis of how hypothesis testing can guide the choice of the privacy parameter ɛ in an interpretable manner for a differentially private mechanism and its variants. Importantly, our findings show that an adversary’s auxiliary information - in the form of prior distribution of the database and correlation across records and time - indeed influences the proper choice of ɛ. Finally, we also show how the perspective of hypothesis testing can provide useful insights on the relationships among a broad range of privacy frameworks including differential privacy, Pufferfish privacy, Blowfish privacy, dependent differential privacy, inferential privacy, membership privacy and mutual-information based differential privacy.