Inference Attacks Research Articles

Membership Inference Attacks and Defenses in Federated Learning: A Survey

Membership Inference Attacks and Defenses in Federated Learning: A Survey

PCDP-CRLPPM: a classified regional location privacy-protection model based on personalized clustering with differential privacy in data management

Abstract Location data management plays a crucial role in facilitating data collection and supporting location-based services. However, the escalating volume of transportation big data has given rise to increased concerns regarding privacy and security issues in data management, potentially posing threats to the lives and property of users. At present, there are two possible attacks in data management, namely Reverse-clustering Inference Attack and Mobile-spatiotemporal Feature Inference Attack. Additionally, the dynamic allocation of privacy budgets emerges as an NP-hard problem. To protect data privacy and maintain utility in data management, a novel protection model for location privacy information in data management, Classified Regional Location Privacy-Protection Model based on Personalized Clustering with Differential Privacy (PCDP-CRLPPM), is proposed. Firstly, a twice-clustering algorithm combined with gridding is proposed, which divides continuous locations into different clusters based on the different privacy protection needs of different users. Subsequently, these clusters are categorized into different spatiotemporal feature regions. Then, a Sensitive-priority algorithm is proposed to allocate privacy budgets adaptively for each region. Finally, a Regional-fuzzy algorithm is presented to introduce Laplacian noise into the centroids of the regions, thereby safeguarding users’ location privacy. The experimental results demonstrate that, compared to other models, PCDP-CRLPPM exhibits superior resistance against two specific attack models and achieves high levels of data utility while preserving privacy effectively.

Shareable artificial intelligence to extract cancer outcomes from electronic health records for precision oncology research.

Databases that link molecular data to clinical outcomes can inform precision cancer research into novel prognostic and predictive biomarkers. However, outside of clinical trials, cancer outcomes are typically recorded only in text form within electronic health records (EHRs). Artificial intelligence (AI) models have been trained to extract outcomes from individual EHRs. However, patient privacy restrictions have historically precluded dissemination of these models beyond the centers at which they were trained. In this study, the vulnerability of text classification models trained directly on protected health information to membership inference attacks is confirmed. A teacher-student distillation approach is applied to develop shareable models for annotating outcomes from imaging reports and medical oncologist notes. 'Teacher' models trained on EHR data from Dana-Farber Cancer Institute (DFCI) are used to label imaging reports and discharge summaries from the Medical Information Mart for Intensive Care (MIMIC)-IV dataset. 'Student' models are trained to use these MIMIC documents to predict the labels assigned by teacher models and sent to Memorial Sloan Kettering (MSK) for evaluation. The student models exhibit high discrimination across outcomes in both the DFCI and MSK test sets. Leveraging private labeling of public datasets to distill publishable clinical AI models from academic centers could facilitate deployment of machine learning to accelerate precision oncology research.

Threshold Filtering for Detecting Label Inference Attacks in Vertical Federated Learning

Federated learning, as an emerging machine-learning method, has received widespread attention because it allows users to train locally during the training process and uses relevant cryptographic knowledge to safeguard the privacy of data during model aggregation. However, existing federated learning is also susceptible to privacy breaches, e.g., label inference attacks against vertical federated learning scenarios, where an adversary is able to reason about the labels of other participants based on the trained model, leading to serious privacy breaches. In this paper, we design a detection method for label inference attacks in vertical federated learning scenarios, which is able to detect the attacks based on the principles of the attacks. We design a threshold-filtering detection method based on the principle of attack to determine that the model is under attack when the threshold value is greater than a set parameter. Furthermore, we have created six threat model classifications based on different a priori conditions of the adversary to comprehensively analyze the adversary’s attacks. In addition to the detection method of attacks, the extent of attacks on the model and the effectiveness of the defense can also be evaluated. The evaluation module will experimentally measure the changes in the relevant metrics such as the accuracy of the attack, the F1 score, and the change in the accuracy after the defense method. For example, detection in the full connected neural network model assesses the attack and defense effectiveness of the model with an attack accuracy of 86.72% in the breast cancer Wisconsin dataset and an F1 score of 0.743, which is reduced to 36.36% after dispersed training. This ensures that users have an overall grasp of the extent to which the training model is under attack before deploying the model.

Datamodel Distance: A New Metric for Privacy

Recent work developing Membership Inference Attacks has demonstrated that certain points in the dataset are often in- trinsically easier to attack than others. In this paper, we intro- duce a new pointwise metric, the Datamodel Distance, and show that it is empirically correlated to and establishes a theoreti- cal lower bound for the success probability for a point under the LiRA Membership Inference Attack. This establishes a connection between the concepts of Datamodels and Member- ship Inference, and also gives new intuitive explanations for why certain points are more susceptible to attack than others. We then use datamodels as a lens through which to investigate the Privacy Onion Efect.

Privacy-Preserving Techniques in Generative AI and Large Language Models: A Narrative Review

Generative AI, including large language models (LLMs), has transformed the paradigm of data generation and creative content, but this progress raises critical privacy concerns, especially when models are trained on sensitive data. This review provides a comprehensive overview of privacy-preserving techniques aimed at safeguarding data privacy in generative AI, such as differential privacy (DP), federated learning (FL), homomorphic encryption (HE), and secure multi-party computation (SMPC). These techniques mitigate risks like model inversion, data leakage, and membership inference attacks, which are particularly relevant to LLMs. Additionally, the review explores emerging solutions, including privacy-enhancing technologies and post-quantum cryptography, as future directions for enhancing privacy in generative AI systems. Recognizing that achieving absolute privacy is mathematically impossible, the review emphasizes the necessity of aligning technical safeguards with legal and regulatory frameworks to ensure compliance with data protection laws. By discussing the ethical and legal implications of privacy risks in generative AI, the review underscores the need for a balanced approach that considers performance, scalability, and privacy preservation. The findings highlight the need for ongoing research and innovation to develop privacy-preserving techniques that keep pace with the scaling of generative AI, especially in large language models, while adhering to regulatory and ethical standards.

DP-Poison: Poisoning Federated Learning under the Cover of Differential Privacy

Federated learning (FL) enables resource-constrained node devices to learn a shared model while keeping the training data local. Since recent research has demonstrated multiple privacy leakage attacks in FL, e.g., gradient inference attacks and membership inference attacks, differential privacy (DP) is applied to serve as one of the most effective privacy protection mechanisms. Despite the benefit DP brings, we observe that the introduction of DP also brings random changes to client updates, which will affect the robust aggregation algorithms. We reveal a novel poisoning attack under the cover of DP, named the DP-Poison attack in FL. Specifically, the DP-Poison attack is designed to achieve four goals: 1) maintaining the main task performance; 2) launching a successful attack; 3) escaping the robust aggregation algorithms in FL; and 4) keeping the effectiveness of DP privacy protection. To achieve these goals, we design multiple optimization goals to generate DP noise through a genetic algorithm. The optimization ensures that while the benign updates change randomly, the malicious updates can change towards the global model after adding the DP noise, so that it is easier to be accepted by the robust aggregation algorithms. Extensive experiments show that DP-Poison achieves a nearly 100% attack success rate while maintaining the proposed four goals.

SAMFL: Secure Aggregation Mechanism for Federated Learning with Byzantine-robustness by functional encryption

Federated learning (FL) enables collaborative model training without sharing private data, thereby potentially meeting the growing demand for data privacy protection. Despite its potentials, FL also poses challenges in achieving privacy-preservation and Byzantine-robustness when handling sensitive data. To address these challenges, we present a novel Secure Aggregation Mechanism for Federated Learning with Byzantine-Robustness by Functional Encryption (SAMFL). Our approach designs a novel dual-decryption multi-input functional encryption (DD-MIFE) scheme, which enables efficient computation of cosine similarities and aggregation of encrypted gradients through a single ciphertext. This innovative scheme allows for dual decryption, producing distinct results based on different keys, while maintaining high efficiency. We further propose TF-Init, integrating DD-MIFE with Truth Discovery (TD) to eliminate the reliance on a root dataset. Additionally, we devise a secure cosine similarity calculation aggregation protocol (SC2AP) using DD-MIFE, ensuring privacy-preserving and Byzantine-robust FL secure aggregation. To enhance FL efficiency, we employ single instruction multiple data (SIMD) to parallelize encryption and decryption processes. Concurrently, to preserve accuracy, we incorporate differential privacy (DP) with selective clipping of model layers within the FL framework. Finally, we integrate TF-Init, SC2AP, SIMD, and DP to construct SAMFL. Extensive experiments demonstrate that SAMFL successfully defends against both inference attacks and poisoning attacks, while improving efficiency and accuracy compared to existing methods. SAMFL provides a comprehensive integrated solution for FL with efficiency, accuracy, privacy-preservation, and robustness.

Topology modification against membership inference attack in Graph Neural Networks

Graph Neural Networks (GNNs) can effectively leverage information from graph data and apply to various downstream tasks, such as recommendation system, or anomaly detection. Previous studies have shown that node-level GNNs are susceptible to membership inference attacks, potentially leaking private information about the nodes. However, current studies on defending against such attacks primarily focuses on image and text data. They may not be directly applicable to graph data. In addition, classic defense methods such as differential privacy, regularization techniques, and adversarial training. They may reduce model availability or require model retraining while protecting privacy. To deal with these problems, we propose a novel defense strategy against membership inference attacks in graph neural networks. The strategy involves augmenting the graph with additional nodes and modifying its topology to create a new privacy-preserving graph, thereby protecting the privacy of the original nodes. We conducted extensive experiments on three representative GNN models and compared them with state-of-the-art baseline methods to support our research. The experimental results demonstrate that our method significantly reduces the success rate of membership inference attacks while maintaining the basic performance of the target model.

On the atout ticket learning problem for neural networks and its application in securing federated learning exchanges

Artificial Neural Networks (ANNs) have become the backbone of many real-world applications, including distributed applications relying on Federated Learning (FL). However, several vulnerabilities/attacks have emerged in recent years, affecting the benefits of using ANNs in FL, such as reconstruction attacks and membership inference attacks. These attacks can have severe impacts on both the societal and professional levels. For instance, inferring the presence of a patient’s private health record in a medical study or a clinic database violates the patient’s privacy and can have legal or ethical consequences. Therefore, protecting the data and model from malicious attacks in FL systems is important. This paper introduces the Atout Ticket Learning (ATL) problem. This new problem consists of identifying sensitive parameters (atout tickets) of a neural network model, which, if modified, will increase the model’s loss by at least a given threshold ϵ. First, we formulate ATL as an ℓ0-norm minimization problem, and we derive a lower bound on the number of atout tickets needed to achieve a model degradation of ϵ. Second, we design the Atout Ticket Protocol (ATP) as an effective solution for privacy-preserving in FL systems using atout tickets, along with the benefit of noise perturbations and secure aggregation techniques. Finally, we experiment ATP against FL reconstruction attacks using new selection strategies, namely Inverting Gradients, Deep Leakage, and Improved Deep Leakage. The results show that ATP is highly robust against these attacks.

Subgraph Structure Membership Inference Attacks against Graph Neural Networks

Graph Neural Networks (GNNs) have been widely applied to various applications across different domains. However, recent studies have shown that GNNs are susceptible to the membership inference attacks (MIAs) which aim to infer if some particular data samples were included in the model’s training data. While most previous MIAs have focused on inferring the membership of individual nodes and edges within the training graph, we introduce a novel form of membership inference attack called the Structure Membership Inference Attack (SMIA) which aims to determine whether a given set of nodes corresponds to a particular target structure, such as a clique or a multi-hop path, within the original training graph. To address this issue, we present novel black-box SMIA attacks that leverage the prediction outputs generated by the target GNN model for inference. Our approach involves training a three-label classifier, which, in combination with shadow training, aids in enabling the inference attack. Our extensive experimental evaluation of three representative GNN models and three real-world graph datasets demonstrates that our proposed attacks consistently outperform three baseline methods, including the one that employs the conventional link membership inference attacks to infer the subgraph structure. Additionally, we design a defense mechanism that introduces perturbations to the node embeddings thus influencing the corresponding prediction outputs by the target model. Our defense selectively perturbs dimensions within the node embeddings that have the least impact on the model's accuracy. Our empirical results demonstrate that the defense effectiveness of our approach is comparable with two established defense techniques that employ differential privacy. Moreover, our method achieves a better trade-off between defense strength and the accuracy of the target model compared to the two existing defense methods.

Maui: Black-Box Edge Privacy Attack on Graph Neural Networks

Graphs are ubiquitous data structures with nodes representing objects and edges representing relationships between them. Graph Neural Networks (GNNs) have recently been proposed to study graph-structured data, but unfortunately, are susceptible to privacy leakage. This issue becomes more urgent as GNNs gain wide deployment in many real-world settings including social network analysis, bioinformatics, and cybersecurity. In this paper, we propose the first link inference attack that can compromise user data under the most difficult security settings, which we call Maui. We demonstrate that private edge information can be inferred by a malicious user with a black-box approach. Extensive experiments on six real-world datasets show our attacks conduct effective link inference attacks in various scopes. Our attack achieves significant performance improvements over the current state-of-the-art. When targeting 2-layer Graph Convolution Networks, for inferring edges of a single node, our attack outperforms the best existing method by 12.0%, increasing from 83.7% to 95.7%; when inferring edges of the entire graph, our attack achieves a 19.6% improvement, from 67.7% to 87.3%. Our results underscore the need for countermeasures against privacy attacks in GNNs, as they can reveal rich information about graph structures.

VFLGAN: Vertical Federated Learning-based Generative Adversarial Network for Vertically Partitioned Data Publication

In the current artificial intelligence (AI) era, the scale and quality of the dataset play a crucial role in training a high-quality AI model. However, good data is not a free lunch and is always hard to access due to privacy regulations like the General Data Protection Regulation (GDPR). A potential solution is to release a synthetic dataset with a similar distribution to that of the private dataset. Nevertheless, in some scenarios, it has been found that the attributes needed to train an AI model belong to different parties, and they cannot share the raw data for synthetic data publication due to privacy regulations. In PETS 2023, Xue et al. [29] proposed the first generative adversary network-based model, VertiGAN, for vertically partitioned data publication. However, after thoroughly investigating, we found that VertiGAN is less effective in preserving the correlation among the attributes of different parties. This article proposes a Vertical Federated Learning-based Generative Adversarial Network, VFLGAN, for vertically partitioned data publication to address the above issues. Our experimental results show that compared with VertiGAN, VFLGAN significantly improves the quality of synthetic data. Taking the MNIST dataset as an example, the quality of the synthetic dataset generated by VFLGAN is 3.2 times better than that generated by VertiGAN w.r.t. the Frechet Distance. We also designed a more efficient and effective Gaussian mechanism for the proposed VFLGAN to provide the synthetic dataset with a differential privacy guarantee. On the other hand, differential privacy only gives the upper bound of the worst-case privacy guarantee. This article also proposes a practical auditing scheme that applies membership inference attacks to estimate privacy leakage through the synthetic dataset.

MemberShield: A framework for federated learning with membership privacy

Federated Learning (FL) allows multiple data owners to build high-quality deep learning models collaboratively, by sharing only model updates and keeping data on their premises. Even though FL offers privacy-by-design, it is vulnerable to membership inference attacks (MIA), where an adversary tries to determine whether a sample was included in the training data. Existing defenses against MIA cannot offer meaningful privacy protection without significantly hampering the model’s utility and causing a non-negligible training overhead. In this paper we analyze the underlying causes of the differences in the model behavior for member and non-member samples, which arise from model overfitting and facilitate MIAs. Accordingly, we propose MemberShield, a generalization-based defense method for MIAs that consists of: (i) one-time preprocessing of each client’s training data labels that transforms one-hot encoded labels to soft labels and eventually exploits them in local training, and (ii) early stopping the training when the local model’s validation accuracy does not improve on that of the global model for a number of epochs. Extensive empirical evaluations on three widely used datasets and four model architectures demonstrate that MemberShield outperforms state-of-the-art defense methods by delivering substantially better practical privacy protection against all forms of MIAs, while better preserving the target model utility. On top of that, our proposal significantly reduces training time and is straightforward to implement, by just tuning a single hyperparameter.

A Zero Auxiliary Knowledge Membership Inference Attack on Aggregate Location Data

Location data is frequently collected from populations and shared in aggregate form to guide policy and decision making. However, the prevalence of aggregated data also raises the privacy concern of membership inference attacks (MIAs). MIAs infer whether an individual's data contributed to the aggregate release. Although effective MIAs have been developed for aggregate location data, these require access to an extensive auxiliary dataset of individual traces over the same locations, which are collected from a similar population. This assumption is often impractical given common privacy practices surrounding location data. To measure the risk of an MIA performed by a realistic adversary, we develop the first Zero Auxiliary Knowledge (ZK) MIA on aggregate location data, which eliminates the need for an auxiliary dataset of real individual traces. Instead, we develop a novel synthetic approach, such that suitable synthetic traces are generated from the released aggregate. We also develop methods to correct for bias and noise, to show that our synthetic-based attack is still applicable when privacy mechanisms are applied prior to release. Using two large-scale location datasets, we demonstrate that our ZK MIA matches the state-of-the-art Knock-Knock (KK) MIA across a wide range of settings, including popular implementations of differential privacy (DP) and suppression of small counts. Furthermore, we show that ZK MIA remains highly effective even when the adversary only knows a small fraction (10%) of their target's location history. This demonstrates that effective MIAs can be performed by realistic adversaries, highlighting the need for strong DP protection.

Safeguarding Privacy in Genome Research: A Comprehensive Framework for Authors.

As genomic research continues to advance, sharing of genomic data and research outcomes has become increasingly important for fostering collaboration and accelerating scientific discovery. However, such data sharing must be balanced with the need to protect the privacy of individuals whose genetic information is being utilized. This paper presents a bidirectional framework for evaluating privacy risks associated with data shared (both in terms of summary statistics and research datasets) in genomic research papers, particularly focusing on re-identification risks such as membership inference attacks (MIA). The framework consists of a structured workflow that begins with a questionnaire designed to capture researchers' (authors') self-reported data sharing practices and privacy protection measures. Responses are used to calculate the risk of re-identification for their study (paper) when compared with the National Institutes of Health (NIH) genomic data sharing policy. Any gaps in compliance help us to identify potential vulnerabilities and encourage the researchers to enhance their privacy measures before submitting their research for publication. The paper also demonstrates the application of this framework, using published genomic research as case study scenarios to emphasize the importance of implementing bidirectional frameworks to support trustworthy open science and genomic data sharing practices.

ArmSpy++: Enhanced PIN Inference through Video-based Fine-grained Arm Posture Analysis

As one of the most common ways for user authentication, Personal Identification Number (PIN), due to its simplicity and convenience, has suffered from plenty of side-channel attacks, which pose a severe threat to people’s privacy and property. The success of existing attacks is usually built upon the premise of no occlusion between the attacker and the victim’s hand gesture, but it increases the difficulty of launching the attack and the possibility of exposure. To overcome such limitation, we propose ArmSpy++, an improved video-assisted PIN inference attack built upon our previous research, ArmSpy. Specifically, ArmSpy++ employs new modules to leverage more features like the keystroke-induced elbow bending, wrist speed variation, and the spatial relationship between different arm joints, to correctly detect Keystrokes. ArmSpy++ delves into the perspective relationship and natural typing habits to ensure a high success rate of PIN inference. We also re-designed the inferred PIN pattern coordination mechanism to accurately deduce the PINs. By using a pre-trained HigherHRNet model for posture estimation ArmSpy++ eliminates the necessity of additional training. The extensive experiments demonstrate that ArmSpy++ can achieve over \(83.1\% \) average accuracy with 3 attempts and even \(92.5\% \) for some victims, indicating the severity of the threat posed by ArmSpy++.

A Maneuver in the Trade-Off Space of Federated Learning Aggregation Frameworks Secured with Polymorphic Encryption: PolyFLAM and PolyFLAP Frameworks

Maintaining user privacy in machine learning is a critical concern due to the implications of data collection. Federated learning (FL) has emerged as a promising solution by sharing trained models rather than user data. However, FL still faces several challenges, particularly in terms of security and privacy, such as vulnerability to inference attacks. There is an inherent trade-off between communication traffic across the network and computational costs on the server or client, which this paper aims to address by maneuvering between these performance parameters. To tackle these issues, this paper proposes two complementary frameworks: PolyFLAM (“Polymorphic Federated Learning Aggregation of Models”) and PolyFLAP (“Polymorphic Federated Learning Aggregation of Parameters”). These frameworks provide two options to suit the needs of users, depending on whether they prioritize reducing communication across the network or lowering computational costs on the server or client. PolyFLAM reduces computational costs by exchanging entire models, eliminating the need to rebuild models from parameters. In contrast, PolyFLAP reduces communication costs by transmitting only model parameters, which are smaller in size compared to entire models. Both frameworks are supported by polymorphic encryption, ensuring privacy is maintained even in cases of key leakage. Furthermore, these frameworks offer five different machine learning models, including support vector machines, logistic regression, Gaussian naïve Bayes, stochastic gradient descent, and multi-layer perceptron, to cover as many real-life problems as possible. The evaluation of these frameworks with simulated and real-life datasets demonstrated that they can effectively withstand various attacks, including inference attacks that aim to compromise user privacy by capturing exchanged models or parameters.

Label-Only Membership Inference Attack Based on Model Explanation

It is well known that machine learning models (e.g., image recognition) can unintentionally leak information about the training set. Conventional membership inference relies on posterior vectors, and this task becomes extremely difficult when the posterior is masked. However, current label-only membership inference attacks require a large number of queries during the generation of adversarial samples, and thus incorrect inference generates a large number of invalid queries. Therefore, we introduce a label-only membership inference attack based on model explanations. It can transform a label-only attack into a traditional membership inference attack by observing neighborhood consistency and perform fine-grained membership inference for vulnerable samples. We use feature attribution to simplify the high-dimensional neighborhood sampling process, quickly identify decision boundaries and recover a posteriori vectors. It also compares different privacy risks faced by different samples through finding vulnerable samples. The method is validated on CIFAR-10, CIFAR-100 and MNIST datasets. The results show that membership attributes can be identified even using a simple sampling method. Furthermore, vulnerable samples expose the model to greater privacy risks.

User Mobility Modeling in Crowdsourcing Application to Prevent Inference Attacks

With the rise of the Internet of Things (IoT), mobile crowdsourcing has become a leading application, leveraging the ubiquitous presence of smartphone users to collect and process data. Spatial crowdsourcing, which assigns tasks based on users’ geographic locations, has proven to be particularly innovative. However, this trend raises significant privacy concerns, particularly regarding the precise geographic data required by these crowdsourcing platforms. Traditional methods, such as dummy locations, spatial cloaking, differential privacy, k-anonymity, and encryption, often fail to mitigate the risks associated with the continuous disclosure of location data. An unauthorized entity could access these data and infer personal information about individuals, such as their home address, workplace, religion, or political affiliations, thus constituting a privacy violation. In this paper, we propose a user mobility model designed to enhance location privacy protection by accurately identifying Points of Interest (POIs) and countering inference attacks. Our main contribution here focuses on user mobility modeling and the introduction of an advanced algorithm for precise POI identification. We evaluate our contributions using GPS data collected from 10 volunteers over a period of 3 months. The results show that our mobility model delivers significant performance and that our POI extraction algorithm outperforms existing approaches.