Differential Privacy Research Articles

Revealing the True Cost of Locally Differentially Private Protocols: An Auditing Perspective

While the existing literature on Differential Privacy (DP) auditing predominantly focuses on the centralized model (e.g., in auditing the DP-SGD algorithm), we advocate for extending this approach to audit Local DP (LDP). To achieve this, we introduce the LDP-Auditor framework for empirically estimating the privacy loss of locally differentially private mechanisms. This approach leverages recent advances in designing privacy attacks against LDP frequency estimation protocols. More precisely, through the analysis of numerous state-of-the-art LDP protocols, we extensively explore the factors influencing the privacy audit, such as the impact of different encoding and perturbation functions. Additionally, we investigate the influence of the domain size and the theoretical privacy loss parameters ϵ and δ on local privacy estimation. In-depth case studies are also conducted to explore specific aspects of LDP auditing, including distinguishability attacks on LDP protocols for longitudinal studies and multidimensional data. Finally, we present a notable achievement of our LDP-Auditor framework, which is the discovery of a bug in a state-of-the-art LDP Python package. Overall, our LDP-Auditor framework as well as our study offer valuable insights into the sources of randomness and information loss in LDP protocols. These contributions collectively provide a realistic understanding of the local privacy loss, which can help practitioners in selecting the LDP mechanism and privacy parameters that best align with their specific requirements. We open-sourced LDP-Auditor in [4].

VFLGAN: Vertical Federated Learning-based Generative Adversarial Network for Vertically Partitioned Data Publication

In the current artificial intelligence (AI) era, the scale and quality of the dataset play a crucial role in training a high-quality AI model. However, good data is not a free lunch and is always hard to access due to privacy regulations like the General Data Protection Regulation (GDPR). A potential solution is to release a synthetic dataset with a similar distribution to that of the private dataset. Nevertheless, in some scenarios, it has been found that the attributes needed to train an AI model belong to different parties, and they cannot share the raw data for synthetic data publication due to privacy regulations. In PETS 2023, Xue et al. [29] proposed the first generative adversary network-based model, VertiGAN, for vertically partitioned data publication. However, after thoroughly investigating, we found that VertiGAN is less effective in preserving the correlation among the attributes of different parties. This article proposes a Vertical Federated Learning-based Generative Adversarial Network, VFLGAN, for vertically partitioned data publication to address the above issues. Our experimental results show that compared with VertiGAN, VFLGAN significantly improves the quality of synthetic data. Taking the MNIST dataset as an example, the quality of the synthetic dataset generated by VFLGAN is 3.2 times better than that generated by VertiGAN w.r.t. the Frechet Distance. We also designed a more efficient and effective Gaussian mechanism for the proposed VFLGAN to provide the synthetic dataset with a differential privacy guarantee. On the other hand, differential privacy only gives the upper bound of the worst-case privacy guarantee. This article also proposes a practical auditing scheme that applies membership inference attacks to estimate privacy leakage through the synthetic dataset.

A Zero Auxiliary Knowledge Membership Inference Attack on Aggregate Location Data

Location data is frequently collected from populations and shared in aggregate form to guide policy and decision making. However, the prevalence of aggregated data also raises the privacy concern of membership inference attacks (MIAs). MIAs infer whether an individual's data contributed to the aggregate release. Although effective MIAs have been developed for aggregate location data, these require access to an extensive auxiliary dataset of individual traces over the same locations, which are collected from a similar population. This assumption is often impractical given common privacy practices surrounding location data. To measure the risk of an MIA performed by a realistic adversary, we develop the first Zero Auxiliary Knowledge (ZK) MIA on aggregate location data, which eliminates the need for an auxiliary dataset of real individual traces. Instead, we develop a novel synthetic approach, such that suitable synthetic traces are generated from the released aggregate. We also develop methods to correct for bias and noise, to show that our synthetic-based attack is still applicable when privacy mechanisms are applied prior to release. Using two large-scale location datasets, we demonstrate that our ZK MIA matches the state-of-the-art Knock-Knock (KK) MIA across a wide range of settings, including popular implementations of differential privacy (DP) and suppression of small counts. Furthermore, we show that ZK MIA remains highly effective even when the adversary only knows a small fraction (10%) of their target's location history. This demonstrates that effective MIAs can be performed by realistic adversaries, highlighting the need for strong DP protection.

PTT: Piecewise Transformation Technique for Analyzing Numerical Data Under Local Differential Privacy

PTT: Piecewise Transformation Technique for Analyzing Numerical Data Under Local Differential Privacy

Adaptive compressed learning boosts both efficiency and utility of differentially private federated learning

In the federated learning (FL) research field, current research is confronted with several pivotal challenges, e.g., data privacy, model utility and communication efficiency. Furthermore, these challenges are further amplified by statistical data heterogeneous in the FL system. Thus, a novel Communication-efficient and Utility-assured Gaussian differential privacy-based Personalized Federated Adaptive Compressed Learning method, called CUG-PFACL, is proposed. Specifically, an end-to-end local adaptive compressed learning strategy is designed, including three crucial modules, namely the measurement matrix, the personalized compressed data transformation and the local model. Especially, jointly training the measurement matrix module and the personalized compressed data transformation module can mitigate the inherent statistical heterogeneity while preserving all important characteristics of the compressed private data of each local client, and alleviate the additional heterogeneity induced by Gaussian differential privacy in each global communication round. Numerous experimental simulation and comparisons demonstrate that CUG-PFACL has three notable advantages: data privacy guarantee, enhanced personalized model utility and high-efficient communication.

Local Differential Privacy for correlated location data release in ITS

The ubiquity of location positioning devices has facilitated the implementation of various Intelligent Transportation System (ITS) applications that generate an enormous volume of location data. Recently, Local Differential Privacy (LDP) has been proposed as a rigorous privacy framework that permits the continuous release of aggregate location statistics without relying on a trusted data curator. However, the conventional LDP was built upon the assumption of independent data, which may not be suitable for inherently correlated location data. This paper investigates the quantification of potential privacy leakage in a correlated location data release scenario under a local setting, which has not been addressed in the literature. Our analysis shows that the privacy guarantee of LDP could be degraded in the presence of spatial–temporal and user correlations, albeit the perturbation is performed locally and independently by the users. This privacy guarantee is bounded by a privacy barrier that is affected by the intensity of correlations. We derive several important closed-form expressions and design efficient algorithms to compute such privacy leakage in a correlated location data. We subsequently propose a Δ-CLDP model that enhances the conventional LDP by incorporating the data correlations, and design a generic LDP data release framework that renders adaptive personalization of privacy preservation. Extensive theoretical analyses and simulations on scalable real datasets validate the security and performance efficiency of our work.

Subgraph Structure Membership Inference Attacks against Graph Neural Networks

Graph Neural Networks (GNNs) have been widely applied to various applications across different domains. However, recent studies have shown that GNNs are susceptible to the membership inference attacks (MIAs) which aim to infer if some particular data samples were included in the model’s training data. While most previous MIAs have focused on inferring the membership of individual nodes and edges within the training graph, we introduce a novel form of membership inference attack called the Structure Membership Inference Attack (SMIA) which aims to determine whether a given set of nodes corresponds to a particular target structure, such as a clique or a multi-hop path, within the original training graph. To address this issue, we present novel black-box SMIA attacks that leverage the prediction outputs generated by the target GNN model for inference. Our approach involves training a three-label classifier, which, in combination with shadow training, aids in enabling the inference attack. Our extensive experimental evaluation of three representative GNN models and three real-world graph datasets demonstrates that our proposed attacks consistently outperform three baseline methods, including the one that employs the conventional link membership inference attacks to infer the subgraph structure. Additionally, we design a defense mechanism that introduces perturbations to the node embeddings thus influencing the corresponding prediction outputs by the target model. Our defense selectively perturbs dimensions within the node embeddings that have the least impact on the model's accuracy. Our empirical results demonstrate that the defense effectiveness of our approach is comparable with two established defense techniques that employ differential privacy. Moreover, our method achieves a better trade-off between defense strength and the accuracy of the target model compared to the two existing defense methods.

Improved Data Privacy with Differential Privacy in Federated Learning

Multiple users may train machine learning models cooperatively using Federated Learning (FL). There is a risk of malicious acquisition of participants' personal data due to the fact that traditional machine learning needs users to provide data for training. Through the use of federated learning, which involves moving the training process from a central server to terminal devices, users' data may be protected. Each participant keeps their dataset local and only exchanges model updates. This research proposed an innovative proposal for the medical industry's differentiated privacy approach for overcoming these problems. When several healthcare organizations work together to develop models that use different and extensive information, clinical applications may be greatly enhanced. Thus, the Local and Centre Differential Privacy (LCDP) on clinical datasets is a feature of our proposed approach. Reason being that the training data is the primary emphasis of the local model, while the machine learning model is the primary focus of the central model. We discover that the local model and the central model are linked in a unique way, changes in the original data lead to changes in the gradient, which in turn lead to changes in the model parameters. Based on this finding, our technique is better than prior central methods since it protects the data, gradient, and model all at once by bridging the gap between the two. Our system provides better privacy protections and even higher performance than some of the best previous central methods, which is an excellent outcome of rigorous evaluation.

AI powered privacy protection: A survey of current state and future directions

The research is conducted to investigate how AI transforms the notion of protection of privacy through discussing the status quo technologies, challenges, and future directions. All due to the sudden rise of digital data, protection of personal information has evolved as a major concern for which AI-enabled solutions are being introduced. Advanced concepts of AI are therefore marking the paradigm shift in how organizations handle sensitive data, letting more secure and privacy-oriented practices lead the way with notions such as differential privacy, federated learning, and anomaly detection. However, despite these advances, formidable challenges remain regarding the opacity of AI models, possible algorithmic biases, and regulatory compliance. The paper further discusses the future of AI for privacy protection, with new developments: XAI, integration of AI with blockchain, and quantum-resistant cryptography. These advances offer great transparency, security, and responsibility in privacy management. It further underlines that the collaboration of governments, industrial leaders, and researchers is required in providing appropriate frameworks for the usage of AI, given the ethical and regulatory concerns around privacy protection as the influence of AI grows. While AI is indeed very promising in improving privacy protections, the degree to which it can actually function depends on the surmounting of present limitations and harmonization of technological development with shifting data privacy criteria. The AI in this research paper will continue to play a leading role in shaping the future of privacy preservation, with answers continuing to emanate from innovation, security, and ethical concerns. It is through continuous improvement and collaboration that AI can ensure effective privacy measures in an increasingly data-driven world.

Efficient private SCO for heavy-tailed data via averaged clipping

AbstractWe consider stochastic convex optimization for heavy-tailed data with the guarantee of being differentially private (DP). Most prior works on differentially private stochastic convex optimization for heavy-tailed data are either restricted to gradient descent (GD) or performed multi-times clipping on stochastic gradient descent (SGD), which is inefficient for large-scale problems. In this paper, we consider a one-time clipping strategy and provide principled analyses of its bias and private mean estimation. We establish new convergence results and improved complexity bounds for the proposed algorithm called AClipped-dpSGD for constrained and unconstrained convex problems. We also extend our convergent analysis to the strongly convex case and non-smooth case (which works for generalized smooth objectives with H$$\ddot{\text {o}}$$ o ¨ lder-continuous gradients). All the above results are guaranteed with a high probability for heavy-tailed data. Numerical experiments are conducted to justify the theoretical improvement.

Privacy-preserving data analysis

With the ever-increasing volume of data being generated and shared across various platforms, the challenge of maintaining privacy while extracting value from this data has become paramount. This paper delves into the realm of Privacy-Preserving Data Analysis (PPDA), examining its current landscape and the pivotal techniques shaping it. Using datasets from diverse domains, we evaluated four leading PPDA techniques—Differential Privacy, Homomorphic Encryption, Secure Multi-Party Computation (SMPC), and Data Obfuscation—to discern their efficacy and trade-offs in terms of data utility and privacy breach risk. Our findings underscore the strengths and constraints of each method, guiding researchers and practitioners in choosing the optimal approach for specific scenarios. As data continues to be an invaluable asset in the digital age, the tools and techniques to analyze it privately will play a critical role in shaping future data-driven decision-making processes.

Future Trends and Technological Innovations of Private AI Deployment

As data protection laws like GDPR tighten, companies must ensure their AI systems comply with regulations, leading to widespread use of encryption, federated learning, and differential privacy to protect data. Privatized AI deployment will transform enterprise operations by integrating AI with core business processes, automating and enhancing efficiency and decision-making. For instance, manufacturing firms can use privatized AI for intelligent monitoring and predictive maintenance, reducing costs and improving quality. The market landscape will shift as companies reevaluate technology and service providers, favoring those offering comprehensive privatization solutions for system security and stability. The growing demand for AI expertise will also drive educational institutions to offer more AI-related courses and training. In summary, the trends in privatized AI deployment will deeply affect business models and market structures, necessitating that companies stay abreast of technological advancements and actively embrace privatized AI opportunities for sustained innovation and growth.

Local differential privacy federated learning based on heterogeneous data multi-privacy mechanism

Federated learning enables the development of robust models without accessing users data directly. However, recent studies indicate that federated learning remains vulnerable to privacy leakage. To address this issue, local differential privacy mechanisms have been incorporated into federated learning. Nevertheless, local differential privacy will reduce the availability of data. To explore the balance between privacy budgets and data availability in federated learning, we propose federated learning for clustering hierarchical aggregation with adaptive piecewise mechanisms under multiple privacy-FedAPCA as a way to balance the relationship between privacy preservation and model accuracy. First, we introduce an adaptive piecewise mechanism that dynamically adjusts perturbation intervals based on the data ranges across different layers of the model, ensuring minimized perturbation variance while maintaining the same level of privacy. Second, we propose two dynamic privacy budget allocation methods, which are allocating the privacy budget based on global accuracy and global loss, and allocating the privacy budget based on local accuracy and loss, to ensure that better model accuracy can be achieved under the same privacy budget. Finally, we propose a clustering hierarchical aggregation method in the model aggregation stage, and the model is updated and aggregated after the unbiased estimation of the disturbance in each cluster according to the variance of each layer. FedAPCA improves the balance between privacy preservation and model accuracy. Our experimental results, comparing FedAPCA with the SOTA multi-privacy local differential privacy federated learning frameworks on the MNIST and CIFAR-10 datasets, demonstrate that FedAPCA improves model accuracy by 1%–2%.

Evaluating small area differential privacy life expectancy

Evaluating small area differential privacy life expectancy

Optimum noise mechanism for differentially private queries in discrete finite sets

The differential privacy (DP) literature often centers on meeting privacy constraints by introducing noise to the query, typically using a pre-specified parametric distribution model with one or two degrees of freedom. However, this emphasis tends to neglect the crucial considerations of response accuracy and utility, especially in the context of categorical or discrete numerical database queries, where the parameters defining the noise distribution are finite and could be chosen optimally. This paper addresses this gap by introducing a novel framework for designing an optimal noise probability mass function (PMF) tailored to discrete and finite query sets. Our approach considers the modulo summation of random noise as the DP mechanism, aiming to present a tractable solution that not only satisfies privacy constraints but also minimizes query distortion. Unlike existing approaches focused solely on meeting privacy constraints, our framework seeks to optimize the noise distribution under an arbitrary (ϵ,δ)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(\\epsilon , \\delta )$$\\end{document} constraint, thereby enhancing the accuracy and utility of the response. We demonstrate that the optimal PMF can be obtained through solving a mixed-integer linear program. Additionally, closed-form solutions for the optimal PMF are provided, minimizing the probability of error for two specific cases. Numerical experiments highlight the superior performance of our proposed optimal mechanisms compared to state-of-the-art methods. This paper contributes to the DP literature by presenting a clear and systematic approach to designing noise mechanisms that not only satisfy privacy requirements but also optimize query distortion. The framework introduced here opens avenues for improved privacy-preserving database queries, offering significant enhancements in response accuracy and utility.

Effective Route Recommendation Leveraging Differentially Private Location Data

The proliferation of GPS-enabled devices and advances in positioning technologies have greatly facilitated the collection of user location data, making them valuable across various domains. One of the most common and practical uses of these location datasets is to recommend the most probable route between two locations to users. Traditional algorithms for route recommendation rely on true trajectory data collected from users, which raises significant privacy concerns due to the personal information often contained in location data. Therefore, in this paper, we propose a novel framework for computing optimal routes using location data collected through differential privacy (DP)-based privacy-preserving methods. The proposed framework introduces a method for accurately extracting transitional probabilities from perturbed trajectory datasets, addressing the challenge of low data utility caused by DP-based methods. Specifically, to effectively compute transitional probabilities, we present a density-adjusted sampling method that enables the collection of representative data across all areas. In addition, we introduce an effective scheme to approximately estimate transitional probabilities based on sampled datasets. Experimental results on real-world data demonstrate the practical applicability and effectiveness of our framework in computing optimal routes while preserving user privacy.

Privacy-Preserving Transfer Learning Framework for Kidney Disease Detection

This paper introduces a new privacy-preserving transfer learning framework for the classification of kidney diseases. In the proposed framework, transfer learning is employed for feature extraction, and differential privacy is used to obtain noisy gradients. A variety of CNN architectures, including Xception, ResNet50, InceptionResNetV2, MobileNet, DenseNet201, InceptionV3, and VGG19 are utilized to evaluate the proposed framework. Analysis of a large dataset of 12,400 labeled kidney CT images shows that transfer learning architectures based on the proposed framework achieve excellent accuracy ratios in privacy-preserving classification. These results demonstrate the effectiveness of the proposed framework in enabling transfer learning models to classify kidney diseases while ensuring privacy. The MobileNet architecture stands out for its exceptional performance, with an impressive accuracy of 99.83% in privacy-preserving classification. Considering the findings of this study, it is evident that the proposed framework is appropriate for the early and private diagnosis of kidney diseases and promotes the achievement of promising results in this field.

Pangenome-Informed Language Models for Privacy-Preserving Synthetic Genome Sequence Generation.

The public availability of genome datasets, such as The Human Genome Project (HGP), The 1000 Genomes Project, The Cancer Genome Atlas, and the International HapMap Project, has significantly advanced scientific research and medical understanding. Here our goal is to share such genomic information for downstream analysis while protecting the privacy of individuals through Differential Privacy (DP). We introduce synthetic DNA data generation based on pangenomes in combination with Pretrained-Language Models (PTLMs). We introduce two novel tokenization schemes based on pangenome graphs to enhance the modeling of DNA. We evaluated these tokenization methods, and compared them with classical single nucleotide and k -mer tokenizations. We find k -mer tokenization schemes, indicating that our tokenization schemes boost the model's performance consistency with long effective context length (covering longer sequences with the same number of tokens). Additionally, we propose a method to utilize the pangenome graph and make it comply with DP privacy standards. We assess the performance of DP training on the quality of generated sequences with discussion of the trade-offs between privacy and model accuracy. The source code for our work will be published under a free and open source license soon.

Trajectory Privacy-Protection Mechanism Based on Multidimensional Spatial–Temporal Prediction

The popularity of global GPS location services and location-enabled personal terminal applications has contributed to the rapid growth of location-based social networks. Users can access social networks at anytime and anywhere to obtain services in the relevant location. While accessing services is convenient, there is a potential risk of leaking users’ private information. In data processing, the discovery of issues and the generation of optimal solutions constitute a symmetrical process. Therefore, this paper proposes a symmetry–trajectory differential privacy-protection mechanism based on multi-dimensional prediction (TPPM-MP). Firstly, the temporal attention mechanism is designed to extract spatiotemporal features of trajectories from different spatiotemporal dimensions and perform trajectory-sensitive prediction. Secondly, class-prevalence-based weights are assigned to sensitive regions. Finally, the privacy budget is assigned based on the sensitive weights, and noise conforming to localized differential privacy is added. Validated on real datasets, the proposed method in this paper enhanced usability by 22% and 37% on the same dataset compared with other methods mentioned, while providing equivalent privacy protection.

Fair and optimal prediction via post‐processing

AbstractWith the development of machine learning algorithms and the increasing computational resources available, artificial intelligence has achieved great success in many application domains. However, the success of machine learning has also raised concerns about the fairness of the learned models. For instance, the learned models can perpetuate and even exacerbate the potential bias and discrimination in the training data. This issue has become a major obstacle to the deployment of machine learning systems in high‐stakes domains, for example, criminal judgment, medical testing, online advertising, hiring process, and so forth. To mitigate the potential bias exhibited by machine learning models, fairness criteria can be integrated into the training process to ensure fair treatment across all demographics, but it often comes at the expense of model performance. Understanding such tradeoffs, therefore, is crucial to the design of optimal and fair algorithms. My research focuses on characterizing the inherent tradeoff between fairness and accuracy in machine learning, and developing algorithms that can achieve both fairness and optimality. In this article, I will discuss our recent work on designing post‐processing algorithms for fair classification, which can be applied to a wide range of fairness criteria, including statistical parity, equal opportunity, and equalized odds, under both attribute‐aware and attribute‐blind settings, and is particularly suited to large‐scale foundation models where retraining is expensive or even infeasible. I will also discuss the connections between our work and other related research on trustworthy machine learning, including the connections between algorithmic fairness and differential privacy as well as adversarial robustness.