Rigorous Privacy Guarantee Research Articles

DP-FedEwc: Differentially private federated elastic weight consolidation for model personalization

Federated learning (FL) has become a prevalent paradigm for training a model collaboratively on multiple clients with the coordination of a central server. As traditional FL suffers from client-drift due to data heterogeneity across clients, many personalized FL (PFL) techniques have been proposed. However, the issue of privacy leakage within PFL remains inadequately addressed. When incorporating differential privacy (DP) directly into PFL to provide rigorous privacy guarantees, it amplifies heterogeneity among clients and introduces high variance in uploaded information, significantly compromising model’s utility. In this paper, we propose a novel privacy-preserving PFL framework called Differentially Private Federated Elastic weight consolidation (DP-FedEwc), to achieve effective model personalization for each client under sample-level DP. We focus on a practical setting where the server is honest-but-curious. We first implement a FedEwc algorithm in a communication-efficient manner, and provide privacy guarantees by perturbing models and their parameter importance (PI). We show that FedEwc is robust to DP-introduced heterogeneity caused by noisy models, especially when the model is a deep neural network. Since excessive noise may render PI invalid, we present an Adaptive Parameter importance Perturbation (APP) method to adaptively add Gaussian noise to different coordinates of PI, thereby alleviating the negative effect of DP noise. Moreover, to accurately calibrate the privacy cost resulting from querying heterogeneous data across various clients when computing PI through APP, we adapt a Bayesian Accountant (BA) method to DP-FedEwc. We conduct experiments on standard benchmark datasets, and the experimental results confirm the superiority of DP-FedEwc over DP-PFL baselines.

Dynamic differential privacy-based dataset condensation

With the continuous expansion of data scale, data condensation technology has emerged as a means to reduce costs related to storage, time, and energy consumption. Data condensation can generate a synthesized dataset of reduced size, enabling the training of models that exhibit high performance comparable to the original dataset. Nevertheless, data condensation has also exposed privacy issues. Although many approaches have been proposed to preserve privacy for data condensation, the privacy protection for data condensation has not been well explored. Furthermore, to the best of our knowledge, none of the existing approaches propose dynamic parameters-based differential privacy dataset condensation considering unnecessary noise introduced by the fixed privacy parameter strategy. Most approaches typically inject constant noise with the fixed variance into gradients across all layers using predefined privacy parameters, which can significantly impact model accuracy. In this paper, we investigate alternative approaches for data condensation with differential privacy (DP) that aim to ensure DP while minimizing the noise added to gradients and improving the model accuracy. First, we develop a dynamic threshold method to reduce the noise added to gradients in the later stages of training by using a clipping threshold that decreases with training rounds. Second, noise injection in our method is not arbitrary as in conventional approaches; instead, it is based on the maximum size of the gradient after clipping. This approach ensures that only minimal noise increments are introduced, thereby mitigating accuracy loss and parameter instability that may arise from excessive noise injection. Finally, our privacy analysis confirms that the proposed method provides a rigorous privacy guarantee. Extensive evaluations on different datasets demonstrate that our approach can improve accuracy compared to existing DP data condensation techniques while adhering to the same privacy budget and applying a specified clipping threshold.

SoK: Can Trajectory Generation Combine Privacy and Utility?

While location trajectories represent a valuable data source for analyses and location-based services, they can reveal sensitive information, such as political and religious preferences. Differentially private publication mechanisms have been proposed to allow for analyses under rigorous privacy guarantees. However, the traditional protection schemes suffer from a limiting privacy-utility trade-off and are vulnerable to correlation and reconstruction attacks. Synthetic trajectory data generation and release represent a promising alternative to protection algorithms. While initial proposals achieve remarkable utility, they fail to provide rigorous privacy guarantees. This paper proposes a framework for designing a privacy-preserving trajectory publication approach by defining five design goals, particularly stressing the importance of choosing an appropriate Unit of Privacy. Based on this framework, we briefly discuss the existing trajectory protection approaches, emphasising their shortcomings. This work focuses on the systematisation of the state-of-the-art generative models for trajectories in the context of the proposed framework. We find that no existing solution satisfies all requirements. Thus, we perform an experimental study evaluating the applicability of six sequential generative models to the trajectory domain. Finally, we conclude that a generative trajectory model providing semantic guarantees remains an open research question and propose concrete next steps for future research.

PKDGAN: Private Knowledge Distillation With Generative Adversarial Networks

The deployment of deep learning applications has to address the increasing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information from either model parameters or accesses to the inference model. Recently, differential privacy (DP) has been proposed to offer provable privacy guarantees by randomizing the training process of neural networks. However, many approaches tend to provide the worst case privacy protection for model publishing, inevitably impairing the accuracy of the trained models. Thus, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but the student models can be released with privacy guarantees. In this paper, a three-player (teacher-student-discriminator) learning framework, Private Knowledge Distillation with Generative Adversarial Networks (PKDGAN), is proposed, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. Moreover, a cooperative learning strategy is also suggested to support the collective training of multiple students against the discriminator when each student is with insufficient unlabelled training data. To enforce rigorous privacy guarantees, PKDGAN applies a Rényi differential privacy mechanism throughout the training process, and use it with a moment accountant technique to track the privacy cost. PKDGAN allows students to be trained with unlabelled public data and very few epochs, which avoids the exposure of training data while ensuring model performance. In the experiments, PKDGAN is found to have consistently good performance on various datasets (MNIST, SVHN, CIFAR-10, and Market-1501). When compared to prior works [1], [2], PKDGAN exhibits 5-82% accuracy loss improvement without compromising any privacy guarantee.

How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy

Machine Learning (ML) models are ubiquitous in real-world applications and are a constant focus of research. Modern ML models have become more complex, deeper, and harder to reason about. At the same time, the community has started to realize the importance of protecting the privacy of the training data that goes into these models. Differential Privacy (DP) has become a gold standard for making formal statements about data anonymization. However, while some adoption of DP has happened in industry, attempts to apply DP to real world complex ML models are still few and far between. The adoption of DP is hindered by limited practical guidance of what DP protection entails, what privacy guarantees to aim for, and the difficulty of achieving good privacy-utility-computation trade-offs for ML models. Tricks for tuning and maximizing performance are scattered among papers or stored in the heads of practitioners, particularly with respect to the challenging task of hyperparameter tuning. Furthermore, the literature seems to present conflicting evidence on how and whether to apply architectural adjustments and which components are “safe” to use with DP. In this survey paper, we attempt to create a self-contained guide that gives an in-depth overview of the field of DP ML. We aim to assemble information about achieving the best possible DP ML model with rigorous privacy guarantees. Our target audience is both researchers and practitioners. Researchers interested in DP for ML will benefit from a clear overview of current advances and areas for improvement. We also include theory-focused sections that highlight important topics such as privacy accounting and convergence. For a practitioner, this survey provides a background in DP theory and a clear step-by-step guide for choosing an appropriate privacy definition and approach, implementing DP training, potentially updating the model architecture, and tuning hyperparameters. For both researchers and practitioners, consistently and fully reporting privacy guarantees is critical, so we propose a set of specific best practices for stating guarantees. With sufficient computation and a sufficiently large training set or supplemental nonprivate data, both good accuracy (that is, almost as good as a non-private model) and good privacy can often be achievable. And even when computation and dataset size are limited, there are advantages to training with even a weak (but still finite) formal DP guarantee. Hence, we hope this work will facilitate more widespread deployments of DP ML models.

Incorporating Prior Knowledge in Local Differentially Private Data Collection for Frequency Estimation

Local differential privacy (LDP) is a prevalent measure of privacy protection as it provides rigorous privacy guarantees and has been widely studied for statistical analysis, especially in frequency estimation. As a representative LDP-enabled frequency estimation algorithm, Google's <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Randomized Aggregation Privacy-Preserving Ordinal Response</i> (RAPPOR) has been put into practice. However, it achieves sub-optimal utility due to the following limitations. Firstly, the adoption of the MD5 hash function inevitably results in the hash collision. Secondly, the application of the randomized response technique leads to randomness. To improve the practical effectiveness of RAPPOR and the utility of frequency-based services, we propose an LDP-enabled frequency estimation method called PK-RAPPOR, in which we devise an effective re-encoding hash function (RE-HF) incorporating prior knowledge (PK) about the rough frequency ranking of items. RE-HF divides items into several cohorts based on the PK and generates a unique hash value set for each item. Compared with the original RAPPOR, the hash collision can be eliminated for items from different cohorts, and the effect of randomness can be decreased by the overlapping of items from the same cohorts. We validate our proposed method with theoretical analysis and demonstrate its effectiveness with experiments on both synthetic and real-world datasets.

RRN: A differential private approach to preserve privacy in image classification

AbstractThe wide application of image classification has given rise to many intelligent systems, such as face recognition systems, which makes our life more convenient. However, the ensuing privacy leakage problem has become increasingly serious. The training of a deep neural network requires lots of data, which may contain sensitive information of users and may be exploited by data collectors. A perturbation algorithm named RRN is proposed for image data based on local differential privacy, which provides a rigorous privacy guarantee. Existing solutions have low accuracy due to the high sensitivity of an image; the authors' method combines the Randomized Response mechanism with the Laplace mechanism to solve this problem. Experiments were conducted on the MNIST and CIFAR‐10 datasets to show the effectiveness of the algorithm. Experimental results shows that the model is better than baseline models. The algorithm was also implemented on the commonly used model in deep learning, the VGG model, which can achieve 96.4% accuracy in the non‐private version on the CIFAR‐10 dataset. The accuracy of the differential private VGG model based on the RRN algorithm is 83% when , which is still excellent. The experimental results show that the RRN algorithm can both preserve privacy and data utility.

PCFed: Privacy-Enhanced and Communication-Efficient Federated Learning for Industrial IoTs

Federated learning (FL) is capable of analyzing tremendous data from smart edge devices in Industrial Internet of Things (IIoTs), empowering numerous industrial applications. However, the increasing privacy concerns and deployment costs of IIoT environment have been posing new challenges for FL. This article proposes PCFed, a novel privacy-enhanced and communication-efficient FL framework to provide higher model accuracy with rigorous privacy guarantees and great communication efficiency. In particular, we develop a sampling-based intermittent communication strategy via a PID (proportional, integral, and derivative) controller on the cloud server to adaptively reduce the communication frequency. In addition, we design a budget allocation mechanism to balance the tradeoff between model accuracy and privacy loss. Then, we develop PCFed+, an enhanced variant for PCFed, with further consideration of infinite data streams on edge servers. Extensive experiments demonstrate that both PCFed and PCFed+ can significantly outperform existing schemes, in terms of communication efficiency, privacy protection, and model accuracy.

Understanding adaptive gradient clipping in DP‐SGD, empirically

Differentially Private Stochastic Gradient Descent (DP-SGD) is a prime method for training machine learning models with rigorous privacy guarantees. Since its birth, DP-SGD has gained popularity and has been widely adopted in both academic and industrial research. One well-known challenge when using DP-SGD is how to improve utility while maintaining privacy. To this end, recently we have seen several proposals that clip the gradients with adaptive thresholds rather than a fixed one. Although each proposal comes with some theoretical justification, the theories often rely on strong assumptions and are not compatible with each other. It is hard to know whether they are good in practice and how good they are. In this paper, we investigate adaptive clipping in DP-SGD from an empirical perspective. With extensive experiments, we were able to gain some fresh insights and proposed two new adaptive clipping strategies based on them. We cross-compared the existing methods and our new strategies experimentally. Results showed that our strategies did provide a substantial improvement in model accuracy, and outperformed the state-of-the-art adaptive clipping methods consistently.

ShareChain: Blockchain‐enabled model for sharing patient data using federated learning and differential privacy

AbstractEvery individual in our technologically evolved world needs proper data security. The procedure of exchanging medical information is increasingly concerned with data privacy. Many techniques have been offered for preserving data security. These techniques use approaches such as ‐anonymity, ‐diversity, and others. However, such solutions are vulnerable to attribute disclosure, homogeneity, and background knowledge risks due to their syntactic nature. In this work, we describe a safe and secure architecture and semantic approach for data sharing that is based on blockchain, local differential privacy (LDP), and federated learning (FL). The proposed framework generates an atmosphere devoid of trust in which data owners are no longer required to have trust in the controllers. The FL models enable the whole network to decentralize its data‐driven learning. Interplanetary file system (IPFS) is used to provide data security in a distributed environment because each file in IPFS has a digital fingerprint that is computed using a cryptographic hash function on the file's whole contents. Due to the rigorous privacy guarantee, data owners no longer need to be worried about the security of their data. The proposed model's assessment parameters include latency, throughput, privacy, and accuracy. The data privacy of the proposed model is protected via LDP and FL, and its latency and throughput communication transactions on permissioned blockchain are calculated and compared with those of the benchmark model. The findings indicate that the proposed model delivers 85% more accurate privacy than the benchmark model.

Skellam mixture mechanism

Deep neural networks have strong capabilities of memorizing the underlying training data, which can be a serious privacy concern. An effective solution to this problem is to train models with differential privacy ( DP ), which provides rigorous privacy guarantees by injecting random noise to the gradients. This paper focuses on the scenario where sensitive data are distributed among multiple participants, who jointly train a model through federated learning , using both secure multiparty computation ( MPC ) to ensure the confidentiality of each gradient update, and differential privacy to avoid data leakage in the resulting model. A major challenge in this setting is that common mechanisms for enforcing DP in deep learning, which inject real-valued noise , are fundamentally incompatible with MPC, which exchanges finite-field integers among the participants. Consequently, most existing DP mechanisms require rather high noise levels, leading to poor model utility. Motivated by this, we propose Skellam mixture mechanism (SMM), a novel approach to enforcing DP on models built via federated learning. Compared to existing methods, SMM eliminates the assumption that the input gradients must be integer-valued, and, thus, reduces the amount of noise injected to preserve DP. The theoretical analysis of SMM is highly non-trivial, especially considering (i) the complicated math of DP deep learning in general and (ii) the fact that the mixture of two Skellam distributions is rather complex. Extensive experiments on various practical settings demonstrate that SMM consistently and significantly outperforms existing solutions in terms of the utility of the resulting model.

A Survey on Differential Privacy for Unstructured Data Content

Huge amounts of unstructured data including image, video, audio, and text are ubiquitously generated and shared, and it is a challenge to protect sensitive personal information in them, such as human faces, voiceprints, and authorships. Differential privacy is the standard privacy protection technology that provides rigorous privacy guarantees for various data. This survey summarizes and analyzes differential privacy solutions to protect unstructured data content before it is shared with untrusted parties. These differential privacy methods obfuscate unstructured data after they are represented with vectors and then reconstruct them with obfuscated vectors. We summarize specific privacy models and mechanisms together with possible challenges in them. We also discuss their privacy guarantees against AI attacks and utility losses. Finally, we discuss several possible directions for future research.

Differentially private data aggregating with relative error constraint

Privacy preserving methods supporting for data aggregating have attracted the attention of researchers in multidisciplinary fields. Among the advanced methods, differential privacy (DP) has become an influential privacy mechanism owing to its rigorous privacy guarantee and high data utility. But DP has no limitation on the bound of noise, leading to a low-level utility. Recently, researchers investigate how to preserving rigorous privacy guarantee while limiting the relative error to a fixed bound. However, these schemes destroy the statistical properties, including the mean, variance and MSE, which are the foundational elements for data aggregating and analyzing. In this paper, we explore the optimal privacy preserving solution, including novel definitions and implementing mechanisms, to maintain the statistical properties while satisfying DP with a fixed relative error bound. Experimental evaluation demonstrates that our mechanism outperforms current schemes in terms of security and utility for large quantities of queries.

DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing

AbstractDeep learning techniques have achieved remarkable performance in wide-ranging tasks. However, when trained on privacy-sensitive datasets, the model parameters may expose private information in training data. Prior attempts for differentially private training, although offering rigorous privacy guarantees, lead to much lower model performance than the non-private ones. Besides, different runs of the same training algorithm produce models with large performance variance. To address these issues, we propose DPlis– Differentially Private Learning wIth Smoothing. The core idea of DPlis is to construct a smooth loss function that favors noise-resilient models lying in large flat regions of the loss landscape. We provide theoretical justification for the utility improvements of DPlis. Extensive experiments also demonstrate that DPlis can effectively boost model quality and training stability under a given privacy budget.

Medical imaging deep learning with differential privacy

The successful training of deep learning models for diagnostic deployment in medical imaging applications requires large volumes of data. Such data cannot be procured without consideration for patient privacy, mandated both by legal regulations and ethical requirements of the medical profession. Differential privacy (DP) enables the provision of information-theoretic privacy guarantees to patients and can be implemented in the setting of deep neural network training through the differentially private stochastic gradient descent (DP-SGD) algorithm. We here present deepee, a free-and-open-source framework for differentially private deep learning for use with the PyTorch deep learning framework. Our framework is based on parallelised execution of neural network operations to obtain and modify the per-sample gradients. The process is efficiently abstracted via a data structure maintaining shared memory references to neural network weights to maintain memory efficiency. We furthermore offer specialised data loading procedures and privacy budget accounting based on the Gaussian Differential Privacy framework, as well as automated modification of the user-supplied neural network architectures to ensure DP-conformity of its layers. We benchmark our framework’s computational performance against other open-source DP frameworks and evaluate its application on the paediatric pneumonia dataset, an image classification task and on the Medical Segmentation Decathlon Liver dataset in the task of medical image segmentation. We find that neural network training with rigorous privacy guarantees is possible while maintaining acceptable classification performance and excellent segmentation performance. Our framework compares favourably to related work with respect to memory consumption and computational performance. Our work presents an open-source software framework for differentially private deep learning, which we demonstrate in medical imaging analysis tasks. It serves to further the utilisation of privacy-enhancing techniques in medicine and beyond in order to assist researchers and practitioners in addressing the numerous outstanding challenges towards their widespread implementation.

Correlated tuple data release via differential privacy

Privacy preserving methods supporting for tuple data release have attracted the attention of researchers in multidisciplinary fields. Among the advanced methods, differential privacy (DP), introducing independent Laplace noise, has become an influential privacy mechanism owing to its provable and rigorous privacy guarantee. Nonetheless, in practice, tuple data to be protected are always correlated while independent noise may cause undesirable information disclosure than expected. Recent researches attempt to optimize the sensitivity function of DP with consideration of the correlation strength between data – but have a drawback in a substantial growth of noise level. Therefore, for correlated tuple data release, how to decrease the noise level incurred by correlation strength is yet to be explored. To remedy this problem, this paper exploits the degradation of DP in expected privacy levels for correlated tuple data and proposes a solution to mitigate it. We first demonstrate a filtering attack, presenting a possibility of using the different dependence between original outputs and perturbations to sanitize a certain level of noise to extract individual’s sensitive information. Secondly, we introduce the notion of correlated tuple differential privacy (CTDP) to preserve expected privacy for correlated tuple data and further propose a generalized Laplace mechanism (GLM) to achieve privacy guarantees in CTDP. Then we design a practical iteration mechanism, including an update function, to conduct GLM when facing large scale queries. Finally, experimental evaluation on real-world datasets over multiple fields show that our solution consistently outperforms state-of-the-art mechanisms in data utility while providing the same privacy guarantee as other approaches for correlated tuple data.

A differentially private distributed data mining scheme with high efficiency for edge computing

A wide range of data mining applications benefit from the low latency offered by edge computing. However, edge computing suffers from limited computing resources, which inhibits the applications of the computationally expensive data mining methods. In the edge-cloud environment, usually, the participants turn to collaboratively train machine-learning models that yield more accurate prediction results. However, data owners may not be willing to sharing the own data for the privacy concerns. To handle such disparate goals, we focus on tree-based distributed data mining scheme with differential privacy, which is computationally friendly. The basic idea of our approach is based on a distributed ensemble strategy. Each participant builds an elegant decision model based on their own data, which has a good tradeoff between the computation and the accuracy of the data distribution, and shares it with other participants after being injected with the elaborate noise. Then the useful knowledge transferred from the decision models is acquired by other participants in an adaptive ensemble strategy. Both the theoretical analysis and the experiments show that our scheme provides an efficient data mining manner that can achieve a good prediction accuracy while providing rigorous privacy guarantee over the distributed data.

Analysis of Application Examples of Differential Privacy in Deep Learning.

Artificial Intelligence has been widely applied today, and the subsequent privacy leakage problems have also been paid attention to. Attacks such as model inference attacks on deep neural networks can easily extract user information from neural networks. Therefore, it is necessary to protect privacy in deep learning. Differential privacy, as a popular topic in privacy-preserving in recent years, which provides rigorous privacy guarantee, can also be used to preserve privacy in deep learning. Although many articles have proposed different methods to combine differential privacy and deep learning, there are no comprehensive papers to analyze and compare the differences and connections between these technologies. For this purpose, this paper is proposed to compare different differential private methods in deep learning. We comparatively analyze and classify several deep learning models under differential privacy. Meanwhile, we also pay attention to the application of differential privacy in Generative Adversarial Networks (GANs), comparing and analyzing these models. Finally, we summarize the application of differential privacy in deep neural networks.

Protecting the Moving User’s Locations by Combining Differential Privacy and k‐Anonymity under Temporal Correlations in Wireless Networks

The rapid development of the Global Positioning System (GPS) devices and location‐based services (LBSs) facilitates the collection of huge amounts of personal information for the untrusted/unknown LBS providers. This phenomenon raises serious privacy concerns. However, most of the existing solutions aim at locating interference in the static scenes or in a single timestamp without considering the correlation between location transfer and time of moving users. In this way, the solutions are vulnerable to various inference attacks. Traditional privacy protection methods rely on trusted third‐party service providers, but in reality, we are not sure whether the third party is trustable. In this paper, we propose a systematic solution to preserve location information. The protection provides a rigorous privacy guarantee without the assumption of the credibility of the third parties. The user’s historical trajectory information is used as the basis of the hidden Markov model prediction, and the user’s possible prospective location is used as the model output result to protect the user’s trajectory privacy. To formalize the privacy‐protecting guarantee, we propose a new definition, L&amp;A‐location region, based on k‐anonymity and differential privacy. Based on the proposed privacy definition, we design a novel mechanism to provide a privacy protection guarantee for the users’ identity trajectory. We simulate the proposed mechanism based on a dataset collected in real practice. The result of the simulation shows that the proposed algorithm can provide privacy protection to a high standard.

Enhancing Differential Privacy for Federated Learning at Scale

Federated learning (FL) is an emerging technique that trains machine learning models across multiple de-centralized systems. It enables local devices to collaboratively learn a model by aggregating locally computed updates via a server. Privacy is a core aspect of FL, and recent works in this area are advancing the privacy guarantee of an FL network. To ensure rigorous privacy guarantee for FL, prior works have focused on methods to securely aggregate local updates and provide differential privacy (DP). In this paper, we investigate a new privacy risk for FL. Specifically, FL may frequently encounter unexpected user dropouts because it is implemented over a large-scale network. We first observe that user dropouts of an FL network may lead to failure in achieving the desired level of privacy protection, i.e., over-consumption of the privacy budget. Subsequently, we develop a DP mechanism robust to user dropouts by dynamically calibrating noise with account of the dropout rate. We evaluate the proposed technique to train convolutional neural network models on MNIST and FEMNIST datasets over a simulated FL network. Our results show that our approach significantly improves privacy guarantee for user dropouts compared to existing DP algorithms on FL networks.