Provable Privacy Guarantees Research Articles

Task-Agnostic Privacy-Preserving Representation Learning for Federated Learning against Attribute Inference Attacks

Federated learning (FL) has been widely studied recently due to its property to collaboratively train data from different devices without sharing the raw data. Nevertheless, recent studies show that an adversary can still be possible to infer private information about devices' data, e.g., sensitive attributes such as income, race, and sexual orientation. To mitigate the attribute inference attacks, various existing privacy-preserving FL methods can be adopted/adapted. However, all these existing methods have key limitations: they need to know the FL task in advance, or have intolerable computational overheads or utility losses, or do not have provable privacy guarantees. We address these issues and design a task-agnostic privacy-preserving presentation learning method for FL (TAPPFL) against attribute inference attacks. TAPPFL is formulated via information theory. Specifically, TAPPFL has two mutual information goals, where one goal learns task-agnostic data representations that contain the least information about the private attribute in each device's data, and the other goal ensures the learnt data representations include as much information as possible about the device data to maintain FL utility. We also derive privacy guarantees of TAPPFL against worst-case attribute inference attacks, as well as the inherent tradeoff between utility preservation and privacy protection. Extensive results on multiple datasets and applications validate the effectiveness of TAPPFL to protect data privacy, maintain the FL utility, and be efficient as well. Experimental results also show that TAPPFL outperforms the existing defenses.

PKDGAN: Private Knowledge Distillation With Generative Adversarial Networks

The deployment of deep learning applications has to address the increasing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information from either model parameters or accesses to the inference model. Recently, differential privacy (DP) has been proposed to offer provable privacy guarantees by randomizing the training process of neural networks. However, many approaches tend to provide the worst case privacy protection for model publishing, inevitably impairing the accuracy of the trained models. Thus, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but the student models can be released with privacy guarantees. In this paper, a three-player (teacher-student-discriminator) learning framework, Private Knowledge Distillation with Generative Adversarial Networks (PKDGAN), is proposed, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. Moreover, a cooperative learning strategy is also suggested to support the collective training of multiple students against the discriminator when each student is with insufficient unlabelled training data. To enforce rigorous privacy guarantees, PKDGAN applies a Rényi differential privacy mechanism throughout the training process, and use it with a moment accountant technique to track the privacy cost. PKDGAN allows students to be trained with unlabelled public data and very few epochs, which avoids the exposure of training data while ensuring model performance. In the experiments, PKDGAN is found to have consistently good performance on various datasets (MNIST, SVHN, CIFAR-10, and Market-1501). When compared to prior works [1], [2], PKDGAN exhibits 5-82% accuracy loss improvement without compromising any privacy guarantee.

DProvDB: Differentially Private Query Processing with Multi-Analyst Provenance

Recent years have witnessed the adoption of differential privacy (DP) in practical database systems like PINQ, FLEX, and PrivateSQL. Such systems allow data analysts to query sensitive data while providing a rigorous and provable privacy guarantee. However, the existing design of these systems does not distinguish data analysts of different privilege levels or trust levels. This design can have an unfair apportion of the privacy budget among the data analyst if treating them as a single entity, or waste the privacy budget if considering them as non-colluding parties and answering their queries independently. In this paper, we propose DProvDB, a fine-grained privacy provenance framework for the multi-analyst scenario that tracks the privacy loss to each single data analyst. Under this framework, when given a fixed privacy budget, we build algorithms that maximize the number of queries that could be answered accurately and apportion the privacy budget according to the privilege levels of the data analysts.

Trajectory Data Collection with Local Differential Privacy

Trajectory data collection is a common task with many applications in our daily lives. Analyzing trajectory data enables service providers to enhance their services, which ultimately benefits users. However, directly collecting trajectory data may give rise to privacy-related issues that cannot be ignored. Local differential privacy (LDP), as the de facto privacy protection standard in a decentralized setting, enables users to perturb their trajectories locally and provides a provable privacy guarantee. Existing approaches to private trajectory data collection in a local setting typically use relaxed versions of LDP, which cannot provide a strict privacy guarantee, or require some external knowledge that is impractical to obtain and update in a timely manner. To tackle these problems, we propose a novel trajectory perturbation mechanism that relies solely on an underlying location set and satisfies pure ε-LDP to provide a stringent privacy guarantee. In the proposed mechanism, each point's adjacent direction information in the trajectory is used in its perturbation process. Such information serves as an effective clue to connect neighboring points and can be used to restrict the possible region of a perturbed point in order to enhance utility. To the best of our knowledge, our study is the first to use direction information for trajectory perturbation under LDP. Furthermore, based on this mechanism, we present an anchor-based method that adaptively restricts the region of each perturbed trajectory, thereby significantly boosting performance without violating the privacy constraint. Extensive experiments on both real-world and synthetic datasets demonstrate the effectiveness of the proposed mechanisms.

Privacy-Preserving federated learning: An application for big data load forecast in buildings

Internet of Everything (IoE) is playing key role to enable smart energy management in buildings. A huge amount of load data are accumulated continuously to control the energy systems inside buildings. Big data analysis and prediction play increasingly vital roles in making the building energy systems smarter. Meanwhile, the big data forecasting technique represents the key element for an efficient energy control mechanism in buildings. However, the load forecast models require a huge amount of consumption data with enough variety to achieve high accuracy. Moreover, the collected IoE data are privacy-sensitive. That is, the fine-grained load data collected at the level of buildings may reveal information about the occupant’s behaviors. In this paper, by using Federating Learning (FL) as a decentralized machine learning technique, buildings occupants’ information can efficiently participate in the decision-making without revealing privacy through a cloud/Edge computing framework. In the FL approach, the big data are widely distributed between buildings, where a shared learning model is trained locally on each of the buildings. Nonetheless, the FL approach does not provide provable privacy guarantees. Moreover, FL requires participating buildings to regularly exchange huge updates resulting in huge bandwidth consumption. The paper presents a new privacy-preserving bandwidth-efficient FL that relies on Differential Privacy to theoretically provide privacy. Furthermore, to ensure the fast convergence of the FL model, a POWER-SELECTION protocol is introduced to select between buildings to participate in the FL round. By numerical analysis, the proposed FL approach is demonstrated to improve the accuracy of load forecast while compromising provable privacy.

Fuzzy Differential Privacy Theory and Its Applications in Subgraph Counting

Transportation networks are essential to the operation of societies and economies. Protecting the privacy of sensitive information is a meaningful conception in sustainable transport when mining the transportation data. In data mining, differential privacy (DP) has provable privacy guarantees for releasing sensitive data by introducing randomness into query results. However, it suffers from significant accuracy loss of outputs when the query has high sensitivity (e.g., triangle counting). The reason is that the range of random perturbation to each query result in DP is too large. It consists of all possible output values for a query that forms a large or even unbounded interval. However, when impose perturbation only in a small neighborhood of the true query result, the similarity measure based on randomness in DP fails. Thereupon, we introduce fuzziness into DP to formulate new models which have smaller disturbance via fuzzy similarity measures. In this article, we establish a novel and general theory of private data analysis, fuzzy differential privacy (FDP). The new theory FDP aims to acquire a more flexible tradeoff between the accuracy of outputs and the privacy-preserving level of data. FDP combines DP with fuzzy set theory by introducing fuzziness into the query results and characterizing similarities between outputs via multiple fuzzy similarity measures. From this perspective, DP can be viewed as a special case of FDP with probabilistic similarity measure. Compared with DP, FDP has three superiorities: 1) most fuzzy similarity measures in FDP support sliding window perturbation strategies we proposed, which refer to perturbation in a small neighborhood of the query results; 2) FDP adds noise to the query results only according to a fraction of all possible neighboring datasets; and 3) the fuzzy similarity with valued in [0,1] quantifies the privacy protection level intuitively. These three points enable more accurate outputs while providing provable and intuitive privacy guarantees. As for subgraph counting, the state-of-the-art method is ladder framework in DP. We illustrate FDP mechanisms by applying them to a common application in subgraph counting–triangle/4-cliques counting. Experiments show that FDP is effective and efficient with smaller output errors than DP.

Privacy-Preserving Database Fingerprinting.

When sharing relational databases with other parties, in addition to providing high quality (utility) database to the recipients, a database owner also aims to have (i) privacy guarantees for the data entries and (ii) liability guarantees (via fingerprinting) in case of unauthorized redistribution. However, (i) and (ii) are orthogonal objectives, because when sharing a database with multiple recipients, privacy via data sanitization requires adding noise once (and sharing the same noisy version with all recipients), whereas liability via unique fingerprint insertion requires adding different noises to each shared copy to distinguish all recipients. Although achieving (i) and (ii) together is possible in a naïve way (e.g., either differentially-private database perturbation or synthesis followed by fingerprinting), this approach results in significant degradation in the utility of shared databases. In this paper, we achieve privacy and liability guarantees simultaneously by proposing a novel entry-level differentially-private (DP) fingerprinting mechanism for relational databases without causing large utility degradation. The proposed mechanism fulfills the privacy and liability requirements by leveraging the randomization nature of fingerprinting and transforming it into provable privacy guarantees. Specifically, we devise a bit-level random response scheme to achieve differential privacy guarantee for arbitrary data entries when sharing the entire database, and then, based on this, we develop an -entry-level DP fingerprinting mechanism. We theoretically analyze the connections between privacy, fingerprint robustness, and database utility by deriving closed form expressions. We also propose a sparse vector technique-based solution to control the cumulative privacy loss when fingerprinted copies of a database are shared with multiple recipients. We experimentally show that our mechanism achieves strong fingerprint robustness (e.g., the fingerprint cannot be compromised even if the malicious database recipient modifies/distorts more than half of the entries in its received fingerprinted copy), and higher database utility compared to various baseline methods (e.g., application-dependent database utility of the shared database achieved by the proposed mechanism is higher than that of the considered baselines).

Differentially private block coordinate descent

Deep learning models have revolutionized AI tasks by producing accurate predictions. These models′ success largely depends on precise training using large-scale datasets, mostly crowdsourced from the target population. The training datasets may contain sensitive personal information, and the model parameters can encode this information on internal wirings of hidden layers, thus bearing the risk of a privacy breach. The modern trend of sharing trained models has grown the privacy breach risk manifold. The performance of existing privacy-preserving deep learning models trying to address this issue is unsatisfactory. Consequently, only a marginal proportion of these privacy-preserved models have been adopted in the industry. Therefore, we developed the first differentially private version of the block coordinate descent (BCD) algorithm. Our proposed mechanism considerably cuts the privacy cost by injecting an appropriate amount of noise in block variables. It achieves high accuracy comparable to its non-private equivalents. It improves the convergence speed and provides a provable privacy guarantee by performing privacy accounting using advanced composition and moments accountant methods. We empirically evaluate the robustness of our proposed mechanism on benchmark datasets. The results demonstrate the competitive performance in terms of both privacy cost reduction and speedy convergence against the state-of-the-art differential privacy-based mechanisms.

No Free Lunch Theorem for Security and Utility in Federated Learning

In a federated learning scenario where multiple parties jointly learn a model from their respective data, there exist two conflicting goals for the choice of appropriate algorithms. On one hand, private and sensitive training data must be kept secure as much as possible in the presence of semi-honest partners; on the other hand, a certain amount of information has to be exchanged among different parties for the sake of learning utility. Such a challenge calls for the privacy-preserving federated learning solution, which maximizes the utility of the learned model and maintains a provable privacy guarantee of participating parties’ private data. This article illustrates a general framework that (1) formulates the trade-off between privacy loss and utility loss from a unified information-theoretic point of view, and (2) delineates quantitative bounds of the privacy-utility trade-off when different protection mechanisms including randomization, sparsity, and homomorphic encryption are used. It was shown that in general there is no free lunch for the privacy-utility trade-off , and one has to trade the preserving of privacy with a certain degree of degraded utility. The quantitative analysis illustrated in this article may serve as the guidance for the design of practical federated learning algorithms.

PADP-FedMeta: A personalized and adaptive differentially private federated meta learning mechanism for AIoT

Powered by edge computing, the last few years have seen a rapid growth in AIoT applications. Federated learning (FL), as a typical machine learning framework for edge intelligence, has attracted a large number of attention since it can protect user privacy. However, recent studies have shown that FL cannot fully ensure privacy. To address this, differential privacy technique is widely used in FL. Nevertheless, existing works neglect that data on devices are non-independent and identically distributed (Non-IID), which largely degrades model accuracy and convergence speed. In this paper, we propose PADP-FedMeta, a personalized and adaptive differentially private federated meta learning mechanism with a provable privacy and convergence guarantee. PADP-FedMeta mitigates the negative effect of Non-IID upon model accuracy by introducing federated meta learning, and significantly improves the convergence speed with an adaptive privacy parameter. Comprehensive experimental results show the effectiveness of our mechanism and its superior performance over the state-of-the-art differentially private FL schemes.

Covariance’s Loss is Privacy’s Gain: Computationally Efficient, Private and Accurate Synthetic Data

AbstractThe protection of private information is of vital importance in data-driven research, business and government. The conflict between privacy and utility has triggered intensive research in the computer science and statistics communities, who have developed a variety of methods for privacy-preserving data release. Among the main concepts that have emerged are anonymity and differential privacy. Today, another solution is gaining traction, synthetic data. However, the road to privacy is paved with NP-hard problems. In this paper, we focus on the NP-hard challenge to develop a synthetic data generation method that is computationally efficient, comes with provable privacy guarantees and rigorously quantifies data utility. We solve a relaxed version of this problem by studying a fundamental, but a first glance completely unrelated, problem in probability concerning the concept of covariance loss. Namely, we find a nearly optimal and constructive answer to the question how much information is lost when we take conditional expectation. Surprisingly, this excursion into theoretical probability produces mathematical techniques that allow us to derive constructive, approximately optimal solutions to difficult applied problems concerning microaggregation, privacy and synthetic data.

Federating recommendations using differentially private prototypes

Machine learning methods exploit similarities in users’ activity patterns to provide recommendations in applications across a wide range of fields including entertainment, dating, and commerce. However, in domains that demand protection of personally sensitive data, such as medicine or banking, how can we learn recommendation models without accessing the sensitive data and without inadvertently leaking private information? Many situations in the medical field prohibit centralizing the data from different hospitals and thus require learning from information kept in separate databases. We propose a new federated approach to learning global and local private models for recommendation without collecting raw data, user statistics, or information about personal preferences. Our method produces a set of locally learned prototypes that allow us to infer global behavioral patterns while providing differential privacy guarantees for users in any database of the system. By requiring only two rounds of communication, we both reduce the communication costs and avoid excessive privacy loss associated with typical federated learning iterative procedures. We test our framework on synthetic data, real federated medical data, and a federated version of Movielens ratings. We show that local adaptation of the global model allows the proposed method to outperform centralized matrix-factorization-based recommender system models, both in terms of the accuracy of matrix reconstruction and in terms of the relevance of recommendations, while maintaining provable privacy guarantees. We also show that our method is more robust and has smaller variance than individual models learned by independent entities.

Fairness and Cost Constrained Privacy-Aware Record Linkage

Record linkage algorithms match and link records from different databases that refer to the same real-world entity based on direct and/or quasi-identifiers, such as name, address, age, and gender, available in the records. Since these identifiers generally contain personal identifiable information (PII) about the entities, record linkage algorithms need to be developed with privacy constraints. Known as privacy-preserving record linkage (PPRL), many research studies have been conducted to perform the linkage on encoded and/or encrypted identifiers. Differential privacy (DP) combined with computationally efficient encoding methods, e.g. Bloom filter encoding, has been used to develop PPRL with provable privacy guarantees. The standard DP notion does not however address other constraints, among which the most important ones are fairness-bias and cost of linkage in terms of number of record pairs to be compared. In this work, we propose new notions of fairness-constrained DP and fairness and cost-constrained DP for PPRL and develop a framework for PPRL with these new notions of DP combined with Bloom filter encoding. We provide theoretical proofs for the new DP notions for fairness and cost-constrained PPRL and experimentally evaluate them on two datasets containing person-specific data. Our experimental results show that with these new notions of DP, PPRL with better performance (compared to the standard DP notion for PPRL) can be achieved with regard to privacy, cost and fairness constraints.

Aggregation and Transformation of Vector-Valued Messages in the Shuffle Model of Differential Privacy

Advances in communications, storage and computational technology allow significant quantities of data to be collected and processed by distributed devices. Combining the information from these endpoints can realize significant societal benefit but presents challenges in protecting the privacy of individuals, especially important in an increasingly regulated world. Differential privacy (DP) is a technique that provides a rigorous and provable privacy guarantee for aggregation and release. The Shuffle Model for DP has been introduced to overcome challenges regarding the accuracy of local-DP algorithms and the privacy risks of central-DP. In this work we introduce a new protocol for vector aggregation in the context of the Shuffle Model. The aim of this paper is twofold; first, we provide a single message protocol for the summation of real vectors in the Shuffle Model, using advanced composition results. Secondly, we provide an improvement on the bound on the error achieved through using this protocol through the implementation of a Discrete Fourier Transform, thereby minimizing the initial error at the expense of the loss in accuracy through the transformation itself. This work will further the exploration of more sophisticated structures such as matrices and higher-dimensional tensors in this context, both of which are reliant on the functionality of the vector case.

Preserving User Privacy in Personalized Networks

The advances in efficient computation and data processing technologies have ushered in an entirely new era in which network connectivity applications are being revolutionized by Artificial Intelligence (AI). One of the emerging AI applications in wireless networks is network personalization. Personalized wireless networks are designed to track user satisfaction levels using Machine Learning (ML) models that are trained and updated using context data collected from users. Nonetheless, user data collection and transmission make sensitive user information prone to various privacy attacks. Therefore, in this letter, we propose a privacy-preserving framework for personalized wireless networks. The proposed framework is enabled by Differential Privacy (DP), which is an influential privacy technology that offers a rigorous and provable privacy guarantee. Besides, the proposed framework is supported by a multi-stage prediction process, which adds a second layer of privacy protection to personal user data. Lastly, we investigate the impact of privacy budget and data size on the quality of user satisfaction predictors in personalized wireless networks.

Secure DNA Motif-Finding Method Based on Sampling Candidate Pruning

With the continuous exploration of genetic research, gradually exposed privacy issues become the bottleneck that limits its development. DNA motif finding is an important study to understand the regulation of gene expression; however, the existing methods generally ignore the potential sensitive information that may be exposed in the process. In this work, we utilize the -differential privacy model to provide provable privacy guarantees which is independent of attackers’ background knowledge. Our method makes use of sample databases to prune the generated candidate motifs to lower the magnitude of added noise. Furthermore, to improve the utility of mining results, a strategy of threshold modification is designed to reduce the propagation and random sampling errors in the mining process. Extensive experiments on actual DNA databases confirm that our approach can privately find DNA motifs with high utility and efficiency.

A Comparison of Algorithms for Solving the Multiagent Simple Temporal Problem

The Simple Temporal Problem (STP) is a popular representation for solving centralized scheduling and planning problems. When scheduling agents are associated with different users who need to coordinate some of their activities, however, considerations such as privacy and scalability suggest solving the joint STP in a more distributed manner. Building on recent advances in STP algorithms that exploit loosely-coupled problem structure, this paper develops and evaluates algorithms for solving the multiagent STP. We define a partitioning of the multiagent STP with provable privacy guarantees, and show that our algorithms can exploit this partitioning while still finding the tightest consistent bounds on timepoints that must be coordinated across agents. We also demonstrate empirically that our algorithms can exploit concurrent computation, leading to solution time speed-ups over state-of-the-art centralized approaches, and enabling scalability to problems involving larger numbers of loosely-coupled agents.

Obfuscation of images via differential privacy: From facial images to general images

Due to the pervasiveness of image capturing devices in every-day life, images of individuals are routinely captured. Although this has enabled many benefits, it also infringes on personal privacy. A promising direction in research on obfuscation of facial images has been the work in the k-same family of methods which employ the concept of k-anonymity from database privacy. However, there are a number of deficiencies of k-anonymity that carry over to the k-same methods, detracting from their usefulness in practice. In this paper, we first outline several of these deficiencies and discuss their implications in the context of facial obfuscation. We then develop a framework through which we obtain a formal differentially private guarantee for the obfuscation of facial images in generative machine learning models. Our approach provides a provable privacy guarantee that is not susceptible to the outlined deficiencies of k-same obfuscation and produces photo-realistic obfuscated output. In addition, we demonstrate through experimental comparisons that our approach can achieve comparable utility to k-same obfuscation in terms of preservation of useful features in the images. Furthermore, we propose a method to achieve differential privacy for any image (i.e., without restriction to facial images) through the direct modification of pixel intensities. Although the addition of noise to pixel intensities does not provide the high visual quality obtained via generative machine learning models, it offers greater versatility by eliminating the need for a trained model. We demonstrate that our proposed use of the exponential mechanism in this context is able to provide superior visual quality to pixel-space obfuscation using the Laplace mechanism.

Differentially private regression analysis with dynamic privacy allocation

In recent years, machine learning has reaped huge fruits in the domain of artificial intelligence. However, during the process of model training, machine learning models may be afflicted with the risk of disclosing sensitive information contained in training data. Therefore, there raises an urgent requirement for decreasing the risk of sensitive information leakage. As a novel privacy-preserving mechanism, differential privacy can tackle the shortcoming of traditional privacy model effectively, while providing a provable privacy guarantee. However, the existing literatures on differentially private regression models are limited and lack of the dynamic privacy allocation methods, which may have an influence on the balance between privacy guarantee and model performance. In this paper, we propose an adaptive differentially private regression model to enhance the security guarantee without sacrificing plenty of model utility, which allocates the privacy budget dynamically by using the relevance-based noise imposition mechanism. We add less noise to objective function when input features greatly contribute to the model output, and vice-versa. Theoretical analysis and rigorous experiments demonstrate that our approach not only retains a desirable model utility under a modest privacy budget, but also reduces the potential risk of privacy disclosure.

Privacy of Dependent Users Against Statistical Matching

Modern applications significantly enhance user experience by adapting to each user’s individual condition and/or preferences. While this adaptation can greatly improve a user’s experience or be essential for the application to work, the exposure of user data to the application presents a significant privacy threat to the users—even when the traces are anonymized—since the statistical matching of an anonymized trace to prior user behavior can identify a user and their habits. Because of the current and growing algorithmic and computational capabilities of adversaries, provable privacy guarantees as a function of the degree of anonymization and obfuscation of the traces are necessary. Our previous work has established the requirements on anonymization and obfuscation in the case that data traces are independent between users. However, the data traces of different users will be dependent in many applications, and an adversary can potentially exploit such. In this paper, we consider the negative impact of dependency between user traces on their privacy. First, we demonstrate that the adversary can readily identify the association graph of the obfuscated and anonymized version of the data, revealing which user data traces are dependent. Next, we demonstrate that the adversary can use this association graph to break user privacy with significantly shorter traces than in the case of independent users, and that obfuscating data traces independently across users is often insufficient to remedy such leakage. In other words, we have shown that inter-user dependency is disastrous to privacy, and any non-negligible dependency between users significantly reduces the effectiveness of anonymization and obfuscation schemes. Finally, we discuss how users can improve privacy by employing joint obfuscation that removes or reduces the data dependency.