Differentially Private Distributed Multi-Task Relationship Learning

Abstract

In many real-world applications, data are distributed across different geographical regions, and may come from different distributions which result in multiple learning tasks. In such cases, distributed multi-task learning is usually used to learn the multiple related tasks to improve the generalization performance for each task. Among the distributed multi-task learning algorithms, distributed multi-task relationship learning (DMTRL) attracts much attention in the community as it learns task relationships from data, instead of imposing a prior task relatedness assumption. To perform DMTRL, task model or its gradient is transferred between task node and central server. There will be a potential privacy violation when the distributed data are possessed by different institutes and contain sensitive information (e.g., medical records). In this paper, we propose a distributed multi-task relationship learning approach under differential privacy called DRUPE, where privacy protection is achieved through perturbing the gradients at each task node. In particular, to reduce the high variance of the perturbed gradients and achieve fast convergence rate, we develop a variance reduction based gradient calibration method, which first estimates the gradient error from the previous perturbed gradients and then calibrates the current perturbed gradient by subtracting the gradient error term. Moreover, to alleviate the negative effect caused by the inaccurate task relationships that are inferred from the private task models, we present a task relationship calibration method which uses the least-squares approximation algorithm to calibrate the inaccurate pairwise relationships between the tasks. Experimental results on real-world datasets confirm the effectiveness of our approach.