An efficient approach for privacy preserving decentralized deep learning models based on secure multi-party computation

Abstract

This paper aims to develop a new efficient framework named Secure Decentralized Training Framework (SDTF) for Privacy Preserving Deep Learning models. The main feature of the proposed framework is its capable of working on a decentralized network setting that does not need a trusted third-party server while simultaneously ensuring the privacy of local data with a low cost of communication bandwidth. Particularly, we first propose a so-called Efficient Secure Sum Protocol (ESSP) that enables a large group of parties to jointly calculate a sum of private inputs. ESSP can work not only with integer number but also with floating point number without any data conversion. We then propose a Secure Model Sharing Protocol that enables a group of parties securely train and share the local models to be aggregated into a global model. Secure Model Sharing Protocol exploits randomization techniques and ESSP to protect local models from any honest-but-curious party even n-2 of n parties colluding. Eventually, these protocols are employed for collaborative training decentralized deep learning models.We conduct theoretical evaluation of privacy and communication cost as well as empirical experiments on balance class image datasets (MNIST) and an unbalance class text dataset (UCI SMS Spam). These experiments demonstrate the proposed approach can obtain high accuracy (i.e. 97% baseline accuracy in only 10 training rounds with MNIST, 100 training rounds with SMS Spam) and robust to the heterogeneity decentralized network, with non-IID and unbalance data distributions. We also show a reduction in required rounds of training to achieve the accuracy baseline by 5× as compared to Downpour SGD. It is shown that the proposed approach can achieve both the privacy at the level of cryptographic approaches and efficiency at the level of randomization techniques, while it also retains higher model’s utility than differential privacy approaches.