Effective Activation Functions for Homomorphic Evaluation of Deep Neural Networks

Srinath Obla,Xinghan Gong,Daniel Takabi,Asma Aloufi,Peizhao Hu

doi:10.1109/access.2020.3017436

Abstract

CryptoNets and subsequent work have demonstrated the capability of homomorphic encryption (HE) in the applications of private artificial intelligence (AI). In convolutional neural networks (CNNs), many computations are linear functions such as the convolution layer which can be homomorphically evaluated. However, there are layers such as the activation layer which is comprised of non-linear functions that cannot be homomorphically evaluated. One of the most commonly used methods is approximating these non-linear functions using low-degree polynomials. However, using the approximated polynomials as activation functions introduces errors which could have a significant impact on accuracy in classification tasks. In this paper, we present a systematic method to construct HE-friendly activation functions for CNNs. We first determine what properties in a good activation function contribute to performance by analyzing commonly used functions such as Rectified Linear Units (ReLU) and Sigmoid. Then, we compare polynomial approximation methods and search for an optimal range of approximation for the polynomial activation. We also propose a novel weighted polynomial approximation method tailored to the output distribution of a batch normalization layer. Finally, we demonstrate the effectiveness of our method using several datasets such as MNIST, FMNIST, CIFAR-10.

Highlights

Deep neural networks have made significant contributions to solving complex tasks, especially in computer vision
convolutional neural networks (CNNs), since we have showed the differentiable property of Softplus, Sigmoid and Tanh, we can state that they are continuous over domain [a, b]
NETWORK ARCHITECTURE AND TRAINING PROCEDURE To train on MNIST and FMNIST with polynomial activations, we adapt the Deep CNN model proposed by Chabanne et al Figure 12(a) shows an overview of the model with 14 layers and 6 activation layers

Summary

Introduction

Deep neural networks have made significant contributions to solving complex tasks, especially in computer vision. In some applications, these networks require a large volume of private data; for example, lung cancer [1] and diabetic retinopathy detection [2]. A. CONVOLUTIONAL NEURAL NETWORKS A CNN is a type of deep neural network that is used primarily for analyzing image data. CONVOLUTIONAL NEURAL NETWORKS A CNN is a type of deep neural network that is used primarily for analyzing image data A CNN transforms image data from the input layer, through layers of computations, into the scores of each label in the output layer

Objectives

Results

Conclusion