RMAF: Relu-Memristor-Like Activation Function for Deep Learning

Patrick Anokye,Nyima Tashi,Yongbin Yu,Kwabena Adu,Mighty Abra Ayidzoe,Xiangxiang Wang

doi:10.1109/access.2020.2987829

Patrick Anokye, Nyima Tashi + Show 4 more

Open Access

https://doi.org/10.1109/access.2020.2987829

Copy DOI

Abstract

Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLUMemristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.

Highlights

Deep learning (DL) has recently shown very good performance on a range of tasks including computer vision, speech processing and natural language processing [1] partly due to the availability of large-scale datasets and high end computational resources [2]
SQUEEZE NETWORK In Table 3, 4, we present the accuracy of squeezeNet with ReLU-Memristor-like activation function (RMAF) and other nonlinear functions trained on CIFAR-10, CIFAR-100, MNIST and ImageNet dataset
The RMAF was based on the properties of memristive window function, where the function can adjust to deep networks

Summary

INTRODUCTION

Deep learning (DL) has recently shown very good performance on a range of tasks including computer vision, speech processing and natural language processing [1] partly due to the availability of large-scale datasets and high end computational resources [2]. In other to achieve this, the neural networks depend on units called activation functions [23]. These functions (such as Sigmoid, Tanh and ReLU etc.) are the backbone of any neural network. Glorot et al [55] found that DNNs with rectifier linear unit (ReLU) in place of traditional sigmoid and tanh can perform much better on image recognition and text classification tasks. Our experiments show that RMAF outperforms ReLU and other standard activation functions on deep networks applied to variety of challenging domains such as image classification and feature recognition. The contributions of this paper are as follows, 1) We highlight an activation function named RMAF to improve the performance of neural networks.

RELATED WORK

THE PROPOSED METHOD

EXPERIMENTAL RESULTS

CONCLUSION AND FUTURE WORK