Generic model implementation of deep neural network activation functions using GWO-optimized SCPWL model on FPGA

Abstract

The implementation of non-linear Activation Functions (AFs) within the Artificial Neural Network (ANN) on the Field Programmable Gate Array (FPGA) is substantial due to the various applications it performs. Accuracy, speed and complexity are the most crucial factors considered in this implementation. Building non-linear AFs in a reconfigurable ANN requires either sequential operations and/or additional complexity. In this paper, a generic model for three types of non-linear AFs (Logistic sigmoid (LogSig), Tan sigmoid (TanSig) and Radial Basis Function (RBF)) has been designed based on Simplicial Canonical Piecewise Linear (SCPWL) model that is optimized using Grey Wolf Optimizer (GWO(Algorithm. The designed model has been achieved by nine segments of the SCPWL model. The input of the AFs is ranging from (−8 to 8). Matlab has been deployed to design, optimize, simulate and validate this model. The maximum errors were 5.2e−3, 15.4e−3 and 7e−3 for LogSig, TanSig and RBF respectively. And, the Mean Square Error (MSE) were 1.81e−6, 1.22e−5 and 1.42e−5 for LogSig, TanSig and RBF respectively. The Matlab/HDL Coder has been used to generate the VHDL codes. The Xilinx Arty A7 (Xc7a35ticsg324-1L) FPGA kit is used to validate the designed model on Vivado Design Suite software. It has been noticed that it takes 581 Look-Up Tables (LUTs), nine DSP slices and a delay of (35.346 ns) to implement the nine SCPWL segments for any linear and non-linear AF. For validation, a complete ANN has been built with three hidden layers, each layer contain with one of the proposed AF models.