An Adaptive Slope Basic Dynamic Node Creation Algorithm for Single Hidden Layer Neural Networks

Abstract

This paper presents an adaptive slope basic dynamic node creation algorithm for single hidden layer neural networks (ASBDNCA). The proposed algorithm is a constructive approach of building a single hidden layer neural network. The ASBDNCA puts emphasis on architectural adaptation and functional adaptation during learning. It uses gradient descent optimization method in sequential mode as the weights update rule of individual hidden node. To achieve functional adaptation, the slope of the sigmoidal activation function (SAF) is adapted during learning. The algorithm determines not only optimal number of hidden nodes, as also optimum value of the slope parameter for the non-linear nodes. One simple variant derived from ASBDNCA in which the slope parameter of SAF is fixed. Both the variants are compared to each other on five function approximation tasks. Simulation results reveal that adaptive slope SAF present several advantages over traditional fixed shape sigmoidal activation function, resulting in increased flexibility, smoother learning, better convergence and better generalization performance.