Adaptive internal activation functions and their effect on learning in feed forward networks

Abstract

Selecting the correct function for each neuron allows greater representational power, and thus smaller more efficient networks. This paper presents a method of dynamically modifying the activation function within each neuron during training. Thus allowing the network designer to use complex activation functions without having to assign the correct one to each individual neuron.