Deep networks with non-static activation function

Abstract

Deep neural networks typically with a fixed activation function at each neuron, have shown breakthrough performances. The fixed activation function is not the optimal choice for different data distributions. Toward this end, this work improves the deep neural networks by proposing a novel and efficient activation scheme called “Mutual Activation” (MAC). A non-static activation function is adaptively learned in the training phase of deep network. Furthermore, the proposed activation neuron cooperating with maxout is a potent higher-order function approximator, which can break through the convex curve limitation. Experimental results on object recognition benchmarks demonstrate the effectiveness of the proposed activation scheme.