PSGU: Parametric self-circulation gating unit for deep neural networks

Abstract

Activation functions are of great importance for the performance and training of deep neural networks. High-performance activation function is expected to effectively prevent the gradient from vanishing and help network converge. This paper provides a novel smooth activation function, called Parameterized Self-circulating Gating Unit (PSGU), aiming to train an adaptive activation function to improve the performance of deep networks. Compared with other works, we propose and study the self-circulation gating property of activation function, and analyze its influence on the signal transmission in network by controlling the flow of information. Specifically, we theoretically analyze and propose the initialization based on PSGU, which adequately explores the properties in neighborhood of the origin. Finally, the proposed activation function and initialization are compared with other methods on commonly-used network architectures, the achieved performances of using PSGU alone or combining with our proposed initialization are over par with the state of the art.