Horseshoe Chaos in a 3D Neural Network with Different Activation Functions

Fangyan Yang,Song Tang,Guilan Xu

doi:10.1155/2013/430963

Abstract

This paper studies a small neural network with three neurons. First, the activation function takes the sign function. Although the network is a simple hybrid system with all subsystems being exponentially stable, we find that it can exhibit very complex dynamics such as limit cycles and chaos. Since the sign function is a limit case of sigmoidal functions, we find that chaos robustly exists with some different activation functions, which implies that such chaos in this network is more related to its weight matrix than the type of activation functions. For chaos, we present a rigorous computer-assisted study by virtue of topological horseshoe theory.

Highlights

Since substantial evidence of chaos is found in biological studies of natural neuronal systems, researchers have realized that chaos is much helpful for neural networks escaping the local minima and may play an essential role in the storage and retrieval of information [1,2,3]
An interesting phenomenon we find in this paper is that the Hopfield neural network (HNN) can demonstrate chaos, it is a switching system that only consisted of stable subsystems; such chaos still exists even when we replace the sign function with many other activation functions
We have studied a 3D Hopfield neural network with the sign activation function

Summary

Introduction

Since substantial evidence of chaos is found in biological studies of natural neuronal systems, researchers have realized that chaos is much helpful for neural networks escaping the local minima and may play an essential role in the storage and retrieval of information [1,2,3]. In order to answer the two questions, this paper will take a limit of the sigmoidal functions by zooming out the input scale and study a small Hopfield neural network (HNN) with hard switches. Such sign function is extremely easy to implement, and of dynamical and biological significance in gene regulatory networks [4, 5].

Chaos in the Small Network with Different Activation Functions

Computer-Assisted Proof of Chaos

B2 D2 C2

Conclusions