The Spatial Complexity of Inhomogeneous Multi-layer Neural Networks

Abstract

Inhomogeneous multi-layer neural networks (IHMNNs) have been applied in various fields, for example, biological and ecological contexts. This work studies the learning problem of IHMNNs with an activation function $$f(x) = \dfrac{1}{2} (|x+1| - |x-1|)$$f(x)=12(|x+1|-|x-1|) that derives from cellular neural networks, which can be adapted to the study of the vision systems of mammals. Applying the well-developed theory of symbolic dynamics, the explicit formulae of the topological entropy of the output and hidden spaces are given. We also demonstrate that, for any $$\lambda \in [0, \log 2]$$??[0,log2] and $$\epsilon > 0$$∈>0, parameters such that the topological entropy $$h$$h of the hidden/output space of IHMNN that satisfies $$|h - \lambda | < \epsilon $$|h-?|<∈ exists. This means that the collection of topological entropies is dense in the closed interval $$[0, \log 2]$$[0,log2], which leads to the fact that IHMNNs are universal machines in some sense and hence are more efficient in learning algorithms. This paper aims to provide a mathematical foundation for the illustration of the capability of machine learning, while the method we have adopted can be extended to the investigation of multi-layer neural networks with other activation functions.