Information geometry on extendable hierarchical large scale neural network model

Abstract

In this paper, an extendable hierarchical large scale neural network model is developed based on the theoretical analysis of information geometry. In a hierarchical set of systems, a lower order system is included in the parameter space of a larger one as a subset. Such a parameter space has rich geometrical structures that are responsible for the dynamic behaviors of learning. Extendable hierarchical large scale neural network divides a task into small tasks, and each task is fulfilled by a small network under the principle of divide and conquer to improve the performance of a single network. By studying the dual manifold architecture for a family of neural networks and analyzing the hierarchical expansion of this model based on information geometry, the paper proposes a new method to construct the extendable hierarchical large scale neural network model that has knowledge-increasable and structure-extendible ability. The method helps to provide explanation of the transformation mechanism of human recognition system and understand the theory of global architecture of neural network.