Abstract
a critical point is a point at which the derivatives of an error function are all zero. It has been shown in the literature that critical points caused by the hierarchical structure of a real- valued neural network (NN) can be local minima or saddle points, although most critical points caused by the hierarchical structure are saddle points in the case of complex-valued neural networks. Several studies have demonstrated that singularity of those kinds has a negative effect on learning dynamics in neural networks. As described in this paper, the decomposition of high- dimensional neural networks into low-dimensional neural networks equivalent to the original neural networks yields neural networks that have no critical point based on the hierarchical structure. Concretely, the following three cases are shown: (a) A 2-2-2 real-valued NN is constructed from a 1-1-1 complex-valued NN. (b) A 4-4-4 real-valued NN is constructed from a 1-1-1 quaternionic NN. (c) A 2-2-2 complex-valued NN is constructed from a 1-1-1 quaternionic NN. Those NNs described above do not suffer from a negative effect by singular points during learning comparatively because they have no critical point based on a hierarchical structure.
Highlights
A neural network is a network composed of neurons, and can be trained to find nonlinear relationships in data
This paper presented a proposal for an implementation process of a neural network (NN) having no critical point based on a hierarchical structure
Results demonstrate that real-valued and complex-valued NNs having no critical point based on a hierarchical structure can be constructed by decomposing a high-dimensional NN into equivalent real-valued or complexvalued NNs
Summary
A neural network is a network composed of neurons, and can be trained to find nonlinear relationships in data. Fukumizu et al mathematically proved the existence of a local minimum resulting from a hierarchical structure in a real-valued NN (ordinary NN handling real-valued signals) They demonstrated that critical points in a three-layer realvalued NN with H 1 hidden neurons behave as critical points in a three-layer real-valued NN with H hidden neurons, and that they are local minima or saddle points. A critical point in a three-layer complex-valued NN behaves in the same manner as that in a three-layer real-valued NN [1]: critical points in a three-layer complex-valued NN with H 1 hidden neurons turn into critical points in a three-layer complex-valued NN wit H neurons, which are saddle points (except for cases meeting rare conditions) Such singular points have been emerging lately as objects of study. This paper presents an attempt to implement an NN having no critical point based on a hierarchical structure
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: International Journal of Advanced Computer Science and Applications
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
