Natural Gradient Method Research Articles

Improving generalization performance of natural gradient learning using optimized regularization by NIC.

Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly,the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

Equivariant nonstationary source separation

Most of source separation methods focus on stationary sources, so higher-order statistics is necessary for successful separation, unless sources are temporally correlated. For nonstationary sources, however, it was shown [Neural Networks 8 (1995) 411] that source separation could be achieved by second-order decorrelation. In this paper, we consider the cost function proposed by Matsuoka et al. [Neural Networks 8 (1995) 411] and derive natural gradient learning algorithms for both fully connected recurrent network and feedforward network. Since our algorithms employ the natural gradient method, they possess the equivariant property and find a steepest descent direction unlike the algorithm [Neural Networks 8 (1995) 411]. We also show that our algorithms are always locally stable, regardless of probability distributions of nonstationary sources.

Adaptive natural gradient learning algorithms for various stochastic models

The natural gradient method has an ideal dynamic behavior which resolves the slow learning speed of the standard gradient descent method caused by plateaus. However, it is required to calculate the Fisher information matrix and its inverse, which makes the implementation of the natural gradient almost impossible. To solve this problem, a preliminary study has been proposed concerning an adaptive method of calculating an estimate of the inverse of the Fisher information matrix, which is called the adaptive natural gradient learning method. In this paper, we show that the adaptive natural gradient method can be extended to be applicable to a wide class of stochastic models: regression with an arbitrary noise model and classification with an arbitrary number of classes. We give explicit forms of the adaptive natural gradient for these models. We confirm the practical advantage of the proposed algorithms through computational experiments on benchmark problems.

Improving stability in blind source separation with stochastic median gradient

A new online learning algorithm is derived for blind separation of mixed signals with symmetric distributions. The stability of the method is analysed and compared with the natural gradient method. It is proved that the set of stability conditions obtained is less stringent. Some experiments are included.

Superefficiency in blind source separation

Blind source separation is the problem of extracting independent signals from their mixtures without knowing the mixing coefficients nor the probability distributions of source signals and may be applied to EEG and MEG imaging of the brain. It is already known that certain algorithms work well for the extraction of independent components. The present paper is concerned with superefficiency of these based on the statistical and dynamical analysis. In a statistical estimation using t examples, the covariance of any two extracted independent signals converges to 0 of the order of 1/t. On-line dynamics shows that the covariance is of the order of /spl eta/ when the learning rate /spl eta/ is fixed to a small constant. In contrast with the above general properties, a surprising superefficiency holds in blind source separation under certain conditions where superefficiency implies that covariance decreases in the order of 1/t/sup 2/ or of /spl eta//sup 2/. The paper uses the natural gradient learning algorithm and method of estimating functions to obtain superefficient procedures for both batch estimation and on-line learning. A standardized estimating function is introduced to this end. Superefficiency does not imply that the error variances of the extracted signals decrease in the order of 1/t/sup 2/ or /spl eta//sup 2/ but implies that their covariances (and independencies) do.